Passing parameters using pack/post, etc.

General issues of interest both for network and
individual cell parallelization.

Moderator: hines

JBall
Posts: 18
Joined: Tue Jun 15, 2010 8:47 pm

Passing parameters using pack/post, etc.

Post by JBall »

Hello,

I've been trying to get some model optimization going in parallel neuron. This involves, among other things, the ability to pass numbers between nodes to set model parameters and to compare fitness scores for different parameter sets. As a starting point, the code below is built to generate an initial matrix of parameters for n = pc.nhost different cell models, and then pass the vectors of those parameters out to each node (I've cut out the parameter generation code for simplicity here). Then, a single simulation should run. As an initial test of the simulation, I'm printing the numbers of spikes for each model to the screen. The whole problem seems to be the way I'm unpacking the messages I send using pack & post. If I comment out the block of code where I retrieve the messages, the whole thing runs fine. I have a hunch that I have a conceptual misunderstanding of how the packing and posting procedure is supposed to go. Any help is very much appreciated.

Thanks!

Update: My mistake, commenting out the section where I take and unpack the posted messages allows me to get past that block, but pc.psolve() gives me a segmentation fault. To the best of my knowledge, the way I'm initializing things should create a copy of each cell and related objects on every node that can then be simulated in parallel. Is this not right?

Update 2(!): Again, my mistake--I'd left out stdinit(). I'm back to my previous state--if I leave out the message take/unpack process, the code runs fine and I get spike numbers for n = pc.host cells. If I try to take the parameters and use them, the simulation hangs.

Code: Select all

{load_file("nrngui.hoc")}
{load_file("template.hoc")}


objref pc
{pc = new ParallelContext()}
cvode_local(1)
cvode.atol(1e-5)
cvode.maxstep(0.05)
cvode.cache_efficient(1)
tstop = 1000
v_init = -66



val = 0
nmems = pc.nhost()

objref r
{r = new Random(pc.time())}

// Setting up things to be simulated----------------

objref cell,nc[2],nil,tvec[2],stim

cell = new Celltemp()
cell.soma stim = new IClamp(0.5)
stim.amp = 0.3
stim.del = 100
stim.dur = 800

cell.soma nc = new NetCon(&cell.soma.v,nil)
cell.dend nc[1] = new NetCon(&cell.dend.v,nil)

tvec[0] = new Vector()
tvec[1] = new Vector()

nc.record(tvec[0])
nc[1].record(tvec[1])
//--------------------------------------------------


objref pars, params
pars = new Vector()
params = new Matrix(18,nmems)


proc initvals() {
        for i=0,nmems-1 {
           // generate matrix of initial parameters
	}
}



proc usevals() {

// Set cell values to passed vector

}



if(pc.id==0) {

initvals()

  for i=0,nmems-1 {
	pars = params.getcol(i)
        pc.pack(i,pars)
        pc.post(i)
  }
}



for i=0,nmems-1 {
  if(pc.id==i) {

   pc.take(i)
   pc.unpack(&val,pars)

   usevals(pars)

  }
}

pc.barrier()


{pc.set_maxstep(0.05)}
{pc.psolve(tstop)}

pc.barrier()

for i=0, nmems-1 {

if (i==pc.id) {
printf("%g\t %g\t %d\n", tvec.size(),tvec[1].size(), pc.gid)
}
pc.barrier() 
}
}

{pc.runworker()}
{pc.done()}
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Passing parameters using pack/post, etc.

Post by hines »

The fundamental problem here is the mixing of MPI parallelism which puts all communication onto the shoulders of the model author, and BulletinBoard parallelism which imposes a master worker style of computation where the master submits tasks to the bulletin board and the (master/workers) execute tasks taken from the bulletin board. The bottom line is that pc. post, take, etc only make sense AFTER pc.runworker is called. At that point only the master returns and all the workers never return but only execute submitted tasks. Since you are using pc.barrier for synchronization (possibly introducing a significant load balance performance waste if each simulation run takes significantly different time), the most general way to exchange information among the processors is via pc.alltoall
(see http://www.neuron.yale.edu/neuron/stati ... n.html#MPI )

Please take a look at
http://www.neuron.yale.edu/neuron/stati ... l#SubWorld
The ParallelContext has undergone three major conceptual extensions.
1. BulletinBoard only when first introduced. The master is in control and communicates with workers through the bulletin board AFTER pc.runworker is called.
2. Neuron Network parallel support suing global cell identifiers. A network is typically simulated PRIOR to pc.runworker and {pc.runworker() pc.done() quit()} is executed to get a proper exit.
3. Introduction of subworlds. Combines neural network, MPI synchronized parallelization, with Bulletin Board parallelization. The prototypical situation considered was parallel optimization of parallel networks. One creates a parallel network model that can be created, simulated, destroyed, as a single function call that returns some complicated value (in Python, typically a tuple of Objects) using function arguments to define the simulation (in Python a tuple of parameters). If only one conceptual network is involved and only a few parameter changes differ between simulations then the creation of the network can be factored out of the function and destrcution can be avoided. I.e. f(param) only involves setting the changed parameters and running the (parallel) simulation.

Now that one has the concept of a parameterized function returning a value, one can easily use the bulletin board to implement an optimization algorithm. As long as the bulletin board is non-empty of tasks to do, all the workers and master will be 100% active.

I have a hoc and python example of the use of subworlds. Send a request to michael dot hines at yale dot edu and let me know if you want the python, hoc, or both versions of the example. I believe the network is a ring of cells and the task return values are more or less useless. But it should give you a good analogy with which to work. ie. the fundamental concept is an "e = efun(args)" which to compute can use pc.nhost processors and the number of these functions that can be computing simultaneously is pc.nhost_world/pc.nhost.
aaronmil
Posts: 35
Joined: Fri Apr 25, 2014 10:54 am

Re: Passing parameters using pack/post, etc.

Post by aaronmil »

I have a hoc and python example of the use of subworlds. Send a request to michael dot hines at yale dot edu and let me know if you want the python, hoc, or both versions of the example.
Hi Michael,
It would be very useful to see a python example of using subworlds to do parallel optimization of parallel networks (ie, the same network is instantiated in each subworld, and not destroyed between iterations), where a function is called in each iteration that scatters a different set of parameters to each subworld and collects some calculated objectives.

Edit: https://github.com/nrnhines/ring

Thank you!
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Passing parameters using pack/post, etc.

Post by hines »

A 4 host subworld example of that ring model can be run by
mpiexec -n 4 nrniv -mpi -python tstsubworld.py
pc.subwrold(1) means that each subworld has 1 host and there are nhost subworlds.
The function that is placed on the bulletin board to do and returns a value when it is done, is defined in ring.py as the
def runring(ncell=5, delay=1, tstop=100):
which instantiates a Ring as the first statement in a local variable with the consequence
that it is destroyed on exit from runring.

So to make the Ring permanent, just factor out the construction statement. In runring
you can then change any parameters of the ring. Just be aware that those changes persist
when the worker takes up its next (nondeterministic) task. (not a problem if all changes
are defined by the parameters.)

Let me know if I somehow missed your point.
aaronmil
Posts: 35
Joined: Fri Apr 25, 2014 10:54 am

Re: Passing parameters using pack/post, etc.

Post by aaronmil »

Hi,
I have a question about using the ParallelContext interface. I am using Python.

If I split a ParallelContext into subworlds, after pc.runworker() is called, pc.submit can be used to post jobs to the bulletin board, but there is no guarantee that all subworlds will end up executing a job. Instead, a single subworld may execute multiple jobs, while some subworlds may execute none.

Is there a way to ensure that all subworlds execute a function after calling pc.runworker() ? I don't need all subworlds to execute the code simultaneously, or do anything collective across subworlds, or pass messages across subworlds. I just need to make sure all the subworlds update a local context. pc.context() does not appear suitable for this either.

Code: Select all

from mpi4py import MPI
from neuron import h
pc = h.ParallelContext()
subworld_size = 2
pc.subworlds(subworld_size)
pc.runworker()
# something like pc.submit or pc.context, but guaranteed that each subworld will execute it once.
pc.done()
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Passing parameters using pack/post, etc.

Post by hines »

Is there a way to ensure that all subworlds execute a function after calling pc.runworker() ?
pc.context almost does that, I see the problem with it is that it only executes on all the ranks of the worker subworlds. And
a direct call after runworker only executes on rank 0 of the master subworld. A submit executes on all the ranks of a subworld
that works on the job. Perhaps that would be enough for you as it is easy in python to pass a context object as part of the submit.
Anyway, it seems that I should change to semantics of pc.context so that it executes on all world ranks (except 0 (rank 0 of the master
subworld)). It may also be infelicitous to have the >0 ranks of subworlds have a id_bbs and nhost_bbs values of -1 instead of the same
values as the 0 rank of a subworld. But I have not thought that through very well yet.

It may be helpful to see the results of a simple pc.context test. I'm using

Code: Select all

$ cat testcontext.py
from neuron import h
import time
pc = h.ParallelContext()
pc.subworlds(2)
nhost_world = int(pc.nhost_world())
id_world = int(pc.id_world())
nhost_bbs = int(pc.nhost_bbs())
id_bbs = int(pc.id_bbs())
nhost = int(pc.nhost())
id = int(pc.id())
def f(arg):
  print ("arg=%d nhost_world=%d id_world=%d nhost_bbs=%d id_bbs=%d nhost=%d id=%d"%
    (arg, nhost_world, id_world, nhost_bbs, id_bbs, nhost, id))

f(1)
time.sleep(.1) #enough time to print

pc.runworker()
print ("after runworker")
pc.context(f, 2)
f(3) #rank 0 of the master subworld

for i in range(1): #time to print and
  pc.post("wait")   # bulletin board to communicate
  time.sleep(.1)
  pc.take("wait")

pc.done()
h.quit()

The weird time wasting fragments are just to get all the printing to come out before going on to the next phase of the code or quitting.
They are entirely unnecessary in terms of execution sequence.
The results of execution are (output from each arg group are non-deterministic).

Code: Select all

$ mpiexec -n 6 nrniv -mpi -python testcontext.py
numprocs=6
NEURON -- VERSION 7.5 master (c693a84) 2017-12-07
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2016
See http://neuron.yale.edu/neuron/credits

arg=1 nhost_world=6 id_world=0 nhost_bbs=3 id_bbs=0 nhost=2 id=0
arg=1 nhost_world=6 id_world=1 nhost_bbs=-1 id_bbs=-1 nhost=2 id=1
arg=1 nhost_world=6 id_world=3 nhost_bbs=-1 id_bbs=-1 nhost=2 id=1
arg=1 nhost_world=6 id_world=4 nhost_bbs=3 id_bbs=2 nhost=2 id=0
arg=1 nhost_world=6 id_world=2 nhost_bbs=3 id_bbs=1 nhost=2 id=0
arg=1 nhost_world=6 id_world=5 nhost_bbs=-1 id_bbs=-1 nhost=2 id=1
after runworker
arg=3 nhost_world=6 id_world=0 nhost_bbs=3 id_bbs=0 nhost=2 id=0
arg=2 nhost_world=6 id_world=2 nhost_bbs=3 id_bbs=1 nhost=2 id=0
arg=2 nhost_world=6 id_world=3 nhost_bbs=-1 id_bbs=-1 nhost=2 id=1
arg=2 nhost_world=6 id_world=4 nhost_bbs=3 id_bbs=2 nhost=2 id=0
arg=2 nhost_world=6 id_world=5 nhost_bbs=-1 id_bbs=-1 nhost=2 id=1
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Passing parameters using pack/post, etc.

Post by hines »

Is there a way to ensure that all subworlds execute a function after calling pc.runworker() ?
Yes. Use the following module

Code: Select all

$ cat submit_impl_of_context.py 
'''
Usage:
from submit_impl_of_context import pccontext

def context(arg):
  ...

# for any pc.subworld organization
#after pc.runworker()
#execute context(arg) on all nhost_world ranks except 0
pccontext(context, arg)

'''

'''
Implementation of what pc.context should be via use of pc.submit. The problem
with pc.context is that it does not execute on any worker of subworld 0.
by submitting pc.nhost_bbs jobs with a context_callable, context pair
of args and arranging them to execute one per pc.id_bbs
we can get the effect of what pc.context should be. Note that
id_world==0 does NOT execute context_callable(context)
'''
from neuron import h
pc = h.ParallelContext()

def _context(context, arg):
  if (int(pc.id_world()) > 0):
    context(arg)
  if (int(pc.id()) == 0): #increment context count
    pc.take("context")
    i = pc.upkscalar()
    pc.post("context", i+1)
    while True:
      if pc.look("context") and pc.upkscalar() == nhost_bbs:
        return # nhost_bbs distinct ranks executed _context

def pccontext(context, arg): # working version of pc.context(context, arg)
  pc.post("context", 0)
  for i in range(int(pc.nhost_bbs())):
    pc.submit(_context, context, arg)
  while pc.working():
    pass
  pc.take("context")

if __name__ == "__main__":
  import time
  pc.subworlds(2)
  nhost_world = int(pc.nhost_world())
  id_world = int(pc.id_world())
  nhost_bbs = int(pc.nhost_bbs())
  id_bbs = int(pc.id_bbs())
  nhost = int(pc.nhost())
  id = int(pc.id())
  def f(arg):
    print ("arg=%d nhost_world=%d id_world=%d nhost_bbs=%d id_bbs=%d nhost=%d id=%d"%
      (arg, nhost_world, id_world, nhost_bbs, id_bbs, nhost, id))

  f(1)
  time.sleep(.1) #enough time to print

  pc.runworker()
  print ("after runworker")
  pccontext(f, 2)
  f(3) #rank 0 of the master subworld

  for i in range(1): #time to print and
    pc.post("wait")   # bulletin board to communicate
    time.sleep(.1)
    pc.take("wait")

  pc.done()
  h.quit()

The test result is

Code: Select all

$ mpiexec -n 6 nrniv -mpi -python submit_impl_of_context.py 
numprocs=6
NEURON -- VERSION 7.5 master (c693a84) 2017-12-07
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2016
See http://neuron.yale.edu/neuron/credits

arg=1 nhost_world=6 id_world=1 nhost_bbs=-1 id_bbs=-1 nhost=2 id=1
arg=1 nhost_world=6 id_world=2 nhost_bbs=3 id_bbs=1 nhost=2 id=0
arg=1 nhost_world=6 id_world=3 nhost_bbs=-1 id_bbs=-1 nhost=2 id=1
arg=1 nhost_world=6 id_world=5 nhost_bbs=-1 id_bbs=-1 nhost=2 id=1
arg=1 nhost_world=6 id_world=0 nhost_bbs=3 id_bbs=0 nhost=2 id=0
arg=1 nhost_world=6 id_world=4 nhost_bbs=3 id_bbs=2 nhost=2 id=0
after runworker
arg=2 nhost_world=6 id_world=2 nhost_bbs=3 id_bbs=1 nhost=2 id=0
arg=2 nhost_world=6 id_world=3 nhost_bbs=-1 id_bbs=-1 nhost=2 id=1
arg=2 nhost_world=6 id_world=4 nhost_bbs=3 id_bbs=2 nhost=2 id=0
arg=2 nhost_world=6 id_world=1 nhost_bbs=-1 id_bbs=-1 nhost=2 id=1
arg=2 nhost_world=6 id_world=5 nhost_bbs=-1 id_bbs=-1 nhost=2 id=1
arg=3 nhost_world=6 id_world=0 nhost_bbs=3 id_bbs=0 nhost=2 id=0
Michaels-MacBook-Air:Parallel hines$ 
aaronmil
Posts: 35
Joined: Fri Apr 25, 2014 10:54 am

Re: Passing parameters using pack/post, etc.

Post by aaronmil »

Thank you, Michael. Is there any particular reason that you were enforcing that your new pccontext is not executed on the master? This module works just as well without the awkwardess of having to separately call the function on rank 0 manually:

Code: Select all

from neuron import h
pc = h.ParallelContext()


def _context(context, arg):
    context(arg)
    if (int(pc.id()) == 0):  # increment context count
        pc.take("context")
        i = pc.upkscalar()
        pc.post("context", i + 1)
        while True:
            if pc.look("context") and pc.upkscalar() == nhost_bbs:
                return  # nhost_bbs distinct ranks executed _context


def pccontext(context, arg):  # working version of pc.context(context, arg)
    pc.post("context", 0)
    for i in range(int(pc.nhost_bbs())):
        pc.submit(_context, context, arg)
    while pc.working():
        pass
    pc.take("context")


if __name__ == "__main__":
    import time

    pc.subworlds(2)
    nhost_world = int(pc.nhost_world())
    id_world = int(pc.id_world())
    nhost_bbs = int(pc.nhost_bbs())
    id_bbs = int(pc.id_bbs())
    nhost = int(pc.nhost())
    id = int(pc.id())

    def f(arg):
        print ("arg=%d nhost_world=%d id_world=%d nhost_bbs=%d id_bbs=%d nhost=%d id=%d" %
               (arg, nhost_world, id_world, nhost_bbs, id_bbs, nhost, id))

    f(1)
    time.sleep(.1)  # enough time to print

    pc.runworker()
    print ("after runworker")
    pccontext(f, 2)

    for i in range(1):  # time to print and
        pc.post("wait")  # bulletin board to communicate
        time.sleep(.1)
        pc.take("wait")

    pc.done()
    h.quit()
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Passing parameters using pack/post, etc.

Post by hines »

It seemed to me most natural that the master context exists prior to the call to pc.context
aaronmil
Posts: 35
Joined: Fri Apr 25, 2014 10:54 am

Re: Passing parameters using pack/post, etc.

Post by aaronmil »

Also, while I've got you, the unique id assigned to a submit call is not equal to the id returned upon job completion. It is its negative.

Code: Select all

from neuron import h
pc = h.ParallelContext()
def func(*args):
	return
pc.runworker()
submit_id = pc.submit(func, *args)
while pc.working():
	return_id = pc.userid()
assert submit_id == return_id, 'submit_id: %i != return_id: %i' % (submit_id, return_id)
pc.done()
produces:

Code: Select all

numprocs=6
NEURON -- VERSION 7.5 master (2450560) 2017-12-11
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2016
See http://neuron.yale.edu/neuron/credits

Traceback (most recent call last):
  File "scratchpad.py", line 14, in <module>
    assert submit_id == return_id, 'submit_id: %i != return_id: %i' % (submit_id, return_id)
AssertionError: submit_id: -1 != return_id: 1
On a related note, it would be nice to have something like
pc.look(id)
to see if a particular pc.submit associated with a particular id is done without removing it from the bulletin board. The problem with
pc.working()
is that it removes the job from the bulletin board.

It would also be nice if the user_id provided with the syntax
pc.submit(user_id, func, *args)
could be a string rather than only an integer (like pc.post()). Thanks for your consideration.
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Passing parameters using pack/post, etc.

Post by hines »

ParallelContext.context has been fixed to execute on all pc.id_world() > 0. The fix is pushed to github. The test is
https://github.com/nrnhines/nrntest/blo ... context.py
It is good to also manually run the test with
pc.suboworld commented out and with arguments of 1 and pc.nhost()

I will look into the submit id issue.

With regard to your other suggestions, I will consider further. I start out with the prejudice that it is a bad idea to specify where a
bulletin board submit should
execute. The idea is to naturally have all the processes active as long as there are jobs to do on the bulletin board. The bulletin board is
a poor man's pattern of Gelernters' Linda system. His book "mirror worlds" is worth reading. My main regret with the bulletin board is that it
is not distributed and so does not scale to large nhost. The primary metaphor is a shared memory (tuple space). Practically, it seems to me
that the flexibility of submitting a python callable with arbitrary pickleable arguments and return value pretty much covers the domain.
It is helpful to add job identification to the return value (which one gets with pc.pyret() )
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Passing parameters using pack/post, etc.

Post by hines »

I see that the explicit first arg to submit userid does work

Code: Select all

from neuron import h
pc = h.ParallelContext()
def func(*args):
        return
pc.runworker()
args=(1, 2, 3)
submit_id = pc.submit(25, func, *args)
while pc.working():
        return_id = pc.userid()
assert submit_id == return_id, 'submit_id: %i != return_id: %i' % (submit_id, r$
pc.done()
This kind of thing was useful for HOC since functions were limited to returning a double scalar. With python, this becomes fairly useless since
python can return complex objects including tuples. In the absence of a userid first arg to submit, I need to review how the submit return value is
constructed and how the pc.userid() is retrieved. It may be the case that I will just update the documentation.
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Passing parameters using pack/post, etc.

Post by hines »

Made a slight change to the github repository so that the pc.submit() return value is the same as the return value of pc.userid().
Note that the pc.working() return value is a unique system generated id (not pc.userid()). that value is passed to the job execution as
h.hoc_ac_

As mentioned, none of this is very important with python job submission as it is simple for the job to return a value that contains some or all
of the
args it received.
aaronmil
Posts: 35
Joined: Fri Apr 25, 2014 10:54 am

Re: Passing parameters using pack/post, etc.

Post by aaronmil »

hines wrote: Thu Dec 21, 2017 8:16 am ParallelContext.context has been fixed to execute on all pc.id_world() > 0. The fix is pushed to github.
Thank you.
hines wrote: Thu Dec 21, 2017 8:16 am With regard to your other suggestions, I will consider further. I start out with the prejudice that it is a bad idea to specify where a bulletin board submit should execute. The idea is to naturally have all the processes active as long as there are jobs to do on the bulletin board.
What you are describing is a "load balanced" version of an asynchronous map operation. the ipyparallel module offers both a "direct" interface, where arguments are mapped to workers in a defined order, as well as a "load balanced" interface, where arguments are mapped to any worker that is ready to receive a job. Both have their use cases.

However, this is completely separate from the question of how you pull completed results off the bulletin board. pc.working() pulls any result off the bottom of a stack. I'm proposing that something like pc.ready(user_id) be used on the master process to query if a particular job is done and return a boolean. Then something like pc.get(user_id) could take that particular result out of the stack, and the next pc.pyret() call would make available the result on the master process. I have hacked together this functionality by always throwing every result returned by pc.working() into a python dictionary indexed by the user_id. Then if I want a true map operation that returns a list of results in the order that the arguments were submitted, I have to basically use another buffer separate from the internals of the pc bulletin board.
hines wrote: Thu Dec 21, 2017 8:16 am Practically, it seems to me that the flexibility of submitting a python callable with arbitrary pickleable arguments and return value pretty much covers the domain. It is helpful to add job identification to the return value (which one gets with pc.pyret() )
By the way, what you have accomplished with the python wrapper of ParallelContext is a way to use a bulletin board for concurrent operations, while also doing collective operations within each subgroup of processes. Currently ipyparallel does not offer this functionality, and neither does mpi4py.futures. They only offer complete partitioning of 1 process per worker. It would be extremely useful to separate out this concurrent collective parallelism into a separate git repository / python module so that it can be installed and used separately from NEURON, since it solves a much more general problem for scientific computing.

If that were the case, the last item on my wishlist would be for the python callable syntax of pc.submit to accept **kwargs in addition to *args.

Thanks for the discussion!
aaronmil
Posts: 35
Joined: Fri Apr 25, 2014 10:54 am

Re: Passing parameters using pack/post, etc.

Post by aaronmil »

hines wrote: Thu Dec 21, 2017 1:29 pm Made a slight change to the github repository so that the pc.submit() return value is the same as the return value of pc.userid().
Great, thank you!
Post Reply