I am using a simple bulletin board setup to parallelize my simulations in hoc. However, I'm struggling with 2 things:
A) I would like to return 2 result values (threshold and conduction speed).
B) I would also like to return/read the input arguments of the pc.submit() corresponding to each set of result values so that I know to which set of parameters they belong.
As you can see, I'm not simply looping through a single parameter, and so I cannot simply sort as shown in the initbatpar.hoc example.
I have a couple of ideas, but not sure if they're doable in hoc:
1) Is there a way to have my function (find_thresh) return multiple values? If so, how do I obtain these values with some variant of pc.retval()? This would address issue A.
2) The Parallel Context intro in the online documentation states that: "The arguments to the function executed by the submit call are also available." If so, how can I access the input arguments (in which case I would know where to store the returned threshold value)? This would address issue B.
3) Both issues could be addressed if the following is allowable: I have a matrix that is defined (and allocated memory) before starting pc.runworker(). Then each job would save its results in its correct location in the results matrix. However, I assume that the parallel processors cannot save a matrix in this fashion, correct?
Barring a more elegant solution, my backup plan is to have each job write its own output file (thus not having to return anything to the master via the bulletin board.
Thank you!
Nikki
BB: Sorting values and returning multiple values
Moderator: hines
-
- Site Admin
- Posts: 6305
- Joined: Wed May 18, 2005 4:50 pm
- Location: Yale University School of Medicine
- Contact:
Re: BB: Sorting values and returning multiple values
Good questions. Strangely enough, the documentation of ParallelContext's submit method discusses these very items. This bears careful reading.nikkip wrote:I am using a simple bulletin board setup to parallelize my simulations in hoc. However, I'm struggling with 2 things:
A) I would like to return 2 result values (threshold and conduction speed).
B) I would also like to return/read the input arguments of the pc.submit() corresponding to each set of result values so that I know to which set of parameters they belong.
http://www.neuron.yale.edu/neuron/stati ... ext.submit
A. How to return multiple result values.
"In the event more than a single scalar return value is required use ParallelContext.post() within the function_name body with a key equal to the id of the task. For example:
Code: Select all
func function_name() {local id
id = hoc_ac_
$o1.reverse()
pc.post(id, $o1)
return 0
}
...
while( (id = pc.working) != 0) {
pc.take(id)
pc.upkvec.printf
}
where the example is a toy problem (the worker is given a Vector that it must reverse and return to the master). If more than one thing is to be returned (i.e. a mix of scalars, Vectors, strdefs, and pickleable Python objects), use the pack() method to compose them into a single message body before posting, then take and unpack the message on the master.
B. How to retrieve submit's arguments
"If there is no explicit userid, then the args (after the function name) are saved locally and can be unpacked when the corresponding working call returns. A local userid (unique only for this ParallelContext) is generated and returned by the submit call and is also retrieved with ParallelContext.userid() when the corresponding working call returns."
Each host would have its own local matrix instance, and no host could access any other host's matrix. All communication between processors is via the bulletin board and must use submit (available only to the master), post, and take.Both issues could be addressed if the following is allowable: I have a matrix that is defined (and allocated memory) before starting pc.runworker(). Then each job would save its results in its correct location in the results matrix. However, I assume that the parallel processors cannot save a matrix in this fashion, correct?
Re: BB: Sorting values and returning multiple values
Thank you Ted! I was just working on understanding pack, post, and retval/upkscalar/upkvec/etc. I now have this working. However, I'm still struggling with retrieving the input arguments. I can obtain the user ID, as suggested in your post, but then to which function to I feed the user ID to unpack the input arguments that were used?
I'm testing this with very simple code:
I'm testing this with very simple code:
Code: Select all
objref pc
pc = new ParallelContext()
objref myvec
myvec = new Vector(3,0)
func myobfunc() { local key, myvar1, myvar2, myvar3
key = $1
myvar1 = $2
myvar2 = $3
myvar3 = $4
myvec.x[0] = 2*myvar1
myvec.x[1] = 3*myvar2
myvec.x[2] = 4*myvar3
pc.pack(myvec.x[0])
pc.pack(myvec.x[1])
pc.pack(myvec.x[2])
pc.pack(myvec)
pc.post(key)
return key
}
{ pc.runworker() }
objref myvec_returned
myvec_returned = new Vector()
proc batchrun() {local K, cntr
cntr = 0
for K = 0, 4 {
pc.submit("myobfunc",cntr,K,K+1,K+2)
cntr = cntr+1
}
while (pc.working) {
x = pc.userid()
print "userID: ", x
pc.unpack(x)
key = pc.retval()
pc.look_take(key)
print "key=", key
var1 = pc.upkscalar()
print "var1=", var1
var2 = pc.upkscalar()
print "var2=", var2
var3 = pc.upkscalar()
print "var3=", var3
myvec_returned = pc.upkvec()
print myvec_returned.x[0], myvec_returned.x[1], myvec_returned.x[2]
}
}
batchrun()
{ pc.done() }
quit()
-
- Site Admin
- Posts: 6305
- Joined: Wed May 18, 2005 4:50 pm
- Location: Yale University School of Medicine
- Contact:
Re: BB: Sorting values and returning multiple values
Apparently there isn't such a function. Here's a very simple test program that can be executed serially, that you can use as a starting point for trying different ways to submit tasks, retrieve return values, and access input arguments (thanks to Michael Hines for providing this).nikkip wrote:I can obtain the user ID, as suggested in your post, but then to which function to I feed the user ID to unpack the input arguments that were used?
Code: Select all
objref pc
pc = new ParallelContext()
func f() {
return $1*$2
}
pc.runworker()
for i=3, 5 pc.submit("f", i, 4)
while ((id = pc.working()) != 0) {
print "id ", id
print "arg1 ", pc.upkscalar()
print "arg2 ", pc.upkscalar()
print "return value ", pc.retval()
}
(1) it returns an automatically generated local userid
and
(2) the arguments arg1 arg2 . . . to the pc.submit that placed this particular task on the bulletin board can be retrieved, one at a time and in the same order as in that pc.submit call, via whichever upk* method is appropriate (upkscalar, upkstr, upkvec, or upkpyobj)
(3) the return value can be retrieved via the retval or pyret method, as appropriate.
If you prefer to call pc.submit with an explicit userid, you'll have to manage arguments yourself. For example, if you have three arguments that are a scalar, a vector, and a string, you could
Code: Select all
objref tmpvec, tmpstr
objref arg1vec, arg2list, arg3list
arg1vec = new Vector() // arg1vec.x[i] will be the scalar arg associated with userid1
arg2list = new List() // arg2list.o(i) will be the vector arg associate with userid1
arg3list = new List() // arg3list.o(i) will be a String (defined in nrn/lib/hoc/stdlib.hoc)
for i = 0,2 {
x = . . . code that returns a scalar . . .
arg1vec.append(x)
tmpvec = . . . code that returns a Vector . . .
arg2list.append(tmpvec)
tmpstr = new String() // String.s is a strdef
. . . code that prints a string to tmpstr.s . . .
arg3list.append(tmpstr)
pc.submit(i, "f", x, tmpvec, tmpstr)
}
objref tmpvec, tmpstr
Code: Select all
while ((id = pc.working()) !=0 {
// id is the userid for the task that returned
// use it as an index into arg1vec, arg2list, and arg3list
// to recall the corresponding arguments
}