Multisplit single cell with many synapses

General issues of interest both for network and
individual cell parallelization.

Moderator: hines

Post Reply
bsuter
Posts: 42
Joined: Fri May 28, 2010 9:04 am

Multisplit single cell with many synapses

Post by bsuter »

Hello,
1. I've made a multi-compartmental single-cell model neuron that runs without error with multisplit over 6 processors (on a single Microsoft Windows/Intel computer).
2. I've added many (e.g. 2000) Exp2Syn point processes to this single cell, distributed across most of the segments, with each synapse driven by a dedicated NetCon and NetStim pair. I intend to drive each synapse independently; the parameters of each NetStim instance will be set before the start of each run. This simulation runs fine without parallelization.
3. I'd like to run this synaptic simulation parallelized over all local processor cores, as in (1) above. My initial attempts - simply selecting the same ParallelComputing GUI settings as in (1) - resulted in an error "usable mindelay is 0". I have not yet carefully investigated this error, but found one forum post that talks about NetCon with multisplit: http://www.neuron.yale.edu/phpbb/viewto ... t=15#p5763

I figure I should take a step back and try to understand any relevant fundamentals ...

A. What are the considerations necessary when adding synapses (NetStim->NetCon->Exp2Syn) to a multisplit single-cell model?
B. Is there a particular published model or other material that would be a good reference?

I've read what I could find on parallelization, much of which seems to pertain to distributing workloads across clusters of machines, and/or to situations where multiple cells exist. Unfortunately, I was not yet able to understand how my situation maps onto those cases. Perhaps additional readings of those papers will get me there. In the meantime, any pointers would be much appreciated.

Thank you.
ted
Site Admin
Posts: 5856
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Multisplit single cell with many synapses

Post by ted »

bsuter wrote:simply selecting the same ParallelComputing GUI settings as in (1) - resulted in an error "usable mindelay is 0".
That's not an error, it's just an automatically generated statement of the minimum usable NetCon delay for your particular model. Zero should be OK for any network model whose axons have subluminal conduction velocities. Besides, I don't think a NetCon will let you specify delay < 0.
bsuter
Posts: 42
Joined: Fri May 28, 2010 9:04 am

Re: Multisplit single cell with many synapses

Post by bsuter »

Hi Ted,
I called it an error because in addition to that message, the simulation does not run:
oc>run()
nrniv: usable mindelay is 0 (or less than dt for fixed step method)
near line 439
run()
^
finitialize(-79.8 )
init( )
stdinit( )
run( )
That's not an error, it's just an automatically generated statement of the minimum usable NetCon delay for your particular model. Zero should be OK for any network model whose axons have subluminal conduction velocities. Besides, I don't think a NetCon will let you specify delay < 0.
The delay was set =0 for each NetCon, when I got that message. There is no error when running without multisplit.

I changed the delay =0.1 for each NetCon, and now the simulation does run, without any messages. However, as I originally wrote, my main goal is to understand what the fundamental considerations are in parallelizing a single-cell+many-synapses model over multiple local processors? For example: under what conditions can I expect a speed-up? For the single-cell alone, with e.g. stimulation at the soma, multisplit produces a significant speed-up. But when there are many synapses distributed all over the dendrites, perhaps that is no longer to be expected? Is there some particular way of setting up the model that might favor a multisplit advantage, over other ways?

For example, enabling multisplit over 6 local processors (after setting NetCon delay to 0.1) in this case resulted in a slowing of execution, from 0.73 sec to 3.72 sec. I was hoping that by asking about the fundamental considerations, I could avoid peppering this board with lots of incremental questions (that likely reflect my ignorance).

I'm not aiming to simulate axon conduction / delays per se, just many synapses with particular spatial distributions, and ultimately a variety of temporal and spatio-temporal activation patterns.

Thank you!
ted
Site Admin
Posts: 5856
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Multisplit single cell with many synapses

Post by ted »

Minimum delay is important for parallelization that involves spike exchange. If you want to have NetCons with delay=0, it is necessary to make sure that the spike source and target are on the same thread.

It may be helpful to read the following papers, which are available from http://www.neuron.yale.edu/neuron/nrnpubs:

The section on NEURON in Brette et al.
Simulation of networks of spiking neurons: a review of tools and strategies. J. Comput. Neurosci. 23:349-398, 2007.

Hines, M.L. and Carnevale, N.T.
Translating network models to parallel hardware in NEURON. J. Neurosci. Methods 169:425-455, 2008.

Hines, M.L., Markram, H. and Schuermann, F.
Fully implicit parallel simulation of single neurons. Journal of Computational Neuroscience 25:439-448, 2008.
bsuter
Posts: 42
Joined: Fri May 28, 2010 9:04 am

Re: Multisplit single cell with many synapses

Post by bsuter »

Thank you for the paper references. I still don't understand what determines the actual minimal NetCon.delay for a given multisplit model, but I've done some empirical tests and was able to adjust my model parameters to run parallelized (multisplit, 6 threads, cache efficient) without errors. In the process, I collected some data on what effect the NetCon.delay has on overall run-time. I'll share those results here, in case it is of interest to others.

Background: The model has roughly ~1000 segments and a complexity of ~23000. Multisplit generates 33 pieces. There are ~2700 Exp2Syn point processes, each driven by a NetCon, which in turn is driven by a NetStim. The workstation has 6 cores. Initially I set NetCon.delay=0 (the default), and used NetStim.start=200 to set the stimulation onset timing. However, this did not run with multisplit, and the NetCon.delay was the culprit.

Note: In my case, I am not interested in modeling/studying axonal conduction delays. Also, I have just one post-synaptic cell, and I am not studying network behavior. Thus, the NetCon.delay parameter is essentially arbitrary in my simulation.

Testing: I activated a fixed subset (N=249) of these synapses either (a) synchronously, or (b) with start time randomized over a 50 ms window.
For each of these configurations, I measured the run-time as I varied the NetCon.delay value (for all ~2700 NetCon instances).
I compared the run-times for single-threaded execution with those for multisplit, 6-threaded execution.

1st column = NetCon.delay in ms
2nd column = single-threaded sim duration in sec
3rd column = parallel sim duration in sec

Synchronous activation of 249 synapses

Code: Select all

 0.0    0.78    error
 0.1    0.78    3.81
 0.2    0.78    2.14
 0.3    0.78    1.63
 0.4    0.78    1.26
 0.5    0.76    1.08
 0.6    0.78    1.06
 0.7    0.78    0.94
 0.8    0.78    0.92
 0.9    0.78    0.84
 1.0    0.78    0.75
 1.5            0.68
 2.0    0.76    0.53
 3.0    0.76    0.48
 4.0    0.80    0.44
 5.0    0.78    0.36
 6.0    0.80    0.41
 7.0    0.78    0.40
 8.0    0.78    0.37
16.0    0.80    0.42
32.0    0.78    0.36
64.0    0.78    0.38
So with synchronous activation, multisplit results in a 2x speed-up as long as the NetCon.delays are all >= 5 ms. Granted, the run time was already very short, but my simulation requires this typical run to be repeated 100's of times, so it's a welcome improvement.

Asynchronous activation (over 50 ms window) of 249 synapses

Code: Select all

 0.1    55.99    18.13
 0.2             15.77
 0.5             14.43
 1.0             13.76
 2.0             13.19
 5.0    52.24    13.16
16.0            13.21
Asynchronous activation (over 50 ms window) of 2719 synapses

Code: Select all

 0.1    66.32    20.55
 1.0             17.03
 5.0             17.39
Finally, I set the NetCon.delay=200, using this parameter to set the onset timing of the stimulation; and I use the NetStim.start parameter to add "jitter" (asynchronous activation) to the set of synapses, by assigning random values from a uniform distribution between 0 and 50 ms. The run time with multisplit for this scenario was 16.62 sec, similar to the

Conclusions:
1. NetCon.delay must be >0 with multisplit.
2. When activating ~250 synapses synchronously, NetCon.delay > 5 ms is required to see a 2x speed-up with multisplit.
3. When activating ~250 synapses asynchronously (whereby the single-threaded run time is ~60 sec), a 4x speed-up is possible with multisplit. NetCon.delay should be > 1 ms.
4. When activating ~2700 synapses, results were about the same as for ~250 synapses.
5. This is likely specific to my conditions (workstation, model); you should probably test your own models to see whether you're close to the optimal speed-up.

Happy to hear any suggestions for improvements to my set-up!

Thanks.
Post Reply