Bulletin-Style Grid-Like Simulations: Indexing Getting Mixed Up

General issues of interest both for network and
individual cell parallelization.

Moderator: hines

Post Reply
agmccrei
Posts: 24
Joined: Thu Sep 10, 2015 1:34 pm

Bulletin-Style Grid-Like Simulations: Indexing Getting Mixed Up

Post by agmccrei » Sun May 12, 2019 1:23 pm

Hello,

I am doing a grid-like set of simulations on a single-neuron model (python/neuron) with increasing holding currents (i.e. using IClamp at the soma; all spike-generating current levels) and increasing synaptic input frequencies (i.e. using Exp2Syn processes at various dendritic locations). I've parallelized my code in a bulletin board style way as follows:

Code: Select all

# Load the model and initiate functions
# "getMeasures" is a function written in python that calls to a hoc function to run the model and analyze the output traces from that simulation

numsteps = 50
modfreqs = numpy.array([0,0.5,1,2,3,4,5,8,9,10,12,15,16,20,25,30]) # Hz
currsteps = numpy.arange(rheobase,rheobase+0.0025*numsteps,0.0025) # nA

pc = h.ParallelContext()
pc.runworker()

if pc.nhost() == 1:
	for y in range(0,numsteps):
		for modfreq in range(0,len(modfreqs)):
				results = getMeasures(y,modfreqs[modfreq],modfreq,currsteps[y])
				# RECORD VARIABLES FROM RESULTS INTO LISTS USING "y" AND "modfreq" TO INDEX

else:
	for y in range(0,numsteps):
		for modfreq in range(0,len(modfreqs)):
			pc.submit(getMeasures,y,modfreqs[modfreq],modfreq,currsteps[y])
	while pc.working():
		results = pc.pyret()
		# RECORD VARIABLES FROM RESULTS INTO LISTS USING "y" AND "modfreq" TO INDEX

pc.done()

When I run it this way, I get some unexpected results that I would not otherwise obtain when running in serial. When looking at the baseline spike rates (i.e. without synaptic inputs - modfreqs[0]) they do not always increase gradually with increasing holding currents. When I plot out the baseline voltage traces (i.e. from the parallelized simulations), it is clear that they are from simulations that include synaptic inputs (or even from different current injections steps), indicating a mismatch between input parameters and output traces. When I print out the input parameters onto the filenames for the voltage plots that are generated, they are the correct input parameters (but of course this does not match the output that is plotted). Strangely, when printing out the lists of the input parameters (which I include as output parameters in my functions), they appear in the correct order, so the output indexing appears fine, suggesting that it has something more to do with the simulations themselves getting mixed up somehow during submission.

A few additional points:
-I run the parallel simulations on NSG, and I have received similar indexing mix-ups when running it on two different HPC resources (Stampede2 and Comet).
-The indexing mix-up is not consistent across repeated runs, and it seems like less indexing mix-ups occur when I have a closer 1:1 spread of simulations:processors (i.e. instead of multiple simulations on a smaller number of processors).
-From the troubleshooting I've done, it seems to me that the indexing mix-ups occur because of how I've parallelized my code, but I am having trouble pin-pointing where exactly I went wrong. Any input would be appreciated.

Thanks,

Alex GM

ted
Site Admin
Posts: 5704
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Bulletin-Style Grid-Like Simulations: Indexing Getting Mixed Up

Post by ted » Mon May 13, 2019 11:22 am

Three suggestions.
1. Keep your code simple. Drop the conditional statement and its "what to do if nhost == 1" clause--it's completely unnecessary. Your parallelized program should produce the same results for any value of nhost > 0; the only difference will be total run time.
2. Make sure that each returned result includes the corresponding values of that job's run parameters (in this case, the values of y and modfreq, or perhaps better to return the actual injected current and synaptic input frequency). You'll need those because you can't assume that jobs will be completed in the order in which they were posted.
3. Don't eat anything bigger than your head. While you're in develop & debug mode, keep runtimes short (just long enough to produce a discernable result), and the number of runs small. If your goal is to iterate over M sets of N parameter values, make sure that things work properly with small values for M and N before trying to scale up.

agmccrei
Posts: 24
Joined: Thu Sep 10, 2015 1:34 pm

Re: Bulletin-Style Grid-Like Simulations: Indexing Getting Mixed Up

Post by agmccrei » Mon May 13, 2019 5:13 pm

Thanks Ted,

I return both the index values as well as the actual parameters values (i.e. injected current and input frequency). The odd thing about my problem is that everything gets indexed in the correct order, despite the results appearing out of order. When I print out output lists (i.e. that were processed by the parallelized function) containing the index values and input parameter values they all appear in perfect order. When I look at the corresponding output spike rates, it looks fine up until about a third of the way through the list where spike rate values start getting mixed up.

Here is an example of the printouts for 1 holding current (0.0625 nA) across all input frequencies (see below - the spike rate should be starting high and mostly just going down as you introduce higher frequency inhibitory inputs - if comparing to the serialized version). What appears to happen is that the baseline spike rate (index 0 in bold) will get replaced by the spike rate at another input frequency (looking across holding currents, I see that this happens across many holding currents, but randomly across job submissions). Also note that it always appears to be the baseline spike rate that gets replaced with another spike rate (also in bold) and never any of the other spike rates. As you go further down the list of holding currents, this indexing error becomes more and more frequent. From looking at the corresponding baseline trace plots, it appears to be the simulations themselves and not the indexing since you can see inhibitory perturbations in the baseline voltage traces (i.e. when there should be not perturbations).

InputFrequency = [ 0. 0.5 1. 2. 3. 4. 5. 8. 9. 10. 12. 15. 16. 20. 25. 30. ]
HoldingCurrent = [ 0.0625 0.0625 0.0625 0.0625 0.0625 0.0625 0.0625 0.0625 0.0625 0.0625 0.0625 0.0625 0.0625 0.0625 0.0625 0.0625]
BaseSpikeRates = [ 6.73763644 11.58658185 11.42230036 11.09430156 11.02388508 10.76028111 10.05856319 8.05810791 9.04482212 10.02964317 9.9031935 7.56412823 8.00142248 6.73763644 4.25603405 1.75078202]

I have also double-checked printouts of the inputs parameters before parallelization, before pc.submit(), as well as right after pc.pyret() to ensure that there are no repeated indices during submission or outputting results and there are none. Since the input parameters appear correct, and the outputted input parameters are indexing results correctly, I'm led to believe that parallel processes are somehow contaminating the baseline trace. Unfortunately, I've been unable to pintpoint exactly where the error is, even going as far as creating a list of recording vectors to try to ensure that every simulation has a self-contained simulation trace that will not get overwritten.

AGM

agmccrei
Posts: 24
Joined: Thu Sep 10, 2015 1:34 pm

Re: Bulletin-Style Grid-Like Simulations: Indexing Getting Mixed Up

Post by agmccrei » Mon May 13, 2019 6:06 pm

UPDATE:
Turns out my error was pretty trivial (as is usually the case). I was using code from a previous project, and I had forgotten to adjust it for conditions where the input spike rate was 0 Hz. The problem was that I had a conditional statement in my hoc code that prevented the NetStim parameters from being adjusted when the input frequency was 0 Hz, which meant that whenever the frequency was 0 Hz, it just used the frequency that was used most recently on that core. This meant that the first few iterations of 0 Hz worked fine since NetStim's parameters were initiated to be at 0 Hz, and once NetStim got adjusted on the processor, it could not go back to 0 Hz and just kept the same input frequency that was used most recently. Apologies for the unnecessary post - this had me confused for a few days now (mainly because of the randomness in the output results). A good reminder for me to always double-check all of my code.

ted
Site Admin
Posts: 5704
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Bulletin-Style Grid-Like Simulations: Indexing Getting Mixed Up

Post by ted » Mon May 13, 2019 7:00 pm

No need to apologize. These threads have enormous potential tutelary value. A great deal can be learned from someone else's mistakes (including the facts that there area many ways for "working" code to fail, and how easy it can be to set a trap for oneself).

Post Reply