Question regarding multisplit and rank assignment

General issues of interest both for network and
individual cell parallelization.

Moderator: hines

Post Reply
neuromancer
Posts: 10
Joined: Mon Apr 13, 2020 4:42 pm

Question regarding multisplit and rank assignment

Post by neuromancer »

Hello, I am attempting to convert my working network model from serial to parallel to cut down the run time. It is implemented on a windows server edition with python code and biophysical single cell neuron models with complex morphologies. It will be run on a multi-processor computer with 16 cores, and 32 logical processors, and 128gb of RAM. This current test network is not large (6 neurons). This network is comprised of 2 subnetworks of 3 neurons each. Only 1 neuron of each subnetwork (I'll call this cell type 1) synapses onto the other 2 cells (cell type 2). Cell type 2 only receives input. Cell type 1 has 3 spike detectors. There is also some external poisson input via vecstim to all the neurons. The two subnetworks are also interconnected via cell type 1.

I have read through all the documentation for parallel context and read a few of Michael Hines' papers from 2008 and 2009. I've also watched a nice youtube video from Michael Hines from a few years ago (HBP education). The code is almost there, but I'm struggling with a few aspects that seem simple, but perhaps I am not understanding it well.

My first question is about set_gid2node(). I believe have enough processors to have one gid per processor so a "round-robin" style of assignment is not necessary. All of my gids in the network are unique (0 through 9). Could I simply create the assigments in this fashion:
self is the python class object: subnetwork1 or subnetwork2

Code: Select all

  
  	self.gidlist = list(range(pc.id()+self.offset, (pc.nhost()/2)+self.offset))
        for gid in self.gidlist:
            pc.set_gid2node(gid+self.offset, pc.id())
            
        pc.cell(self.celltype_1._gid, self.celltype_1._spike_detector)
        pc.cell(self.celltype_1._gid2, self.celltype_1._release_detector)
        pc.cell(self.celltype_1._gid3, self.celltype_1._release_detector2)
        for cell in self.celltype2cells:
            pc.cell(cell._gid, cell._spike_detector)
            
mpiexec -n 10 python diffrun.py <-my script to run a simulation. 10 processors for 10 gids (3 for cell type 1 + 2 for cell type 2)*2 subnetworks. I put this code is the class object scripts and so I'm guessing I would have to offset the rank numbers (pc.id()) each time I instantiate a subnetwork. i.e. (rank 0-4 for subnetwork1 and rank 5-9 for subnetwork2). I have already tried this and I am getting errors like: 2 NEURON: gid=0 has not been set on rank 2. I'm confused because I don't understand how gid 0 is being forced to be on rank 2. I must be missing something...

My next question is regarding multiple spike detectors in one cell and multisplit. I am having trouble understanding why multisplit is necessary when I have created unique gids for each spike detector and placed these gids on different processors (assuming I can get my issue above to work). It seems to me that it is a method to create another id (sid) for a split node location, but why must this split node location be created? If it is necessary to include pc.multisplit() in my code, I am unsure where to place it in my code.

Any help is much appreciated. Thanks a lot and thank you to the moderators for creating these tools for scientists.
ted
Site Admin
Posts: 6286
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Question regarding multisplit and rank assignment

Post by ted »

First two questions are:
Is all this complexity necessary at this stage?
Will it ever be?
to cut down the run time
How long is run time now? What do you anticipate doing that will make it longer? More cells? Cells with more complexity (more compartments, more mechanisms that involve differential equations or kinetic schemes)? Bigger tstop?
Cell type 1 has 3 spike detectors.
Do all three NetCons monitor the same presynaptic spike trigger zone, or do they have to monitor different presynaptic trigger zones? If all 3 are monitoring the same presynaptic variable at the same location, then all three NetCons share a single spike detector, and there won't be any need for sids. Just assign a gid to each presynaptic cell, and associate that with a single NetCon attached to the presynaptic cell's spike trigger zone.
I believe have enough processors to have one gid per processor so a "round-robin" style of assignment is not necessary.
The fewer cells there are per host, the harder it can be to achieve load balance. Load balance is essential for efficient CPU usage (otherwise the hosts that finish first will sit around waiting for the slower hosts to complete their tasks). If you only need to execute just one simulation, and will never increase the complexity of the network, then fine--use as many processors as you like. But if you need to execute a family of simulations, e.g. for parameter optimization, exploration of parameter space, evaluation of response to stochastic afferrent drive, then load balance is important.

And if programmer time and effort are expensive compared to computer time, it would be a good idea to follow the KISS principle--introduce implementational complexity only when it is absolutely necessary. In other words, don't jump right to multisplit. Instead, see how far you can get with number of hosts < number of cells. If it turns out that multisplit is necessary, then fine, add it.
mpiexec -n 10 . . .
Get your parallelized code to run properly with n=1 (i.e. should produce results identical to what the serial code did). Then make sure it runs properly with n=2, and only after that works go on to n=whatever.
neuromancer
Posts: 10
Joined: Mon Apr 13, 2020 4:42 pm

Re: Question regarding multisplit and rank assignment

Post by neuromancer »

Thanks for your response!

I plan to make a network of 12 neurons with realistic morphological reconstructions. When I ran a 1 second simulation, it took about 2-3hrs. I will need to do 10 trials of a certain randomized input spike train simulation, but with about 50 different input spike trains types (~500s total). I thought it would be worth it to cut down the run speed in this case. If you don't think it would change much, then I can figure out a way to reduce the tstop and number of input spike train types, but even if parallelization cuts it down by half, it would help a lot.

The 3 NetCons in cell type1 are not in the same spike trigger zone. So I still don't understand the purpose of multisplit very well. Since my 3 NetCons on cell type 1 are on different spike trigger zones, does this now necessitate multisplit and the creation of sids? It seems to me that multisplit separates the morphology at nodes and gives these nodes ids but I don't understand why one would want to do that and what you would use the ids for. If I'm able to represent my 3 different spike trigger zones on 3 gids and not use multisplit, I'm okay with that. But I would still like to understand multisplit just because I'm really curious about it and to learn more about parallel neuron modeling.

I see what you're saying about having fewer hosts than cells and load balance. Will definitely follow your advice with n=1, n=2 and so on.

Thanks a lot!
Last edited by neuromancer on Mon May 11, 2020 5:44 am, edited 1 time in total.
neuromancer
Posts: 10
Joined: Mon Apr 13, 2020 4:42 pm

Re: Question regarding multisplit and rank assignment

Post by neuromancer »

to follow up with my questions regarding multisplit here is a cartoon diagram of one instance of cell type 1.

link to image: https://imgur.com/dyY1kS7

Is it necessary to somehow “alert” MPI that one neuron is separated? If it is necessary, then I’m not sure how I can specify that only one neuron type is split, while the others are not. Would it be correct to do the following syntax:

pc = h.ParallelContext()
splitNodeLocation = celltype1.dend[17](0.5) # just an example
splitNodeLocation2 = celltype1.dend[5](0.5) # just an example
sid = 0 # splid id
pc.multisplit(splitNodeLocation,sid, 1)
pc.multisplit(splitNodeLocation2,sid, 1)

I’m assuming the third argument, 1, is important so that the 3 gids are owned by different processors rather than remaining all on one processor? And using the cartoon as an example, would I create 2 split locations to separate these 3 gids and do the 2 sids need to be the same? I think this is what is specified in the documentation for “backbone style”.
Post Reply