Segfault when running long simulation with many neurons

When Python is the interpreter, what is a good
design for the interface to the basic NEURON
concepts.

Moderator: hines

Post Reply
jgosmann
Posts: 3
Joined: Tue Jun 24, 2014 12:55 pm
Contact:

Segfault when running long simulation with many neurons

Post by jgosmann » Wed Jun 25, 2014 12:35 pm

I am using the Python interface to run simulations of about 200 (compartmental) neurons and a whole bunch of synapses. That works fine for about the first 3 seconds, but shortly afterwards I get a segmentation fault. The output does not help much:

Code: Select all

NEURON -- VERSION 7.3 ansi (1106:9f00197f3a2d) 2014-06-19
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2014
See http://www.neuron.yale.edu/neuron/credits

loading membrane mechanisms from ../BahlEtAl2012/channels/x86_64/.libs/libnrnmech.so
Additional mechanisms from files
 IKM.mod SlowCa.mod cad.mod h.mod kca.mod kfast.mod kslow.mod nap.mod nat.mod
        1 
exp(119900) out of range, returning exp(700)
exp(43101.4) out of range, returning exp(700)
exp(119900) out of range, returning exp(700)
exp(119900) out of range, returning exp(700)
No more errno warnings during this execution
exp(15859.5) out of range, returning exp(700)
exp(5697.09) out of range, returning exp(700)
exp(15859.5) out of range, returning exp(700)
exp(15859.5) out of range, returning exp(700)
No more errno warnings during this execution
exp(4436.34) out of range, returning exp(700)
exp(1590.26) out of range, returning exp(700)
exp(4436.34) out of range, returning exp(700)
exp(4436.34) out of range, returning exp(700)
No more errno warnings during this execution
exp(2802.81) out of range, returning exp(700)
exp(1002.98) out of range, returning exp(700)
exp(2802.81) out of range, returning exp(700)
exp(2802.81) out of range, returning exp(700)
No more errno warnings during this execution
exp(1766.39) out of range, returning exp(700)
exp(1766.39) out of range, returning exp(700)
exp(1766.39) out of range, returning exp(700)
exp(1766.39) out of range, returning exp(700)
No more errno warnings during this execution
exp(2157.57) out of range, returning exp(700)
exp(771.003) out of range, returning exp(700)
exp(2157.57) out of range, returning exp(700)
exp(2157.57) out of range, returning exp(700)
No more errno warnings during this execution
exp(4700.02) out of range, returning exp(700)
exp(1685.05) out of range, returning exp(700)
exp(4700.02) out of range, returning exp(700)
exp(4700.02) out of range, returning exp(700)
No more errno warnings during this execution
exp(15226.9) out of range, returning exp(700)
exp(5469.65) out of range, returning exp(700)
exp(15226.9) out of range, returning exp(700)
exp(15226.9) out of range, returning exp(700)
No more errno warnings during this execution
exp(71216.3) out of range, returning exp(700)
exp(25598.8) out of range, returning exp(700)
exp(71216.3) out of range, returning exp(700)
exp(71216.3) out of range, returning exp(700)
No more errno warnings during this execution
exp(403193) out of range, returning exp(700)
exp(144950) out of range, returning exp(700)
exp(403193) out of range, returning exp(700)
exp(403193) out of range, returning exp(700)
No more errno warnings during this execution
exp(2.59729e+06) out of range, returning exp(700)
exp(933766) out of range, returning exp(700)
exp(2.59729e+06) out of range, returning exp(700)
exp(2.59729e+06) out of range, returning exp(700)
No more errno warnings during this execution
exp(1.8068e+07) out of range, returning exp(700)
exp(6.49573e+06) out of range, returning exp(700)
exp(1.8068e+07) out of range, returning exp(700)
exp(1.8068e+07) out of range, returning exp(700)
No more errno warnings during this execution
exp(1.34135e+08) out of range, returning exp(700)
exp(4.82239e+07) out of range, returning exp(700)
exp(1.34135e+08) out of range, returning exp(700)
exp(1.34135e+08) out of range, returning exp(700)
No more errno warnings during this execution
[1]    5271 segmentation fault (core dumped)  python sq_pure.py
Any ideas how to track down/fix/work around this problem?

Some other information which might be relevant:
  • I am repeatedly call the run function and run the model for an additional 1ms with each call.
  • To each neuron's soma an APCount object is attached, recording spikes into individual vectors.
  • Before each call to run all vectors will be resized to a size of 0 (and their size will be retrieved afterwards).
I tested both the current 7.3 release and current source code from the repository with the same results.

ted
Site Admin
Posts: 5572
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Segfault when running long simulation with many neurons

Post by ted » Wed Jun 25, 2014 1:47 pm

jgosmann wrote:works fine for about the first 3 seconds, but shortly afterwards I get a segmentation fault.
. . .
I am repeatedly call the run function and run the model for an additional 1ms with each call.
Both statements can't be true. run() resets t to 0.
Before each call to run all vectors will be resized to a size of 0 (and their size will be retrieved afterwards).
Are you using the standard run system, or are you implementing your own run control system that calls finitialize() and fadvance() directly?

If your model is scalable, you might start by trying to discover if the problem lies in one of your model cells, and if so, which one.

jgosmann
Posts: 3
Joined: Tue Jun 24, 2014 12:55 pm
Contact:

Re: Segfault when running long simulation with many neurons

Post by jgosmann » Thu Jun 26, 2014 4:07 pm

ted wrote:run() resets t to 0.
Then the documentation is misleading. It says
Run the simulation (advance the solver) until tstop [ms]
Are you using the standard run system, or are you implementing your own run control system that calls finitialize() and fadvance() directly?
Right now I am calling

Code: Select all

neuron.init()
the beginning and then in every timestep

Code: Select all

neuron.run(neuron.h.t + dt)
dt is 1ms and might differ from the Neuron timestep.

ramcdougal
Posts: 154
Joined: Fri Nov 28, 2008 3:38 pm
Location: Yale School of Medicine

Re: Segfault when running long simulation with many neurons

Post by ramcdougal » Thu Jun 26, 2014 4:29 pm

There is a subtle and unfortunate distinction between h.run and neuron.run.

The full code of neuron.run (defined here) is:

Code: Select all

def run(tstop):
    """
    function run(tstop)
    
    Run the simulation (advance the solver) until tstop [ms]

    """
    h('tstop = %g' % tstop)
    h('while (t < tstop) { fadvance() }')
You'll notice that this literally just keeps fadvancing until tstop is passed.

h.run (defined here), by contrast, is more sophisticated: it re-initializes the model, and calls continuerun which includes code to allow the gui to stop a simulation and to facilitate the live-updating of graphs.

h.run is a library function, which you can load by executing: h.load_file("stdrun.hoc")

(Alternatively if you're using the graphical interface, you can just do: from neuron import h, gui)

Post Reply