Is it possible to track "value explosions" and out of range errors

When Python is the interpreter, what is a good
design for the interface to the basic NEURON
concepts.

Moderator: hines

Post Reply
duytanph
Posts: 16
Joined: Wed Jul 10, 2019 5:03 pm

Is it possible to track "value explosions" and out of range errors

Post by duytanph »

Hello,

I am working on creating my own mechanism .mod file and running into large value errors such as:

Code: Select all

exp(1.86734e+184) out of range, returning exp(700)
No more errno warnings during this execution
Without getting into too much detail, the mechanism works completely fine without any "out of range" errors when running on a single processor without any parallelization.

However, once I start simulating over a large neuronal network (across anywhere between 50 to 100 CPUs), I begin seeing multiple out of range errors. The main problem is that the errors are not specifying which variable is "exploding" making it difficult for me to know where the origin of the error is. I've tried using h.Vector() to record the different variables, but it seems that only some instantiations (over millions) of the mechanism are exploding making it hard to pinpoint the problem.

Is there anything I can do to help specify or print out where these out of range values are actually originating from? Thanks.
ted
Site Admin
Posts: 6287
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Is it possible to track "value explosions" and out of range errors

Post by ted »

Interesting problem. How big does the network itself have to be (total # cells, total # synapses)? At what point (time) in the simulation does it occur?

The first thing to do is to see if there are any peculiarities about the NMODL code itself that might cause problems, and the quickest way to do that is for you to email the mod file to me
ted dot carnevale at yale dot edu

Hopefully that will be sufficient; if not, further diagnosis will be much more involved.
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Is it possible to track "value explosions" and out of range errors

Post by hines »

The exp function is often used for channel rates where the arg is some function of voltage. As the arg is 1.8...e184 that could be far downstream from the cause of such a large arg. Also it sometimes occurs under conditions of temporary instability while the variable time step method is busy reducing dt to a numerically stable value. But I doubt you have cvode enabled during your simulations. Without knowing precisely the best diagnostic sequence for your specific condition I would start by determining the rank on which the warning occurred. This presumes that you are using a spike coupled network in which you save the spike raster. With that and the rank, you can retrospectively simulate on a single process desktop the subnet that was on the rank while stimulating it with PatternStim that feeds the raster into the cells of the subnet so that the cells reproduce exactly all the state trajectories that resulted in the warning. From there you can use gdb and work backward to the original cause.

Determining the rank requires that you build again from the sources after modifying nrn/src/oc/math.c where you add a line after line 48

Code: Select all

        }else if (x > 700) {
which would look like

Code: Select all

    hoc_execerror("hoc_exp with arg > 700", NULL);
A lot of the above may be unfamiliar usage for you (e.g. PatternStim; running a fake rank, nhost sim on a single process; and gdb) and if you would like further help, contact me by email at michael dot hines at yale dot edu.
Post Reply