Modifying tstop in FInitializeHandler

When Python is the interpreter, what is a good
design for the interface to the basic NEURON
concepts.

Moderator: hines

Post Reply
iraikov
Posts: 17
Joined: Wed Mar 18, 2015 11:53 am
Location: University of California, Irvine

Modifying tstop in FInitializeHandler

Post by iraikov »

Hello,

I am in the process of converting parallel network code from HOC to Python. The HOC code installs a handler via FInitializeHandler that compares the elapsed wall time with the maximum time for the job, estimates whether there is sufficient time to complete the simulation, and reduces tstop if necessary (code below). The HOC version of the code used psolve and it seemed that whenever the handler modified tstop, psolve accordingly only integrated until that new value of tstop. But it looks like when calling psolve and FInitializeHandler from Python, but still using the HOC handler, psolve keeps using the original tstop value. What is the correct way to port this code? Should the handler routine be written in Python as well? Thanks!

Python invocation of FInitializeHandler:

Code: Select all

...
h('objref fi_status')
if pc.id() == 0:
  h.fi_status          = h.FInitializeHandler("simstatus()")
h.stdinit()
pc.psolve(h.tstop)
HOC FInitializeHandler routine:

Code: Select all

walltime = 0

dt_status = 1.0

proc simstatus() { local wt
    wt = startsw()
    if (walltime > 0) {
        printf("*** computation time at t=%g ms was %g s\n", t, wt-walltime)
        checksimtime(wt, wt-walltime)
    }
    walltime = wt
    if (t + dt_status < tstop) {
        cvode.event(t + dt_status, "simstatus()")
    }
}


tcsum = 0
tcma = 0
nsimsteps = 0

proc checksimtime() { local wt, tt, trem, tsimrem, tsimneeded, tstop1
    wt = $1
    tt = $2
    // cumulative moving average simulation time per time step
    tcma = tcma + (tt - tcma) / (nsimsteps + 1)
    tcsum = tcsum + tt
    // remaining physical time
    trem = tstop - t
    // remaining simulation time
    tsimrem = max_walltime_hrs*3600 - tcsum - mkcellstime - connectcellstime - connectgjstime
    // simulation time necessary to complete the simulation
    tsimneeded = trem*tcma+results_write_time
    printf("*** remaining computation time is %g s and remaining simulation time is %g ms\n", tsimrem, trem)
    printf("*** estimated computation time to completion is %g s\n", tsimneeded)
    if (tsimneeded > tsimrem) {
        tstop1 = int((tsimrem - results_write_time)/tt) + t
        printf ("*** not enough time to complete %g ms simulation, simulation will likely stop around %g ms\n", tstop, tstop1)
        tstop = tstop1
    }
    nsimsteps = nsimsteps + 1
 }

ted
Site Admin
Posts: 6286
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Modifying tstop in FInitializeHandler

Post by ted »

Given working hoc code that controls tstop, why bother reimplementing it in Python? What's wrong with just executing the hoc code from Python?
iraikov
Posts: 17
Joined: Wed Mar 18, 2015 11:53 am
Location: University of California, Irvine

Re: Modifying tstop in FInitializeHandler

Post by iraikov »

Hi Ted,

Thanks for your reply, but the point is precisely that calling the HOC code from Python does not produce the desired result. That is, the HOC handler routine modifies tstop, but psolve continues to integrate using the original value of tstop. So how can I can use the same HOC routine, but ensure that it can modify the value of tstop used by psolve? Thanks,

-Ivan
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Modifying tstop in FInitializeHandler

Post by hines »

ParallelContext.psolve(stoptime) should do the same thing independent of whether it was called from hoc or python. The only thing that can
make that call return earlier than stoptime is if somebody sets h.stoprun = 1 (returns as soon as the present step completes) or if someone during
a simulation executes cvode.event(another_stop_time) which will return when delivered.
consider

Code: Select all

Michaels-MacBook-Air:tmp hines$ cat temp.py
from neuron import h
h.load_file("stdgui.hoc")
pc = h.ParallelContext()

s = h.Section()

def run(et, tstop):
  pc.set_maxstep(10)
  h.stdinit()
  h.cvode.event(et)
  pc.psolve(tstop)
  print ("et=%g tstop=%g h.t=%g" % (et, tstop, h.t))

run(20, 10)
run(6, 10)

Michaels-MacBook-Air:tmp hines$ nrniv -python temp.py
NEURON -- VERSION 7.5 master (2ae566b) 2017-01-23
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2016
See http://neuron.yale.edu/neuron/credits

et=20 tstop=10 h.t=10
et=6 tstop=10 h.t=6
>>> 
iraikov
Posts: 17
Joined: Wed Mar 18, 2015 11:53 am
Location: University of California, Irvine

Re: Modifying tstop in FInitializeHandler

Post by iraikov »

Hi Michael,

So what you are saying is that the original HOC code should not have worked in the first place. I will modify my code to use cvode.event for the new stop time then. Thank you!
hines wrote:ParallelContext.psolve(stoptime) should do the same thing independent of whether it was called from hoc or python. The only thing that can
make that call return earlier than stoptime is if somebody sets h.stoprun = 1 (returns as soon as the present step completes) or if someone during
a simulation executes cvode.event(another_stop_time) which will return when delivered.
consider

Code: Select all

Michaels-MacBook-Air:tmp hines$ cat temp.py
from neuron import h
h.load_file("stdgui.hoc")
pc = h.ParallelContext()

s = h.Section()

def run(et, tstop):
  pc.set_maxstep(10)
  h.stdinit()
  h.cvode.event(et)
  pc.psolve(tstop)
  print ("et=%g tstop=%g h.t=%g" % (et, tstop, h.t))

run(20, 10)
run(6, 10)

Michaels-MacBook-Air:tmp hines$ nrniv -python temp.py
NEURON -- VERSION 7.5 master (2ae566b) 2017-01-23
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2016
See http://neuron.yale.edu/neuron/credits

et=20 tstop=10 h.t=10
et=6 tstop=10 h.t=6
>>> 
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Modifying tstop in FInitializeHandler

Post by hines »

By the way, in an mpi context with nhost>1 all ranks must participate in a cvode.event with the same delivery time and
rank 0 should figure out the step event time and then broadcast to all the other ranks which then all do a
cvode.event(et .. i.e. all ranks have to stop at the same time or spike exchange will hang.
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Modifying tstop in FInitializeHandler

Post by hines »

As an example of what I'm talking about consider:

Code: Select all

Michaels-MacBook-Air:tmp hines$ cat temp.py
from neuron import h
h.load_file("stdgui.hoc")
pc = h.ParallelContext()
rank = int(pc.id())
nhost = int(pc.nhost())

def rank0decide(et):
  x = h.Vector(1)
  if rank == 0:
    x.x[0] = et
  pc.broadcast(x, 0)
  print ("%d rank0decide x.x[0] = %g h.t=%g" % (rank, x.x[0], h.t))
  h.cvode.event(x.x[0]) # the stop event


def run(et1, et2, tstop):
  pc.set_maxstep(10)
  h.stdinit()
  h.cvode.event(et1, (rank0decide,et2))
  pc.psolve(tstop)
  print ("%d et1=%g et2=%g tstop=%g h.t=%g" % (rank, et1, et2, tstop, h.t))

run(2, 20, 10)
run(2, 6, 10)

pc.barrier()
h.quit()
Michaels-MacBook-Air:tmp hines$ mpiexec -n 3 nrniv -mpi -python temp.py
numprocs=3
NEURON -- VERSION 7.5 master (2ae566b) 2017-01-23
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2016
See http://neuron.yale.edu/neuron/credits

0 rank0decide x.x[0] = 20 h.t=2
1 rank0decide x.x[0] = 20 h.t=2
2 rank0decide x.x[0] = 20 h.t=2
0 et1=2 et2=20 tstop=10 h.t=10
1 et1=2 et2=20 tstop=10 h.t=10
2 et1=2 et2=20 tstop=10 h.t=10
0 rank0decide x.x[0] = 6 h.t=2
1 rank0decide x.x[0] = 6 h.t=2
2 rank0decide x.x[0] = 6 h.t=2
0 et1=2 et2=6 tstop=10 h.t=6
1 et1=2 et2=6 tstop=10 h.t=6
2 et1=2 et2=6 tstop=10 h.t=6
Michaels-MacBook-Air:tmp hines$
iraikov
Posts: 17
Joined: Wed Mar 18, 2015 11:53 am
Location: University of California, Irvine

Re: Modifying tstop in FInitializeHandler

Post by iraikov »

Oh I see, I do seem to recall that the previous HOC code would actually sometimes hang if tstop was modified to an earlier time. But if rank 0 decides that tstop should be decreased, how can it tell the other ranks it intends to do a broadcast? It seems that broadcasting tstop at each dt_status timestep would incur unnecessary communication costs... Is there something more clever we could do here?
hines wrote:By the way, in an mpi context with nhost>1 all ranks must participate in a cvode.event with the same delivery time and
rank 0 should figure out the step event time and then broadcast to all the other ranks which then all do a
cvode.event(et .. i.e. all ranks have to stop at the same time or spike exchange will hang.
Post Reply