Page 1 of 1

Modifying tstop in FInitializeHandler

Posted: Thu Mar 02, 2017 10:54 am
by iraikov
Hello,

I am in the process of converting parallel network code from HOC to Python. The HOC code installs a handler via FInitializeHandler that compares the elapsed wall time with the maximum time for the job, estimates whether there is sufficient time to complete the simulation, and reduces tstop if necessary (code below). The HOC version of the code used psolve and it seemed that whenever the handler modified tstop, psolve accordingly only integrated until that new value of tstop. But it looks like when calling psolve and FInitializeHandler from Python, but still using the HOC handler, psolve keeps using the original tstop value. What is the correct way to port this code? Should the handler routine be written in Python as well? Thanks!

Python invocation of FInitializeHandler:

Code: Select all

...
h('objref fi_status')
if pc.id() == 0:
  h.fi_status          = h.FInitializeHandler("simstatus()")
h.stdinit()
pc.psolve(h.tstop)
HOC FInitializeHandler routine:

Code: Select all

walltime = 0

dt_status = 1.0

proc simstatus() { local wt
    wt = startsw()
    if (walltime > 0) {
        printf("*** computation time at t=%g ms was %g s\n", t, wt-walltime)
        checksimtime(wt, wt-walltime)
    }
    walltime = wt
    if (t + dt_status < tstop) {
        cvode.event(t + dt_status, "simstatus()")
    }
}


tcsum = 0
tcma = 0
nsimsteps = 0

proc checksimtime() { local wt, tt, trem, tsimrem, tsimneeded, tstop1
    wt = $1
    tt = $2
    // cumulative moving average simulation time per time step
    tcma = tcma + (tt - tcma) / (nsimsteps + 1)
    tcsum = tcsum + tt
    // remaining physical time
    trem = tstop - t
    // remaining simulation time
    tsimrem = max_walltime_hrs*3600 - tcsum - mkcellstime - connectcellstime - connectgjstime
    // simulation time necessary to complete the simulation
    tsimneeded = trem*tcma+results_write_time
    printf("*** remaining computation time is %g s and remaining simulation time is %g ms\n", tsimrem, trem)
    printf("*** estimated computation time to completion is %g s\n", tsimneeded)
    if (tsimneeded > tsimrem) {
        tstop1 = int((tsimrem - results_write_time)/tt) + t
        printf ("*** not enough time to complete %g ms simulation, simulation will likely stop around %g ms\n", tstop, tstop1)
        tstop = tstop1
    }
    nsimsteps = nsimsteps + 1
 }


Re: Modifying tstop in FInitializeHandler

Posted: Fri Mar 03, 2017 12:05 pm
by ted
Given working hoc code that controls tstop, why bother reimplementing it in Python? What's wrong with just executing the hoc code from Python?

Re: Modifying tstop in FInitializeHandler

Posted: Fri Mar 03, 2017 1:42 pm
by iraikov
Hi Ted,

Thanks for your reply, but the point is precisely that calling the HOC code from Python does not produce the desired result. That is, the HOC handler routine modifies tstop, but psolve continues to integrate using the original value of tstop. So how can I can use the same HOC routine, but ensure that it can modify the value of tstop used by psolve? Thanks,

-Ivan

Re: Modifying tstop in FInitializeHandler

Posted: Fri Mar 03, 2017 1:50 pm
by hines
ParallelContext.psolve(stoptime) should do the same thing independent of whether it was called from hoc or python. The only thing that can
make that call return earlier than stoptime is if somebody sets h.stoprun = 1 (returns as soon as the present step completes) or if someone during
a simulation executes cvode.event(another_stop_time) which will return when delivered.
consider

Code: Select all

Michaels-MacBook-Air:tmp hines$ cat temp.py
from neuron import h
h.load_file("stdgui.hoc")
pc = h.ParallelContext()

s = h.Section()

def run(et, tstop):
  pc.set_maxstep(10)
  h.stdinit()
  h.cvode.event(et)
  pc.psolve(tstop)
  print ("et=%g tstop=%g h.t=%g" % (et, tstop, h.t))

run(20, 10)
run(6, 10)

Michaels-MacBook-Air:tmp hines$ nrniv -python temp.py
NEURON -- VERSION 7.5 master (2ae566b) 2017-01-23
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2016
See http://neuron.yale.edu/neuron/credits

et=20 tstop=10 h.t=10
et=6 tstop=10 h.t=6
>>> 

Re: Modifying tstop in FInitializeHandler

Posted: Fri Mar 03, 2017 2:12 pm
by iraikov
Hi Michael,

So what you are saying is that the original HOC code should not have worked in the first place. I will modify my code to use cvode.event for the new stop time then. Thank you!
hines wrote:ParallelContext.psolve(stoptime) should do the same thing independent of whether it was called from hoc or python. The only thing that can
make that call return earlier than stoptime is if somebody sets h.stoprun = 1 (returns as soon as the present step completes) or if someone during
a simulation executes cvode.event(another_stop_time) which will return when delivered.
consider

Code: Select all

Michaels-MacBook-Air:tmp hines$ cat temp.py
from neuron import h
h.load_file("stdgui.hoc")
pc = h.ParallelContext()

s = h.Section()

def run(et, tstop):
  pc.set_maxstep(10)
  h.stdinit()
  h.cvode.event(et)
  pc.psolve(tstop)
  print ("et=%g tstop=%g h.t=%g" % (et, tstop, h.t))

run(20, 10)
run(6, 10)

Michaels-MacBook-Air:tmp hines$ nrniv -python temp.py
NEURON -- VERSION 7.5 master (2ae566b) 2017-01-23
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2016
See http://neuron.yale.edu/neuron/credits

et=20 tstop=10 h.t=10
et=6 tstop=10 h.t=6
>>> 

Re: Modifying tstop in FInitializeHandler

Posted: Fri Mar 03, 2017 2:16 pm
by hines
By the way, in an mpi context with nhost>1 all ranks must participate in a cvode.event with the same delivery time and
rank 0 should figure out the step event time and then broadcast to all the other ranks which then all do a
cvode.event(et .. i.e. all ranks have to stop at the same time or spike exchange will hang.

Re: Modifying tstop in FInitializeHandler

Posted: Fri Mar 03, 2017 2:31 pm
by hines
As an example of what I'm talking about consider:

Code: Select all

Michaels-MacBook-Air:tmp hines$ cat temp.py
from neuron import h
h.load_file("stdgui.hoc")
pc = h.ParallelContext()
rank = int(pc.id())
nhost = int(pc.nhost())

def rank0decide(et):
  x = h.Vector(1)
  if rank == 0:
    x.x[0] = et
  pc.broadcast(x, 0)
  print ("%d rank0decide x.x[0] = %g h.t=%g" % (rank, x.x[0], h.t))
  h.cvode.event(x.x[0]) # the stop event


def run(et1, et2, tstop):
  pc.set_maxstep(10)
  h.stdinit()
  h.cvode.event(et1, (rank0decide,et2))
  pc.psolve(tstop)
  print ("%d et1=%g et2=%g tstop=%g h.t=%g" % (rank, et1, et2, tstop, h.t))

run(2, 20, 10)
run(2, 6, 10)

pc.barrier()
h.quit()
Michaels-MacBook-Air:tmp hines$ mpiexec -n 3 nrniv -mpi -python temp.py
numprocs=3
NEURON -- VERSION 7.5 master (2ae566b) 2017-01-23
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2016
See http://neuron.yale.edu/neuron/credits

0 rank0decide x.x[0] = 20 h.t=2
1 rank0decide x.x[0] = 20 h.t=2
2 rank0decide x.x[0] = 20 h.t=2
0 et1=2 et2=20 tstop=10 h.t=10
1 et1=2 et2=20 tstop=10 h.t=10
2 et1=2 et2=20 tstop=10 h.t=10
0 rank0decide x.x[0] = 6 h.t=2
1 rank0decide x.x[0] = 6 h.t=2
2 rank0decide x.x[0] = 6 h.t=2
0 et1=2 et2=6 tstop=10 h.t=6
1 et1=2 et2=6 tstop=10 h.t=6
2 et1=2 et2=6 tstop=10 h.t=6
Michaels-MacBook-Air:tmp hines$

Re: Modifying tstop in FInitializeHandler

Posted: Fri Mar 03, 2017 2:32 pm
by iraikov
Oh I see, I do seem to recall that the previous HOC code would actually sometimes hang if tstop was modified to an earlier time. But if rank 0 decides that tstop should be decreased, how can it tell the other ranks it intends to do a broadcast? It seems that broadcasting tstop at each dt_status timestep would incur unnecessary communication costs... Is there something more clever we could do here?
hines wrote:By the way, in an mpi context with nhost>1 all ranks must participate in a cvode.event with the same delivery time and
rank 0 should figure out the step event time and then broadcast to all the other ranks which then all do a
cvode.event(et .. i.e. all ranks have to stop at the same time or spike exchange will hang.