Efficient way of debugging Python scripts with MPI library

General issues of interest both for network and
individual cell parallelization.

Moderator: hines

Post Reply
itaru
Posts: 14
Joined: Wed Jan 29, 2020 10:15 pm
Contact:

Efficient way of debugging Python scripts with MPI library

Post by itaru »

Hi,

If someone knows an efficient way of debugging a Python script (with MPI library enabled) on a parallel system, let me know.

Thanks,
Itaru.
ted
Site Admin
Posts: 5886
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Efficient way of debugging Python scripts with MPI library

Post by ted »

General suggestions that you probably already know:
1. Do as much development/debugging as possible on a multicore desktop machine. Defer executing code on specialized parallel hardware until absolutely necessary.
2. Write code that will produce the same results regardless of the number of hosts on which it is executed. If the code is a model specification, implement it it so that it will produce the same results regardless of which cells are being simulated by any given host. If you are dealing with a network model, include a procedure that reports the model's connectivity matrix (including connection weights and delays) after model setup is complete.
3. If the script involves stochasticity, make sure that you have explicit control the pseudorandom number generator (RNG). In particular, the RNG's "seed" (or "sequence parameters") should be explicitly specified by you, not taken from the date, time of day, number of runs that have been executed etc.. Verify that each host is using the correct seed/sequence parameters.
itaru
Posts: 14
Joined: Wed Jan 29, 2020 10:15 pm
Contact:

Re: Efficient way of debugging Python scripts with MPI library

Post by itaru »

Hi Ted,
I take all of your suggestions. I kept adding an exit() call to check where MPI library starts to failing, but took a while
to understand that I was accessing a cell that is not associated with the rank it was made; I was wishing some handy tool
like a debugger.

Itaru.
ted
Site Admin
Posts: 5886
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Efficient way of debugging Python scripts with MPI library

Post by ted »

My suggestions pertain to development and debugging of NEURON models. It looks like your question is centered on MPI itself or NEURON's internals.
hines
Site Admin
Posts: 1619
Joined: Wed May 18, 2005 3:32 pm

Re: Efficient way of debugging Python scripts with MPI library

Post by hines »

I was accessing a cell that is not associated with the rank it was made
This wording puzzles me as I don't know anyway for a cell not to be associated with the rank it was made on. (although if you are using multisplit there may be parts of a cell that are on ranks that are different from the rank associated with the gid of the cell's spike output location)
Post Reply