Segmentation faults on Bluegene/P

General issues of interest both for network and
individual cell parallelization.

Moderator: hines

Post Reply
biswa
Posts: 11
Joined: Tue Jun 07, 2005 4:29 pm

Segmentation faults on Bluegene/P

Post by biswa » Thu Nov 22, 2012 1:30 pm

I have been trying to install Neuron 7.3 on a Bluegene/P machine using the two step process that I had earlier used for the compilation of Neuron on a Cray XE6 machine. The specific steps are:

Code: Select all

./configure --prefix=`pwd` --enable-bluegeneP --with-nmodl-only
followed by make, make install and

Code: Select all

./configure --prefix=`pwd` --enable-bluegeneP --without-nmodl --with-paranrn MPICC=mpixlc MPICXX=mpixlcxx CC=mpixlc CXX=mpixlcxx --host=powerpc64
(with and without thread-safe compilers) again followed by make and make install.

Everything works great without the usual configure or build failure hurdles. As I call the nrnivmodl (using absolute paths so as to avoid relative path issues on Bluegene), the mod files are compiled successfully as well. The problem occurs as soon as I submit the jobs using qsub and the associated cobalt-mpirun. The error logs throws the following error:

Code: Select all

<Nov 22 17:54:58.054533> BE_MPI (ERROR): The error message in the job record is as follows:
<Nov 22 17:54:58.054584> BE_MPI (ERROR):   "killed with signal 6"
This generally means that the executable is bigger than the memory available for each process. Given that I have 512 MB/process and my executable is roughly 3 MB, this can not be the true source of my problem. Moreover, if I were to use 2GB/process using a SMP, the segmentation fault still persists (regardless of thread-safe compilers). Makes me believe that there is something uncanny that I have done during the installation's configure step. STDOUT ejects the following segmentation errors:

Code: Select all

NEURON -- VERSION 7.3 (744:d0b867b1a597) 2012-11-20
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2012
See http://www.neuron.yale.edu/credits.html

Additional mechanisms from files
kv.mod
/test/powerpc64/special: Segmentation violation
5  near line 0

hines
Site Admin
Posts: 1574
Joined: Wed May 18, 2005 3:32 pm

Re: Segmentation faults on Bluegene/P

Post by hines » Fri Nov 23, 2012 8:09 am

I'm guessing the problem is --enable-bluegeneP in the first step. That sets the compiler to
mpixlc and mpixlcxx (see nrn/m4/nrnbg.m4)
When I build on a BG/P, I use:

Code: Select all

if test "$sdir" = "" ; then
  sdir=/home/hines/neuron/nrn
fi

$sdir/configure --prefix=`pwd` \
 --without-x --with-nmodl-only
make -j 4 install

$sdir/configure --prefix=`pwd` \
 --enable-bluegeneP --with-paranrn \
 --host=powerpc64-suse-linux
make -j 4 install

hines
Site Admin
Posts: 1574
Joined: Wed May 18, 2005 3:32 pm

Re: Segmentation faults on Bluegene/P

Post by hines » Fri Nov 23, 2012 8:24 am

On a BG/P the option --enable-nrnpython also works. For a BG/Q I build with:

Code: Select all

../nrn/configure --prefix=$IDIR --without-x --with-nmodl-only
make install

../nrn/configure --prefix=$IDIR --enable-bluegeneQ \
  --with-paranrn --with-nrnpython \
  --host=powerpc64-unknown-linux-gnu

make -j 4 install
However, I also found it necessary to make the following patch to allow python to work on
the particular machine I used.

Code: Select all

diff -r 2a5b8e5c32a5 -r 9fbb2f7bbe9b src/nrnpython/nrnpython.cpp
--- a/src/nrnpython/nrnpython.cpp	Thu Nov 15 09:30:38 2012 -0500
+++ b/src/nrnpython/nrnpython.cpp	Fri Nov 23 14:22:03 2012 +0100
@@ -91,6 +91,7 @@
 	static int started = 0;
 //	printf("nrnpython_start %d started=%d\n", b, started);
 	if (b == 1 && !started) {
+		Py_NoSiteFlag = 1;
 		p_nrnpy_pyrun = nrnpy_pyrun;
 		Py_Initialize();
 		started = 1;


Post Reply