Open MPI error with h.nrnmpi_init(): mca_base_component_repository_open: unable to open mca_patcher_overwrite

Post Reply
tony-azevedo
Posts: 4
Joined: Fri May 29, 2020 9:09 am

Open MPI error with h.nrnmpi_init(): mca_base_component_repository_open: unable to open mca_patcher_overwrite

Post by tony-azevedo »

Hi. I have a bedeviliing issue setting up parallelism in neuron on linux (Ubuntu 18.04.4). When I run h.nrnmpi_init(), I get this error:

Code: Select all

mca_base_component_repository_open: unable to open mca_patcher_overwrite: /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi/mca_patcher_overwrite.so: undefined symbol: mca_patcher_base_patch_t_class (ignored).
 
I have found this commit to git hub by nrnhines, https://github.com/neuronsimulator/nrn/ ... d7d4aa690b, which in turn links to this page https://github.com/open-mpi/ompi/issues/3705. If I understand correctly from these issue pages, then dynamically loading some of the mpi libraries is failing. Steps that I have taken to get this far:
  • Building neuron from source (./configure --with-iv --with-paranrn --with-nrnpython=python3)
  • Compiling openmpi (v4.0.3) with configure/make: ./configure --prefix=/usr/local --disable-dlopen --disable-mca-dso
I configured openmpi with those options to try to get around the dynamic loading issue, as described in the openmpi FAQs here https://www.open-mpi.org/faq/?category= ... #avoid-dso. The ompi issue page https://github.com/open-mpi/ompi/issues/3705 has the following script, named ompi-dlopen.py, to test whether MPI_Init() is working

Code: Select all

import ctypes
libdir = "/usr/local/lib/"

lib = ctypes.CDLL(libdir+"libmpi.so", ctypes.RTLD_LOCAL)
ierr = lib.MPI_Init(None,None)
assert ierr==0

r = ctypes.create_string_buffer(4096)
n = ctypes.c_int()
ierr = lib.MPI_Get_library_version(r, ctypes.byref(n))
assert ierr==0
print(r[0:n.value])

ierr = lib.MPI_Finalize()
assert ierr==0
Output from "python ompi-dlopen.py" is:

Code: Select all

b'Open MPI v4.0.3, package: Open MPI tony@tony-desktop Distribution, ident: 4.0.3, repo rev: v4.0.3, Mar 03, 2020\x00'
So it's working, and finding the correct version of open mpi (v4.0.3). This suggests to me that I shouldn't need to run the patch suggested in that issue resolution, that nrnhines refers to in his git hub commit. However, the issue still seems to be that the wrong libraries are found, since the error message above points to /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi/mca_patcher_overwrite.so, which has to be the preinstalled open mpi libraries.

Any suggestions you have would be helpful. A couple things I'm considering trying are 1) re-making openmpi without the options and then using the patch from the ompi git hub issue, and 2) building neuron with --with-paranrn=dynamic as nrnhines mentions. Would #2 matter?

Thanks, I'm stuck!
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Open MPI error with h.nrnmpi_init(): mca_base_component_repository_open: unable to open mca_patcher_overwrite

Post by hines »

Let's see if this problem disappears with a cmake build. If you don't have cmake, then

Code: Select all

sudo apt install cmake
Then with a fresh nrn repository clone

Code: Select all

git clone https://github.com/neuronsimulator/nrn nrncmake
cd nrncmake
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=install
make -j install
export PATH=`pwd`/install/bin:$PATH
mpiexec -n 2 nrniv -mpi ../src/parallel/test0.hoc
We can continue from there if necessary. Do you have two installations of openmpi?
tony-azevedo
Posts: 4
Joined: Fri May 29, 2020 9:09 am

Re: Open MPI error with h.nrnmpi_init(): mca_base_component_repository_open: unable to open mca_patcher_overwrite

Post by tony-azevedo »

Ha! That worked! Output from 'mpiexec -n 2 nrniv -mpi ../src/parallel/test0.hoc':

Code: Select all

numprocs=2
NEURON -- VERSION 8.0.dev-109-g6d23a393 master (6d23a393) 2020-06-02
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits

I am 1 of 2
I am 0 of 2
Thank you! I guess that I should have rebuilt neuron after installing open mpi anyways? Maybe that would have helped. What steps should I now take to have a single version of neuron? Thanks for your help!
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Open MPI error with h.nrnmpi_init(): mca_base_component_repository_open: unable to open mca_patcher_overwrite

Post by hines »

I guess that I should have rebuilt neuron after installing open mpi anyways?
Building from sources with mpi enabled requires a pre-existing version of mpi, ie openmpi or mpich. Installing a binary distribution which we built
with --DNRN_ENABLE_MPI_DYNAMIC=ON does not require an mpi on the user machine, but you can subsequently install mpi and then it will work. The only requirement is that the version of mpi that you install on the user machine must be one of the versions of mpi that we dynamically built against.
Generally that has been openmpi, but we are in the process of extending to openmpi or mpich.
What steps should I now take to have a single version of neuron?
It is not necessary that you have a single version of neuron. Just have PATH and PYTHONPATH point to your desired installation. ie.

Code: Select all

export PATH=<CMAKE_INSTALL_PREFIX>/bin;$PATH
export PYTHONPATH=<CMAKE_INSTALL_PREFIX>/lib/python;$PYTHONPATH
If you wish to have only a single installation, then remove all the files installed by your other installations and rebuild from sources one more time with your desired CMAKE_INSTALL_PREFIX>
tony-azevedo
Posts: 4
Joined: Fri May 29, 2020 9:09 am

Re: Open MPI error with h.nrnmpi_init(): mca_base_component_repository_open: unable to open mca_patcher_overwrite

Post by tony-azevedo »

The directions for pointing to the correct neuron installation make sense.
Thank you for the guidance on the openmpi issue. I'm still a little unclear, was the --DNRN_ENABLE_MPI_DYNAMIC=ON option part of the build I just did? I should have said that I installed openmpi v4.0.3, over an older version installed with Ubuntu, but the output from cmake included a line indicating /usr/local/lib/libmpi.so, where the new version is installed.
In any case, it worked. Onwards. Thanks again!
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Open MPI error with h.nrnmpi_init(): mca_base_component_repository_open: unable to open mca_patcher_overwrite

Post by hines »

was the --DNRN_ENABLE_MPI_DYNAMIC=ON option part of the build I just did?
No. Default of NRN_ENABLE_MPI_DYNAMIC is OFF. Dynamic MPI and PYTHON (and InterViews) is most useful for binary installations where one does not
a priori know if mpi or python (or XQuartz for mac) is installed at all on the user machine, or what versions of those are installed.
I installed openmpi v4.0.3, over an older version installed with Ubuntu
It may be best at this point to let sleeping dogs lie. I am semi-surprised that your second installer (from openmpi.org? or did you install from sources?)
overwrote the ubuntu installed version (via sudo apt install openmpi?) The latter generally installs in /usr and there is a database of what is installed so that one can sudo apt remove... The former by default generally installs in /usr/local.
tony-azevedo
Posts: 4
Joined: Fri May 29, 2020 9:09 am

Re: Open MPI error with h.nrnmpi_init(): mca_base_component_repository_open: unable to open mca_patcher_overwrite

Post by tony-azevedo »

Yeah, you're right, I'm going to just proceed without worrying about openmpi anymore. Sorry, what I said wasn't clear, openmpi was not installed "over" the old version, that is still there in /usr/bin (e.g. old open_info is there). I installed openmpi from sources (a tarball from from openmpi.org), with '--prefix=/usr/local', so that is where v4.0.3 is. But I did not try to remove the older version.

But everything seems to be working. Thank again.
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Open MPI error with h.nrnmpi_init(): mca_base_component_repository_open: unable to open mca_patcher_overwrite

Post by hines »

openmpi was not installed "over" the old version
That is better. One can safely update or remove the system version with apt and one can safely update your source compiled version without them
interfering with each other. From the perspective of a NEURON build, one by default always gets the first one in the PATH.
Post Reply