possible memory leak with random in python3

When Python is the interpreter, what is a good
design for the interface to the basic NEURON
concepts.

Moderator: hines

Post Reply
lneisenman
Posts: 20
Joined: Wed Dec 16, 2009 10:26 am

possible memory leak with random in python3

Post by lneisenman » Fri Nov 22, 2013 10:43 am

In debugging a batch simulation, I ran into what looks like a memory leak in python3 with the randomstream object from Brette 2007. Attached is a simple example to illustrate the issue. The leak does not seem to be present in python2. I'm using Ubuntu 12.04 64bit running in virtualbox on a Win7 64 bit machine. Python3.3 is from the deadsnakes PPA

Here is the output in python2:
NEURON -- Release 7.3 (980:01f9410e41bc) 2013-10-31
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2013
See http://www.neuron.yale.edu/neuron/credits

Python version 2.7.3
memory 21.504
memory 22.032
memory 22.032
memory 22.032
memory 22.032
memory 22.032
memory 22.032
memory 22.032
memory 22.032
memory 22.032
all done

and in python3:
NEURON -- Release 7.3 (980:01f9410e41bc) 2013-10-31
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2013
See http://www.neuron.yale.edu/neuron/credits

Python version 3.3.3
memory 55.464
memory 87.408
memory 118.56
memory 149.976
memory 181.392
memory 212.808
memory 244.224
memory 275.64
memory 307.056
memory 338.472
all done


Here is the code:

Code: Select all

# -*- coding: utf-8 -*-
"""
demonstrate py3 memory leak
Use ranstream from Brette et al 2007
"""

from __future__ import print_function
from __future__ import division

import resource
import sys
from neuron import h


h.load_file("ranstream.hoc")    # load the randomstream object


class Test():
    """
    simple list of ranstreams
    """

    def __init__(self, N=1000):
        self.N = N
        self.ran_list = list()
        for i in range(self.N):
            self.ran_list.append(h.RandomStream(i))

        self.use_streams()

    def use_streams(self):
        """
        pick some random numbers from each stream
        """
        h.mcell_ran4_init(1)
        for rs in self.ran_list:
            rs.start()
            rs.r.uniform(0, 1)
            for i in range(self.N):
                rs.repick()


if __name__ == '__main__':

    print('Python version', sys.version.split()[0])
    for i in range(10):
        net = Test()
        mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1000
        print('memory', mem)

    print("all done")
And here is my RandomStream.hoc

Code: Select all

// random_stream_offset_ = 1000
// to match the statement in Brette /commmon/init.hoc
random_stream_offset_ = 500

begintemplate RandomStream
public r, repick, start, stream
external random_stream_offset_
objref r
proc init() {
	stream = $1
	r = new Random()
	start()
}
func start() {
	return r.MCellRan4(stream*random_stream_offset_ + 1)
}
func repick() {
	return r.repick()
}
endtemplate RandomStream

lneisenman
Posts: 20
Joined: Wed Dec 16, 2009 10:26 am

Re: possible memory leak with random in python3

Post by lneisenman » Fri Dec 20, 2013 9:48 am

Replacing MCellRan4 with Random123 does not eliminate the leak so it is likely something related to the python3 interface (e.g. http://www.neuron.yale.edu/phpBB/viewto ... 818#p11360) rather than MCellRan4.

Larry

hines
Site Admin
Posts: 1575
Joined: Wed May 18, 2005 3:32 pm

Re: possible memory leak with random in python3

Post by hines » Thu Dec 26, 2013 2:24 pm

I believe this is fixed in
http://www.neuron.yale.edu/hg/neuron/nr ... 017ff66654
The interface used PyString_AsString many times to access the char* buffer but this no longer exists in Python-3 and instead a more elaborate process needed (?) to be
carried out which involved the possibility of PyUnicode_AsASCIIString followed by PyBytes_AsString followed by a copy into char* buffer and return the buffer.
Unfortunately I failed to free the buffer when the caller no longer needed it and hence the memory leak. I wish I knew a better way to do this without a char* copy.

lneisenman
Posts: 20
Joined: Wed Dec 16, 2009 10:26 am

Re: possible memory leak with random in python3

Post by lneisenman » Wed Jan 08, 2014 2:43 pm

Sorry it took so long to respond. The memory leak appears to be fixed. However, I now get errors from trying to use references to record time in a vector.

Larry

Code: Select all

Python 3.3.3 (default, Dec 27 2013, 19:27:19) 
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from neuron import h
NEURON -- Release 7.3 (902:aa017ff66654) 2013-12-26
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2013
See http://www.neuron.yale.edu/neuron/credits

>>> vec = h.Vector()
>>> vec.record(h._ref_t)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'hoc.HocObject' object has no attribute '_ref_t'

hines
Site Admin
Posts: 1575
Joined: Wed May 18, 2005 3:32 pm

Re: possible memory leak with random in python3

Post by hines » Wed Jan 08, 2014 9:19 pm

Darn. The python-3 memory leak fix introduced that bug. Pushed the fix for the fix.
http://www.neuron.yale.edu/hg/neuron/nr ... 325edb3544

lneisenman
Posts: 20
Joined: Wed Dec 16, 2009 10:26 am

Re: possible memory leak with random in python3

Post by lneisenman » Fri Jan 10, 2014 10:18 am

That did the trick, thanks.

Larry

Post Reply