Numpy?

When Python is the interpreter, what is a good
design for the interface to the basic NEURON
concepts.

Moderator: hines

hines
Site Admin
Posts: 1576
Joined: Wed May 18, 2005 3:32 pm

Re: Numpy?

Post by hines » Thu Jul 03, 2008 6:16 pm

--enable-numpy is supposed to be valid in order to get slightly faster performance.
However that implementation is coming in the future and for now I would leave
it out. The present implementation is reasonably fast though vectors are being
copied between numpy and the hoc Vector.

sec6
Posts: 39
Joined: Sat Mar 15, 2008 10:32 am

Re: Numpy?

Post by sec6 » Tue Aug 05, 2008 6:41 pm

For large arrays, copying between numpy and the hoc Vector is not only time-inefficient, it is also memory inefficient. (There are, at least transiently, two copies of the same data in memory, one as numpy.array, the other as hoc Vector.) Eventually, I hope, Vector.from_python() will allow hoc Vectors to share memory with numpy arrays which would eliminate both types of inefficiency.

Until then, here is a proposed feature which would sometimes avoid the memory-inefficiency: How about allowing a Python generator to be the argument of from_python().

1) There's no question that this would be a feature of a hypothetical utopian future totally-Pythonic version of NEURON.
2) In some circumstances this would avoid keeping two copies of data in memory. For example, suppose the generator reads data from a file on disk, or computes a function f(x) over a range of values of x.
3) I don't know much about the internals of Python or hoc, but it seems like this might be easier to implement than memory sharing. I think from_python() just needs to check if its argument is a list or numpy.array, and then, if not, check if its argument has a next() method, and then, if so call the next() method repeatedly, and finally, catch a StopIteration exception.
4) This would allow any Python enumerable type to be copied into a hoc Vector, without first copying it into a list or numpy.array

Just write a generator expression:

Code: Select all

myVec.from_python((x for x in myEnumerableObject))
As it is, you have to write:

Code: Select all

myVec.from_python([x for x in myEnumerableObject])
Note () generator expression, rather than [] list expansion. The list expansion generates a temporary, anonymous list, which could be memory-expensive if myEnumerableObject is large.

hines
Site Admin
Posts: 1576
Joined: Wed May 18, 2005 3:32 pm

Re: Numpy?

Post by hines » Tue Aug 05, 2008 7:50 pm

Just write a generator expression:
You're right. I wasn't thinking of generators explicitly
and have never used them
but it should work for any object that allows the sequence
protocol. I'll look into generalizing to that case.
I believe the other direction works fine. ie

Code: Select all

import neuron
v = neuron.h.Vector(10).indgen().add(1)
[x for x in v] # [1.0, ..., 10.0]
y = (x for x in v) # y is a generator object

hines
Site Admin
Posts: 1576
Joined: Wed May 18, 2005 3:32 pm

Re: Numpy?

Post by hines » Wed Aug 06, 2008 11:13 am

I committed the iterator protocol addition for Vector.from_python
to the subversion repository. The performance is not bad; more toward the sequence protocol than an explicit loop in python.
Times below are in seconds for a 1GHz mswin laptop.
a = range(1000000) 0.06
v = neuron.Vector(a) 23.3
v = neuron.h.Vector()
v.from_python(a) 0.13
v.sum() 0.02
499999500000.0
l = v.to_python() 0.08
len(l)
1000000
no numpy
v.from_python(x for x in xrange(1000000)) 0.481
l = [x for x in v] 0.621

sec6
Posts: 39
Joined: Sat Mar 15, 2008 10:32 am

Re: Numpy?

Post by sec6 » Thu Aug 07, 2008 7:09 pm

Wow that was fast.
I realize, now, that, by making the suggestion, I've sort of incurred a moral obligation to download the latest source & try it out. I'm embarrassed to say that it's going to take me longer to do that than it took to implement the feature.

Post Reply