Having python read from a NEURON binary file

When Python is the interpreter, what is a good
design for the interface to the basic NEURON
concepts.

Moderator: hines

Post Reply
Annik
Posts: 27
Joined: Fri Sep 07, 2012 1:02 pm

Having python read from a NEURON binary file

Post by Annik » Mon May 05, 2014 5:02 pm

Hi, apologies if this is the incorrect place for this post.
I've been having a bit of trouble with python correctly reading a binary file output by neuron. Neuron records the data as follows:

Code: Select all

strdef outdir, fprefix
outdir  = "./data"
fprefix = "test"
strdef filename
objref runfile
runfile = new File()
sprint(filename,"%s/%s.dat",outdir,fprefix) 
		runfile.wopen(filename)
		runfile.seek(0) 
// for each section, output header and data
	//output somatic section header
runfile.printf("%s", "soma")       // sec_list_name
runfile.printf("%d", 1)        // num_sections
I have written a bit of python code to read the binary file,

Code: Select all

def read_list_header(self):
	sec_list_name = self.file.read(4)
	self.file.seek(4,1)
	num_sections = struct.unpack('i', self.file.read(4))[0] 
	self.file.seek(4,1)
	return (sec_list_name, num_sections)
sec_list_name is correct; "soma" is printed to the screen. num_sections should be 1, but it spits out 1024.

I'm not sure if the issue is with Neuron storing this value, or with python retrieving it.

ted
Site Admin
Posts: 5645
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Having python read from a NEURON binary file

Post by ted » Mon May 05, 2014 9:09 pm

This has nothing to do with "binary" files. The hoc code writes a plain text file. You can examine its contents just as you would any other plain text file. That will tell you whether the problem lies in the hoc code or in the python code.

Annik
Posts: 27
Joined: Fri Sep 07, 2012 1:02 pm

Re: Having python read from a NEURON binary file

Post by Annik » Tue May 06, 2014 11:58 am

Sorry, I should have given more information. The hoc code continues as follows:

Code: Select all

			
//output somatic section subheader
runfile.printf("%s", "soma")
runfile.printf("%d", 1)
for i = 0,soma_listv.count()-1 {
	 for j = 0,soma_listv.o(i).count()-1 {
		 soma_listv.o(i).o(j).vwrite(runfile)
		 soma_listicl.o(i).o(j).vwrite(runfile)
		 soma_listcli.o(i).o(j).vwrite(runfile)
	}
}

//output axonal section subheader
	runfile.printf("%s", "axon")
	runfile.printf("%d", 1)

for i = 0,axon_listv.count()-1 {
	 for j = 0,axon_listv.o(i).count()-1 {
	        axon_listv.o(i).o(j).vwrite(runfile)
		axon_listicl.o(i).o(j).vwrite(runfile)
		axon_listcli.o(i).o(j).vwrite(runfile)
	 }
}

// etc....
I thought vwrite write the vector in binary format to an already opened fileobj, in this case test.dat, so I assumed this was all going into a binary file. Is this not the case?

ted
Site Admin
Posts: 5645
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Having python read from a NEURON binary file

Post by ted » Tue May 06, 2014 12:39 pm

I thought vwrite write the vector in binary format to an already opened fileobj, in this case test.dat, so I assumed this was all going into a binary file. Is this not the case?
See for yourself if it did. Try examining the contents of the file. If you're using UNIX/Linux/OS X, use
od -a filename | more
If you're using MSWin and don't have anything like od, try opening the file with a text editor.

It may help to know that all hoc variables are double precision floating point (see 12.4.3 Variables starting on page 353 of The NEURON Book).

Annik
Posts: 27
Joined: Fri Sep 07, 2012 1:02 pm

Re: Having python read from a NEURON binary file

Post by Annik » Thu May 08, 2014 11:03 am

It is indeed a binary file. If I run the python code with

Code: Select all

def read_list_header(self):
   sec_list_name = self.file.read(4)
   self.file.seek(4,1)
   num_sections = struct.unpack('d', self.file.read(8))[0] 
   self.file.seek(8,1)
   return (sec_list_name, num_sections)

Code: Select all

(sec_name, num_sections) = self.read_list_subheader()
print sec_name
print num_sections
I get

Code: Select all

soma
5.05923221341e-321
Something is up by I don't know where to go from here.

ted
Site Admin
Posts: 5645
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Having python read from a NEURON binary file

Post by ted » Thu May 08, 2014 11:48 am

First, a question: are you quite sure that the actual advantages of using binary files (in theory, speed of writing/reading data, reduced file size) for your particular problem will be so great that it's worth all the effort you're putting into implementing binary file i/o? Not to mention the future problems that you're creating for yourself and your collaborators ("Look at all these binary files--does anybody remember the file format? Too bad that Excel/SigmaPlot/whatever can't read these things.").

If you are still determined to use binary files, you need to create a binary "test" file whose contents are known. If you can't do that, you're sunk. If you try to write the test file with hoc, make sure you can read it with hoc.

Once you have a file whose contents are known, you need to execute this loop
REPEAT
try to read the file with your Python code
if the results aren't correct, revise your Python code
UNTIL the results are correct

Annik
Posts: 27
Joined: Fri Sep 07, 2012 1:02 pm

Re: Having python read from a NEURON binary file

Post by Annik » Thu May 08, 2014 12:02 pm

I'm somewhat new to this, so I was only using a binary file because I was using vwrite. I'm certainly not set on it. How can I get neuron to record data in another file type?

ted
Site Admin
Posts: 5645
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:

Re: Having python read from a NEURON binary file

Post by ted » Thu May 08, 2014 1:17 pm

The File class's printf is very useful for writing strings and numerical values. If you are looking for ideas about file formats, here are a couple of simple ones.

This is generated by proc clipboard_save() in nrn/share/lib/hoc/stdlib.hoc, and would be easy to implement in your own hoc code.

Code: Select all

label:cai*1e4           << name of the dependent variable
201                        << number of time val pairs
0       0.03               << time0  value0
0.025   0.0296997
 . . . etc. . . .
This is generated from a graph that shows plots of 5 different variables, by the Print & File Window Manager's "Print / Ascii" feature. The first line doesn't seem real useful to me, but the rest of it would be easy to implement with a bit of hoc.

Code: Select all

Graph addvar/addexpr lines
5 201
x ica ica_nacax ica_capump ica_cachan ica_pmp_cadifpmp
0       -0.000422814    -0.00028025     0       -0.000142564    0
0.025   -0.000422814    -0.00028025     0       -0.000142564    0.00083607
 . . . etc. . . .
It is possible to be much more elaborate, e.g. including metadata such as date and time, "experiment number" or even the values of key parameters. But simplicity has advantages (easier to develop, debug, and use with other programs).

Post Reply