Page 1 of 1

Reinforcement Learning with NetPyNE: a closed-loop with external stimulation

Posted: Sat Jul 18, 2020 4:01 pm
by ziemek
Hi there,

After a great tutorial of NetPyNE done by Salvador Dura I wanted to extend my question from the chat :)

I would like to make a simulation where I will stimulate a network in a closed-loop with external stimulation similar to Reinforcement Learning paradigm.

So let's say I have:
  • The network called the Agent
  • The Agent is implemented in NetPyNE
  • I would like to to simulate the network, on demand in 2 ways: 1) as a regular run for n ms; 2) as the observation run for n ms, during which the Agent receives an external stimulation
  • I would like to collect spikes from the output neurons

eg. pseudocode:

Code: Select all

# input (10,10) matrix which means 100 input cells, output -> 2 cells
agent = NetPyNEAgent(input_shape=(10,10), output_shape=2)

while True:
    obs = environment.get_observation()
    # make a external stimulation and run simulation for 100 ms
    agent.observe(observation=obs, time=100)
    # run the agent just for 100 ms without stimulation

    # number of output spike 
    spikes = agent.get_output_spikes()
I have 2 problems here:
  • The method observe() is the most critical one, since I don't know how to make an external stimulation in NetPyNE
  • I have seen only NetPyNE examples where simulation were entirely defined and run without interruption. I don't know how to make run(), stim() and another run()
Any simple code example would be sufficient for me to just get started.

Re: Reinforcement Learning with NetPyNE: a closed-loop with external stimulation

Posted: Mon Jul 20, 2020 10:53 am
by salvadord
Hi Ziemowit,

Thanks for your email and interest in the tool! Yes, everything you mention is possible in netpyne and we actually have a closely related example, where we control a simple virtual arm using RL and STDP: ... les/RL_arm

Please have a look at the example and if you have any particular questions feel free to post them in the forum (or in this other forum, or we could also have a quick videochat