A simulated neuro-robotic environment for bi-directional closed-loop experiments

Abstract We have developed a new simulation environment, called NeuVision, that is able to perform neuro-robotic experiments in a closed-loop architecture, by simulating a large-scale neuronal network bi-directionally connected to a robot. We conceived it primarily as a support tool to be used in the context of the ‘embodied electrophysiology’, a growing field that could help, in the future, to realize innovative bi-directional and adaptive Brain-Machine Interfaces. The main features of our system are related to the efficient visualization of the neural activity, the possibility to define different connectivity rules and stimulation points, and the integration of statistical analysis tools for fast neural dynamics characterization. Our preliminary results show that we are able to reproduce both spontaneous and evoked activity of cultured networks. Hence, by defining a decoding strategy based on the Center of Activity, we carried out experiments with a simulated neuronal network in a closed-loop with a robot. Our results suggest the NeuVision simulated environment could be used as a tool to support in vitro experiments with real systems.


Introduction
In recent years, there has been a growing recognition of the crucial role played by an animal's body and the surrounding environment in understanding the neural basis of behavior [1]. According to this view, behavior arises only through the interaction of neural activity, the body, and the environment, an idea whose roots go back to cybernetics [2].
Given the importance on the embodiment of a neural circuit and the ''situatedness'' of that body within an environmental context, it was quite natural to extend this approach to the electrophysiology field (i.e. embodied electrophysiology) [3,4]. More specifically, researchers have begun to explore the possibility to create new systems at the interface between neuroscience and robotics, with the primary aim at understanding the mechanisms of adaptive behaviors and neural coding. This has been done: at single neuron level, by interfacing artificial and actual neurons [5]; at a population level, by controlling the dynamic regime of neuronal populations [6], of its adaptive properties [7,8]; at the level of a ''kind of whole organism'', in experiments in which portions of nervous tissue are connected to artificial devices [9,10], virtual bodies [3], or to form artificial/hybrid ''animals'' [11,12]. Such bi-directional systems have often been referred as neurally controlled`animats', or hybrots'. In all these experiments, the underlying rationale is that the dynamic and adaptive properties of neural systems can be understood by looking at their interaction, in a bi-directional closed-loop, with the external environment. * E-mail: michela.chiappalone@iit.it, corresponding author Cultures of neurons dissociated from di erent districts of the brain (e.g. cortex, hippocampus, thalamus), with no a erent sensory inputs or efferent motor outputs, are a good candidate as experimental substrate for embodied electrophysiology [11,13]. A complex hw/sw system has to be implemented to support the execution of real neuro-robotic experimental protocols. The physical body provided to the culture could be any device able to explore an environment, like a small mobile robot equipped with sensors that collect information about the surrounding world. Starting from the results previously obtained by the members of our lab [12], we propose here an innovative simulated neuro-robotic environment, named NeuVision, to be specifically used in the framework of embodied electrophysiology (i.e. closed-loop experiments). The system is composed of two main software modules (i.e., one network and one robotic module) which can be independently programmed, giving the user the ability to define the experimental protocol that best suits their needs. The network module is able to reproduce the electrophysiological activity typically found in large-scale neuronal networks, such as primary cultures from the cortex of embryonic/post-natal rodents [14]. The implemented software provides great flexibility to define neuron models and connectivity rules as well as stimulation protocols, allowing the user to visualize the neural activity in both standard and novel ways. The robotic module was developed with the purpose of reproducing the sensor data of real commercial mobile robots. We refer in particular to the Khepera robot series by K-Team (Yverdon-les-bains, Switzerland), equipped with infra-red sensors to estimate obstacle distances. User-friendly graphical interfaces allow to easily build virtual worlds and di erent algorithms are available to pre-process sensory information. The NeuVision code is freely available and can be downloaded at the following web-site http://www.neuvisionsimulator.org under the GNU GPL license. According to our`new vision', the simulated and the real environment could contribute to the same experimental goal in a very synergistic way: the simulated network or the real biological network can communicate with either a real or a simulated robot, simply by exchanging the modules of the working environment. In this paper we present the first results related to the fully simulated neuro-robotic environment, which allows us to test hypotheses and experimental protocols that can then be implemented on a real system.

Neuro-robotic Simulator Architecture
The high level interactions among the components in a typical closedloop configuration are depicted in diagram A of Figure 1. During an experimental session, the robot moves inside an arena exploring the environment through its sensors, and thus generating a stream of sensory information. The sensory information is processed by the Sensory Information Processor which extracts the most suitable features which will serve for stimulating the network. The Stimulation Pattern Generator translates the sensory information in stimulation patterns, which will be received by the Neural Network. The electrical activity of the Neural Network is continuously recorded by the Neural Activity Recorder. The collected data are then processed by the Neural Activity Decoder, while the motor commands controlling the robot movements are generated by the Motor Command Generator, on the basis of the recorded neural activity.
NeuVision simulator has been implemented by interfacing di erent object-oriented languages, supported by the .NET Framework developed by Microsoft ® . More precisely, we used C# to write the simulation environment and the data analysis tools, and C++/CLI to run the neural network model in order to have a strict control on memory management and to optimize code performance. A diagram of the interactions between the main components of the neuro-robotic simulator is shown in Figure 1B, where we highlight the main data flows. The Neuro-robotic Simulator object manages the simulation progress iteration by iteration, and contains (i) the Robotic Module, (ii) the Stimulation Manager and (iii) the Neural Network Model. The user can control all three components by means of graphical user interfaces (GUIs) that allow configuring the parameters, and visualizing output data, illustrated more in depth in the next section. Each component is able to store and retrieve data from storage, such as user configuration files or processing outputs.
The Robotic Module manages all necessary aspects to simulate a robot and its environment. We have reproduced a Khepera II/III robot (K-Team, Yverdon-les-bains, Switzerland), which is the same device we use for performing real experiments with biological preparations. In the framework of the`embodied electrophysiology', we did not consider it important to realistically simulate the actual motor dynamics of the robot (taking into account, for example, the robot inertia), but simply to give our network a body, and thus be able to acquire information about the surrounding world without being a ected by noise or measure errors.
The sensory information, i.e., the distances between the obstacles and the robot as they are acquired by the sensors, are thus conveniently processed and provided to the Stimulation Manager. This module works as an interface between the Robotic Module and the Neural Network Model. It generates the stimulus patterns that code the sensory information. These patterns represent the input of the Neural Network Model which computes the status of the network as a consequence of the provided stimulation. Finally, the simulated neural activity is used by the Robotic Module to generate the motor commands for the robot.

Neuro-robotic Simulator Features
The neuro-robotic environment implemented within NeuVision o ers several features of interest from a computational point of view. We conceived it primarily as a support tool for experimental investigations with a neuro-robotic set-up. For this reason NeuVision has been tailored to this particular application, i.e., to study the interactions between neurons and a small robot moving in a virtual environment. According to this perspective, we did not consider it a primary goal to optimize simulation performances or to introduce biophysically detailed models. We strategically chose to keep things simple for both the developer and the user, avoiding the integration of di erent tools, that could be readily usable but less flexible from the architecture point of view.
Within NeuVision we realized innovative tools in order to visualize and analyze, in an e ective way, the dynamics of neuronal networks. Some examples of the available methods for visualizing the neuronal signals are shown in Figure 2. As an example, the connectivity of a network can be displayed showing all the links between neurons as reported in Figure 2B ure 2A) or by the instantaneous spiking rate ( Figure 2C). To compute the spiking rate, the user chooses to keep in consideration all spikes, or only spikes fired by either excitatory, or inhibitory, neurons. The electrophysiological activity of the network can be displayed by means of a raster plot (on the bottom of Figure 2A), that makes it easy to distinguish between spiking and bursting activity. More precisely, we classify bursting as the activity of a neuron that fires at least 3 times within a 20 ms time frame. Otherwise we consider the neuron activity as spiking. Moreover a network burst is a widespread synchronous activation of almost the entire network, and it is detected whenever at least 30% of neurons are bursting at the same time [15]. In Figure 2A, an example of a spontaneous network burst is shown. At the top of the figure, six snapshots of the membrane potentials are shown.
NeuVision allows to define di erent stimulation protocols in order to simulate evoked activity. For example the user can select some particular neurons and stimulate them with periodic pulses. On the other hand, the graphic interface shown in Figure 3 is used for configuring a neuro-robotic experiment. More precisely it lets the user to define how sensory information is translated into electrical stimuli for the network.
NeuVision o ers tools to perform classical analysis for a fast characterization of the neural dynamics. In particular it is possible to detect on-the-fly bursts and network bursts. Other tools allow computation of the Post-Stimulus Time Histogram (PSTH), the Inter-Spike Interval histogram (ISI), the Inter-Burst Interval histogram (IBI) and the Center of Activity Trajectory (CAT) [16].

Network Module
NeuVision supports three types of neuronal model: Izhikevic, Integrate and Fire and Hodgkin-Huxley. Nevertheless, the simulation results presented in the following sections derive only from networks made up of Izhikevich neurons [17]. We considered two di erent types of neurons to model excitatory and inhibitory populations, respectively: regular spiking and fast spiking neurons [18]. To preserve some features of the structure of in vitro cortical neurons, we set the ratio between excitatory and inhibitory neurons to 4:1, as reported in [19,20]. The neuron dimension is defined by the length of a square placed in the network grid. The inter-neuronal propagation speed of the action potential (presynaptic conduction) was set in the range 0.1-1.14 m/s, in accordance with the literature [21]. We stimulate each neuron in the network with a random sub-threshold current in order to elicit an ongoing spontaneous activity. The user configures this current by setting its mean intensity and standard deviation. Each neuron is modeled as a point placed in a cell of a square matrix. The connections between two neurons are described by a single synaptic weight which defines the variation of the membrane potential of the post-synaptic neuron when a spike arrives; a weight can be positive, if the presynaptic neuron is excitatory, or negative if it is inhibitory. The synaptic weights are initialized according to a uniform random distribution. The user can choose not to change such values during the simulation, or take into account plasticity e ects by means of a spike timing dependent plasticity (STDP) rule. Such synaptic transmission is also characterized by a synaptic delay, which was set at 2 ms for the presented simulations. The connectivity of the network is determined by two parameters that define, for each couple of neurons, the probability that there is a connection between them. The probability of connections P C between two neurons placed at a distance d is calculated according to the following equation, inspired from the Waxman algorithm for the random networks [22]: where P C MAX is the maximum probability of connection (which corresponds to the situation of two neurons at distance 0), and D P C is a decay constant of the probability of connection.

Robotic Module
The main purpose of the developed software is the simulation of neurorobotic experiments, which can support the development of experimental protocols to be performed in a real closed-loop system. For this reason we implemented editors that allow the user to create simple world models. More precisely, it is possible to define a two-dimensional virtual arena containing obstacles and robots as shown in Figure 3, hence reproducing our real set-up, as described in previous works [12]. Obstacles can have a simple geometric shape, like square, circular or triangular, or a complex shape defined by the user (by means of another editor). The robots are equipped with range finder sensors that provide information about the distance of the robot from any obstacle, or from the arena's wall. The user can choose the number of range finders and their directions. The robot simulation was developed with the purpose of reproducing sensor data of real commercial mobile robots, such as the Khepera, manufactured by K-Team (Yverdon-les-bains, Switzerland).

Validation of the Neural Simulator
NeuVision is able to reproduce the electrophysiological activity composed of synchronized bursts, mixed with random spikes, as typically exhibited experimentally by dissociated cortical networks [23,24]. The results presented here are representative of several simulations but refer in particular to the analyses we performed on 15 minutes of simulated spontaneous activity. The network is made up of 1024 neurons, with one inhibitory neuron for every four excitatory neurons, randomly distributed on a 32×32 grid.
The IBI histogram showing the simulated spontaneous activity is shown in Figure 4A (red trace) compared with the IBI profile coming from the spontaneous activity of a rat cortical culture at 30 day in vitro (DIV), reported in the same panel (black trace). The activity is spread quite uniformly over the entire time, without a dominant peak, as already reported in the literature [25]. The analysis of the PSTH highlights a relevant peak of activity about 60 ms after stimulating an excitatory neuron as shown in Figure 4B. This peak is due to the network burst that the stimulation evokes.

Spiking and Bursting Activity
We computed the Mean Firing Rate (MFR, spikes/s) of the network, finding a value of 3.41 ± 6.32 spikes/s (mean ± standard deviation), comparable with what we can find in our experimental recordings [25]. This high standard deviation is due to the di erent dynamics between excitatory and inhibitory neurons, respectively modeled as regular and fast spiking. This di erence is clearly shown in Figure 5 where raster plots of spontaneous activity are displayed. In the figure, the red points represent the activity of the regular spiking neurons (i.e. excitatory), while the blue ones of the fast spiking neurons (inhibitory). It is worth noting that the activity elicited during a network burst is dominated by the excitatory component. The firing rates of excitatory and inhibitory neurons during network bursts are more similar than during spiking activity, even if there is a great variability in the spike frequency between any two neurons, as indicated by high standard deviations. The mean bursting rate (MBR) is 3.12 bursts/min. The mean network burst rate (MNBR) is 7.07 network bursts/min and the mean duration of a network burst is 71.29 ± 6.36 ms.
In order to characterize the stimulus-evoked dynamics expressed by a neuronal network, we stimulated di erent types of neurons from different sites. In particular we analyzed the di erence between stimulating only one neuron and four adjacent neurons, and only excitatory neurons or only inhibitory neurons. To study the network response, we reproduced the typical low-frequency stimulation protocols experimentally used in multi-recording set-ups [20,26]. More precisely, we delivered a monophasic current stimuli pulse (width 10 ms; amplitude 100 µA; frequency 0.2 Hz).

Network Burst Propagations
To characterize the propagation of network bursts both spontaneous and evoked, we made use of the Center of Activity Trajectory (CAT) [27,28]. CAT describes the global neural dynamics and is particularly useful when propagation phenomena involving large part of a network occur. CAT is computed similarly to the center of mass, where we consider the spatial coordinates of each neuron in the network and its firing rate instead of the mass. More precisely we made use of the following formula: where N is the total number of neurons in the network and the firing rate at time t of the neuron n at coordinates (r n c n ) is calculated as: where NS n (t 0 , t 1 ) returns the number of spikes fired by the neuron n during the time interval (t 0 , t 1 ).
CAT is obtained by linking temporally consecutive centers of activity. Figure 6A shows the CAT of a single evoked network burst due to the stimulation of four excitatory neurons, with a time scale of 50 ms. It can be clearly seen that the origin of the network burst is in the bottom right corner. After about 20 ms, the center of activity begins to shift towards the opposite corner of the network. For evoked network bursts this is a typical path, as confirmed by Figure 6B that shows the CATs of all the evoked network bursts superimposed. This regularity is totally missing for spontaneous network bursts ( Figure 6C) and this is particularly evident in Figure 6D where the CATs of all the spontaneous network bursts are shown. As can be observed, centers of activity seem to follow no predefined path, resulting in an unstructured pattern.

A Simulated Neuro-robotic Experiment
We fully simulated a closed-loop experiment in which a robot, controlled by the electrical activity of a network of neurons, moves in a circular arena containing some obstacles (see Figure 7A). The obstacles and the arena's walls are perceived as real by the sensors of the robot but they are not able to stop the robot trajectory. In order to bi-directionally interface the neural network, we defined a decoding and a coding scheme. The decoding scheme is used to generate motor commands dependent on the neural activity. The coding scheme allows translation of the sensory information picked up by the robot sensors into electrical stimulations for the neurons.

Decoding scheme
In order to generate motor commands we used the Center of Activity (CA), as previously suggested in the work of Chao [29]. If the CA is located at the center of the network, the neural activity is uniformly spatially distributed, and the robot keeps on moving in the same direction at a minimum speed. Otherwise, if the CA is located at another point, such as for example the red one in Figure 7B, then the speed of the robot is proportional to the distance from the center d, and its angular speed of rotation is proportional to the θ angle. In Figure 7B, the robot is turning to the right.   Figure 8 shows the points used to stimulate the network on the basis of the sensory information provided by the robot moving in a controlled environment. Their positions were chosen according to a somatotopic arrangement, i.e., they topologically correspond to the spatial distribution of the sensors on the robot's body. For each stimulation point, the intensity of the supplied stimulus is proportional to the distance between the robot and the obstacle measured by the corresponding sensor. At each iteration the stimulation current I s n due to the sensory input, is equal to:

Coding scheme
where n is the neuron in correspondence of the stimulation point of the sensor s, d s is the distance between the robot and the obstacle measured by sensor s and A is a proportional factor, chosen by the user.
The somatotopic arrangement of the stimulation points introduces a correspondence between the orientations of the neural network and the robot body. As previously illustrated in the`Decoding scheme' section, we synthesized the activity of the network by means of the CA, that is a sort of center of mass of the neural dynamics. Because sensors that detect farther obstacles are responsible for stronger stimulations, they tend to increase the network activity around their corresponding stimulation points. As a consequence the CA is shifted towards those points and the robot drives towards the farthest obstacles. Actually, this coding scheme lets a sort of attraction for empty spaces emerge. Of course we are aware that these rules are far from what could be found in nature. In other contexts, for example in unbounded environments, it could be inappropriate to make the robot explore its world in an e ective way, because of over-stimulations. Nevertheless we adopted it to show the possibility to exploit the properties of a neural network to control a robot, and to illustrate how it is possible to use NeuVision to implement a neuro-robotic interface.

Application of coding/decoding schemes to a simulated bi-directional system
Figures 9A-D show four examples of trajectory of the robot controlled using the aforementioned rules. As it can be observed, the control is not perfect, but the robot always moves inside the arena and always avoids obstacles except for one single case ( Figure 9C). A completely di erent situation is depicted by Figures 9E-H which show robot trajectories obtained only by the spontaneous activity of the simulated network, without any sensory information (i.e. without stimulation and, as a consequence, without any feedback information from the robot). In this case, the robot moves as if it were blind, without taking into account the positions of the obstacles or the arena boundaries.
The described results suggest that, to let a simple intelligent behavior emerge, the bi-directional communication between a neuronal network and an artificial body must be provided through specific algorithms of coding/decoding.

Discussion and Conclusions
We have presented a new simulation environment, named NeuVision, able to reproduce neuro-robotic experiments. The software was primarily designed as a support tool for conducting real experiments within the field of the embodied electrophysiology, i.e. biological neuronal net- works bi-directionally connected to an external artifact moving in a controlled environment. The software architecture has been designed and implemented within the .NET framework (Microsoft). As a matter of fact this development platform is still little used for simulation because of its recent introduction, but we think it could o er important advantages in that context, especially in relation to the thread management and the graphical user interface development. The modularity of the architecture makes it easy to interface simulated large-scale neural networks, and virtual/real mobile robots, within the same experiment. Thus, with the goal to improve the complexity of the experimental design, we have made it possible to add new functionalities to the software, which has been designed to be simple for both users and developers. For the above reasons, we chose to develop a completely new environment rather than making use of available network simulators [30--36] or interfacing our computational model with more sophisticated robot models [37--39], already presented in the literature and freely available. This gave us a strict control on the definition of both the network connectivity and the stimulation protocol and the possibility to easily add all the functionalities we need to manage the experimental phases of a closed-loop session, such as specific visualization and analysis tools.
The implemented software provides great flexibility for defining neuronal networks and connectivity rules as well as stimulation protocols, also allowing the user to visualize the neural activity in both standard and novel ways [14]. The validation results show that the neural simulator is able to reproduce both the spontaneous and evoked activity of neuronal networks. We computed the firing and the bursting rate (i.e. MFR and MBR), obtaining statistics comparable with those found during real experiments [25]. We characterized the neural dynamics considering excitatory and inhibitory neurons separately. Our Inter-Spike-Interval (ISI) analyses highlighted that neurons in the modeled network tend to fire with the same type of activity, either spiking or bursting, found in real networks. In order to validate the stimulus-evoked activity, we tried to reproduce low-frequency stimulation protocols, by stimulating di erent types of neurons from di erent sites in the network. Moreover we also used non-standard analysis tools, such as the Center of Activity Trajectory [16], to characterize the propagation of network bursts, finding a clear distinction between spontaneous and stimulus-evoked network bursts. More precisely, we found that evoked network bursts follow few predefined propagation paths, whereas spontaneous ones are much less regular; this has also been demonstrated for biological networks of neurons cultured over multi-electrode arrays [28].
In this work we also presented a fully simulated neuro-robotic experi-ment, to show the potentialities of NeuVision. The obtained results highlight the importance of a feedback signal from the environment, given through the`eyes' of the robot, to e ectively control the robot's trajectory. The great advantages of such a neuro-robotic simulator mostly rely on the possibility to explore di erent experimental protocols and di erent coding/decoding schemes, to let the culture interact with an external environment. The same experimental procedures could then be implemented in a real closed-loop system. In the future we plan to extend the available schemes of coding and decoding to make possible the emergence of di erent intelligent behaviors. Forthcoming software improvements will focus on the implementation of a more general network topology editor that will allow users to define di erent types of connectivity, such as small-world and scale-free. Di erent STDP rules will soon be included in the simulations to enable synaptic plasticity studies. At a later time we plan to introduce the support for three-dimensional visualization and for parallel computing, with particular regard to the latest GPU technology. Moreover, we will add the possibility to use Microsoft Robotics to configure more realistic robotic simulations.
We are convinced that a synergistic approach is crucial to increase the e ectiveness of closed-loop electrophysiological experiments in order to understand how networks encode information [13]. As a matter of fact, simulations can help in overcoming the technical limitations of real experiments, allowing users to monitor the neural activity with a level of detail unreachable by present recording technologies. This makes it possible to highlight phenomena that otherwise would go unobserved during an experiment. At the same time, without the constraints of physical stimulation through microelectrodes, sensory information can be provided to the network in a more detailed and reliable way. Moreover, because there is no risk of killing a simulated culture, or of damaging virtual electrodes, it is possible to freely try a wide variety of experimental protocols. This makes more likely the appearance of unexpected results that can suggest new real experiments specifically designed to reproduce the simulated emergent behaviors. A better understanding of the processes leading to biological cognition can, in turn, facilitate progress in understanding neural pathologies, designing neural prosthetics, and creating fundamentally di erent types of artificial intelligence.