Evolution of central pattern generators for the control of a five-link bipedal walking mechanism

Abstract Central pattern generators (CPGs), with a basis is neurophysiological studies, are a type of neural network for the generation of rhythmic motion. While CPGs are being increasingly used in robot control, most applications are hand-tuned for a specific task and it is acknowledged in the field that generic methods and design principles for creating individual networks for a given task are lacking. This study presents an approach where the connectivity and oscillatory parameters of a CPG network are determined by an evolutionary algorithm with fitness evaluations in a realistic simulation with accurate physics. We apply this technique to a five-link planar walking mechanism to demonstrate its feasibility and performance. In addition, to see whether results from simulation can be acceptably transferred to real robot hardware, the best evolved CPG network is also tested on a real mechanism. Our results also confirm that the biologically inspired CPG model is well suited for legged locomotion, since a diverse manifestation of networks have been observed to succeed in fitness simulations during evolution.


I. INTRODUCTION
The field of humanoid robotics has received an increasing interest over the last few decades [1].In contrast to conventional robotic designs working in specialized environments (e.g.highly customized robotic arms in assembly lines), robots with anthropomorphic features are expected to be more adept in a growing number of environments in which they are expected to perform, as most of the real environments have been initially designed to suit human needs and anatomy [2].Humanoid designs are also desirable from a human-robot interaction point of view: Humans tend to interact and communicate better with human-like entities.
Bipedal locomotion is a principal part of the research efforts in the field of humanoid robotics.The main motivation for studying bipedal locomotion, and walking robots in general, is that it is in many ways superior to conventional wheeled approaches on real terrain [3] and in situations where robots need to accompany, or replace, humans.Another motivation for the research on bipedal walking robots is to gain a better understanding of the physiology of human locomotion [1].
It has been suggested that bipedal walking mechanisms are more flexible in coping with obstacles in complex environments when compared to other walking mechanisms (quadruped, insectoid etc.) [4].But this comes with the cost of substantially reduced stability, which in turn asks * atilim@alumni.chalmers.sefor more sophisticated control approaches.While the issue of stability is the main incentive for designing better control methods, recent research has also made progress with control methods focused on reducing the impact of falling in case the locomotion system fails [5].
The intention of this study has been to investigate how well the so called five-link bipedal walking mechanism works under the control of a central pattern generator -a type of artificial neural network with a firm basis in neurophysiologic experiments-subject to optimization using evolutionary algorithms.After experiments with realistic physical simulation, the results have been also put to test in real-world dynamics, on a hardware implementation.
After presenting a rather brief background information on central pattern generators and the five-link planar walking mechanism in the rest of this section, the article continues in section II with the details of the built physical simulation, the design of the CPG network, and the hardware on which the results are tested.This is followed by a selection of obtained results in section III and the conclusions in section IV.

Central pattern generators.
Neurophysiologic studies on animals suggest that their nervous systems incorporate specialized oscillatory neural pathways, named central pattern generators (CPG), that are responsible for most of the rhythmic movements produced by the organism, including locomotion [6].The defining charac-teristic of these neural circuits is that they are inherently capable of producing oscillatory activity on their own, without depending on oscillations in the sensory input.In experiments on living animals, it has been shown for numerous cases that these neural circuits produce sustained rhythmic activation patterns even when they are in isolation from external stimuli [7].
In the field of robotics, CPGs are increasingly being used as a model for the control of legged robot locomotion for bipedal, quadrupedal, and other designs.They have been successfully applied to non-legged cases such as serpentine locomotion [8] and swimming [9].It has also been demonstrated that CPG controllers can also be successfully implemented as analog electronic circuits [10].The control of legged robotic locomotion with CPGs has the advantage of being biologically inspired and more adaptive to changes in the environment, as compared to classical analytical control approaches like the widely used zero moment point (ZMP) method.
Mathematically, CPGs are modelled as systems of coupled differential equations, similar to models for other continuous-time artificial neural networks.A CPG network is composed of oscillatory units, an arrangement of two mutually inhibiting neurons each becoming active in turn.An oscillatory unit has a natural frequency and amplitude of oscillation on its own (depending on oscillation parameters), but when several such units are interconnected and in turn connected to an external input, they tend to tune in to the frequency of the present input.By connecting these oscillatory units in different ways, networks with complex frequency and phase relationships can be constructed, making these very suitable for the control of walking mechanisms.For the purposes of this study, a half-center CPG model is used, introduced by Matsuoka [14,15], the details of which will be given in section II.
Five-link planar bipedal walking mechanism.It has been recently demonstrated that anthropomorphic mechanisms even without any actuation can stably walk down slopes in three-dimensions, by a controlled release of stored gravitational potential energy [11].These so-called passive dynamic walkers suggested that stable human-like gaits on level ground or upward slopes can be achieved with a smaller number of actuators than previously considered.It has been suggested that a five-link mechanism (Fig. 3), lacking feet and having only four actuators (two for the hips and two for the knees), can be used to produce stable human-like gaits.Mechanisms of this type have been used on several occasions to study bipedal walking [11,12] and running [13].
The five-link mechanism is often planar, restricted to run in two dimensions by means of an attached lateral boom [12,13] as is the case in this study.This has the advantage of making the physical simulation and mathematical analysis of the gait less complicated.It may also be argued that studies with simple planar mechanisms can provide a more direct insight on the fundamental processes involved in bipedal locomotion.The five-link implementation in this study, as a physical simulation and constructed hardware, is described in detail in section II.

II. EXPERIMENTAL SETUP A. CPG controller
The basis for the CPG controller used in this study is the half-center oscillatory unit known as the Matsuoka's oscillator [14,15].The model is described by the following set of equations: where u i is the inner state, v i is the variable of self inhibition, y i is the output of the ith neuron, u 0 is a constant excitatory input to the oscillator, τ and τ are time constants, β is the coefficient of self inhibition, and w is the weight of inhibitory connection between neuron 1 and neuron 2 (Fig. 1).The value of parameter u 0 has an effect on the oscillation amplitude, and the values of τ and τ determine the natural oscillation frequency of the unit oscillator in the absence of an oscillatory input from other sources (which might be introduced by z i and f i , as described further below).
One such oscillatory unit is responsible for the control of one mechanical joint in the walking mechanism, making a total of four unit oscillators in the controller network.Neuron 1 and neuron 2 of an oscillatory unit are respectively denoted flexor and extensor neurons for that joint, drawing on an analogy with the anatomy of muscular action in real joints.The output y = y 1 − y 2 of the oscillatory unit (Fig. 1) is used as the angular speed of the corresponding joint, after a linear scaling that is described in the following section.
z i in Eq. ( 1) represent the total input from other CPG unit oscillators in the controller network to the ith neuron of this unit oscillator, which might be excitatory (positive) or inhibitory (negative).This can be written as with y j representing the output of the jth neuron in the set of remaining unit oscillators in the network, and w ij is the connection weight existing in-between.
Often, the input to the components of an oscillatory unit is arranged such that z 1 = −z 2 , meaning that an internal network connection having the effect of, say, promoting the flexion movement of a joint (or equally, inhibiting the extension movement), should excite the flexor neuron and at the same time inhibit the extensor neuron of the corresponding oscillatory unit.The condition z 1 = −z 2 is not implicitly imposed in this study, and the genetic algorithms (GA) implementation (described in section II C below) is free to determine the connection paths and types of connection to each of the neurons in a unit oscillator independently.
f i in Eq. ( 1) represent the total feedback input to the ith neuron, in a similar fashion to z i described above.Feedback paths provide a means to maintain an adaptive mutual coordination, called entrainment, between the CPG network and the walking mechanism subject to physics of the environment [16].This is achieved by the continuous modification of oscillation characteristics and phase relations of the CPG network by the external inputs, and in turn, the commands sent by the CPG network driving the walking mechanism through the environment, and again, the effect of this on the CPG network through the feedback pathways (Fig. 2).The feedback pathways existing between the walking mechanism and the CPG controller network are described in the following section.

B. Walking mechanism and the physical simulation
The five-link planar walking mechanism considered in this study consists of four actuators and the five links inbetween: Two actuators in the hip and two in the knee joints, two links for each leg, and the fifth link connecting the two legs.The mechanism has no supporting feet, i.e. no ankle joints.
The study depends on a physical simulation model for the five-link walking mechanism, comparable to the one introduced in the seminal paper by Taga et al. [16], which has been designed and programmed from scratch.The model essentially consists of the application of Newton-Euler equations to the five rigid bodies constituting the mechanism in two dimensions and an impact model between the feet and the ground profile described as a spring / damper, similar to [17].
An overview of the model is presented in Fig. 3.The angular relationships of the constituent links of the mechanism are all written with respect to Link 1, the link between the two hip joints standing at angle θ 1 measured from the vertical.The thighs, Link 2 and Link 3, stand at θ 2 and θ 3 with respect to Link 1, and the shanks, Link 4 and Link 5, stand at θ 4 and θ 5 measured from the corresponding thighs they are attached to, respectively.The CPG controller network, described previously, is tied to the walking mechanism by direct coupling of the output y i from the corresponding oscillatory unit of the CPG network (Fig. 1) as the angular speed θi of each joint.θi = min(max(y i , −1), 1)ω max (3) Before driving the joint, the output y i is scaled such that its absolute value is bounded within the maximum attainable rotational speed ω max of the joints, based on the servo motor specifications of the hardware implementation described in section II D.
The mass of Link 1 connecting the hip joints is denoted m, and the link is simulated as a point mass on the simulation plane.m 1 and l 1 denote the mass and the length of thighs and m 2 and l 2 denote the mass and the length of the shanks for both legs.For both thighs and shanks, the center of mass is assumed to be halfway through their length.k g and b g represent the elasticity and damping coefficients of the ground impact model.In total, the implemented model has nine parameters describing the walking mechanism and its interaction with the environment, with the addition of gravitational acceleration g and the maximum rotational speed ω max of the joints.
The function p(x) defines the ground profile y = p(x), which provides a simple means to describe the environment the mechanism needs to be tested in, and makes it possible to define any form of terrain including slopes and simple obstacles.Even more complex cases can be introduced by defining p(x) as a Fourier series with appropriate parameters.
The feedback pathways from the walking mechanism to the CPG controller network are based on six different features: The deviation from the vertical of each of the four joints, and the state of the two feet.The feedback pathways are described by the following set of equations.
Left hip: where t l represents whether the left foot is on the ground (t l = 1) or not (t l = 0); and the same holds for t r corresponding to the right foot.The presence and strength of feedback to hips and knees are regulated by two coefficients, a 1 and a 2 , and the value of these are in turn included in the GA optimization process described in the following section.This minimalistic approach has been observed to produce results of comparable success with the approach used in former studies of this kind ( [16] as an example), involving greater numbers of coefficients (up to four for each feedback pathway).

C. Genetic algorithms
For optimizing the parameters of the CPG controller network, including the internal connection structure between the neurons and the presence and strength of feedback pathways from the walking mechanism, a standard genetic algorithms (GA) implementation has been employed.Fitness evaluations of the population were done in the physical simulation of the five-link mechanism with a set of parameters not subject to evolution, arranged to match dimensions and masses of the physical parts for the intended hardware implementation.
The encoding scheme used for the chromosomes consists of 25 real numbers describing the parameters of the CPG network, the presence, strength, and nature (i.e.inhibitory or exhibitory) of internal connectivity between different oscillatory units, and the presence and strength of the feedback pathways described in the previous section.The chromosome encodes: w, u 0 , τ , τ , and β as the internal parameters of the unit oscillators, w ij , the elements of the connectivity matrix, and also a 1 and a 2 , the feedback coefficients described in the previous section.The values of τ and τ are directly used in the hip unit oscillators, whereas in the knee unit oscillators, these values are taken as τ /2 and τ /2, resulting in an oscillation frequency twice that of the hip joints, as is the case in a real human's gait [16].
There is a predetermined structure imposed on where connections might be present between the unit oscillators of the CPG network.The connectivity matrix presented in Fig. 4 shows, for a certain neuron (a row), the weights of incoming connections from other neurons (columns) in the network.The structure presented here makes use of the expected symmetry in the CPG controller network (corresponding to the bilateral symmetry of the walking mechanism and the expected behavioral symmetry of the human-like gait).The parameters w 1 to w 8 are encoded together with a set of multipliers w * 1 to w * 8 (filtered with a threshold in the decoding process of the chromosome to give either 0 or 1), providing a simple way for the evolution process to turn a given connection on or off.The parameters a 1 and a 2 describing the feedback pathways are also coupled with parameters a * 1 and a * 2 in a similar way.The connection weight between coupled flexor and extensor neurons is given by w, which is the same for every unit oscillator.
Note that the connections between the hip and knee unit oscillators of a side are unidirectional, as in [16]: There might be an effect on knee unit oscillators from the corresponding hip unit oscillator on its side, but not the other way (correspondingly, the connectivity matrix in Fig. 4

is asymmetric).
A summary of parameters used with the GA implementation are summarized in Table I.With this parameter set, stable walking gaits were common to occur after just a few generations of fitness evaluations.The used fitness functions are described together with the results in section III.

D. Hardware
With the intention of seeing how well the results achieved by the physical simulation and the GA runs behave when subject to real-world dynamics, a hardware implementation of the five-link mechanism has been con- structed.As with the physical simulation described previously, this mechanism needs to be run in two dimensions; and this was achieved, as in [18], [12] and [13], by an attached lateral boom freely rotating around a pivot (Fig. 5).The boom restricts the movement of the mechanism to a spherical surface approximating motion in two dimensions, given the boom radius is sufficiently large.This support has no effect on the vertical stability of the mechanism in the sagittal plane (the plane perpendicular to hip joint axes, dividing the body into left and right halves), and the dynamics are assumed to be identical with the two-dimensional simulation.
The body of the robot was constructed out of hard plastic parts as found fit for the purpose.Four standard servo motors were used as actuators (rated with ω max = 5.51 radian/s, see section II B), with the motors in hip joints having a range of movement of 180 • , and the motors in the knee joints 90 • (Fig. 3).The resulting mechanism was comparable to the commercially available five-link robot "Red-Bot", used in [12] and other studies, manufactured by Iguana Robotics.The finished mechanism (Fig. 6) was about 20 cm (7.87 inches) high and weighed slightly more than 200 grams (0.44 pounds).
The hardware was driven by a dedicated interface circuit attached to the host computer (running the CPG controller and GA evaluations), based on the Parallax Basic Stamp 2 TM microcontroller.It communicated with the host computer using RS232 serial communications protocol.

A. Simulation and GA
At first, a hand-tuning approach was tried to see if a stable gait walking on level ground can be produced by just experimenting with CPG parameters, using the previous results by Taga et al. [16] as the starting point.This proved to be very hard to achieve, and in the very rare cases where the mechanism could be made to walk, the gait seemed unnatural, and eventually destabilized into a fall after just three or four steps in all cases.
The trials with the genetic algorithm were started using a fitness function measuring the horizontal displacement of Link 1 (see Fig. 3) between the beginning and the end of the evaluation.At the start of fitness evaluations, the mechanism was released with a straight upright posture (θ 1 = θ 4 = θ 5 = 0, θ 2 = θ 3 = π), from slightly above the ground.During the course of GA runs, several other objectives were introduced to the fitness measure (like promoting an upright posture and putting an upper boundary to the height of the mechanism from the ground, to prevent jumping), but in the end the improvement introduced by these were minimal in all cases, and each additional fitness objective introduced additional exceptions to deal with.
The best gait evolved on level ground with the simple objective of maximizing the distance moved, is represented in Fig. 7.The gait has also been tested on slightly inclined or declined (up to 5 • ) ground profiles p(x) with success.The parameter values describing this gait are given in Table II and the CPG network structure of the individual is given in Fig. 8.The network seems to be fairly optimal and have no redundant connections.Note that the knee oscillatory units make use of feedback more than the hip units (a 2 > a 1 ).
The overall observation is that the CPG approach to human-like bipedal walking shows great capacity.Many different gaits thus produced were observed during the evaluations of GA runs, and it was particularly striking that even in the first generation (where the CPG parameters / connections are completely random by definition) there were individuals with stable-looking gaits, even if these were generally not sustained long enough.This is suggestive of an inherent aptness of the CPG network for the task of walking.Virtually in all runs (with level, inclined, and declined ground profiles), a stable gait was eventually established after five or six generations of the GA evaluation.Also, occasionally during the course of GA evaluations, there emerged individuals not making any use of the feedback pathways (i.e. with parameters a 1 and a 2 almost zero), yet still able to exhibit a stable gait.While the dynamics of these give an impression of being less humanlike in comparison to gaits utilizing the feedback mechanism, the existence of such individuals suggests that-at least on level terrain and without any obstructions-the CPG network approach is capable of producing stable gaits with or without feedback from the environment.

B. Hardware
To observe how well the successful gaits from the GA runs perform under real-world dynamics, these were executed on the constructed five-link hardware implementation.The mechanism was run on level ground, and correspondingly, results evolved for the level ground were tried on the hardware.The runs generally reproduced to a good degree what was achieved in the physical simulation, after slight adjustments to the low-level interface code (running on the microcontroller) involving the exact timing of servo motor signals, which is crucial for their accurate angular positioning.
One particular drawback in the hardware runs is introduced by the laterally attached support structure.Even if there is an expected degree of asymmetry caused by the slightly unequal distances covered by the left and right feet (Fig. 9), the impact of this on the gait balance was larger than what was anticipated.While this can easily be averted by using a sufficiently long boom as compared to the distance between the legs (with the disadvantage of introducing more weight onto Link 1, prompting to change in parameter m in Fig. 3), another solution, such as a linear support structure with a pin-in-slot connection might prove to be a better option.

IV. CONCLUSIONS
This study was an investigation on how well the fivelink planar bipedal walking mechanism can be controlled using a CPG network design evolved using a genetic algorithm.The parameters determining the oscillation characteristics of the network together with its internal connection structure and feedback pathways were subject to GA optimization, evaluations of which were done in a realistic physical simulation.
The ability of the CPG control approach to produce stable human-like bipedal gaits was demonstrated.Furthermore, it was observed that evolution was progressing with an apparent ease in all cases.This suggests that the CPG approach to walking, describing a gait in terms of phase relations within a simple network of mutually inhibiting oscillators, with little or no higher level supervision to the minute details of walking, is a feasible one.
Individuals lacking feedback pathways, but still able to walk on level terrain are observed during the GA runs.It may be argued that the CPG mechanism is inherently able to sustain a stable gait, while the feedback pathways are needed to provide a means to adapt to irregularities in the environment, such as varying ground slope or obstacles.Analyzing the performance of the CPG approach subject to evolutionary algorithms under the presence of obstacles or in dynamically changing environments would be a straightforward extension to this study.It would be also very promising to include the physical parameters of the walking mechanism in the evolutionary process and then attempting to find optimal bipedal walking hardware designs in environments with different characteristics.

FIG. 1 .
FIG.1.Matsuoka's half-center model, as employed in the current study.Note that the inputs fi and zi to each neuron, both shown here to be excitatory, may either be excitatory or inhibitory (see the text).

FIG. 2 .
FIG.2.An illustration of the concept of entrainment between the CPG network and the walking mechanism / environment.

FIG. 5 .
FIG. 5.The five-link planar walking mechanism and the accompanying support structure.

FIG. 7 .
FIG. 7. The best evolved human-like gait.Right leg is shown in solid black, left leg in dashed gray.The blue dot near the center indicates the trajectory of the center of mass.The orientation of Link 1, hence the whole structure, is represented by the short gray line originating upwards from the hip.The marks below the ground line represent contacts by the left foot (gray) and right foot (black).Direction of movement is to the right.

FIG. 9 .
FIG. 9.The walking trajectory of the five-link mechanism, viewed from above.

TABLE I .
GA parameters used in the evolution of CPG networks.Made up of individuals reproduced from the preceding generation and individuals created by crossover (determined by the crossover probability).Elitism was used (the best individual was always kept). 1

TABLE II .
Parameter set of the CPG network producing the best resulting human-like gait.