Urn models have been studied for nearly one century. In 1931, G. Pólya provided the first probabilistic result on the game consisting in drawing a ball from an urn initially containing one red ball and one black ball (see ). At each time step, a ball is drawn and put back in the urn with an additional ball of the same color. The probability to draw a red ball is the proportion of red balls in the urn. Pólya proved that, as the number of draws tends to infinity, the proportion of red balls tends to a random variable following the uniform distribution on .
The Pólya urn is easily generalizable to a large class of two-colored urn models characterized by a choice function . Let and be the numbers of red and black balls in the urn after draws. Note that, by construction, . The probability that the -th ball is red is given by
Consequently, the probability to draw a black ball at time is .
In this paper we will consider a parametric model for the choice function . That is, we will assume that is known up to a finite dimensional parameter . For instance, the parameter can be the initial quantity of balls of each color, in which case would be a -dimensional vector in . The previous equation becomes
Our goal will be to propose a valid statistical methodology to estimate the parameter . An important issue must be raised. In certain urn models, consistent estimation of the parameter based on only one path of length may be impossible, even if tends to infinity. This will be proved formally in Theorem 8. Therefore, we will suppose that independent paths are observed, each consisting of a sequence of balls drawn according to the model defined in eq. (1). The statistical theory is developed as is fixed and tends to infinity. This framework may be unusual but is in accordance with data from real life experiments such as those which will be reported in Section 2. These data sets consist of a large number of replications of finite paths.
We define estimators for that we prove to be consistent and asymptotically normal under some usual regularity assumptions on the model (1). We study more precisely two particular cases: the maximum likelihood estimator (MLE) and the weighted least squares estimators (WLSE).
We have applied these statistical tools to the problem of path formation by an ant colony. One of the fundamental factors affecting an organism survival is its ability to optimally and dynamically exploit its environment. For example, in order to take advantage of the best sites of resources, housing or reproduction, these areas must be discovered and exploited at the earliest opportunity. Many species of ants rise to this challenge by developing a network of paths, which connects different strategic sites such as nests and food sources. These paths consist of pheromones, attractive chemical substances. We focus on a specific aspect of this phenomenon: the selection of a path among pre-existing channels. When exploring their environment, ants often face bifurcations and must bypass obstacles. As shown by experimental studies, the laying of pheromones by ants passing successively through a bifurcation results in two possible outcomes: either one branch is eventually selected and the other abandoned, or both branches end up being uniformly chosen (see ). The analysis of the spontaneous path formation by a colony of ants is made difficult by the absence of any means to measure precisely the quantity of, or even detect, the pheromones laid by the ants.
It is commonly assumed that when approaching a bifurcation, ants choose a branch and lay a certain constant amount of pheromone without ever turning back. Consequently, the quantity of pheromone laid on a branch is proportional to the number of ants which passed through it. Thus this phenomenon can be described by an urn model as proposed by . More precisely they define a choice function with a two-dimension parameter such that the probability for an ant to choose the right branch after passages through the right branch and passages in total is given by
The probability to choose the left branch is consequently . The parameter makes this model non-linear with respect to the proportion of passages through one branch. It models the sensitivity of the ant to the concentration of pheromone. The parameter is the intrinsic attractiveness of each branch and can also be interpreted as the inverse of the attractiveness (or the strength) of the pheromone deposit laid by each ant.
Several probabilistic studies provide the asymptotic behavior of in terms of and (See [1, 1, 4]). The influence of the two parameters is on different time scales, but they can contribute to the same effect (selection of one branch or unifomization of the traffic on the two branches) or have antagonistic effects. The model is thus characterized by four phases, according to the values of and : slow or fast uniformization; slow or fast selection. The phase most commonly considered in the literature is slow selection. This corresponds in our model to and larger than : selection of one branch will eventually happen, though slowly because of weak pheromone deposits (see [2, 2, 2]). However, the model may account for other possibilities, such as fast uniformization, which occurs when and . One purpose of this paper is to investigate more thoroughly these possibilities which have been more or less overlooked in the previous literature.
Ethological studies have already provided values for the parameter , but the methods used mainly consisted of calibration without control of the statistical validity of these methods and resultsV (see [2, 5, 6, 7]) and in particular these methods do not produce confidence regions. However this type of information supplies interesting and important elements to the behavioral discussion. In this paper, we define the MLE and the WLSE for the ant behavior model (2). We assess the quality of these estimators in a simulation experiment. We then use them on experimental data provided by a real life experiment performed to this purpose with ants. We also compute a confidence region by a Bootstrap algorithm.
The model (2) can be applied, mutatis mutandis, to many other fields. For instance we can consider a fork with two branches as a neuron having two axons. During each period of time, the length of one of the two axons increases. The longer an axon is, the higher the probability that it will further grow. Thus the dynamic of this biological system may also be modeled by eq. (2) (see ). Furthermore, using the notion of a choice function (see Section 1.3), the model (1) can be adapted and applied to many situations where a binary choice occurs, or to even more complex situations such as networks with several nodes (see [9, 10]), for instance network exploration by an ant colony (see [7, 11, 12, 13, 14, 1516]).
The paper is organized as follows. The model (1) and the statistical framework is rigorously defined in Section 1. Section 1.1 is focused on the MLE and Section 1.2 on the WLSE. Under some usual regularity conditions, we prove that both estimators are consistent and asymptotically normal. Moreover the MLE is asymptotically efficient. The numerical implementation of the MLE may be difficult and unstable (and lengthy), therefore, a WLSE is considered and theoretically studied. This estimator does not match the theoretical performances of the MLE, but is easier to compute and is popular among practitioners. Section 1.3 proposes an extension of the general urn model to a class of vertex reinforced random walk on graphs. Section 2 is an adaptation of the statistical framework to the ethological problem. In Section 3, we first introduce the assumptions on the ant behavior and then we define the model proposed by . The four phases of the model are described more precisely and are interpreted from the point of view of ethology. Section 4 rewrites the MLE and the WLSE for this particular case. In Section 5, we show that it is not possible to consistently estimate the parameter of the model on a single experiment of length , even if tends to infinity. In order to assess the performance of the estimators, a short simulation experiment is reported in Section 2.4. Section 3 reports the study on the experimental data. The experimental protocol and the data produced are described in Section 3.1 and Section 3.2. Section 3.3 supplies the estimation results (computation of the estimators and their confidence regions). We provide some concluding remarks in Section 4 and prove the theoretical statistical results of this paper in Section 5.
1 Parameter estimation
Let us first write precisely the model studied and statistical framework used. Let , we assume that is a sequence of Bernoulli random variables (representing the colors of the balls drawn: 1 for red and 0 for black) and that there exists a function called the choice function of the urn and a “true value” of the parameter such that, for all integers ,
where and for , and is the sigma-field generated by . The random walk is an inhomogeneous Markov chain. For and a sequence , applying the Markov property, we obtain
where with , by convention.
We assume that we observe experiments, each consisting in a path of length of the model (3). In other words, we have a set of sequences of consecutive draws. For , and , let denote the color of the -th ball drawn in the -th experiment. Let and , be the total number of red balls drawn at time during the -th experiment, so that . In all the paper, will be fixed and our asymptotic results will be obtained with (the number of experiments) tending to .
The choice function and the true value of the parameter is the same for all experiments, i.e. for , and ,
For conciseness of notation, we will hereafter simply write for . To prove the consistency and the asymptotic normality of the estimators introduced below, we need the following assumptions on the choice function . For any function defined on , we denote and the gradient and Hessian matrix with respect to , the partial derivative with respect to the -th component of , and the transpose of the vector or matrix .
(Regularity) The set is compact with non-empty interior. For , and the function is twice continuously differentiable on .
(Identifiability) If for all , then ,
-dimensional vectors , , are linearly independent in .
Assumption 1 ensures that is the unique maximizer of and that the Fisher information matrix
1.1 Maximum likelihood estimation (MLE)
The structure of the model (3) allows to have an explicit expression of likelihood . The independence of the experiments yields the following multiplicative form
The log-likelihood function based on paths, is thus given by
Let be the maximum likelihood estimator of , that is
Define (where the dependence on is omitted). Then, (6)
Let denote the Gaussian distribution with mean and covariance .
If Assumptions 1–(i) and 1–(ii) hold then the maximum likelihood estimator is a strongly consistent estimator of . If moreover Assumption 1–(iii) holds and is an interior point of , then, as tends to , converges weakly towards .
The proof is in Section 5.4. It is the consequence of a more general result stated and proved therein.
1.2 Weighted least squares estimation (WLSE)
Least squares estimators are very popular among practitioners. Moreover for some urn models, the MLE may be numerically unstable hence difficult (and lengthy) to compute so the WLSE constitutes a convenient alternative. In order to describe this estimator, we introduce some notation. For , define
with the convention . The quantity is the empirical probability that red balls have been drawn at time and is the empirical conditional probability that a red ball is again chosen at time given red balls were drawn at time .
We further define and . Let be a sequence of weights and define the contrast function
The weighted least squares estimator minimizes , that is (9)
If moreover Assumption 1–(iii) holds and is an interior point of , then as tends to , converges weakly towards , where is a definite positive covariance matrix.
If moreover , for all , the estimator is asymptotically efficient, i.e. .
The proof is in Section 5.4, where an explicit expression for is supplied.
It is possible to extend the statistical framework introduced in the previous sections to a large class of reinforced random walks on graphs. For instance, let be a non-oriented graph, such that each vertex is connected to a finite number of vertices and, with the set of its vertices and the set of its non-oriented edges. We denote by , if . Let be the set of integer vectors indexed on . We define a random walk on , i.e. a sequence of vertices, such that, for all , . The vector is such that, for all vertex , is the number of times the walk has visited up to time . Let , , we suppose that the walk is vertex-reinforced and that there exists a choice function and a parameter such that, for all and ,
For instance, the choice function can be similar to the one proposed by , for and ,
where and .
The statistical framework introduced for the general urn model is easily adaptable to this vertex reinforced random work. Under some adequate regular conditions, it would not be difficult to prove the consistency and the asymptotic normality of the MLE and the WLSE by establishing a theorem similar to the general result proved in Section 5.3.
2 Application to an ethological problem
In 1990, Deneubourg et al. used a particular urn model to reproduce the sequences of consecutive choices made by ants at a fork . Let the urn filled with balls of two colors be replaced by a fork with two branches. Drawing a ball and adding a ball of the same color in the urn is equivalent to an ant choosing a branch and reinforcing it with pheromone by going throw it. Then the probability to draw a red ball depending on the previous draws is equal to the probability to choose the right branch depending on the previous passages.
We first introduce the model proposed by  in the statistical framework described in the previous section. Futher, we provide a description of the model behavior depending on the parameter value. We also supply an ethological interpretation of the parameter. We then prove that it is impossible to estimate the parameter on a single path. Finally we report the study of the estimator accuracy that we performed on simulated data.
2.1 Formalization and results
We first introduce the hypotheses assumed on the ant behavior.
Each ant regularly deposits a constant amount of pheromone as it walks.
The ants are strictly identical which means that every ant has the same reaction to the same amount of pheromone.
Pheromone trails do not evaporate during the experiment.
Each ant reaches the fork alone, chooses a branch and leaves the bifurcation by crossing only once into the chosen branch without passing through or reinforcing the non-chosen branch.
Under these assumptions, the quantity of pheromone laid on each branch is proportional to the number of passages through it. In path formation modeling, these assumptions are commonly made. However, because of the inter-individual variability in ant behavior, the first two assumptions are unrealistic. For instance, the pheromone perception noise implies that each ant could detect a different signal from the same quantity of pheromone. These assumptions are an approximation of the real ant behavior. The implicit hypothesis here is that the inter-individual variability is small enough to consider that all ants are identical. For the third assumption, we suppose that the persistence of the pheromone trails allows to ignore the evaporation of the pheromone. Experimental protocols are designed to make the four assumptions more acceptable by choosing the ant species adequately and by placing them in an appropriate situation (see Section 3).
The random variable , introduced previously, is the choice of the -th ant going through the fork (1 for right and 0 for left). Consequently, for , is the number of passages through the right branch after passages. For and all integers , we define the choice function
We assume that the probability that an ant chooses the right branch at time given the first choices is given by:
where is the intrinsic attractiveness of each branch and is the possible non-linearity of the choice. This process is an urn model which has been exhaustively investigated in the probability literature. We recall here its main features.
If , then \; a.s.
If , then converges almost surely to a random limit with a Beta distribution with density with respect to Lebesgue’s measure on , and is the Gamma function.
If , then eventually only one branch will be chosen, i.e.
Ethological interpretation of the parameters and properties of the model
The parameter characterizes the ant differential sensitivity to the pheromone. When , the ants can detect better and better increasing amounts of pheromones laid on each branch and thus are more likely to choose the branch with the most pheromones. Moreover, after a random but almost surely finite number of passages, one branch will eventually be selected, i.e. all ants will afterwards choose this branch (see Theorem 4–(iii)). In the opposite case, when , the ants are less able to perceive the differences between the amounts of pheromones laid on each branch as these amounts increase. This minimization effect is so strong that the proportion of passages on each branch converges to (see Theorem 4–(i)). It is important to note that, when , the asymptotic behavior of the proportion of passages through each branch depends only on and not on . Furthermore the larger is or the closer is to zero, the faster these effects will happen.
The role of is clearer when eq. (11) is rewritten as follows:
The parameter is the inverse of the reinforcement incrementation and thus can be interpreted as the inverse of the attractiveness (or the strength) of the pheromones laid at each passage. Consequently when is neither very close to zero nor very large, has a strong short term influence. When is small compared to , the first passage strongly reinforces the first chosen branch. Thus during the first few passages, a branch will be highly favored even if (in which case the branches will eventually be uniformly crossed). When is large, the first passages weakly reinforce the chosen branches. Then if (in which case a branch will eventually be selected), a large number of passages must be observed before the clear emergence of a preference. Naturally, the larger is or the closer is to zero, the longer these effects will be seen.
When , the asymptotic behavior of the passage proportion is determined by (see Theorem 4–(ii)). As grows from zero to infinity, the limiting distribution of (as ) evolves continuously from two Dirac point masses at 0 and 1 to a single Dirac mass at 1/2. To illustrate this point, we show in Figure 1 the density of the Beta distribution for and . We make some further comments.
If , a strong asymmetry in the choices of the branches appears. One branch is eventually chosen much more frequently than the other. Furthermore as tends to , the Beta distribution tends to the distribution with two point masses at and . This limit case corresponds to the situation in which a branch is selected, i.e. .
If , the limiting distribution is uniform on .
If , appears to be much more concentrated around . This is similar to what is observed in the case .
To summarize, the model possesses four phases: fast and slow selections and fast and slow uniformizations. These phases are delimited by two phase transitions: a discontinuous one between (branch selection) and (branch uniformization) with a critical state and a smooth one between (strong pheromone deposits) and (weak pheromone deposits). These properties are summarized in the phase diagram in Figure 2. When is small, it is very likely that one branch will be favored during the first passages, thus the empirical distribution of the choices resembles the Beta distribution with a small (the grey solid line in Figure 1). As the number of experiments increases, the shape of the empirical distribution will be closer and closer to its limit: a Dirac mass at if and two Dirac masses at 0 and 1 if . When is large, the earlier passages do not show any preference between the branch. Again, when the number of experiments increases, the asymptotic behavior is progressively revealed.
To date, the most commonly used behavioral state in the model is the slow selection of a branch (see [2, 18]). But at least two other states are interesting. The fast uniformization can describe the case where none of the branches are preferred. The slow uniformization could reproduce the saturation phenomenon. There exists a threshold concentration of pheromone upon which ants can no longer detect the pheromone concentration variations (see ). In experiments involving many ants, one can first observe the favorization of a branch. But when this branch is saturated (its attractiveness stops increasing), ants go more and more through the other branch, whose attractiveness still increases. Eventually, the two branches are uniformly chosen.
In the context of an estimation procedure, the similar effects of and (favorization/selection of a branch or not) induce an identifiability issue. Indeed, we observe a finite number of choices and consequently we only see the short term behavior. We have seen that the favorization of a branch in the first passages could be due to a pair of parameter values with small compared to and close to or to a pair with close to and large compared to .
Thus, we can expect that the estimation of the parameters will be difficult when both parameters contribute to the same effect, e.g. large and small (fast selection of one branch) or small and large (no selection); and also when the parameters have competing effects: very small and , or very large and . The statistical procedure that we introduce in this paper partially circumvents this difficulty, since it focuses on the transition probabilities rather than on the general shape of a curve, which is what calibration methods do. This will be illustrated in Section 2.4.
2.2 Parameter estimation
We start with the maximum likelihood estimator, that is defined by eq. (5). The Fisher information matrix has the following expression.
with and for ,
It is important to note that the Fisher information matrix is not diagonal. Thus the estimation of each parameter has an effect on the estimation of the other.
Let be a compact subset of which contains . Then the maximum likelihood estimator is consistent and asymptotically normal and efficient, i.e. converges weakly to .
As mentioned above, we also use weighted least squared estimators defined by eq. (9) for several different weight sequences , such that converges almost surely to , for all and .
Let be a compact subset of which contains and assume that converges almost surely to positive weights . Then the weighted least squared estimator is consistent and asymptotically normal, i.e. converges weakly to , where is a positive definite covariance matrix. It is efficient, if , for all and .
2.3 Estimation on a single path
The main feature of the binary choice model for is that only one branch will be crossed eventually. It seems clear then that a statistical procedure based on only one path (one sequence of choices) cannot be consistent, since no new information will be obtained after one branch is eventually abandoned. This intuition is true and more surprisingly, it is also true in the case . This is translated in statistical terms in the following theorem. Let denote the log-likelihood based on a single path of length and its gradient. The model is regular, so the Fisher information is .
If and , then .
If , .
If , then converges almost surely to a random variable as .
The proof is in Section 5.6. Statement (i) means that, when , the Fisher information is bounded. This implies that the parameter cannot be estimated on a single path. This also implies that the length of each path should be taken as large as possible (theoretically infinite) in order to minimize the asymptotic variance of the estimators. Statements (ii) and (iii) imply that the maximum likelihood estimator is inconsistent, since the likelihood does not tend to a constant.
2.4 Simulation experiment
In order to assess the quality of the estimators proposed, we have made a short simulation study. For several pairs , we have simulated 1,000 experiments of paths of length (recall that is the number of ants going through the bifurcation). These are reasonable values in view of the practical experiments with actual ants. We compare the performance of the maximum likelihood estimator (MLE) defined in eq. (5) and of the weighted least squares estimator (WLSE) defined in eq. (9) with the weights defined in eq. (7). The asymptotically efficient WLSE with the weights provides a severely biased estimation of and always estimates a very small value of with a very small dispersion. This is caused by the fact that the empirical and vanish frequently, so that the weights are infinite. We will not report the study for this estimator.
The theoretical standard deviation
We first evaluate numerically some values of the theoretical standard deviations of both estimators for several values of and . We have chosen arbitrary values of and in the range . We have also chosen values of and which correspond to those found in the literature cited and to those that we have estimated in the real life experiment described in Section 3. These results are reported in Table 1 and in Figure 3 and their features are summarized in the following points.
As theoretically expected, the asymptotic variance of the MLE, which is the Fisher information bound, is smaller than the variance of the WLSE, but the ratio between the variances of the two estimators is never less than one fourth. Moreover, their overall behavior is similar.
The variance of the estimators of is smaller when both parameters do not contribute to the same effect. The worst variance is for large and small, that is when the parameter values imply fast selection of a branch. The variance tend to infinity when tends to infinity.
The variance of the estimators of increases with and tends to infinity when tends to 0 and to .
These effects are explained by the fact that the coefficients of the Fisher information matrix tend to zero when tends to zero, except the coefficient corresponding to . See Formula (12).
Performance of the estimators
Recall that we have simulated 1000 experiments, each of paths of length . Because of the length of the computations, each MLE was computed only 500 times. Table 1 reports square root mean squared error (MSE) of both estimators based on the simulated data for the same values of the parameters and their features are summarized in the following points.
For most values of the parameters, the MSE are close to the theoretical standard deviation.
The MSE increase significantly when is large or when both parameters contribute to the same effect.
This increase is more noticeable for the WLSE than for the MLE.
This increase is in part due to the skewness of these estimators. For some values of the parameters, both estimators tend to overestimate the parameters.
For the MLE, the MSE is much larger in the case of non-selection than in the case of selection where the empirical performance of the MLE nearly matches the theoretical value.
These effects are always stronger for the estimation of than for the estimation of .
This degraded performance for some specific or extreme values of the parameters is in part due to numerical issues.
In the case where selection of a branch is fast, many of the empirical weights used to compute the WLSE vanish, and the least squares method uses very few points to fit the curve. The MLE is not affected by this problem.
In the case where both parameters concur to non-selection, the probability of choosing one branch converges very fast to 1/2, and thus the experiment brings very little information. This affects both the MLE and the WLSE, and in addition, many of the empirical weights vanish so the WLSE is even less efficient.
The degraded performance may also be caused by the identifiability problem explained in Remark 5, i.e., similar effects of the two parameters make the estimation more difficult.
Bootstrap confidence intervals
Since the asymptotic variance depends on the unknown parameters, we have computed the pivotal Bootstrap 95% confidence intervals for the parameters based on one simulation of paths of length and a Bootstrap sample size of (see , Section 8.3, for details on this method). In Table 2, we have compared these Bootstrap intervals with the corresponding Monte-Carlo intervals, based on 500 simulations. The match is nearly perfect for the MLE for , but as before, the performance is poorer for the estimation of . The intervals for are noticeably skewed to the right but always contain the true value. For further comparison, we only show here the results corresponding to the values of the parameters estimated in the real life experiment reported below and those corresponding to values found in the earlier literature.
3 Real life experiment with ants
In this section, we apply the previous estimators on data from a path selection experiment by a colony of ants.
3.1 Experiment description
This experiment was done in the Research Center on Animal Cognition (UMR 5169) of Paul Sabatier University Toulouse under the supervision of Guy Theraulaz, Hugues Chaté and the first author. A small laboratory colony (approximately 200 workers) of Argentine ants Linepithema humile was starved for two days before the experiment. During the experiment, the colony had access to a fork carved in a white PVC slab, partially covered by a Plexiglas plate (see Figure 4). The angle between the branches was . The fork galleries had a cm square section. The entrance of the maze was controlled by a door. Food was never present during the experiment. The maze was initially free of any pheromone trail.
Each trial () consisted in introducing separately each ant to the entrance of the fork (see Figure 4) one at a time. Once inside, an ant must choose between the left or the right branch of the fork. As soon as the ant had made a choice and stepped into one branch, it was removed from the setup and another ant was introduced. All the choices were recorded and a trial ended when 100 ants had passed through the fork.
This experimental protocol was designed to strengthen the behavioral assumptions described in Section 3. Any return to the fork is forbidden so that we can consider that each ant passed only one time. There was never more than one ant in the setup. This implies that each ant in the maze received no other cue about the previous passages than the pheromone that was been laid. The species Linepithema humile was in part chosen to justify the assumption of identical pheromone deposits. Indeed, these ants may deposit regularly the same type of pheromone on their trajectory (see [21, 22]). All ants were prepared the same way before the experiments to increase the credibility of the assumption stating that each ant behaved by the same way. The length of the experiments was limited to stay close to the half-life duration of the pheromone trails (see ).
3.2 Data representation
Figures 5(a) shows the 50 paths of length , that is, choice sequences of 100 ants that went through the bifurcation. The paths are represented as random walks with increment when the right branch is chosen, and when the left one is chosen. In less than ten experiments, a branch seemed to be selected, whereas in the others, selection of a branch was not obvious. Figure 5(b) shows the histogram of the distribution of , that is the final proportion of the choices of the right branch. There is no clear visual evidence that as it is claim in the literature (see [2, 5, 6]).
3.3 Parameter estimation
Several values of these parameters have been proposed in the applied literature.  proposed , and more recently  suggested and . It must be noted however that these values are not obtained by a statistical method but by the calibration of a curve to a plot. Therefore, these methods do not lead to confidence intervals. Moreover, a calibration method has an inherent risk of over fitting, because of the identifiability problem explained in Remark 5. As illustrated in Figure 2, if for instance and are both small, then both branches will be asymptotically equally chosen, but paths of finite length might be misleading and the calibration will suggest values of and corresponding to the selection of a branch. The statistical procedure is based on the dynamics of the process and is thus less prone to this type of error. Nevertheless, we will see that our results do not contradict those of [2, 6], but complement them.
Table 3 shows the results of the maximum likelihood estimation and the weighted least squares estimation. Both estimates of are close to 1.1 and the estimates of are between 3 and 7. The confidence intervals are slightly larger than the simulated ones (see Table 2). This increased variability may be due to the extreme paths which seem to show a very fast selection of one branch (see Figure 5). This may suggest that the ants did not have the same behavior and that the distribution of could be a mixture of two distributions.
For both methods, the 95% Bootstrap confidence intervals of contain the value 1. More precisely, as shown in Figure 6, approximately 1/3 of the bootstrap parameters gives weak pheromone deposits () and a weak differential sensitivity (), which means that branches are eventually uniformly crossed. In almost all the others cases, we conclude for weak pheromone deposits () and a strong differential sensitivity (), which means that a branch will be eventually, though slowly, selected. In only a few cases do the estimators give strong pheromone deposits (), but a weak differential sensitivity (), which means that a branch is chosen more than the other at the beginning of the experiment, but branches are eventually uniformly crossed. Finally, there are no values which imply both strong pheromone deposits () and a strong differential sensitivity (). Therefore, we can conclude that pheromone deposits are weak with a good confidence but we cannot confidently decide for .
The values obtained by  (, ) and more recently by  (, ) are both in the confidence intervals for the WLSE found in Table 3. But the values of suggested by these authors are out of the 95% confidence interval for the MLE. Thus these parameters, which decide for a slow branch selection, are no more likely than a parameter set which would yield non-selection of a path.
Figure 6 illustrates the fact that the two estimators are strongly positively correlated. There seems to be two cutoff values for : if , then , and if , then . The above mentioned values reported by [2, 6] exhibit these features: they both have and and increase with .
If we fix the value of and estimate only , then the 95% Bootstrap confidence intervals for are smaller. Figure 7 shows the estimated values of and the confidence intervals as functions of the fixed value of . We see that if is greater than for the MLE (or than for the WLSE), then the confidence intervals of lie entirely above . Furthermore if is less than for the MLE (or than for the WLSE), then the confidence intervals of lie entirely under . This shows that if the deposits are weak enough, i.e. , we can conclude that a slow selection of a branch will occur with probability 1. On the other hand, if the deposits are strong enough, i.e. , we can conclude that branches will eventually be uniformly crossed with probability 1.
4 Concluding remarks
In the literature no parameter estimation methods for reinforced random walks can be found. To partially fill this void, this article proposes a statistical framework to estimate the parameter of a general two-colored urn model. We define the maximum likelihood estimator (MLE) and the weighted least squares estimators (WLSE) for the parameter of this model and prove their consistency and their asymptotically normality under some usual regularity conditions. The proof lies on a general result for a large class of estimators called minimum contrast estimators. The MLE is asymptotically efficient, but can be difficult (lengthy) to compute, which can be an issue specially while using Bootstrap algorithms. The WLSE is a suitable alternative. Moreover this estimator is popular among practitioners.
We apply this statistical tool to the problem of path selection by an ant colony. To this purpose, we performed experiments with actual ants to collect data. The experiment consisted of introducing one hundred ants into a shaped device, one at a time, and observing their successive choices. We also consider the particular urn model introduced by  to describe this phenomenon. This urn has two parameters, and , which have distinct biological interpretations, but contribute to the same effect: either selection of a branch or uniformization of the choices. The parameter influences the short term behavior, whereas determines the asymptotic behavior. Consequently the model exhibits four phases which are illustrated by Figure 2. The case most commonly considered in the literature is the case of slow selection, which corresponds to and : the ants will eventually always choose the same branch, but this selection will take a long time. For instance,  provides the values and . However other phases can be relevant to describe the ant behavior. For instance the fast uniformization, corresponding to and , can model the less likely but not negligible case in which ants do not select a branch.
After assessing the accuracy of the MLE and the WLSE on simulated data, we estimate the value of and with the two estimators. We also evaluate confidence regions by Bootstrap proceeding. The estimated values of and ranged between 1.1 and 3 and between 3 and 7, respectively. This tends to imply that slow selection of a branch will occur. However, the Bootstrap sample gives a confidence level of for the hypothesis of slow selection, while the hypothesis of fast uniformization has a confidence of .
This low level of confidence for the commonly assumed slow selection phase might be explained by technical reasons. The number of experiments (50) is relatively small; increasing the number of replicas will reduce the confidence regions. Moreover the competition between the parameters for the same effect induces an identifiability issue. For instance the apparent preference of a branch may be due to or small with respect to 1. Therefore, the model, which is biologically relevant, is statistically difficult to estimate. Indeed, for an ethological study, discriminating the ant pheromone sensitivity from the pheromone deposit strength is meaningful. But for a statistical procedure, the similarity of effect of the two parameters scales down the estimation performance.
However, the uncertainty may not come from an inefficiency of the statistical procedure, but from shortcomings of the ethological hypotheses. Indeed, the estimated confidence intervals computed from the experimental data are larger than the ones computed from the simulated data (for similar parameter values). Moreover the assumption that the inter-individual variability is negligible is strong. For instance, it may be necessary to consider that the pheromone deposit varies at each passage, i.e. that is random.
These ethological considerations will be further discussed in a forthcoming paper which will analyze more elaborate experimental designs. The ants will be observed while freely evolving in a network with several nodes. In addition to a data analysis, we will model the experiment with a reinforced random walk on a finite graph for which we have provided probabilistic results (see ). The statistical methodology introduced in this paper will be extended to a larger class of reinforced random walks.
5.1 Distribution of , for
In order to compute the distribution of , we introduce some notations. Let be the set of sequences of length of integers such that and for . For let . Then we have
A central limit theorem for the empirical conditional probabilities
For , recall the definition of and in eq. (7) and that .
converges weakly to a Gaussian vector with diagonal covariance matrix with diagonal elements (14)
Define , the empirical estimate of and . Write then
Since the paths , are i.i.d., the multivariate central limit holds for the sequence of dimensional vectors . The proof is concluded by tedious computations using the Markov property, which we omit.
We can prove that the covariance matrix is diagonal by a statistical argument. If we consider the tautological model , i.e. and is the true value. Then the log-likelihood is where . Thus we see that is the maximum likelihood estimator of . This model is a regular statistical model, thus converges weakly to the Gaussian distribution with covariance matrix , where is the Fisher information matrix of the model. It is easily seen that is the dimensional diagonal matrix with diagonal elements given by eq. (14).
5.3 A general result for minimum contrast estimators
Theorems 2 and 3 are a consequence of the general result we prove in this section. More precisely we demonstrate the consistency and the asymptotic normality of a general estimator of which the MLE and the WLSE are particular cases.
For , recall the definition of , in eq. (7) and that . Let , be a sequence of random weights and let be function defined on . Define the empirical contrast function by
For instance, choosing and yields
so that minimizing is equivalent to maximizing the likelihood , defined in eq. (4). Choosing yields the weighted least squares contrast function , defined in eq. (8). We now define the minimum contrast estimator of by
In order to prove the consistency and asymptotic normality of , we make the following assumptions on and on the weights . Let and denote the first and second derivatives of with respect to its second argument.
The function is non-negative, twice continuously differentiable on with if , and .
For all , converge almost surely to and .
The exact expression of the variance is given in the proof.
Under Assumption 11, the strong law of large numbers shows that converges almost surely to
Moreover the convergence of to is uniform, since is compact and the function is twice continuously differential with respect to , its first variable. This yields the consistency of . For the sake of completeness, we give a brief proof. Since minimizes and minimizes , we have
Since is the unique minimizer of , for , we can find such that if and , then . Thus
The central limit theorem is a consequence of the consistency and Lemma 9. A first order Taylor extension of at yields
where . Setting , we have
Let be the mixed second derivative of . Note that
Thus, by the delta-method (see , Theorem 3.3.11) and since converges almost surely to , we obtain that converges weakly towards
where are independent Gaussian random variables with zero mean and variance defined in eq. (14). Equivalently, converges weakly to a Gaussian vector with zero mean and covariance matrix defined by
By the law of large numbers, converges almost surely to and this convergence is also locally uniform. Thus, converges almost surely to . Using again the fact that , we obtain
Denote for brevity . Then, for any , we have
We can now conclude that for large enough , is invertible and we can write
The right hand side converges weakly to the Gaussian distribution with zero mean and covariance matrix .
Assumption 12 holds for the weights and , .
2. As mentioned above, the maximum likelihood estimator minimizes the contrast function written with the function and the weights . Thus the proof of Theorem 2 consists in checking Assumption 11 and 12 to apply Theorem 13. Lemma 14 implies that Assumption 12 holds.
The function considered here satisfies Assumption 11. Indeed, for , define
Remark that is the Kullback-Leibler distance between the Bernoulli measures with respective success probabilities and . Then it is well known that except if . Indeed, by Jensen’s inequality,
and by strict concavity of the log function, equality holds only if . Moreover, so and . ■
Proof of theorem
So that . ■
If the weights are chosen as , then and
5.5 Proofs of Corollaries 6 and 7
Corollaries 6 and 7 are a consequence of Theorem 13. The assumptions on the weights and on the functions have been already verified in the previous section. We have to prove the Assumption 1 on the choice function defined in (10). Hypothesis 1-(i) is obvious.
By elementary computations, we have, for , (19)
Plugging the pairs and into (19) yields
We now prove that if , the vectors and are linearly independent in . For , we have,
Let and assume that for all such that , it holds that
Replacing for instance successively by and yields
If , this implies
By strict convexity of the function on , this is impossible. Thus and Assumption 1-(iii) holds.
5.6 Proof of Theorem 8
Proof of Theorem
8, case In this case the model is Pólya’s urn, and we have (21)
The distribution of is given by
For any , there exists constants such that, for all integers ,
Therefore, there exists a constant such that for all ,
In all three cases, we obtain that the first series in eq. (21) is summable. By symmetry, the sum of the second expectations is also finite. ■
Proof of Theorem
Proof of Theorem
8, case . Let be the event that color 1 is eventually selected, which happens with probability 1/2 by Theorem 4. Then, on , and if , then and . Thus for large enough , the log-likelihood on one path becomes
As , for any ,
If the series is divergent and thus . If then the series is convergent and thus, on ,
This implies that and that this argmax is a random variable which is a function of the whole path, and does not depend on the true value . □
We thanks Guy Theraulaz and Hugues Chaté for providing their material framework and their field expertise to allow the first author to collect the data of the Argentine ants experiments. These experiments are part of the project TRACES supported by the CNRS. They were done during two visits in April and July 2012 of the first author to the Centre de Recherches sur la Cognition Animale (CRCA, Centre de Recherches sur la Cognition Animale, UMR 5169, Paul Sabatier University, Toulouse), whose hospitality is gratefully acknowledged. Line C. Le Goff has been supported by a grant from the French Ministry of Education, Research and Technology and then by a Fyssen Foundation post-doctoral grant.
Pólya G. Sur quelques points de la théorie des probabilités. Ann IHP 1931;1:117–61. Google Scholar
Tarrès P. Localization of reinforced random walks, 2011. Available at 5536 arXiv:1103.5536 http://arxiv.org/abs/1103. Google Scholar
Davis B. Reinforced random walk. Probab Theory Related Fields 1990;84:203–29. Google Scholar
Vittori K, Talbot G, Gautrais J, Fourcassié V, Araujo A, Theraulaz G. Path efficiency of ant foraging trails in an artificial network. J Theor Biol 2006;239:507–15. Google Scholar
Garnier S, Guérécheau A, Combe M, Fourcassié V, Theraulaz G. Path selection and foraging efficiency in argentine ant transport networks. Behav Ecol Sociobiol 2009;63:1167–79. CrossrefWeb of ScienceGoogle Scholar
Thienen W, Metzler D, Choe DH, Witte V. Pheromone communication in ants: a detailed analysis of concentration-dependent decisions in three species. Behav Ecol Sociobiol 2014;68:1611–27. Web of ScienceCrossrefGoogle Scholar
Arganda S, Nicolis S, Perochain A, Péchabadens C, Latil G, Dussutour A. Collective choice in ants: The role of protein and carbohydrates ratios. J Insect Physiol 2014;69:19–26. Web of ScienceCrossrefGoogle Scholar
Aron S, Deneubourg JL, Goss S, Pasteels J. Functional self-organisation illustrated by inter-nest traffic in ants : the Case of the Argentine ant In: Alt W, Hoffmann G, editors. Biological Motion: Proceedings of a Workshop held in Königswinter, Germany, March 16–19, 1989. Berlin, Heidelberg: Springer; 1990:533–47.
Beckers R, Deneubourg JL, Goss S. Modulation of trail laying in the ant lasius niger (hymenoptera: formicidae) and its role in the collective selection of a food source. J Insect Behav 1993;6:751–9. CrossrefGoogle Scholar
Dussutour A, Deneubourg JL, Fourcassié V. Amplification of individual preferences in a social context: the case of wall-following in ants. Proc R Soc London Ser B 2005;272:705–14. Google Scholar
Nicolis S, Deneubourg JL. Emerging patterns and food recruitment in ants: an analytical study. J Theor Biol 1999;198:575–92. Google Scholar
Beckers R, Deneubourg JL, Goss S. Trails and u-turns in the selection of a path by the ant lasius niger. J Theor Biol 1992;159:397–415. Google Scholar
Pasteels J, Deneubourg JL, Goss S. Transmission and amplification of information in a changing environment: the case of insect societies. In: Prigogine I, Sanglier M, editors. Laws of nature and human conduct: Specificities and unifying themes. Brussels: Gordes, Brussels; 1987. Google Scholar
Wasserman L. All of statistics: a concise course in statistical inference. New York: Springer; 2004. Google Scholar
Van Vorhis Key S, Baker T. Trail-following responses of the argentine ant, iridomyrmex humilis (mayr), to a synthetic trail pheromone component and analogs. J Chem Ecol 1982;8:3–14. CrossrefGoogle Scholar
Aron S, Pasteels J, Deneubourg JL. Trail-laying behaviour during exploratory recruitment in the argentine ant, iridomyrmex humilis (mayr). Biol Behav 1989;14:207–17. Google Scholar
Dacunha-Castelle D, Duflo M. Probability and statistics. vol. II. New York: Springer; 1986. Google Scholar