Show Summary Details
More options …

# Statistical Applications in Genetics and Molecular Biology

Editor-in-Chief: Sanguinetti, Guido

6 Issues per year

IMPACT FACTOR 2017: 0.812
5-year IMPACT FACTOR: 1.104

CiteScore 2017: 0.86

SCImago Journal Rank (SJR) 2017: 0.456
Source Normalized Impact per Paper (SNIP) 2017: 0.527

Mathematical Citation Quotient (MCQ) 2017: 0.04

Online
ISSN
1544-6115
See all formats and pricing

Access brought to you by:

provisional account

More options …
Volume 14, Issue 3

# Weighted Kolmogorov Smirnov testing: an alternative for Gene Set Enrichment Analysis

Konstantina Charmpi
• Université Grenoble Alpes, France
• Laboratoire Jean Kuntzmann, CNRS UMR5224, Grenoble, France
• Laboratoire d’Excellence TOUCAN, Toulouse, France
• Other articles by this author:
/ Bernard Ycart
• Corresponding author
• Université Grenoble Alpes, France
• Laboratoire Jean Kuntzmann, CNRS UMR5224, Grenoble, France
• Laboratoire d’Excellence TOUCAN, Toulouse, France
• Email
• Other articles by this author:
Published Online: 2015-05-30 | DOI: https://doi.org/10.1515/sagmb-2014-0077

## Abstract

Gene Set Enrichment Analysis (GSEA) is a basic tool for genomic data treatment. Its test statistic is based on a cumulated weight function, and its distribution under the null hypothesis is evaluated by Monte-Carlo simulation. Here, it is proposed to subtract to the cumulated weight function its asymptotic expectation, then scale it. Under the null hypothesis, the convergence in distribution of the new test statistic is proved, using the theory of empirical processes. The limiting distribution needs to be computed only once, and can then be used for many different gene sets. This results in large savings in computing time. The test defined in this way has been called Weighted Kolmogorov Smirnov (WKS) test. Using expression data from the GEO repository, tested against the MSig Database C2, a comparison between the classical GSEA test and the new procedure has been conducted. Our conclusion is that, beyond its mathematical and algorithmic advantages, the WKS test could be more informative in many cases, than the classical GSEA test.

AMS Subject Classification: Primary 62F03; Secondary 60F17

## 1 Introduction

Since its definition by Subramanian et al. (2005), Gene Set Enrichment Analysis (GSEA) has been very successful, and it may now be considered as the most basic tool of genomic data treatment: see Bild and Febbo (2005), Nam and Kim (2008), Huang et al. (2009) for reviews. GSEA aims at comparing a vector of numeric data indexed by the set of all genes, to the genes contained in a given smaller gene set. The numeric data are typically obtained from a microarray experiment. They may consist in expression levels [see Barbie et al. (2009)], p-values, correlations, fold-changes (FC), t-statistics, signal-to-noise ratios, etc. The number associated to any given gene will be referred to as its weight. For technical reasons, weights are supposed to be positive hereafter (they can always be transformed to be so). Many examples of such data can be downloaded from the Gene Expression Omnibus (GEO) repository (Edgar et al., 2002). The gene set may contain genes known to be associated to a given biological process, a cellular component, a type of cancer, etc. Thematic lists of such gene sets are given in the Molecular Signature (MSig) database (Subramanian et al., 2005). The question to be answered is: are the weights inside the gene set significantly different from weights in a random gene set of the same size? As customary in statistical testing, three alternatives to the null hypothesis can be considered, corresponding here to the gene set having significantly higher weights (enrichment), lower weights (depletion) or both (two-tailed test). In what follows, only the development of the two-tailed test will be detailed. The one-tailed tests are easily deduced.

Denote by N the total number of genes (N≃20,000 for the human genome). It will be convenient to identify the genes to N regularly spaced points on the interval [0, 1], and their weights to the values of a positive valued function g, defined on [0, 1]: gene number i corresponds to point i/N, and its weight wi to g(i/N). In Subramanian et al. (2005), the numbering of the genes is chosen so that weights are ranked in decreasing order. Thus, the weights usually appear to vary smoothly between contiguous genes, and the function g can be assumed to be continuous.

The gene set is included in the set of all genes. Let n be its size. In practice, n ranges from a few tens to a few hundreds: n is much smaller than N. With the identification above, it is considered as a subset of size n of the interval [0, 1], say {U1, …, Un}. If there is no particular relation between the weights and the gene set (null hypothesis), then the gene set must be considered as a uniform random sample without replacement of the set of all genes. The fact that the gene set size n is much smaller than N, justifies identifying the distribution of a uniform n-sample without replacement of {1/N, …, N/N} to that of a n-sample of points, uniformly distributed on [0, 1]. Therefore, the null hypothesis is:

H0: The gene set is a n-tuple (U1, …, Un) of independent, identically distributed (i.i.d.) random variables, uniformly distributed on the interval [0, 1].

The basis of the test statistics is the following step function, cumulating the proportion of weights inside the gene set, along the interval [0, 1]. It is defined for all t between 0 and 1 by:

$Sn(t)=∑k=1ng(Uk)IUk≤t∑k=1ng(Uk), (1)$(1)

where 𝕀 denotes the indicator of an event. The absolute test statistic proposed by Subramanian et al. (2005) is:

$Tn=supt∈[0,1]|Sn(t)−t|. (2)$(2)

The motivation is best understood in the particular case where the weights wi are constant. Then the function g is also constant, and:

$Sn(t)=∑k=1n1nIUk≤t.$

This is the empirical Cumulative Distribution Function (CDF) of the sample (U1, …, Un). The test statistic Tn is the maximal distance between that empirical CDF and the theoretical CDF of the uniform distribution on the interval [0, 1]. In other terms, $nTn$is the Kolmogorov Smirnov (KS) test statistic for the goodness-of-fit of the uniform distribution on [0, 1] to the sample (U1, …, Un) (Arnold and Emerson, 2011). The constant weight case was initially proposed by Mootha et al. (2003), who explicitly referred to the KS statistic [see also (Subramanian et al., 2005, Supporting text, p. 5, 6, 11), Tarca et al. (2013) and Ycart et al. (2014)]. In the general case where the weights are not constant, the distribution of the test statistic Tn under the null hypothesis is unknown. In the current implementations, it is approximated by Monte-Carlo simulation on 1000 random samples (Subramanian et al., 2007).

Our first remark is that in the non constant case, the limit of Sn(t) as n tends to infinity is not t, as (2) seems to suggest, but instead:

$limn→∞ Sn(t)=∫0tg(u)du∫01g(u)du.$

Thus the GSEA test statistic Tn is not appropriately centered, unless the weights are constant. Instead, the following test statistic should be used:

$Tn∗=nsupt∈[0,1]|Sn(t)−∫0tg(u)du∫01g(u)du|. (3)$(3)

The objective of this paper is to derive the asymptotic distribution of $Tn∗$ under the null hypothesis, then deduce from the mathematical result a practical testing procedure, and compare the outputs of that procedure to those of the classical GSEA test.

Our theoretical result is the following.

Theorem 1.1. Let g be a continuous, positive function from [0, 1] into ℝ. Denote by G its primitive: $G(t)=∫0tg(u)du,$ and assume that G(1)=1. Let (Un)n∈ℕ be a sequence of i.i.d. random variables, uniformly distributed on [0, 1]. For all n≥1, and for all t in [0, 1], consider the random variable Sn(t) defined by (1). Let

$Zn(t)=n(Sn(t)−G(t)). (4)$(4)

As n tends to infinity, the stochastic process {Zn(t), t∈[0, 1]} converges weakly in ([0, 1]) to the process {Z(t), t∈[0, 1]} where:

$Z(t)=∫0tg(u)dWu−G(t)∫01g(u)dWu, (5)$(5)

and {Wt, t∈[0, 1]} is the standard Brownian motion.

The hypothesis $∫01g(u)du=1$ induces no loss of generality: since g is continuous and positive, its integral is positive; g can be divided by its integral without changing the values of the cumulated proportion of weights Sn(t). The proof of Theorem 1.1 will be given in Section 2. It is based on the theory of empirical processes, for which Shorack and Wellner (1986) and Kosorok (2008) will be used as general references.

The first consequence of Theorem 1.1 for GSEA, is that as n increases, the distribution of the proposed test statistic $Tn∗$ under the null hypothesis, tends to that of the following random variable T:

$T=supt∈[0,1]|Z(t)|,$

where the random process Z is defined by (5). Denote by F its CDF: for all x>0,

$F(x)=Prob(T≤x). (6)$(6)

Observe that F(x) only depends on g, i.e., on the weights of the vector to be tested. Except in the classical KS case of constant weights, F does not have a closed-form expression, but a Monte-Carlo approximation is easily obtained. The testing procedure generalizes that of the classical KS test: since the test statistic $Tn∗$ has asymptotic CDF F under the null hypothesis, the p-value of an observation $Tn∗=x$ is 1−F(x). That testing procedure will be referred to as Weighted Kolmorov Smirnov (WKS) test. Using the terminology of Goeman and Bühlmann (2007), the WKS test is a “competitive” method. The WKS test assumes that the random variables are independent and identically distributed and thus suffers from the same limitations as other well known methods [see for instance Kim and Volsky (2005), Irizarry et al. (2009)]. The technique of incorporating a variance inflation factor, as proposed by Wu and Smyth (2012), could be employed. As was shown in Ycart et al. (2014), partitioning the genes according to their frequencies should also reduce the false detection rate. A crucial feature is that, since F only depends on the weights, the same evaluation of F can be repeatedly used for many gene sets, which saves computing time. Of course, the repeated application of a test to a full database of several thousand gene sets poses the problem of false discovery rate (FDR) correction. In applications, we have used the method of Benjamini and Yekutieli (2001): see Dudoit and van der Laan (2007) for multiple testing procedures in genomics.

Like the KS test, the WKS test is based on an asymptotic result. In practice, it is used for finite values of n. Therefore, it is necessary to determine for which size n of gene sets, the test can be applied with good precision. A Monte-Carlo comparison of the cumulative distribution function of $Tn∗$ to its limit F for different values of n was conducted. Our conclusion is that the asymptotic test can be safely applied for gene set sizes n larger than 40. For lower values, Monte-Carlo estimation of the p-value can be used instead. Beyond validation through simulated values, it was necessary to compare the outputs of the WKS test to those of the classical GSEA test, on real data. Inside the GEO dataset GSE36133 of Barretina et al. (2012), we have selected vectors (samples) from different types of tumors. These vectors were tested against all gene sets of MSig database C2, calculating for each sample the p-values of both tests. The gene sets known to be related to the same type of cancer as the initial vector were of particular interest. An example corresponding to a sample of liver tumor will be reported; we consider it as typical of the observations that were made with other samples. The obtained results are encouraging: the WKS test tends to output less significant gene sets than the classical GSEA test out of the whole database, but more out of those gene sets related to the correct type of cancer. Our conclusion is that, beyond its mathematical and algorithmic advantages, the WKS test could be more informative in many cases, than the classical GSEA test.

The article is organized in the following way. In Section 2, Theorem 1.1 is proved, and the asymptotic distribution of $Tn∗$ is deduced. Section 3 is devoted to the statistical application, beginning with the description of the Monte-Carlo algorithm of calculation of p-values. Results of simulated tests are reported next. Finally, examples of comparison of the WKS test with the GSEA test on real data are discussed.

## 2 Theoretical background

The notations and results of Kosorok (2008) will be used. In particular, throughout the section, ⇝ denotes the weak convergence of processes in ([0, 1]). We first give the proof of Theorem 1.1, which asserts the convergence ZnZ, where Zn is the empirical process defined by (4), and Z is the Gaussian bridge defined by (5).

Proof. The idea is the following. Consider:

$Zn1(t)=∑k=1ng(Uk)nZn(t). (7)$(7)

Using the general results on empirical processes and Donsker classes, exposed in Section 9.4 of Kosorok (2008), it will be proved that $Zn1⇝Z.$ By the law of large numbers,

$limn→∞∑k=1ng(Uk)n=∫01g(u)du=1, a.s.$

The convergence ZnZ follows as an application of Slutsky’s theorem: Theorem 7.15 of (Kosorok, 2008, p. 112).

The random variable $Zn1(t)$ can be written as follows:

$Zn1(t)=1n(∑k=1ng(Uk)I{Uk≤t}−G(t)∑k=1ng(Uk))=1n(∑k=1ng(Uk)(I{Uk≤t}−G(t))),$

denoting by G the primitive of g, as before. Empirical processes are customarily written as function-indexed processes. Define the class of functions ℱ by:

$ℱ={g(⋅)(I[0,t](⋅)−G(t)) ; t∈[0, 1]}.$

Denote by ℙn the empirical measure of (U1, …, Un), by ℙ the uniform distribution on [0, 1], by ℙnf and ℙf the integrals of f with respect to ℙn and ℙ (Kosorok, 2008, p. 11). For f ∈ℱ, define $Z˜n1(f)$ by:

$Z˜n1(f)=n(ℙnf−ℙf). (8)$(8)

Obviously, for all t∈[0, 1],

$Zn1(t)=Z˜n1(g(⋅)(I[0,t](⋅)−G(t))). (9)$(9)

Let us prove that ℱ is a Donsker class. Firstly, observe that the following class ℱ1 is Donsker.

$ℱ1={I[0,t](⋅)−G(t) ; t∈[0, 1]}.$

Indeed, for f ∈ℱ1, the process $n(ℙnf−ℙf)$ converges weakly to the standard Brownian bridge. Since all functions in ℱ1 take values between −1 and 1, the supremum of |ℙf| over ℱ1 is not larger than 1. The function g, being continuous on a compact interval, is bounded and measurable. From Corollary 9.32, p. 173 of Kosorok (2008), it follows that ℱ is also Donsker. The convergence of $Z˜n1$ now follows from the result of (Kosorok, 2008, p. 11). The limit $Z˜1$ is a zero mean, ℱ -indexed, Gaussian process. Its covariance function is defined, for all f1, f2 in ℱ by:

$E[Z˜1(f1)Z˜1(f2)]=ℙ(f1f2)−ℙf1ℙf2. (10)$(10)

Through (9), the convergence of $Z˜n1$ induces the convergence of $Zn1,$ to a zero mean, [0, 1] -indexed process Z1. Let us compute the covariance function of Z1. For s, t in [0, 1], let:

$f1(⋅)=g(⋅)(I[0,s](⋅)−G(s)) and f2(⋅)=g(⋅)(I[0,t](⋅)−G(t)).$

Applying (10) to these functions f1 and f2 yields:

$E[Z1(t)Z1(s)]=∫0min(t,s)g2(u)du−G(t)∫0sg2(u)du−G(s)∫0tg2(u)du+G(s)G(t)∫01g2(u)du. (11)$(11)

There remains to be proved that Z1 and Z have the same distribution, where Z is defined by the representation (5) in terms of the standard Brownian motion W:

$Z(t)=∫0tg(u)dWu−G(t)∫01g(u)dWu.$

It is a well known fact that the primitive of a deterministic function with respect to the Brownian motion is Gaussian: therefore Z is a Gaussian process. The covariance function is easily calculated, using formula (32), p. 128 of Shorack and Wellner (1986): it is indeed defined by (11). The processes Z1 and Z are both Gaussian, their means and covariance are equal, therefore they have the same distribution.      □

As explained in the introduction, the random variable of interest for GSEA is the supremum of the process |Z| over the interval [0, 1].

Corollary 2.1. Under the notations and hypotheses of Theorem 1.1, let

$Tn∗=supt∈[0,1]|Zn(t)|.$

Then $Tn∗$ converges in distribution to

$supt∈[0,1]|Z(t)|=supt∈[0,1]|∫0tg(u)dWu−∫0tg(u)du∫01g(u)dWu|,$

where W denotes the standard Brownian motion.

Proof. The mapping f↦supt∈[0, 1]|f(t)|, from ([0, 1]) into ℝ+, is continuous. From Theorem 1.1, ZnZ. The conclusion follows as an application of Theorem 7.7, p. 109 of Kosorok (2008).      □

## 3.1 Implementation

The R code (R Core Team, 2013) implementing the WKS test has been made available online, together with a user manual and samples of data. Several issues regarding the implementation are discussed here. The essential step is the evaluation of the cumulative distribution function distribution F defined by (6), or else:

$F(x)=Prob(supt∈[0,1]|∫0tg(u)dWu−G(t)∫01g(u)dWu|≤x). (12)$(12)

A Monte-Carlo calculation has to be used. First of all, sample paths for the stochastic process

${∫0tg(u)dWu ; t∈ [0, 1]}$

must be simulated. This is done using a standard Euler-Maruyama scheme: see Sauer (2013) for a review of numerical methods for stochastic integrals and differential equations. A regular subdivision of the interval [0, 1] into m intervals is chosen:

$ti=im, i=0, …, m.$

Recall that in practice, the function g is known at points i/N representing the genes. Hence it is natural to choose m=N. The stochastic integral is approximated by a sum:

$∫0tg(u)dWu≈∑i=0m−1g(ti)(Wti+1∧t−Wti∧t). (13)$(13)

The increments Wti+1Wti are easily simulated as i.i.d. centered Gaussian variables, with variance 1/m. An estimate of the CDF F is obtained by simulating nsim discretized trajectories of Z, taking the maximum of the absolute value of each, then returning the empirical CDF of the obtained sample. The algorithm can be written as follows.

Actually, since F(x) is evaluated as the proportion of a sample below x, the result must take the uncertainty into account. We propose to return the lower bound of the 95% left-sided confidence interval, instead of the point estimate. This gives an upper bound for the p-value, which is a conservative evaluation. As stated before, the CDF F only depends on the weight function g. The relation between g and F is illustrated on Figure 1. Five different CDF’s have been computed, for gk(x)=(k+1)(1−x1/k), k=0, 1, 2, 3, 4. Denote them by F0, …, F4. The case k=0 is that of constant weights, and can be used as a validation for the algorithm above: F0 is the Kolmogorov Smirnov CDF, which has an explicit expression. It can be checked that the estimate output by Algorithm 1 is close to the known exact function. The curves of Figure 1 were obtained via 20,000 Monte-Carlo simulations, over 15,000 discretization points. It turns out that for all x, F0(x)>…>F4(x): the steeper g, the smaller F, and the larger the p-values. The differences between the curves are sizable: calculating sup|FkF0| for k=1, …, 4 gives 0.199, 0.271, 0.324, 0.356. Theoretical functions g may seem of little practical interest. This is not so, for two reasons. The first reason is the use of robust statistics [see Héritier et al. (2009) as a general reference, and Tsodikov et al. (2002) for application to expression data]. If the initial values are replaced by their ranks, then the weights are N, N−1, …, 2, 1. Therefore, the weight function is g1(x)=2(1−x). This justifies calculating F1 with good precision, which makes the WKS test fast and precise, for all uses over rank statistics. We have done so, using 106 Monte-Carlo simulations, and 105 discretization points. The second reason is the observation of F when the weights come from real data. Eight different GEO datasets were considered: GSE36382 (Mayerle et al., 2013), GSE48348 (Westra et al., 2013), GSE36809 (Xiao et al., 2011), GSE31312 (Frei et al., 2013), GSE48762 (Obermoser et al., 2013), GSE37069 (Seok et al., 2013), GSE39582 (Marisa et al., 2013), and GSE9984 (Mikheev et al., 2008). Several samples of expression levels in each study were selected. In each sample, the expression levels were ranked in decreasing order, and Algorithm 1 was applied in order to obtain an estimation of F. For all real datasets, the estimated F was such that F4(x)<F(x)<F0(x). It seems to be the case in practice that F4 and F0 provide lower and upper bounds for F.

Figure 1:

Cumulated distribution functions Fk corresponding to gk(x)=(k+1)(1−x1/k), for k=0, 1, 2, 3, 4. The highest curve corresponds to k=0 (constant weights, classical Kolmogorov Smirnov CDF). The CDF’s decrease as k increases: the steeper g, the smaller F, and the larger the p-values.

Algorithm 1

Approximation of F

The next algorithmic point concerns the calculation of the test statistic, that is the value of $Tn∗$ defined by (3) for a given set of weights and a gene set of size n:

$Tn∗=nsupt∈[0,1]|Sn(t)−G(t)|,$

where

$Sn(t)=∑k=1ng(Uk)IUk≤t∑k=1ng(Uk).$

The values g(Uk) are the weights of genes inside the gene set. Observe that, if the same vector has to be tested against many gene sets, the calculation of G(t) (cumulated sums of all weights) must be done only once. The value of $Tn∗$ is returned by a procedure similar to that of the classical KS test. Consider two non-decreasing functions f and h where f is a step function with jumps on the set {x1, …, xn} and h is continuous. The supremum of the difference between f and h is computed as follows (Arnold and Emerson, 2011, p. 35).

$supx|f(x)−h(x)|=maxi{max{|h(xi)−f(xi)|, |h(xi)−f(xi−1)|}}.$

As mentioned in the introduction, the WKS has been made one-sided, by testing the signed difference between Sn(t) and G(t): a gene set for which the infimum of Sn(t)−G(t) is significantly negative, contains genes whose weights tend to be small (depletion). Conversely, gene sets for which the supremum of Sn(t)−G(t) is significantly positive, contain genes whose weights tend to be large (enrichment).

## 3.2 Validation of asymptotics on simulated data

Since the WKS test relies on a convergence theorem, it is necessary to determine the values of n (the gene set size) for which the procedure yields precise enough results. Such a validation is standard. For a given n, a sample of gene sets of size n is simulated, under the null hypothesis. For each of them, the test statistic is computed, thus a sample of values of the test statistic under the null hypothesis is obtained. The goodness-of-fit of the theoretical CDF F to the empirical CDF of the sample is tested by the (classical) KS test. Figure 2 shows results that were obtained for two functions g: one is g1(x)=2(1−x) (left panel), the other one comes from real data: a sample in GSE36133 of Barretina et al. (2012) (right panel). The evaluation of F1 was done over 106 Monte-Carlo simulations, and 105 discretization points, as explained in the previous section. For the real data, the number of discretization points was m=N=18,638, and the number of Monte-Carlo simulation was nsim=20,000. The values of n range from 5 to 1100 by step 5. For each n, 1500 uniform random gene sets of size n were simulated. The negative logarithm in base 10 of the KS p-value is plotted. On each plot the horizontal line corresponding to a 5% p-value has been added. The p-values are small until n=40, they stay above 5% after. This is coherent with what is observed for most asymptotic tests, and in particular the classical KS test. Beyond statistical validation, the comparison of the exact CDF, estimated over random gene sets, with the theoretical asymptotic F reveals an interesting feature of the WKS test: the exact CDF tends to be smaller than F. This implies that the asymptotic p-value tends to be larger than the true one, or else that the procedure is conservative: small gene sets are less likely to be declared significant by WKS. However, if a better precision is needed for gene set sizes smaller than 40, then an estimation of the distribution of $Tn∗$ through Monte-Carlo simulation should be preferred.

Figure 2:

Goodness-of-fit of simulated WKS test statistic $Tn∗$ over simulated gene sets. The function g is g(x)=2(1−x) on the left panel. It comes from real data on the right panel. The gene set size (abscissa) ranges from 5 to 1100 by step 5. For each n the ordinate is the negative logarithm in base 10 of the KS goodness-of-fit p-value, over a sample of 1500 gene sets. The dashed lines have ordinate −log10(0.05).

On Figure 2, there is no clear difference between the theoretical g (left), and real data (right). However, it must be recalled that the null hypothesis H0, under which simulations have been done in both cases, is that the gene set is a sample of uniform random variables on the interval [0, 1]. However, in practice, the gene set should be considered instead as a random subset without replacement of the set of all genes. If the gene set size n is small compared to the total number of genes N, the difference is negligible. We have conducted another set of experiments, where gene sets were simulated by extracting random samples without replacement from {1/N, …, N/N}. The results (not reported here), show a good agreement with those of Figure 2, until n=1000. Beyond that value, the asymptotics becomes less precise. It must be observed that gene sets of size larger than 1000 are relatively rare (28 out of the 4722 gene sets of C2).

## 3.3 Comparison with classical GSEA

In this section, a very limited selection of real data is presented. It is only intended as an illustration of the method. Many more tests have been carried out. We consider the results reported below as representative of the observations that can be obtained on different situations.

Several vectors coming from the GEO repository were tested against all 4722 gene sets in the MSig C2 database, using the classical GSEA, and the WKS tests. The vectors that were used came from the GEO dataset GSE36133 of Barretina et al. (2012), annotated using the org.Hs.eg.db package of Carlson (2012). This gave N=18,638 different gene names. Observe that applying the tests, the gene sets are necessarily reduced to those N genes. Out of the 21,047 different gene symbols present in C2, only 16,683 were common with the N genes of the chosen vectors.

For a given vector, two sets of 4722 p-values were obtained, one with the GSEA test, the other with the WKS test. Results that can be considered as typical are represented on Figure 3. In that case, the vector contained expression data from liver tumor tissue. Out of the 4722 gene sets of C2, 129 have “liver” in their title. They were considered as related to liver cancer, and the corresponding points are represented as red triangles on the figure. The negative logarithms in base 10 of the p-values of both tests have been plotted, thus the figure displays 4722 points corresponding to p-value pairs. For comparison sake, only raw p-values are considered, without FDR adjustment. A p-value of 5% is marked by a dashed black line: points on the right of the vertical line are significant for the classical GSEA test, points above the horizontal line are significant for the WKS test. For the WKS test, the CDF F was calculated over m=N=18,638 discretization points, and the number of Monte-Carlo simulations was nsim=105. For the classical GSEA test, the number of Monte-Carlo simulations had to be limited to 104.

Figure 3:

Test of a liver tumor expression vector against the 4722 gene sets of the MSig C2 database. Each point corresponds to a gene set, the coordinates being the negative logarithm in base 10 of the p-values, for the classical GSEA and the WKS tests. Gene sets related to liver cancer in the database are represented as red triangles. The horizontal and vertical dashed lines correspond to 5% p-values.

The vertical dotted lines appearing on the right of the graphic are artefacts, due to the Monte-Carlo method for the GSEA test: the rightmost line corresponds to cases where the point-estimated p-value is equal to 0. Apart from these artefacts, it must be observed that the results of both tests are globally coherent: 2501 database gene sets were significant (p-value smaller than 5%) for the WKS test, 2764 for GSEA, 2268 for both. There are no points in the bottom right corner of the graphics: when a p-value is very small for GSEA, it is never large for WKS. The converse is not true: many points in the upper left corner correspond to gene sets with a large p-value for GSEA, small for WKS.

More interesting is the analysis of liver-related gene sets. Out of 129, 76 were declared significant by the WKS test; 70 by the GSEA test, 66 by both. Therefore, 10 gene sets were declared significant by WKS only, and 4 by GSEA only. Figure 4 plots the cumulated proportions of weights Sn(t) for those 14 gene sets. On the same plot, the functions t (bisector), to which the classical GSEA test compares Sn(t), and G(t), used as a centering by WKS, also appear. On the graphic, the reason why a gene set may be declared significant by one test and not the other, is clear. The 4 gene sets declared significant by GSEA and not WKS, are represented by blue step functions; they are above the G curve. They are indeed far from the bisector, but not far enough from G. Inside the corresponding gene sets, the weights of the genes tend to be representative of the global distribution of weights, and declaring them as significant by comparing to the bisector can be regarded as a bias. Moreover, it should be observed that 3 out of the 4 have size below 19. As already explained, when dealing with very small sizes, the WKS test tends to underrate significance.

Figure 4:

Plots of the cumulated weight function Sn(t) for vectors declared significant by WKS and not GSEA (red step functions) and conversely (blue step functions). The functions t (to which the classical GSEA test compares Sn(t)), and G(t) (used as a centering by WKS), are dashed.

Conversely, the 10 gene sets declared significant by WKS and not GSEA are represented by red step functions. They are relatively close to the bisector as expected, but clearly below the G curve, to which WKS compares. This means that in the corresponding gene sets, the genes tend to have significantly smaller weights, i.e., they are significantly underexpressed. An interesting example is the gene set named Acevedo_methylated_in_liver_cancer_dn. As indicated by the two letters dn, it contains genes which are known to be down-regulated in case of liver cancer (Acevedo et al., 2008). On Figure 3, it appears on the upper left corner: it has p-value close to 0 for WKS, close to 1 for GSEA. Thus WKS has detected it as significantly related to the tested vector, whereas GSEA has not. The case is not unique: 3 gene sets had p-value larger than 0.5 for GSEA, smaller than 10−3 for WKS.

As already stated, these results were consistently observed for different expression vectors, from different types of cancers. In all cases, WKS declared less significant pathways than GSEA in a proportion of about 10% from the whole database, whereas it tended to detect more significant gene sets among those related to the correct type of cancer.

Next, the GEO dataset GSE3165 of Herschkowitz et al. (2007) was studied. An analysis, similar to the one conducted by Wu and Smyth (2012), has been carried out. The dataset was annotated using the hgug4110b.db package of Carlson, which yielded N=12,747 different gene names, since only the platform GPL887 was considered. Subsequently, the 94 samples were divided into two groups: those with basal-like breast cancer and those with one of the other five subtypes of cancer (luminal A, luminal B, Her2, normal-like and claudin-low). The row means were computed over the two submatrices yielding two vectors of size 12,747. The difference between these two vectors was then calculated, giving a vector containing the log(FC) between basal-like and non-basal tumors. This vector was tested against the C2 database, using the classical GSEA, and the WKS tests.

Out of the 4722 gene sets, 1229 were declared significant by GSEA and 836 by WKS. Out of the 4722 gene sets, 159 have “breast” in their title. Regarding the breast-related gene sets, 89 were detected as significant by GSEA, 80 by WKS and 79 by both. Thus, 1 gene set is declared significant by WKS only, and 10 by GSEA only. Although GSEA detects more breast-related gene sets, the WKS test still outperforms GSEA, in the sense that the proportion of the breast-related gene sets declared significant over the total number of gene sets declared significant remains higher. Figure 5 plots the cumulated proportions of weights Sn(t) for those 11 gene sets, together with the bisector and the function G. Regarding the gene sets declared significant by GSEA only, 7 out of 10 have size below 39 and there, the WKS test tends to underrate significance. Regarding the remaining 3 gene sets, they do not lie very far from the G curve. On the contrary, from the right panel, we see a gene set with clear enrichment at the top of the list that GSEA fails to detect.

Figure 5:

Plots of the cumulated weight function Sn(t) for vectors declared significant by GSEA and not WKS (blue step functions, left panel) and conversely (red step function, right panel). The functions t (to which the classical GSEA test compares Sn(t)), and G(t) (used as a centering by WKS), are dashed.

To validate further the fact that the WKS test can be really informative from a biological perspective, the vector of the |log(FC)| was also considered. Then, the one-sided WKS test (corresponding to the test statistic sup(Sn(t)−G(t))) was applied. The 10 most significant gene sets so obtained, are summarized on Table 1.

Table 1

Ten most significant gene sets declared by the WKS test, applied to the vector of |log(FC)| derived from the GEO dataset GSE3165 and the C2 database.

From Table 1, we see that all gene sets, but the last one, are related to breast cancer. More importantly, the two basal-like signatures come first. On the contrary, the two basal-like signatures come in positions 20 and 25, when GSEA is applied on the same data.

## 4 Conclusion

A new method for testing the relative enrichment of a gene set, compared to a vector of numeric data over the whole genome, has been proposed. Like the classical GSEA test of Subramanian et al. (2005), it is based on cumulated proportions of weights, but here, it is both centered and scaled. A convergence result that generalizes the classical Kolmogorov Smirnov theorem, has been obtained. The corresponding testing procedure extends the standard Kolmogorov Smirnov test and has been called Weighted Kolmogorov Smirnov (WKS). A major advantage of the WKS test is that the calculation of p-values only depends on the vector to be tested, and not on the gene set. Therefore, the same distribution function can be used for calculating p-values over many gene sets. A Monte-Carlo evaluation has shown that the procedure is precise for values of the gene set size larger than 40. For a set of less than 40 genes, the WKS test is conservative, in the sense that the p-value is increased, and therefore the gene set is less likely to be declared significant. For statistical coherence, the gene set size should not be larger than 1000. The WKS test has been compared with the classical GSEA test over expression vectors of tumors coming from the GEO dataset GSE36133 of Barretina et al. (2012), tested against the MSig database C2 (Subramanian et al., 2005). The comparison has shown that the results of both tests are globally coherent. The WKS test tends to output less significant gene sets out of the whole database, but more out of gene sets specifically related to the same type of tumor. In particular, the WKS test detects sets of underexpressed genes which are not significant for GSEA. Another comparison has been reported, on the GEO dataset GSE3165 of Herschkowitz et al. (2007): similar conclusions hold true. Examples where WKS detects clearly enriched gene sets that GSEA fails to detect were given. These encouraging results need to be consolidated, by using the WKS test over different types of vectors, and more databases of gene sets.

Like the GSEA test, the WKS test can be used on any type of numeric data. In particular, a transformation can be applied to the raw expression levels before testing. In particular, the initial data can be replaced by their ranks, in which case the test has low computing cost, for a good precision. If, over the same database, the p-values of the initial vector, and the vector of ranks are compared, a good agreement is observed; yet less gene sets are declared significant against the rank vector. Here, we have considered mainly the two sided version of the test: gene sets are declared significant when their cumulated proportion of weights Sn(t) is too far from the theoretical value G(t). The WKS was also made one-sided, by testing the signed difference between Sn(t) and G(t): a gene set for which inf(Sn(t)−G(t)) is significantly negative, contains genes whose weights tend to be small (depletion). Conversely, gene sets for which sup(Sn(t)−G(t)) is significantly positive, contain more genes with large weights (enrichment).

Both the GSEA and the WKS tests have been implemented in a R script. It is available online, together with data samples, and a user manual, from the following address.

http://ljk.imag.fr/membres/Bernard.Ycart/publis/wks.tgz

We hope this will encourage further testing of the tool, and validation in new biological studies.

## Acknowledgments

This work was supported by Laboratoire d’Excellence TOUCAN (Toulouse Cancer). The authors are grateful to Sophie Rousseaux and Jean-Jacques Fournié for using the WKS test on many different datasets; their feedback helped improving the code. They are also indebted to the reviewers for important remarks and suggestions.

Funding: Labex Toucan (Toulouse Cancer).

## References

• Acevedo, L. G., M. Bieda, R. Green and P. J. Farnham (2008): “Analysis of the mechanisms mediating tumor-specific changes in gene expression in human liver tumors,” Cancer Res., 68(8), 2641–2651.

• Arnold, T. B. and J. W. Emerson (2011): “Nonparametric goodness-of-fit tests for discrete null distributions,” R Journal, 3/2, 34–39.Google Scholar

• Barbie, D. A., P. Tamayo, J. S. Boehm, S. Y. Kim, S. E. Moody, I. F. Dunn, A. C. Schinzel, P. Sandy, E. Meylan, C. Scholl, S. Fröhling, E. M. Chan, M. L. Sos, K. Michel, C. Mermel, S. J. Silver, B. A. Weir, J. H. Reiling, Q. Sheng, P. B. Gupta, R. C. Wadlow, H. Le, S. Hoersch, B. S. Wittner, S. Ramaswamy, D. M. Livingston, D. M. Sabatini, M. Meyerson, R. K. Thomas, E. S. Lander, J. P. Mesirov, D. E. Root, D. G. Gilliland, T. Jacks and W. C. Hahn (2009): “Systematic RNA interference reveals that oncogenic KRAS-driven cancers require TBK1,” Nature, 462(7269), 108–112.

• Barretina, J., G. Caponigro, N. Stransky, K. Venkatesan, A. A. Margolin, S. Kim, C. J. Wilson, J. Lehár, G. V. Kryukov, D. Sonkin, A. Reddy, M. Liu, L. Murray, M. F. Berger, J. E. Monahan, P. Morais, J. Meltzer, A. Korejwa, J. Jané-Valbuena, F. A. Mapa, J. Thibault, E. Bric-Furlong, P. Raman, A. Shipway, I. H. Engels, J. Cheng, G. K. Yu, J. Yu, P. Aspesi Jr., M. de Silva, K. Jagtap, M. D. Jones, L. Wang, C. Hatton, E. Palescandolo, S. Gupta, S. Mahan, C. Sougnez, R. C. Onofrio, T. Liefeld, L. MacConaill, W. Winckler, M. Reich, N. Li, J. P. Mesirov, S. B. Gabriel, G. Getz, K. Ardlie, V. Chan, V. E. Myer, B. L. Weber, J. Porter, M. Warmuth, P. Finan, J. L. Harris, M. Meyerson, T. R. Golub, M. P. Morrissey, W. R. Sellers, R. Schlegel and L. A. Garraway (2012): “The cancer cell line encyclopedia enables predictive modelling of anticancer drug sensitivity,” Nature, 483(7391), 603–607.

• Benjamini, Y. and D. Yekutieli (2001): “The control of the false discovery rate in multiple testing under dependency,” Ann. Statist., 29(4), 1165–1188.Google Scholar

• Bild, A. and P. G. Febbo (2005): “Application of a priori established gene sets to discover biologically important differential expression in microarray data,” PNAS 102(43), 15278–15279.

• Carlson, M. (2012): “org.Hs.eg.db: Genome wide annotation for Human,” R package version 2.8.0.Google Scholar

• Carlson, M. “hgug4110b.db: Agilent Human 1A (V2) annotation data (chip hgug4110b),” R package version 2.14.0.Google Scholar

• Dudoit, S. and M. van der Laan (2007): Multiple testing procedures with applications to genomics, New York: Springer.Google Scholar

• Edgar, R., M. Domrachev and A. E. Lash (2002): “Gene expression omnibus: NCBI gene expression and hybridization array data repository,” Nucleic Acids Res., 30(1), 207–210.

• Frei, E., C. Visco, Z. Y. Xu-Monette, S. Dirnhofer, K. Dybkær, A. Orazi, G. Bhagat, E. D. Hsi, J. H. van Krieken, M. Ponzoni, R. S. Go, M. A. Piris, M. B. Møller, K. H. Young and A. Tzankov (2013): “Addition of rituximab to chemotherapy overcomes the negative prognostic impact of cyclin E expression in diffuse large B-cell lymphoma,” J. Clin. Pathol., 66(11), 956–961.

• Goeman, J. J. and P. Bühlmann (2007): “Analyzing gene expression data in terms of gene sets: methodological issues,” Bioinformatics, 23(8), 980–987.

• Héritier, S., E. Cantoni, S. Copt and M. P. Victoria-Feser (2009): Robust methods in biostatistics, New York: Wiley.Google Scholar

• Herschkowitz, J. I., K. Simin, V. J. Weigman, I. Mikaelian, J. Usary, Z. Hu, K. E. Rasmussen, L. P. Jones, S. Assefnia, S. Chandrasekharan, M. G. Backlund, Y. Yin, A. I. Khramtsov, R. Bastein, J. Quackenbush, R. I. Glazer, P. H. Brown, J. E. Green, L. Kopelovich, P. A. Furth, J. P. Palazzo, O. I. Olopade, P. S. Bernard, G. A. Churchill, T. Van Dyke and C. M. Perou (2007): “Identification of conserved gene expression features between murine mammary carcinoma models and human breast tumors,” Genome Biol., 8(5), R76.

• Huang, D. W., B. T. Sherman and R. A. Lempicki (2009): “Bioinformatics enrichment tools: paths toward the comprehensive functional analysis of large gene lists,” Nucleic Acids Res., 37(1), 1–13.

• Irizarry, R. A., C. Wang, Y. Zhou and T. P. Speed (2009): “Gene set enrichment analysis made simple,” Stat. Methods Med. Res., 18(6), 565–575.

• Kim, S. Y. and D. J. Volsky (2005): “PAGE: parametric analysis of gene set enrichment,” BMC Bioinformatics, 6, 144.

• Kosorok, M. R. (2008): Introduction to empirical processes and semiparametric inference, New York: Springer.Google Scholar

• Marisa, L., A. de Reyniès, A. Duval, J. Selves, M. P. Gaub, L. Vescovo, M. C. Etienne-Grimaldi, R. Schiappa, D. Guenot, M. Ayadi, S. Kirzin, M. Chazal, J. F. Fléjou, D. Benchimol, A. Berger, A. Lagarde, E. Pencreach, F. Piard, D. Elias, Y. Parc, S. Olschwang, G. Milano, P. Laurent-Puig and V. Boige (2013): “Gene expression classification of colon cancer into molecular subtypes: characterization, validation, and prognostic value,” PLoS Med., 10(5), e1001453.

• Mayerle, J., C. M. den Hoed, C. Schurmann, L. Stolk, G. Homuth, M. J. Peters, L. G. Capelle, K. Zimmermann, F. Rivadeneira, S. Gruska, H. Völzke, A. C. de Vries, U. Völker, A. Teumer, J. B. van Meurs, I. Steinmetz, M. Nauck, F. Ernst, F. U. Weiss, A. Hofman, M. Zenker, H. K. Kroemer, H. Prokisch, A. G. Uitterlinden, M. M. Lerch and E. J. Kuipers (2013): “Identification of genetic loci associated with Helicobacter pylori serologic status,” J. Am. Med. Assoc., 309(18), 1912–1920.Google Scholar

• Mikheev, A. M., T. Nabekura, A. Kaddoumi, T. K. Bammler, R. Govindarajan, M. F. Hebert and J. D. Unadkat (2008): “Profiling gene expression in human placentae of different gestational ages: an OPRU network and UW SCOR study,” Reprod. Sci., 15(9), 866–877.Google Scholar

• Mootha, V. K., C. M. Lindgren, K. F. Eriksson, A. Subramanian, S. Sihag, J. Lehar, P. Puigserver, E. Carlsson, M. Ridderstråle, E. Laurila, N. Houstis, M. J. Daly, N. Patterson, J. P. Mesirov, T. R. Golub, P. Tamayo, B. Spiegelman, E. S. Lander, J. N. Hirschhorn, D. Altshuler and L. C. Groop (2003): “PGC-1alpha-responsive genes involved in oxidative phosphorylation are coordinately downregulated in human diabetes,” Nat. Genet., 34(3), 267–273.Google Scholar

• Nam, D. and S. Y. Kim (2008): “Gene-set approach for expression pattern analysis,”Brief. Bioinform., 9(3), 189–197.

• Obermoser, G., S. Presnell, K. Domico, H. Xu, Y. Wang, E. Anguiano, L. Thompson-Snipes, R. Ranganathan, B. Zeitner, A. Bjork, D. Anderson, C. Speake, E. Ruchaud, J. Skinner, L. Alsina, M. Sharma, H. Dutartre, A. Cepika, E. Israelsson, P. Nguyen, Q. A. Nguyen, A. C. Harrod, S. M. Zurawski, V. Pascual, H. Ueno, G. T. Nepom, C. Quinn, D. Blankenship, K. Palucka, J. Banchereau and D. Chaussabel (2013): “Systems scale interactive exploration reveals quantitative and qualitative differences in response to influenza and pneumococcal vaccines,” Immunity, 38(4), 831–844.

• R Core Team (2013): R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria, URL http://www.R-project.org/, ISBN 3-900051-07-0.

• Sauer, T. (2013): “Computational solution of stochastic differential equations,” WIREs Comput. Stat., 5(5), 362–371.

• Seok, J., H. S. Warren, A. G. Cuenca, M. N. Mindrinos, H. V. Baker, W. Xu, D. R. Richards, G. P. McDonald-Smith, H. Gao, L. Hennessy, C. C. Finnerty, C. M. López, S. Honari, E. E. Moore, J. P. Minei, J. Cuschieri, P. E. Bankey, J. L. Johnson, J. Sperry, A. B. Nathens, T. R. Billiar, M. A. West, M. G. Jeschke, M. B. Klein, R. L. Gamelli, N. S. Gibran, B. H. Brownstein, C. Miller-Graziano, S. E. Calvano, P. H. Mason, J. P. Cobb, L. G. Rahme, S. F. Lowry, R. V. Maier, L. L. Moldawer, D. N. Herndon, R. W. Davis, W. Xiao and R. G. Tompkins; Inflammation and Host Response to Injury, Large Scale Collaborative Research Program (2013): “Genomic responses in mouse models poorly mimic human inflammatory diseases,” PNAS, 110(9), 3507–3512.

• Shorack, G. R. and J. A. Wellner (1986): Empirical processes with applications to statistics, New York: Wiley.Google Scholar

• Subramanian, A., P. Tamayo, V. K. Mootha, S. Mukherjee, B. L. Ebert, M. A. Gillette, A. Paulovich, S. L. Pomeroy, T. R. Golub, E. S. Lander and J. P. Mesirov (2005): “Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles,” PNAS, 102(43), 15545–15550, URL http://www.pnas.org/content/102/43/15545.full.

• Subramanian, A., H. Kuehn, J. Gould, P. Tamayo and J. P. Mesirov (2007): “Gsea-P: a desktop application for gene set enrichment analysis,” Bioinformatics, 23(23), 3251–3253.

• Tarca, A. L., G. Bhatti and R. Romero (2013): “A comparison of gene set analysis methods in terms of sensitivity, prioritization and specificity,” PLoS One, 8(11), e79217.Google Scholar

• Tsodikov, A., A. Szabo and D. Jones (2002): “Adjustments and measures of differential expression for microarray data,” Bioinformatics, 18(2), 251–260.

• Westra, H. J., M. J. Peters, T. Esko, H. Yaghootkar, C. Schurmann, J. Kettunen, M. W. Christiansen, B. P. Fairfax, K. Schramm, J. E. Powell, A. Zhernakova, D. V. Zhernakova, J. H. Veldink, L. H. Van den Berg, J. Karjalainen, S. Withoff, A. G. Uitterlinden, A. Hofman, F. Rivadeneira, P. A. 't Hoen, E. Reinmaa, K. Fischer, M. Nelis, L. Milani, D. Melzer, L. Ferrucci, A. B. Singleton, D. G. Hernandez, M. A. Nalls, G. Homuth, M. Nauck, D. Radke, U. Völker, M. Perola, V. Salomaa, J. Brody, A. Suchy-Dicey, S. A. Gharib, D. A. Enquobahrie, T. Lumley, G. W. Montgomery, S. Makino, H. Prokisch, C. Herder, M. Roden, H. Grallert, T. Meitinger, K. Strauch, Y. Li, R. C. Jansen, P. M. Visscher, J. C. Knight, B. M. Psaty, S. Ripatti, A. Teumer, T. M. Frayling, A. Metspalu, J. B. van Meurs and L. Franke (2013): “Systematic identification of trans eQTLs as putative drivers of known disease associations,” Nat. Genet., 45(10), 1238–1243.

• Wu, D. and G. K. Smyth (2012): “Camera: a competitive gene set test accounting for inter-gene correlation,” Nucleic Acids Res., 40(17), e133.

• Xiao, W., M. N. Mindrinos, J. Seok, J. Cuschieri, A. G. Cuenca, H. Gao, D. L. Hayden, L. Hennessy, E. E. Moore, J. P. Minei, P. E. Bankey, J. L. Johnson, J. Sperry, A. B. Nathens, T. R. Billiar, M. A. West, B. H. Brownstein, P. H. Mason, H. V. Baker, C. C. Finnerty, M. G. Jeschke, M. C. López, M. B. Klein, R. L. Gamelli, N. S. Gibran, B. Arnoldo, W. Xu, Y. Zhang, S. E. Calvano, G. P. McDonald-Smith, D. A. Schoenfeld, J. D. Storey, J. P. Cobb, H. S. Warren, L. L. Moldawer, D. N. Herndon, S. F. Lowry, R. V. Maier, R. W. Davis and R. G. Tompkins; Inflammation and Host Response to Injury Large-Scale Collaborative Research Program (2011): “A genomic storm in critically injured humans,” J. Exp. Med., 208(13), 2581–2590.

• Ycart, B., F. Pont and J. J. Fournié (2014): “Curbing false discovery rates in interpretation of genome-wide expression profiles,” J. Biomed. Inform., 47, 58–61.

Corresponding author: Bernard Ycart, 51 rue des Mathématiques, 38041 GRENOBLE cedex 9, France; Université Grenoble Alpes, France; Laboratoire Jean Kuntzmann, CNRS UMR5224, Grenoble, France; and Laboratoire d’Excellence TOUCAN, Toulouse, France, e-mail:

Published Online: 2015-05-30

Published in Print: 2015-06-01

Citation Information: Statistical Applications in Genetics and Molecular Biology, Volume 14, Issue 3, Pages 279–293, ISSN (Online) 1544-6115, ISSN (Print) 2194-6302,

Export Citation

## Citing Articles

[1]
Mei Ma, Xiao Liang, Shiqiang Cheng, Lu Zhang, Bolun Cheng, Jian Yang, Miao Ding, Yan Zhao, Ping Li, Yanan Du, Li Liu, Yan Wen, Awen He, QianRui Fan, Xiong Guo, and Feng Zhang
Behavioural Brain Research, 2018, Volume 353, Page 137
[2]
Giovanni Fiorito, Jelle Vlaanderen, Silvia Polidoro, John Gulliver, Claudia Galassi, Andrea Ranzi, Vittorio Krogh, Sara Grioni, Claudia Agnoli, Carlotta Sacerdote, Salvatore Panico, Ming-Yi Tsai, Nicole Probst-Hensch, Gerard Hoek, Zdenko Herceg, Roel Vermeulen, Akram Ghantous, Paolo Vineis, and Alessio Naccarati
Environmental and Molecular Mutagenesis, 2017
[3]
Nino Kordzakhia, Alexander Novikov, and Bernard Ycart
Statistics and Computing, 2017, Volume 27, Number 6, Page 1513
[4]
Marie Tosolini, Christelle Algans, Frédéric Pont, Bernard Ycart, and Jean-Jacques Fournié
OncoImmunology, 2016, Volume 5, Number 7, Page e1188246