A guide to value of information methods for prioritising research in health impact modelling

General health impact model: estimation and decision problems

The model is represented by a deterministic function Y = f( θ ) which maps input parameters θ to (possibly multiple) health outcomes of interest Y. The model might be used to answer two kinds of questions: estimation or decision problems. In either case, the input parameters are uncertain, and we wish to determine the sensitivity of the estimate or decision to each of the inputs, hence to prioritise further research or data collection.

Uncertainty in health impact models

The parameters θ are uncertain, and we assume that estimates are based on the expectation of outcomes over θ , E _θ (Y( θ )), or in decision problems, we choose the action a that maximises E(NB _a ( θ )). Therefore the estimate or decision may be “wrong”, i.e. different from the true θ or the decision we would make if we knew the true θ .

We take a Bayesian perspective, that is, uncertainty about the input parameters θ is represented by a joint probability distribution, representing beliefs under all current information. In health impact models, the information might come from a variety of sources, including analyses of primary data, published information, and judgements. While we do not go into detail on how probability distributions on the model parameters might be derived from the data, we give some examples in later sections, and there are guidelines for the related field of health economic modelling (Briggs et al. 2012). In practice, Monte Carlo simulation is used to produce a sample from the uncertainty distribution of Y which is implied by the distributions stated for θ , and E(Y) is estimated by the mean of this sample.

We wish to learn which of the parameters comprising θ most influence the uncertainty in the estimate or decision, and therefore prioritise further research or data collection. There is a broad literature on sensitivity analysis in epidemiological models (Greenland 1996; Lash et al. 2014), and mathematical models (Frey and Patil 2002; Oakley and O’Hagan 2004; Saltelli et al. 2008). Most commonly in health impact models (de Sá et al. 2017; Woodcock et al. 2014) uncertainty is explored by one-way sensitivity analysis, in which model outputs are compared with uncertain inputs set to alternative values, potentially after uncertainties have been quantified with probability distributions. While this can give an informal guide to what further research may be beneficial, a more formal framework for research prioritisation is given by Value of Information (VoI) methods.

Value of information

The expected value of information is the expected benefit from learning the unknowns θ better, hence producing a more precise estimate or reducing the chance of a wrong decision. The main VoI measures are described briefly here, and full theoretical definitions are given in Section 3.

1. The expected value of partial perfect information (EVPPI) is the maximum expected benefit from further research on an input parameter. This is a measure of sensitivity of the conclusions to the parameter, though answers a slightly different question from one-way sensitivity analysis. The expected value of perfect information (EVPI) is the expected value of learning all parameters, hence eliminating all uncertainty.

2. The expected value of sample information (EVSI) measures the expected benefits from collecting a specific dataset (e.g. of a specific sample size) to inform an uncertain parameter.

The definition of “benefit” depends on the problem and the outcome definition. VoI methods are based on Bayesian decision-theoretic principles, hence can be used for any problem which can be formulated in terms of a decision under uncertainty. As we now describe, this includes estimation problems as well as explicit decisions between policies.

VoI methods are well-established in models that are used to make decisions between policies in health economics (Briggs, Sculpher, and Claxton 2006; Fenwick et al. 2020; Rothery et al. 2020). In decision models, the benefits from further information can be measured in the same unit of health or currency as the NB _a . Hence EVPPI and EVSI measure expected monetary or health gains from further information, which will reduce the chance of making a wrong decision. To use VoI for research prioritisation, if NB _a is a per person net benefit, it should be converted to a population net benefit by multiplying by the size of the current and future population expected to be affected by the decision, though in practice this size will be uncertain and need to be subjected to sensitivity analysis (see, e.g. Rothery et al. 2020). Then if the benefits are measured in monetary terms, a proposed study can be judged as cost-effective if the EVSI is greater than the cost of sampling, and we might choose the study sample size with the greatest expected net benefit of sampling (ENBS, equal to the EVSI minus the study cost).

VoI methods, however, have not been routinely used in health impact models, which are used for both estimation and decision problems. While VoI measures are constructed using decision theory, they can also be used for estimation problems, by conceiving the problem as the “decision” of what estimate of an uncertain quantity to report. In this case, the benefits of further information can be quantified by reductions in variance of the probability distribution representing uncertainty about the quantity of interest, as described in Section 3. The EVPPI in this context is equivalent to a common measure used for sensitivity analysis in mathematical modelling (Borgonovo et al. 2021; Saltelli et al. 2008).

Yokota and Thompson (2004) discussed the challenges of using VoI in health impact models, that include the difficulty of quantifying all uncertainties and the difficulty of computing VoI measures. Recently-developed computational methods, however, have made it straightforward to calculate VoI in very general situations, including both decision-making (Heath, Manolopoulou, and Baio 2017; Heath et al. 2020) and estimation (Jackson et al. 2019).

Overview of the rest of the article

This paper gives a practical tutorial in the usage of Value of Information methods for epidemiologists and public health researchers developing health impact simulation models. A general-purpose R package is provided to implement the methods, and full code is provided in the supplementary material to reproduce all the analyses.

In Section 2 we describe a simple health impact model and illustrate sources of uncertainty in it. In Section 3 we describe the theory and definitions of various Value of Information measures for both estimation and decision-making problems, and describe straightforward algorithms that can be used to compute them. In Section 4 we illustrate a VoI calculation for the example model, and extend this example model in Section 5 to handle cases where we want to prioritise the collection of different kinds of evidence, demonstrating the corresponding VoI analyses in novel situations. These situations include estimating the relative value of collecting biased vs. unbiased evidence in the presence of different amounts of bias, the value of information in different settings when estimates are derived from hierarchical modelling of multiple similar settings, and the value of obtaining “imperfect” information from literature when we cannot collect primary data.

While a deliberately simple model is illustrated for tutorial purposes, the concepts and methods of VoI analysis that we describe are identical for more complex models, and the computation does not become substantially harder. The only requirements are that the model can be defined as a deterministic function of inputs, with all sources of uncertainty quantified as probability distributions on those inputs, and that we can obtain a sample from the joint uncertainty distribution of all model inputs and outputs.

Illustration of model parameter and output uncertainty

Firstly we will define the distributions used for the parameters in the example model. In a later section “Structuring and parameterising health impact models to enable VoI analysis” we describe variants of the model that might be used in situations where the available evidence is different, thus we wish to prioritise the collection of different kinds of data.

A log-normal distribution, with mean 2.7 and standard deviation 0.3 on the log scale, is assumed for the background PM2.5 concentration μ, which ranges from about 10 to 30 μg/m³ (Figure 1). A Beta(5.7,8.9) distribution is used for the proportion of PM2.5 π due to transport (ranging from around 0.1 to 0.7) and the baseline stroke incidence is fixed at I ₀ = 18,530 cases per year. These distributions are based on those used in the ITHIM (Woodcock, Givoni, and Morgan 2013) series of models, which were determined from published estimates from similar contexts and judgements of their relevance. In practice these could vary widely between different settings.

Figure 1:

Input and output uncertainty distributions in the example health impact model.

The joint distribution for the parameters d = (α, β, γ, τ) relating average PM2.5 exposure to stroke risk was obtained by Burnett et al. (2014) as a random sample as follows. The parameters d of curve g ₂(x, d) were estimated from data given by the published relative risks at different exposures x. A sample from the joint distribution for d was obtained by repeating this estimation with alternative plausible relative risks simulated using the published estimates and their standard errors.

Figure 1 (top row and bottom left) then illustrates the assumed distributions for all of the inputs in the example model. Given these assumptions, the corresponding distribution for the model outputs is obtained by Monte Carlo sampling. For a scenario where transport PM2.5 emissions are D = 0.5 times the current amount, the 95% credible interval for averted stroke cases is 118–806 (Figure 1, bottom right). We might then want to know which of the model inputs contribute the most to this uncertainty. A simple way to demonstrate that is by one-way sensitivity analysis. Given alternative values of a parameter, say the 95% credible limits for μ, denoted (μ _L, μ _U), we calculate the resulting values of the model outputs, to indicate the uncertainty in the outputs that is driven by that specific input. Denote the parameters other than μ as ν, so that θ = (μ, ν) is the full set of parameters.

Commonly these resulting outputs are calculated with the ν fixed at point estimates ν ̂ , as f ( μ L , ν ̂ ) , f ( μ U , ν ̂ ) , but this neglects the uncertainty about ν (McCabe et al. 2020). This uncertainty can be accounted for probabilistically, by calculating the expected model output under these lower and upper limits, E(f( θ )|μ = μ _L), E(f( θ )|μ = μ _U), where the expected value is calculated with respect to the uncertainty distribution of ν. This can be computed straightforwardly by Monte Carlo if the distributions of ν are independent of μ. Repeating this procedure for all parameters, and plotting the resulting intervals, gives a tornado plot (see, e.g. de Sá et al. 2017).

Tornado plots, using both non-probabilistic and probabilistic methods, are illustrated for the example in Figure 2. In the left panel, parameters not being varied are fixed at point estimates (medians of their distributions) and in the right panel, their uncertainty is accounted for probabilistically. The first row shows the base-case result of the model, a simple point estimate under the non-probabilistic model, and an estimate and credible interval under the probabilistic model. The remaining rows show one-way sensitivity analyses: estimates of expected stroke cases averted, contrasted between models that assume different extreme values of particular parameters. The first sensitivity analysis contrasts estimates where the proportion of PM2.5 due to transport is fixed to its lower 95% credible limit of μ _L = 17%, or to the upper credible limit of μ _U = 64%. The second sensitivity analysis compares estimates with the background PM2.5 fixed to its lower credible limit of μ _L = 8, or the upper credible limit of μ _U = 27.

Figure 2:

Tornado plot for one-way sensitivity analysis in the example health impact model.

Left: non-probabilistic version where input parameters not being varied are fixed at median values. Right: probabilistic version with central estimate defined by the mean output over the distributions of the inputs. The principal results of the models are shown at the top: for the non-probabilistic model this is a point estimate without a credible interval, and for the probabilistic model this is a median and 95% credible interval. The remaining rows illustrate the sensitivity of the model result to variations in the three model inputs.

For the parameters d = (α, β, γ, τ) governing the dose–response curve g ₂(x, d), instead of varying each parameter separately, they are considered jointly for sensitivity analysis, since their joint distribution is highly correlated, and the ultimate aim is to determine the value of further information about them. Since any data informing dose–response would be likely to inform all four parameters together, varying each parameter independently of the others would not represent the value of further information on dose–response.

Instead, we construct a sensitivity analysis for these parameters by defining two “extreme” instances of the whole curve, which define contrasting strengths of dose–response, hence will produce contrasting estimates when used in a health impact model. To do this, we define the “strength” of dose–response (for a curve with given parameters d) as the ratio between the relative risks ρ = g ₂(x _L, d)/g ₂(x _U, d) for low and high values x _L, x _U of PM2.5. High and low values are defined, arbitrarily, as average PM2.5 concentrations under the base case and for the scenario with transport emissions halved, which are around x _U = 15 and x _L 10 μg/m³ respectively. A sample of values is drawn for d, which results in a corresponding sample for ρ. The joint sample is arranged in order of ρ, and the 2.5 and 97.5% quantiles of ρ are determined. The two “extreme” dose–response curves are defined by the samples of d = (α, β, γ, τ) that produced the 2.5 and 97.5% quantiles of ρ. These curves are shown in red in Figure 1 (bottom left). These are used to produce the third row of the tornado plots in Figure 2.

The parameter describing the proportion of PM2.5 attributable to transport appears to have the greatest influence on the health impact of reduced emissions, when varied between its credible limits. The impacts also vary widely between “strong” and “weak” dose responses, while the impact of the background PM2.5 μ on the expected stroke cases averted appears to be small and non-monotonic. The final row of the plot (labelled “Overall”) shows the principal result of the analysis, the expected number of stroke cases averted, which is 329 if parameter uncertainty is neglected (left), and 375 when calculated more appropriately as an expected value with respect to parameter uncertainty, with an associated 95% credible interval (bottom right, 120 to 800). Note that if the model is nonlinear, as in this case, neglecting parameter uncertainty and reporting the model output f(E( θ )) evaluated at the parameter estimates gives a biased estimate of the expected model output E(f( θ )), which is more appropriately obtained as the mean of the Monte Carlo sample.

One-way sensitivity analysis gives an indication of the sensitivity of a model to its inputs, and a decision-maker will often be interested what the result would be if an uncertain parameter were to take a specific value (see, e.g. McCabe et al. 2020). Implementing it, however, relies on arbitrary choices, particularly if parameters are correlated. Nonlinear relationships between inputs and outputs may also not be detected. One-way sensitivity analysis might be used informally to guide further research, by prioritising research on the parameters whose uncertainty most affects the model output. However, a parameter judged unimportant by one-way sensitivity analysis might sometimes still be valuable to learn better.

A richer and more formal framework for research prioritisation is given by Value of Information analysis, as we introduced in Section 1.3. The expected value of partial perfect information (of knowing a parameter exactly) measures sensitivity of a model to an uncertain parameter as the maximum value of further research on that parameter. The expected value of sample information measures the expected value of a specific study that will give imperfect information about one or more uncertain parameters. We now give formal definitions of Value of Information measures, and in Section 4 we explain how they are computed.

Models for decisions between finite actions

For decisions among a discrete set of actions, we choose the action that maximises the expected net benefit E(NB _a ( θ )). In the health impact model example, there are two possible decisions: the proposed action a = 2, or the status quo a = 1. The expected value of further information is measured as expected gains in net benefit.

Value of information definitions in models for estimation

Estimation problems can be expressed in terms of decision theory. The “action” a is the choice of point estimate Y ̂ for a quantity of interest, typically defined as a function of uncertain parameters, Y = f θ , and the set A of alternative actions is the “support” or set of possible values for Y. The “decision problem” is framed in terms of minimising an expected loss, rather than maximising an expected net benefit. The loss function is defined to represent a preference for estimates of Y closer to the truth. Here, we use the squared error function L ( Y ̂ , Y ) = ( Y ̂ − Y ) 2 . By minimising a quadratic equation, it can be shown that the “decision” that minimises the expected squared loss is Y ̂ = E θ Y , the mean of the belief distribution of Y, and the expected loss under the optimal decision is the variance of the belief distribution var Y = E [ ( Y ̂ − Y ) 2 ] . (Note that with an absolute error loss function, the point estimate would be the median of Y, and the expected loss would be the mean absolute deviation.)

Since the expected value of further information equals the expected loss under current information minus the expected loss under the further information, the VoI is expressed in terms of reductions in variance of the quantity of interest given better knowledge. Hence the EVPI for estimation problems with squared error loss is simply the variance of the uncertainty distribution of Y – an upper bound on any reductions in variance we could achieve by further research to learn θ .

A more useful measure for sensitivity analysis is the EVPPI, which for estimation problems is the expected reduction in variance of the model output from perfect information about a particular uncertain parameter ϕ,

Likewise, the expected value of sample information is the expected reduction in the variance of the model output after the study data Z is observed

Obtaining the expected net benefit of sampling for a proposed study requires variance reductions to be converted to monetary values, by determining the willingness-to-pay for a particular reduction in variance. An example is described in Jackson et al. (2019).

Computing EVPPI for a single parameter

The principle is illustrated in the example by plotting the sampled estimates of the expected stroke cases averted, y r = f ( θ 1 r ) , against a corresponding sample of values θ 1 r from the marginal distribution of one of the parameters θ ₁ that comprise θ , e.g. the proportion π of PM2.5 pollution due to transport (Figure 3). For larger values of θ ₁ = π, the health impact y r is greater, and this relationship can be approximated with a regression model

where h is a sufficiently flexible regression function. While a linear regression provides an adequate estimate of the mean outcome given the predictor in this case, the relationship between a model input and output will not be linear in general – therefore a nonparametric regression function h should be used. In this case, a penalised cubic spline was used (Wood 2017).

Figure 3:

Illustration of using regression to estimate EVPPI as expected reduction in variance of a model output after learning a model input.

The fitted value of the function, h ̂ x , at a value x, estimates E Y | π = x , the expected outcome if the uncertain parameter π were known to equal x. Conversely, the residuals y r − h ̂ x r represent the variability in the outcome that is not explained by π. In other words, the residual variance σ quantifies the uncertainty about f θ that would remain if we were to learn the value of π.

Therefore we can estimate the expected value of partial perfect information for an estimation problem, var Y − E π var Y | π Y = var π E Y | π Y (by the “law of total variance”) in two related ways,

1. as the empirical variance of the fitted values h ̂ x r , an estimate of var π E Y | π Y , roughly interpreted as the variance “explained by” π.

2. as the variance of the y r minus the residual variance, an estimate of var Y − E π var Y | π Y , the reduction in variance on learning π.

Note that the variances and expectations are with respect to the uncertainty distribution of the unknown parameters, hence they are estimated by the empirical variance or mean over samples generated from this distribution.

As described by Strong, Oakley, and Brennan (2014), a similar procedure works for decision problems. Instead of regressing a single model output y r on the parameters ϕ r , we regress the net benefit value NB a ( θ r ) on ϕ r for each alternative action a, which produces a sample of fitted values NB ̂ a r for each a. The mean of the fitted values, 1 R ∑ r = 1 R NB ̂ a r , estimates E θ | ϕ { NB a θ } , the expected net benefit under decision a, given perfect information about ϕ. The first term in the EVPPI formula (2) is then estimated first by taking the minimum of this quantity over a for each r (mimicking the decision taken with perfect information), and then taking the mean of these over samples r. The second term in formula (2) is estimated by the minimum over a of the means over samples r of NB a ( θ r ) , which corresponds to the decision taken under current information (on the basis of maximum expected net benefit).

Computing EVPPI for a group of parameters

The method generalises immediately to estimate the EVPPI corresponding to the value of learning p ≥ 2 parameters jointly. A nonparametric regression is used with more than one predictor,

As p increases, the computation becomes more challenging. To form the nonparametric regression function h , various methods have been suggested. Strong, Oakley, and Brennan (2014), in the context of health economic models, found that generalized additive models (commonly based on splines) performed adequately up to about p = 4, and for higher p, Gaussian process regression performed better. Heath, Manolopoulou, and Baio (2016) devised an efficient Gaussian process method based on spatial modelling principles. We have found multivariate adaptive regression splines (MARS) (Friedman 1991; Milborrow 2011) to be efficient and accurate in many situations, however no method is universally reliable, sensitivity analysis is advised, and this is the subject of ongoing research. We would also recommend a sufficient number of Monte Carlo samples are used so that there is enough “data” to be able to reliably identify the nonparametric regression shape when there are several parameters of interest – over 5000 were required in our example.

These methods are all implemented in the voi R package, which provides a common interface to the generalized additive modelling implementation of Wood (2017), the MARS implementation of Milborrow (2011), and the Gaussian process implementations of Strong, Oakley, and Brennan (2014) and Heath, Manolopoulou, and Baio (2016). Each of these by default chooses a best-fitting regression function through a predictive training procedure. Given data frames of the sampled model inputs and outputs, the joint EVPPI for a pair of parameters named “x1” and “x2” is computed with a single command, e.g.

evppi(outputs, inputs, pars = c(“x1”,“x2”))

for decision models, and a similar function evppivar for estimation models. Additional arguments are available to select and fine-tune the computational method. See the package vignette at https://chjackson.github.io/voi/articles/voi.html for worked examples in both decision and estimation problems.

Computing EVSI

A nonparametric regression method can also be used to estimate the expected value of sample information for a proposed study of a particular design and sample size that aims to estimate one or more of the uncertain parameters (Strong et al. 2015). As for the EVPPI method, this needs only a single Monte Carlo sample for the parameters and model outputs.

The idea is to generate a sample from the predictive distribution of data Z arising from the study (this is the “prior predictive distribution” if the current information is considered as a prior). For each parameter replicate θ r , we generate Z r from the assumed sampling distribution of Z | θ r . We then consider how the data would be analysed. Let T Z be a summary statistic of the data, that contains all information that Z provides about the parameters θ . For each r, the simulated data Z r are summarised using this statistic. The model outcomes Y r are then regressed on the sample T Z r to estimate EVSI, in the same manner as they were regressed on the parameters θ r to estimate EVPPI.

If the study information can be summarised in a single scalar, then we regress on a single predictor defined by samples of this summary statistic,

or if the study provides p different pieces of summary information, a regression with p predictors is used

A simple example is where we wish to learn an uncertain probability p of some outcome, and we propose to observe the number Z out of N individuals who have this outcome. Then for each sample p r from the uncertainty distribution of p, an observation Z r is generated from a Binomial distribution with probability p r and denominator N. The summary statistic is simply T Z r = Z r , or equivalently we could use the point estimate of p, T Z r = Z r / N .

If a summary statistic is not available in closed form, but we can estimate the parameter or parameters of interest ϕ from the data Z, e.g. by numerically maximising a likelihood, we could define T Z r to be the estimate of ϕ produced from the simulated data Z r .

The voi R package implements EVSI estimation in this way, allowing the same nonparametric regression methods as for EVPPI. Again, a single command can be used to invoke the EVSI computation, for example,

evsi(outputs, inputs, study =“binary”, n = c(10,100,1000))

for decision models, and a similar function evsivar for estimation models. outputs and inputs are data frames containing samples of model outputs and parameter values as before. The argument study specifies the design of the proposed study, which is “binary” for the simple Binomial example above, and n is a set of alternative sample sizes to compare the EVSI for. Any study design can be specified by supplying a custom R function to simulate summarised data from the study. Worked examples are given in the package vignette at https://chjackson.github.io/voi/articles/voi.html for both decision and estimation problems.

Some other methods of estimating EVSI have been proposed, see Heath et al. (2020) or Kunst et al. (2020) for a review, though the computation for these is more involved, and work is ongoing to implement them in the voi package.

Describing uncertainty about bias from indirect evidence

Consider π, the proportion of PM2.5 attributable to transport. Previously we supposed that there was published information that provides an estimate, with some measure of uncertainty, for this quantity. We assumed we could use this parameter with no further adjustment in the model, since we were confident that this estimate applied directly to our population of interest.

However in practice, we may be less confident in an estimate, either because we know it is based on a different context such as geographical area, time period or population, or because the details of the context are incompletely reported. For example, we may not be confident to apply our estimate of π to a different location where industrial sources are expected to contribute a greater proportion of PM2.5 emissions, compared to transport sources. In clinical research, this issue is described as directness or generalisability (Guyatt et al. 2011), for example when generalising results to older patients not included within a clinical trial. This uncertainty could be quantified by extending the model to include a bias parameter.

First we replace the parameter π by two different quantities, π ₀, the proportion of PM2.5 due to transport in our area of interest, and π _a , representing the average of this quantity over all areas in a region. In the original example, the evidence about π was represented by a Beta(5.6, 7.8) distribution. Suppose now we considered this to be biased, and renamed it π _a . We could define the “debiased” context-specific parameter, π 0 = expit logit π a + δ , then choose the distribution of δ to represent our beliefs about the extent of bias. For the above example, we would assume δ to be negative, reflecting a lower share of emissions due to transport. If we were unsure of the direction of bias, we might assume δ is normally distributed with mean 0 and standard deviation σ, which defines a distribution over a valid range of probabilities for π ₀, since a logit transform is used. Then we would choose σ to represent a transparent belief about the extent of potential bias (in either direction). For example, if we were 90% sure that the target estimate was no more than 20% larger than the published estimate, then we could determine σ such that the upper 90% quantile of π ₀/π _a is 1.2 (this can be done numerically, and an R function is provided for this in the accompanying code). Expert elicitation (O’Hagan et al. 2006; Tuomisto et al. 2008) might help to quantify similar kinds of uncertainties.

This is an example of probabilistic bias modelling. While there is a wide literature about bias modelling of various kinds (see, e.g. Lash et al. 2014) an advantage of a probabilistic approach is that it directly connects to estimating the value of further research using VoI methods. For example, if we could obtain data about the average over areas for the proportion of PM2.5 due to transport, this value of this would be bounded by the EVPPI of π _a . On the other hand, if we could obtain area-specific data, this would have a value of up to the EVPPI of π ₀.

The EVPPI analysis under a probabilistic bias model is illustrated in the example in Figure 4. In the middle plot, we are 90% sure there is no more than 20% bias. In the right hand plot, this probable extent of bias is increased from 20 to 50%. The standard deviation of the outcome under current information increases with the extent of bias. However, the expected improvements in precision given further data on the true fraction π ₀ in the target setting, relative to the current uncertainty, increases with the extent of bias, while the relative value of biased information (parameter π _a ) goes down.

Hierarchical modelling

More generally, if there were estimates for a parameter from multiple similar contexts, such as areas or time periods, then variations between them might be represented with a hierarchical (or multilevel) model. Consider the situation where we do not have an estimate of the proportion of PM2.5 concentrations due to transport for the target area π ₀, but we have estimates of the same quantity for M other areas. We could estimate the target π ₀ using a meta-analysis or hierarchical model of the published data (in a similar way to, e.g. Heydari et al. 2020) followed by using VoI methods to estimate the value of further research on the target area.

To illustrate with a simple example, we could specify a sampling model for the published point estimates x _i , i = 1, …, M, given the true value π _i of the quantity in area i, e.g. logit x i ∼ N logit π i , s i 2 , where s i 2 is a measure of uncertainty around the published estimate, that could be derived from a published credible interval (or elicited belief). Then (after conditioning on any area-level predictors) we might model unexplained variations in the true value between areas i, as exchangeable random effects, e.g. logit π i ∼ N μ m , σ m 2 if there were no predictors. Assigning prior distributions to the hyperparameters μ m and σ m would then define a Bayesian hierarchical model. The evidence about π ₀ would then be described by the posterior distribution of π ₀ given the data from all areas. This distribution could then be employed in the health impact model, followed by a VoI calculation to estimate the value of further research about π ₀.

Figure 5 shows a hypothetical example, where there are published estimates and credible intervals from six areas. In the left panel, there is large heterogeneity between areas, and in the right panel, there is low heterogeneity. A Bayesian hierarchical model is fitted (using the Stan software, see the supplementary code). The posterior variance of π ₀ is higher in the left-hand example. In consequence, when this posterior distribution is used in the example health impact model, the standard deviation of stroke cases averted is 669 under high heterogeneity, compared to 247 under low heterogeneity. An EVPPI calculation then predicts the standard deviation after learning perfect information about π ₀ would be 244 and 103 under high and low heterogeneity respectively. Thus when the variability in the external data is low, and we assume area-level effects are exchangeable, the absolute gains from further information are more modest, since the external data are assumed to be more representative of the required context.

Figure 5:

Example health impact model informed by data from multiple areas under a hierarchical model.

This simple example serves to illustrate the principles of using VoI in the context of a health impact model informed by evidence from a hierarchical model. See, for example, Shaddick et al. (2018) or Heydari et al. (2020) for more sophisticated examples of hierarchical modelling for air pollution exposures.

Flexible modelling of functional dependencies

Commonly, in health impact models, there are quantities that vary as a continuous function of other quantities. An example here is the “dose–response” curve that describes the relative risk of stroke as a smooth function of the exposure to PM2.5. In other models, there might be quantities that vary as a function of age, or from year to year. The data for particular exposures (or ages, or years) might be weak, thus we aim to “borrow strength” from adjacent data via the fitted curve, while avoiding excessive extrapolation.

We may then want to determine whether additional data on the weakly-informed part of the curve is expected to improve the estimate or decision from the model. For a Value of Information analysis, a functional form should be used which is flexible enough to represent the range of plausible curves given both current data and prior judgements about the curve’s true form. This might be informed by substantive beliefs about the mechanism, as in the dose–response function we illustrated from Burnett et al. (2014). Splines (Wood 2017) or fractional polynomials (Royston and Sauerbrei 2008) can also represent arbitrarily flexible functional relationships.

Once a flexible curve has been defined and fitted to current data, the joint EVPPI for all parameters describing the curve represents the maximum value of further exposure-response data of any kind, as illustrated in the example. Though this assumes that the current data are representative. If additional biases are expected, then these might be introduced as extra parameters through a probabilistic bias model.

Using EVSI to estimate the expected value of imperfect published information

Once we have determined the maximum value of further data informing a particular uncertain parameter, it might then be of interest to determine the expected value of a study of a specific design. In the example, source apportionment studies might give information on the proportion of pollution attributable to transport, and the dose–response curve might be informed by cohort studies of individuals exposed to particular concentrations of pollution. In principle, if we can define a statistical model that is assumed to generate the data, in terms of our unknown parameters, we can estimate the expected value of sample information (EVSI).

Designing a realistic primary data collection exercise in this example would involve a complicated extension to the model. However, we can still illustrate some of the principles by calculating the expected value of imperfect information that comes with a particular level of uncertainty. For the transport PM2.5 fraction π, we might want to determine the expected benefits of information from a study that achieves a certain standard error σ for this estimate. We can use the EVSI to do this, by estimating an “effective sample size” behind such a study. Even though a pollution source apportionment study would not involve observing a sample of individuals, information consisting of an estimated proportion with a standard error is approximately equivalent, mathematically, to the information provided by observing a binary outcome from some set of individuals.

This equivalence could be drawn by interpreting the standard error as the standard deviation of the Beta(a,b) posterior that would result from combining a vague Beta(0,0) prior with a “dataset” of z events out of n observations, giving n = m 1 − m / σ 2 − 1 , where m is the posterior mean a / a + b . Then, estimating m by the mean of the current uncertainty distribution of π, we can determine n, and determine the expected value of a study with standard error σ as the EVSI for a study of a binary outcome with n observations.

For example, given an estimate of m = 0.2 and a standard error of σ = 0.1, we would derive n = 15, the effective sample size of a Beta(a = 15 × 0.2, b = 15 × 1 − 0.2 ) posterior distribution. In contrast, a smaller standard error of 0.01, for the same estimate, would have led to a larger effective sample size of n = 1599.

We illustrate this technique in the health impact example by estimating the expected value of a second source of information on the proportion of PM2.5 π due to transport. The first source of information was represented by a Beta(5.7,8.9) distribution which has a mean of 0.4 and standard deviation of 0.12 (see Figure 1). We estimate the expected value of obtaining an additional source of information to supplement the first source, which has the same level of precision as the first source. Using the technique above, we deduce that obtaining an estimate of a proportion with a standard error of 0.12 would be equivalent to performing a study of n = 42 observations of a binary outcome.

Hence we can use the technique described in Section 4.3 to calculate the expected value of sample information for this second source. For each Monte Carlo sample π 1 , π 2 , … from the original Beta(5.7,8.9) distribution, we generate a value Z r from a Binomial distribution with probability π r and denominator n = 42. This represents a potential result of the proposed study, sampled from the predictive distribution of the binary outcome. The “summary statistic” of the study data is simply T Z = Z . A regression is then fitted with the sampled model outputs Y r as the outcome and the set of sampled Z r as the explanatory variable. The variance of the fitted values from this regression then estimates the expected variance of the model output that would remain after the second source of data is collected (as in Section 4.1). We predict the remaining standard deviation to be 151 for the stroke cases saved after obtaining the second source of information. Thus the information would have a substantial value, in the context of the estimated SD of 135 remaining under perfect information, and the original SD of 188.