Skip to content
Licensed Unlicensed Requires Authentication Published by De Gruyter July 4, 2019

A new indirect estimation of reference intervals: truncated minimum chi-square (TMC) approach

Werner Wosniok and Rainer Haeckel EMAIL logo

Abstract

All known direct and indirect approaches for the estimation of reference intervals (RIs) have difficulties in processing very skewed data with a high percentage of values at or below the detection limit. A new model for the indirect estimation of RIs is proposed, which can be applied even to extremely skewed data distributions with a relatively high percentage of data at or below the detection limit. Furthermore, it fits better to some simulated data files than other indirect methods. The approach starts with a quantile-quantile plot providing preliminary estimates for the parameters (λ, μ, σ) of the assumed power normal distribution. These are iteratively refined by a truncated minimum chi-square (TMC) estimation. The finally estimated parameters are used to calculate the 95% reference interval. Confidence intervals for the interval limits are calculated by the asymptotic formula for quantiles, and tolerance limits are determined via bootstrapping. If age intervals are given, the procedure is applied per age interval and a spline function describes the age dependency of the reference limits by a continuous function. The approach can be performed in the statistical package R and on the Excel platform.

Acknowledgment

The authors are grateful to Dr. Jakob Zierk (for providing a haemoglobin data set of paediatric patients), Dr. Alexander Bertram (γ-GT), Dr. Alexander Krebs (sodium), Dr. Rainer Klauke (alanine aminotransaminase) and Dr. Antje Torge (total protein, thrombocytes, hs cardiac troponin) for providing “big data” pools of patients’ data.

  1. Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.

  2. Research funding: None declared.

  3. Employment or leadership: None declared.

  4. Honorarium: None declared.

  5. Competing interests: The funding organization(s) played no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the report for publication.

Appendix

Assumptions of the TMC approach

Central assumptions of the TMC approach are: (i) The data contains an interval, in which the measured values follow a partial PND. These values are assumed to stem from non-diseased persons or from (few) persons who are diseased but have values of the measurand which do not disturb the PND in the interval. This interval, called non-pathological interval, does not necessarily contain a complete PND. (ii) Values are stochastically independent from another (no multiple values from the same subject).

These assumptions are shared by all indirect methods. The first one is more general than in many other approaches, which assume a normal or a log-normal distribution of values from non-diseased persons.

The power normal distribution has density [25]

(A1)f(x;λ,μ,σ)=1Kσ2πxλ1exp(12(yμσ)2)

where

(A2)K=1φ(1μσ+μσ)

with φ denoting the standard normal (Gaussian) probability density function, and y is the Box-Cox transformation of x with parameter λ: y=(xλ–1)/λ for 0<λ≤1 and y=ln(x) for λ=0. The corresponding cumulative distribution function is

(A3)F(x;λ,μ,σ)=0xf(s;λ,μ,σ)ds

The TMC approach considers reported values as describing intervals on the real line. A reported value “x” is interpreted as the interval [x−d/2, x+d/2), where d is the rounding unit. The notation [a, b) describes an interval with the left limit (a) belonging to the interval, but the right limit (b) not. As example: d=0.1 means that values are reported up to one decimal place, and a reported value of 61.4 g/L corresponds to the interval [61.35, 61.45). A reported value “<x” is understood as the interval [0, x), where usually x is the limit of detection. Data of this kind can be presented as a histogram.

Procedural steps of the TMC approach

The TMC approach has 6 steps, which are implemented in the R script “TMC” (see “TMC implementation” below):

  1. Selection of an age/sex stratum to analyse.

  2. Construction of a histogram for the stratum

  3. Obtain an initial estimate for the PND parameters λ, μ, σ from a sequence of QQ plots

  4. Obtain improved estimates of the PND parameters by the TMC procedure for various truncation interval candidates

  5. Identify the optimal PND parameter among the candidates considered in step 4

  6. Calculate the RILs from the optimal PND parameters from step 5

These steps are described in detail below.

  1. Select an age/sex stratum to analyse

    Age groups are defined by the user. If age is given in years, a typical definition of age groups is (18–29, 30–39, 40–49, 50–59, 60–69, 70–79, 80–89, 90–100) years. Sex is typically coded by F and M. With eight age groups and both sexes in the data there are 16 age/sex strata in the data, which are analysed independently and consecutively.

  2. Construction of a histogram for the stratum

    The data used for TMC estimation corresponds to the data that is needed to construct a histogram for the data. A histogram consists of k bins with limits ci and ci+1, i=1, 2, …, k, and a set of counts n1, n2, …, nk, where ni, i=1, 2, …, k, are the number of reported values lying between ci and ci+1, i=1, 2, …, k. The lower limit is included in each bin, for nk also the upper limit. Initial values for the ci are the mean values between reported values, completed by the maximum of (reported minimum-0.5 rounding units, 0) and (reported maximum+0.5 rounding units) for the outer intervals. All reported values lie between the outer interval limits c1 and ck+1. If the rounding unit is small, this initial construction generates many bins with small counts ni, which is unfavourable for the subsequent calculation of χ2 contributions from the bins. Therefore, bins containing less than the pre-specified minimum of 25 values are aggregated with their neighbours until the required minimum count is achieved. If the aggregation of bins leaves less than 12 bins, a warning is issued, and the actual stratum is not evaluated. Both conditions together imply a minimal number of 300 values per stratum. Bin limits need not be, and usually are not, equidistant. Note that there are k+1 bin limits, but only k bins.

    The aggregated data can be represented graphically as a histogram with the ci as bin limits and the bar heights proportional to ni (see Figure 3). Where needed in the equations below, bins are indexed by the index of their lower limit: the bin [ci, ci+1) has index i.

  3. Obtain an initial estimate for the PND parameters λ, μ, σ from a sequence of QQ plots

    Rationale: Under the assumptions of the TMC approach there must be an interval of the data which appears as a (nearly) straight line in a QQ plot, if the (nearly) correct λ is used to transform the data. This λ and the corresponding data interval are searched for by a simple grid search, using the coefficient of determination as measure for linearity. Regression parameters from the interval provide initial values for the subsequent TMC estimation. These initial values cannot serve as final estimates for the PND parameters for reasons that are outlined under “A remark on QQ plot regression for incomplete data” below.

    1. Set λ=0

    2. Set rmax2=0

    3. Transform the data in the actual stratum by the Cox-Box transformation:

      yi=(xiλ1)/λ   if  λ>0,yi=lnxi if  λ=0

    4. Construct a QQ plot of yi

      abscissa: expected values of the order statistics of a standard Gaussian distribution E(x[i])=Φ−1((i–0.5)/n) with Φ−1 denoting the inverse standard Gaussian distribution

      ordinate: yi

    5. Define the first regression window to consider as the interval between the minimum of the data and the 70% percentile of the data. The 70% percentile is appropriate if the prevalence of contaminated data is not larger than 30%. Otherwise the percentile and the limit in 3.8 must be adjusted accordingly.

    6. Calculate a linear regression for the data in the actual regression window

    7. If the coefficient of determination r2 from 3.6 is >rmax2:

      Set rmax2=r2,λini=λ, μini=β0, σini=β1. β0 and β1 are the regression coefficients (intercept and slope) from 3.6

    8. Move the regression window 5% upwards. If the lower limit is now below or at the 30% percentile of the data, continue with step 3.6, otherwise continue with step 3.9

    9. Increase λ by 0.05. If the resulting λ is <=1, continue with step 3.3, otherwise continue with step 3.10.

    10. The values λini, μini and σini recorded in 3.7 are the initial values for the subsequent TMC procedure. Figure 9 shows an example for an optimal QQ plot resulting at step 3.10.

  4. Obtain improved estimates of the PND parameters by the TMC procedure for various truncation interval candidates

    Rationale: Results from step 3 are only preliminary for the reasons given under “A remark on QQ plot regression for incomplete data” below. This holds for the parameter estimates and the location of the regression window. The TMC approach does not suffer from the problems that the QQ plot has. However, the TMC approach requires more computational effort than a QQ plot, because it is an iterative procedure. The TMC approach is executed for a sequence of truncation interval candidates. A truncation interval candidate has a similar function as the regression window in step 3: it is used to obtain estimates for λ, μ, σ, and the properties of the corresponding distributions are used in step 5 to select the optimal truncation interval among the truncation interval candidates.

    1. Define the first truncation interval candidate as the interval between the minimum of the data and the 70% percentile of the data. If this interval comprises less than eight bins, increase it to eight bins. The 70% percentile is appropriate if the prevalence of contaminated data is not larger than 30%. Otherwise the percentile must be adjusted accordingly.

    2. Calculate estimates for the PND parameters λ, μ, σ on the basis of the actual candidate truncation interval. The rationale is to choose the parameters in a way that the estimated distribution fits as good as possible to the empirical histogram in the truncation interval, while predictions outside the truncation interval should not produce illogical results (like predicting the uncontaminated part of the dataset being larger than the total dataset).

      For the actual truncation interval, the parameters λ, μ, σ are estimated by a Newton-Raphson procedure. The criterion to minimise is the penalised χ2 distance

      (A4)D(λ,μ,σ)=iTgi+jTwj

      Here, T is the set of bin indices [ci, ci+1) contained in the truncation interval [thi, tlo], gi is the chi-square contribution from interval i, and wj is the penalty term for interval j. The χ2 contribution of interval [ci, ci+1) is defined by

      (A5)gi=(niNi(λ,μ,σ))2Ni(λ,μ,σ)

      where Ni is the expected number of values in interval i, given by

      (A6)Ni(λ,μ,σ)=F(ci+1;λ,μ,σ)F(ci;λ,μ,σ)F(thi;λ,μ,σ)F(tlo;λ,μ,σ)kTnk

      This is a conditional expectation, as only the distribution of values in the truncation interval is considered. The sum of all χ2 contributions from the truncation interval has an asymptotical χ2 distribution and is used to test the goodness of fit in the truncation interval. The corresponding p-value pfit is

      (A7)pfit=P(iTgi>χ|T|42)

      It is an approximate measure, because the previous operations for finding the truncation interval are not accounted for.

      The penalty term for interval j in (A4) is defined by

      (A8)wj=ε P(χ12<δj)

      where P(χ12<δj) is the χ2 distribution function with 1 degree of freedom, δj is the χ2 contribution of bin j to (A4). This contribution is >0 only if the expected count in bin j is larger than the observed count, otherwise it is zero:

      (A9)δj=(Njnj)2Nj1Nj>nj

      The factor ε in (A9) is a weighting factor (default: 1.0). The penalty term contributes to the optimality criterion (A4) only in those data intervals, for which the predicted count is larger than the observed one. Also, wj gives a considerable contribution only if δ is outside the range of random fluctuation of a χ2 random variable.

      The iterative Newton-Raphson procedure uses the results λini, μini and σini from 3.10 as initial values.

      Figure 10 displays a histogram with marked truncation interval, observed and expected counts, and Table 2 provides the values involved in the estimation process.

    3. Record the truncation interval limits, the parameter estimates λ, μ, σ for the actual truncation interval, the fit parameters (D(λ, μ, σ) from A4), (pfit from A7) and the degrees of freedom df=|T|–4 for the fit parameter (A7).

    4. If the right limit of the actual truncation interval is below the maximum of the data, shift the truncation interval one bin to the right and continue with step 4.2, otherwise continue with step 5.1

  5. Identify the optimal PND parameter among the candidates considered in step 4

    1. Sort the table set up in step 4.3 by D(λ, μ, σ). The truncation interval with smallest D(λ, μ, σ) is the optimal truncation interval, the corresponding parameters, denoted by λ^,μ^,σ^ define the optimal PND for unaffected values.

    2. If the goodness of fit pfit from (A7) for the optimal parameters is <0.05, issue a warning. This situation indicates an insufficient fit of the estimated to the observed data. Possible reasons are inappropriate stratification or a violation of the assumptions formulated in the beginning of the Appendix.

  6. Calculate the RILs from the optimal PND parameters from step 5

    1. The RILs are calculated as quantiles of the optimal PND found in step 5.1. Typically, the 2.5% and the 97.5% quantiles are used. Quantiles are calculated by inversion of equation (A3).

Figure 9: A QQ plot for a subset of the data from Figure 5 (females, 70–79 years, n=11,056).The horizontal dashed lines include the optimal window, the regression line is fit to data in this window only. The data is transformed by λini=0.5, the optimal value according to the grid search from step 3. Further initial values are μini=6.45 and σini=1.60.
Figure 9:

A QQ plot for a subset of the data from Figure 5 (females, 70–79 years, n=11,056).

The horizontal dashed lines include the optimal window, the regression line is fit to data in this window only. The data is transformed by λini=0.5, the optimal value according to the grid search from step 3. Further initial values are μini=6.45 and σini=1.60.

Figure 10: Histogram for a subset of the data from Figure 3 (females, 70–79 years, n=11056).Grey bins indicate the truncation interval, white bins lie outside the truncation interval. The blue PND probability density curve is fitted by the TMC approach. Solid red and green rectangles indicate the differences between observed and expected counts which contribute to the χ2 criterion (A5). Red rectangles indicate bins in which the expected count is larger than the observed. These rectangles contribute to (A5) inside and outside the truncation interval. Bins outside the truncation interval with expected count smaller than observed, marked by green hatched rectangles, do not contribute to (A5). The vertical dashed blue lines indicate the 2.5% and 97.5% RILs. Details of the calculation are given in Table 2. Observed bin values are white coloured areas+green areas or white coloured areas without red areas.
Figure 10:

Histogram for a subset of the data from Figure 3 (females, 70–79 years, n=11056).

Grey bins indicate the truncation interval, white bins lie outside the truncation interval. The blue PND probability density curve is fitted by the TMC approach. Solid red and green rectangles indicate the differences between observed and expected counts which contribute to the χ2 criterion (A5). Red rectangles indicate bins in which the expected count is larger than the observed. These rectangles contribute to (A5) inside and outside the truncation interval. Bins outside the truncation interval with expected count smaller than observed, marked by green hatched rectangles, do not contribute to (A5). The vertical dashed blue lines indicate the 2.5% and 97.5% RILs. Details of the calculation are given in Table 2. Observed bin values are white coloured areas+green areas or white coloured areas without red areas.

Table 2:

Example for calculating the contributions to the optimisation criterion (A4) for the data shown in Figure 3.

Bin numberLeft bin limit, U/LRight bin limit, U/LObserved CountExpected countObserved minus expectedχ2, unweightedδP(χ12<δ)χ2, weighted
109287122.8164.2219.670.000.000.00
2911380375.34.70.06
31113750753.2−3.20.01
4131510421079.8−37.81.32
5151712541239.714.30.17
6171912261222.63.40.01
7192111021083.518.50.32
82123849889.0−40.01.801.800.821.48
92325677689.5−12.50.230.230.370.08
102527520512.97.10.100.000.000.00
112729432369.962.110.430.000.000.00
122931347260.686.428.650.000.000.00
133133268180.487.642.510.000.000.00
143337414206.8207.2207.580.000.000.00
15374126993.7175.3327.870.000.000.00
16414728452.9231.11010.090.000.000.00
17475524417.8226.22871.420.000.000.00
1855692424.2237.8133670.000.000.00
19694854690.3468.77042480.000.000.00
  1. The grey shaded cells refer to bins in the truncation interval. The optimisation criterion is the sum of the unweighted χ2 contributions from bins in the truncation interval plus the sum of the weighted χ2 contributions from bins outside the truncation interval (the penalty term). These contributions are shown in boldface. The weighting factor ε from (A9) is set to 1 and therefore omitted from the table.

A remark on QQ plot regression for incomplete data

A QQ plot is a convenient method to display graphically distributional properties of a data set. In a normal QQ plot, normally (Gaussian) distributed data fluctuate around a straight line with intercept and slope approximating mean and standard deviation of the distribution. If the data consists of two normally and not overlapping distributions, the QQ plot shows two different straight lines. If the distributions overlap, curved structures arise, but there may still be (nearly) linear components. In either of these situations one might try to estimate intercepts and slopes and use them as means and standard deviations of the distributions involved. However, this is only an approximation, as is easily seen from the construction of a QQ plot. The horizontal axis carries the expected positions of the ordered values. These positions, denoted by E(x[i]), i=1, 2, …, n, where n is the dataset size, are calculated as

(A10)E(x[i])=Φ1((i0.5)/n)

where Φ is the standard normal distribution function. This formula contains the sample size n, which is clearly available if only one distribution is involved. If two (different) distributions are involved, a QQ plot will no more show points fluctuating around straight lines, because all data points have wrong positions on the abscissa. This effect can be seen with artificial test data consisting of two datasets which each contain the expected order statistics (A10) as values. A QQ plot showing correctly the associated straight lines with the correct parameters of the underlying distributions could only be constructed if the sample sizes were known as well as the origin of each point (first or second distribution?). For indirect methods of RL estimation, this information is not available. However, as indirect methods use to operate with large datasets, the error in estimating distributional parameters from a QQ plot is small enough to allow using these estimates as starting values for a Newton-Raphson procedure.

Implementation of the TMC approach

The TMC approach described above is implemented in the script “TMC” that runs under the generally accessible R software language [26]. This programme can be gratuitously obtained from the authors. It automatically considers the influence of gender and age, allows specifying age intervals, and presents the spline function for the relation between RILs and age. The user may choose the reference interval level (e.g. 2.5% and 97.5% quantiles, or 1% and 99% quantile).

The user can also choose whether confidence or tolerance limits for RLs are calculated. Although tolerance limits are more appropriate, they require a much longer computation time.

The R script processes routinely comma separated files (*.csv), where columns represent variables. Required input variables are the measured value, and if a stratified analysis is required, also age and sex. Additional filtering (exclusion of intensive care units, selection of first values only) is possible, if the corresponding further input variables are supplied.

References

1. Haeckel R, Wosniok W, Arzideh F. A plea for intra-laboratory decision limits. Part 1 General consideration and concepts for determining decision limits. Clin Chem Lab Med 2007;45:1033–42.10.1515/CCLM.2007.249Search in Google Scholar PubMed

2. Jones GR, Haeckel R, Loh TP, Sikaris K, Streichert Th, Katayev A, et al. Indirect methods for reference interval determination – review and recommendations. Clin Chem Lab Med 2019;57:20–9.10.1515/cclm-2018-0073Search in Google Scholar PubMed

3. Ozarda Y. Reference intervals: current status, recent developments and future considerations. Biochem Med 2016;26:5–16.10.11613/BM.2016.001Search in Google Scholar PubMed PubMed Central

4. Hoffmann RG. Statistics in the practice of medicine. J Am Med Assoc 1963;185:864–73.10.1001/jama.1963.03060110068020Search in Google Scholar PubMed

5. Hoffmann G, Lichtinghagen R, Wosniok W. Simple estimation of reference intervals from routine laboratory tests. J Lab Med 2016; advanced online publication: DOI: 10.1515/labmed-2015-0104.10.1515/labmed-2015-0104Search in Google Scholar

6. Bhattacharya CG. A simple method of resolution of a distribution into Gaussian components. Biometrics 1967; 23:115–35.10.2307/2528285Search in Google Scholar

7. Arzideh F, Wosniok W, Gurr E, Hinsch W, Schumann G, Weinstock N, et al. A plea for intra-laboratory reference limits. Part 2. A bimodal retrospective concept for determining reference intervals from intra-laboratory data bases demonstrated by catalytic activity concentrations of enzymes. Clin Chem Lab Med 2007;45:1043–57.10.1515/CCLM.2007.250Search in Google Scholar PubMed

8. Kristiansen S, Friis-Hansen L, Jensen CA, Hansen SI. Validation study on the NORIP LDH reference intervals with a proposed new upper limit. Scand J Clin Lab Invest 2018; in print10.1080/00365513.2018.1481223Search in Google Scholar PubMed

9. Lykkeboe S, Nielsen CG, Christensen PA. Indirect method for validating transference of reference intervals. Clin Chem Lab Med 2018;56:463–70.10.1515/cclm-2017-0574Search in Google Scholar PubMed

10. Den Elzen WP, Brouwer A, CobbaerThelen MH, Haagen IA, Cobbaert CM. Number: standardized reference intervals in the Netherlands using a ‘big data’ approach. Clin Chem Lab Med 2019;57:42–56.10.1515/cclm-2018-0462Search in Google Scholar PubMed

11. Ozcurumez M, Haeckel R. Biological variables influencing the estimation of reference limits. Scand J Clin Invest Lab Med 2018;78:337–45.10.1080/00365513.2018.1471617Search in Google Scholar PubMed

12. Zierk J, Arzideh F, Haeckel R, Rauh M, Metzlerr M, Ganslandt Th, et al. Indirect determination of hematology reference intervals in adult patients on Beckman Coulter UniCell DxH 800 and Abbott CELLDyn Sapphire devices. Clin Chem Lab Med 2019;57:730–9.10.1515/cclm-2018-0771Search in Google Scholar PubMed

13. Sonntag O. Is this normal? – This is normal implication and interpretation of the so-called normal values. J Lab Med 2003;8:302–10.Search in Google Scholar

14. Shah SA, Ishihara K, Dherah AJ, Ashavald TF. Reference intervals for 33 biochemical analytes in a healthy Indian population: C-RIDL IFCC initiative. Clin Chem Med Lab 2018;56:2093–103.10.1515/cclm-2018-0152Search in Google Scholar PubMed

15. Reed AH, Cannon DC, Winkelman JW, Bhasin YP, Henry RJ, Pileggi VJ. Estimation of normal ranges from a controlled sample. Clin Chem 1978;18:57–61.10.1093/clinchem/18.1.57Search in Google Scholar

16. Grossi E, Columbo R, Cavuto S, Franzini C. The REALAB project: a new method for the formulation of reference intervals based on current data. Clin Chem 2005;51:1232–40.10.1373/clinchem.2005.047787Search in Google Scholar PubMed

17. Ceriotti F, Henny J, Queralto J, Ziyu S, Özarda Y, Chen B, et al. Common reference intervals for aspartate aminotransferase (AST), alanine aminotransferase (ALT) and γ-glutamyl transferase (GGT) in serum: results from an IFCC multicentre study. Clin Chem Lab Med 2010;48:1593–601.10.1515/CCLM.2010.315Search in Google Scholar PubMed

18. Bakan E, Polat H, Ozarda Y, Ozturk N, Baygutalp NK, Umudum FZ, et al. A reference interval study for common biochemical analytes in Eastern Turkey: a comparison of a reference population with laboratory data mining. Biochem Med 2016;26:210–23.10.11613/BM.2016.023Search in Google Scholar PubMed PubMed Central

19. Biino G, Balduni CL, Casula L, Cavallo P, Vaccargiu S, Parracciani D, et al. Analysis of 12517 inhabitants of a Sardinian geographic isolate reveals that predisposition to thrombocytopenia and thrombocytosis are inherited traits. Haematologia 2011;96:96–101.10.3324/haematol.2010.029934Search in Google Scholar PubMed PubMed Central

20. Balduini CL, Noris P. Platelet count and aging. Haematologica 2014;99:953–5.10.3324/haematol.2014.106260Search in Google Scholar PubMed PubMed Central

21. Masuch A, Ittermann T, Greinacher A, Lubenow N, Kohlmann Th, Nauck M, et al. High-sensitivity cardiac troponin T: association of creatine kinase with the 99th percentile. Clin Chem 2018;64:973–7.10.1373/clinchem.2017.285775Search in Google Scholar PubMed

22. Monneret D, Gellerstedt M, Bonnefont-Rousellot D. Determination of age- and sex-specific 99th percentiles for high-sensitive troponin T from patients: an analytical imprecision- and partitioning-based approach. Clin Chem Lab Med 2018;56:818–29.10.1515/cclm-2017-0256Search in Google Scholar PubMed

23. Zierk J, Arzideh F, Haeckel R, Rascher W, Rauh M, Metzler M. Indirect determination of pediatric blood count reference intervals. Clin Chem Lab Med 2013;51:863–72.10.1515/cclm-2012-0684Search in Google Scholar PubMed

24. Haeckel R. The influence of age and other biological variables on the estimation of reference limits of cardiac troponin T. Clin Chem Lab Med 2018;56:685–7.10.1515/cclm-2017-1082Search in Google Scholar PubMed

25. Freeman J, Modarres R. Inverse Box-Cox: the power-normal distribution. Stat Probabil Lett 2006;76:764–72.10.1016/j.spl.2005.10.036Search in Google Scholar

26. R Core Team (2017). R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org.Search in Google Scholar

Received: 2018-12-18
Accepted: 2019-05-19
Published Online: 2019-07-04
Published in Print: 2019-11-26

© 2019 Walter de Gruyter GmbH, Berlin/Boston

Downloaded on 28.11.2022 from frontend.live.degruyter.dgbricks.com/document/doi/10.1515/cclm-2018-1341/html
Scroll Up Arrow