Jump to ContentJump to Main Navigation
Show Summary Details

Clinical Chemistry and Laboratory Medicine (CCLM)

Published in Association with the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM)

Editor-in-Chief: Plebani, Mario

Ed. by Gillery, Philippe / Lackner, Karl J. / Lippi, Giuseppe / Melichar, Bohuslav / Payne, Deborah A. / Schlattmann, Peter / Tate, Jillian R.

12 Issues per year

IMPACT FACTOR increased in 2015: 3.017
Rank 5 out of 30 in category Medical Laboratory Technology in the 2014 Thomson Reuters Journal Citation Report/Science Edition

SCImago Journal Rank (SJR) 2015: 0.873
Source Normalized Impact per Paper (SNIP) 2015: 0.982
Impact per Publication (IPP) 2015: 2.238

See all formats and pricing


Select Volume and Issue


30,00 € / $42.00 / £23.00

Get Access to Full Text

Statistical methods for assessment of added usefulness of new biomarkers

Michael J. Pencina
  • Department of Biostatistics, Boston University, Boston, MA, USA
  • NHLBI's Framingham Heart Study, Framingham, MA, USA
  • Harvard Clinical Research Institute, Boston, MA, USA
/ Ralph B. D'Agostino
  • Department of Mathematics and Statistics, Boston University, Boston, MA, USA
  • NHLBI's Framingham Heart Study, Framingham, MA, USA
/ Ramachandran S. Vasan
  • School of Medicine, Boston University, Boston, MA, USA
  • NHLBI's Framingham Heart Study, Framingham, MA, USA
Published Online: 2010-08-18 | DOI: https://doi.org/10.1515/CCLM.2010.340


The discovery and development of new biomarkers continues to be an exciting and promising field. Improvement in prediction of risk of developing disease is one of the key motivations in these pursuits. Appropriate statistical measures are necessary for drawing meaningful conclusions about the clinical usefulness of these new markers. In this review, we present several novel metrics proposed to serve this purpose. We use reclassification tables constructed on the basis of clinically meaningful disease risk categories to discuss the concepts of calibration, risk separation, risk discrimination, and risk classification accuracy. We discuss the notion that the net reclassification improvement (NRI) is a simple yet informative way to summarize information contained in risk reclassification tables. In the absence of meaningful risk categories, we suggest a ‘category-less’ version of the NRI and integrated discrimination improvement as metrics to summarize the incremental value of new biomarkers. We also suggest that predictiveness curves be preferred to receiver operating characteristic curves as visual descriptors of a statistical model's ability to separate predicted probabilities of disease events. Reporting of standard metrics, including measures of relative risk and the c statistic, is still recommended. These concepts are illustrated with a risk prediction example using data from the Framingham Heart Study.

Clin Chem Lab Med 2010;48:1703–11.

Keywords: calibration; discrimination; integrated discrimination improvement; net reclassification improvement; reclassification; risk prediction

Corresponding authors: Michael J. Pencina, PhD, Department of Biostatistics, Boston University, Framingham Heart Study, Harvard Clinical Research Institute, 111 Cummington St., Boston, MA 02215, USA Phone: +1 617-358-3386, Fax: +1 617-353-4767, Ramachandran S. Vasan, MD, DM, FACC, Boston University School of Medicine, Framingham Heart Study, 73 Mount Wayte Avenue, Suite 2, Framingham, MA 01702-5803, USA Phone: +1 508-935-3450,

Received: 2010-02-11

Accepted: 2010-06-18

Published Online: 2010-08-18

Published in Print: 2010-12-01

Citation Information: Clinical Chemistry and Laboratory Medicine. Volume 48, Issue 12, Pages 1703–1711, ISSN (Online) 1437-4331, ISSN (Print) 1434-6621, DOI: https://doi.org/10.1515/CCLM.2010.340, August 2010

Citing Articles

Here you can find all Crossref-listed publications in which this article is cited. If you would like to receive automatic email messages as soon as this article is cited in other publications, simply activate the “Citation Alert” on the top of this page.

Mahnaz Barkhordari, Mojgan Padyab, Mahsa Sardarinia, Farzad Hadaegh, Fereidoun Azizi, and Mohammadreza Bozorgmanesh
International Journal of Endocrinology and Metabolism, 2016, Volume Inpress, Number Inpress
Peter M. Burch, David Greg Hall, Elizabeth G. Walker, William Bracken, Richard Giovanelli, Richard Goldstein, Richard E. Higgs, Nicholas M.P. King, Pamela Lane, John-Michael Sauer, Laura Michna, Nagaraja Muniappa, Michael L. Pritt, Katerina Vlasakova, David E. Watson, Debra Wescott, Tanja S. Zabka, and Warren E. Glaab
Toxicological Sciences, 2015, Page kfv328
Bent Müller, Arndt Wilcke, Anne-Laure Boulesteix, Jens Brauer, Eberhard Passarge, Johannes Boltze, and Holger Kirsten
Human Genetics, 2016
Joseph Yeboah, Rebekah Young, Robyn L. McClelland, Joseph C. Delaney, Tamar S. Polonsky, Farah Z. Dawood, Michael J. Blaha, Michael D. Miedema, Christopher T. Sibley, J. Jeffrey Carr, Gregory L. Burke, David C. Goff, Bruce M. Psaty, Philip Greenland, and David M. Herrington
Journal of the American College of Cardiology, 2016, Volume 67, Number 2, Page 139
Beth A. Payne, Jennifer A. Hutcheon, Dustin Dunsmuir, Garth Cloete, Guy Dumont, David Hall, Joanne Lim, Laura A. Magee, Rozina Sikandar, Rahat Qureshi, Erika van Papendorp, J. Mark Ansermino, and Peter von Dadelszen
Journal of Obstetrics and Gynaecology Canada, 2015, Volume 37, Number 1, Page 16
Jing Zhang, Zhi-De Hu, Jia Song, and Jiang Shao
Medicine, 2015, Volume 94, Number 47, Page e2158
Arash Aghajani Nargesi, Behnam Heidari, Sadaf Esteghamati, Nima Hafezi-Nejad, Sara Sheikhbahaei, Atieh Pajouhi, Manouchehr Nakhjavani, and Alireza Esteghamati
Atherosclerosis, 2015
Crystel M. Gijsberts, Hester M. den Ruijter, Dominique P.V. de Kleijn, Albert Huisman, Maarten J. ten Berg, Richard H.A. van Wijk, Folkert W. Asselbergs, Michiel Voskuil, Gerard Pasterkamp, Wouter W. van Solinge, and Imo E. Hoefer
Medicine, 2015, Volume 94, Number 45, Page e1992
Petter Bjornstad, Robert H. Eckel, Laura Pyle, Marian Rewers, David M. Maahs, and Janet K. Snell-Bergeon
The American Journal of Cardiology, 2015
Behnam Heidari, Arash Aghajani Nargesi, Nima Hafezi-Nejad, Sara Sheikhbahaei, Atieh Pajouhi, Manouchehr Nakhjavani, and Alireza Esteghamati
American Heart Journal, 2015
Roberto Elosua, Gavin Lucas, and Carla Lluis-Ganella
Current Cardiovascular Risk Reports, 2013, Volume 7, Number 1, Page 33
Gethin W. Hodges, Casper N. Bang, Kristian Wachtell, Jesper Eugen-Olsen, and Jørgen L. Jeppesen
Canadian Journal of Cardiology, 2015
Koki Mise, Junichi Hoshino, Toshiharu Ueno, Ryo Hazue, Keiichi Sumida, Rikako Hiramatsu, Eiko Hasegawa, Masayuki Yamanouchi, Noriko Hayami, Tatsuya Suwabe, Naoki Sawa, Takeshi Fujii, Shigeko Hara, Kenichi Ohashi, Kenmei Takaichi, and Yoshifumi Ubara
Diabetes/Metabolism Research and Reviews, 2015, Page n/a
Michael C. Kontos, Benjamin M. Scirica, Anita Y. Chen, Laine Thomas, Monique L. Anderson, Deborah B. Diercks, James G. Jollis, and Matthew T. Roe
American Heart Journal, 2015, Volume 169, Number 4, Page 515
Mohammadreza Bozorgmanesh, Mahsa Sardarinia, Farhad Hajsheikholeslami, Fereidoun Azizi, and Farzad Hadaegh
European Journal of Nutrition, 2015
Setor K. Kunutsor, Stephan J.L. Bakker, Jenny E. Kootstra-Ros, Ronald T. Gansevoort, and Robin P.F. Dullaart
Atherosclerosis, 2015, Volume 238, Number 2, Page 356
I. Drion, N. Kleefstra, G. W. D. Landman, A. Alkhalaf, J. Struck, K. H. Groenier, S. J. L. Bakker, and H. J. G. Bilo
Diabetes Care, 2012, Volume 35, Number 11, Page 2354
Frederick R. Appelbaum, Claudio Anasetti, Joseph H. Antin, Harold Atkins, Stella Davies, Steven Devine, Sergio Giralt, Helen Heslop, Ginna Laport, Stephanie J. Lee, Brent Logan, Marcelo Pasquini, Michael Pulsipher, Edward Stadtmauer, John R. Wingard, and Mary M. Horowitz
Biology of Blood and Marrow Transplantation, 2015, Volume 21, Number 2, Page 202
Justin L. Grodin, Sarah Neale, Yuping Wu, Stanley L. Hazen, and W.H. Wilson Tang
The American Journal of Medicine, 2015, Volume 128, Number 3, Page 276
Andrei P. Drabovich, Eduardo Martínez-Morillo, and Eleftherios P. Diamandis
Biochimica et Biophysica Acta (BBA) - Proteins and Proteomics, 2015, Volume 1854, Number 6, Page 677
Thor Ueland, Lars Gullestad, Ståle H. Nymo, Arne Yndestad, Pål Aukrust, and Erik T. Askevold
Clinica Chimica Acta, 2015, Volume 443, Page 71
Ewout W. Steyerberg, Moniek M. Vedder, Maarten J. G. Leening, Douwe Postmus, Ralph B. D'Agostino, Ben Van Calster, and Michael J. Pencina
Biometrical Journal, 2014, Page n/a
Gerard D. van Rijsbergen, Huibert Burger, Steven D. Hollon, Hermien J. Elgersma, Gemma D. Kok, Jack Dekker, Peter J. de Jong, and Claudi L.H. Bockting
Psychiatry Research, 2014, Volume 220, Number 1-2, Page 287
Petter Bjornstad, David M. Maahs, Christopher J. Rivard, Laura Pyle, Marian Rewers, Richard J. Johnson, and Janet K. Snell-Bergeon
Acta Diabetologica, 2014, Volume 51, Number 5, Page 783
Tariq Ahmad, Mona Fiuzat, Michael J. Pencina, Nancy L. Geller, Faiez Zannad, John G.F. Cleland, James V. Snider, Stephan Blankenberg, Kirkwood F. Adams, Rita F. Redberg, Jae B. Kim, Alice Mascette, Robert J. Mentz, Christopher M. O'Connor, G. Michael Felker, and James L. Januzzi
JACC: Heart Failure, 2014, Volume 2, Number 5, Page 477
Petra G. van Peet, Anton J.M. de Craen, Jacobijn Gussekloo, and Wouter de Ruijter
AGE, 2014, Volume 36, Number 3
Vanessa Xanthakis, Lisa M. Sullivan, Ramachandran S. Vasan, Emelia J. Benjamin, Joseph M. Massaro, Ralph B. D’Agostino, and Michael J. Pencina
Statistics in Medicine, 2014, Volume 33, Number 15, Page 2577
Alireza Esteghamati, Nima Hafezi-Nejad, Ali Zandieh, Sara Sheikhbahaei, Maryam Ebadi, and Manouchehr Nakhjavani
Journal of Cardiology, 2014, Volume 64, Number 4, Page 290
Zhi-De Hu, Xiao-Fei Liu, Xiao-Cui Liu, Chun-Mei Ding, and Cheng-Jin Hu
Clinica Chimica Acta, 2014, Volume 433, Page 44
Lori B. Daniels, Paul Clopton, Gail A. Laughlin, Alan S. Maisel, and Elizabeth Barrett-Connor
American Heart Journal, 2014, Volume 167, Number 5, Page 674
A. Massot, D. Giralt, A. Penalba, T. Garcia-Berrocoso, M. Navarro-Sobrino, J.F. Arenillas, M. Ribó, C.A. Molina, J. Alvarez-Sabín, J. Montaner, and P. Delgado
Atherosclerosis, 2014, Volume 233, Number 1, Page 186
Xiao-Fei Liu, Zhi-De Hu, Xiao-Cui Liu, Yuan Cao, Chun-Mei Ding, and Cheng-Jin Hu
Clinical Biochemistry, 2014, Volume 47, Number 3, Page 196
E. Giannitsis and H. A. Katus
European Heart Journal, 2014, Volume 35, Number 5, Page 268
Andre Pascal Kengne, Katya Masconi, Vivian Nchanchou Mbanya, Alain Lekoubou, Justin Basile Echouffo-Tcheugui, and Tandi E. Matsha
Critical Reviews in Clinical Laboratory Sciences, 2014, Volume 51, Number 1, Page 1
Alejandro Bustamante, Carmen Dominguez, Víctor Rodriguez-Sureda, Angel Vilches, Ana Penalba, Dolors Giralt, Teresa García-Berrocoso, Victor Llombart, Alan Flores, Marta Rubiera, Carlos Molina, José Alvarez-Sabín, and Joan Montaner
International Journal of Stroke, 2014, Volume 9, Number 7, Page 910
Richa Sharma, Stephanie Macy, Kara Richardson, Yuliya Lokhnygina, and Daniel T. Laskowitz
Journal of Stroke and Cerebrovascular Diseases, 2014, Volume 23, Number 5, Page 910
Steven Grover, Jean Grégoire, and Todd J. Anderson
Canadian Journal of Cardiology, 2014, Volume 30, Number 2, Page 248.e5
Sophie Paczesny, Nisha Raiker, Sam Brooks, and Christy Mumaw
International Journal of Hematology, 2013, Volume 98, Number 3, Page 275
J. M. T. Hendriksen, G. J. Geersing, K. G. M. Moons, and J. A. H. de Groot
Journal of Thrombosis and Haemostasis, 2013, Volume 11, Page 129
Stefania Mondello, Kara Schmid, Rachel P. Berger, Firas Kobeissy, Domenico Italiano, Andreas Jeromin, Ronal L. Hayes, Frank C. Tortella, and Andras Buki
Medicinal Research Reviews, 2014, Volume 34, Number 3, Page 503
Michael G. Shlipak and Erica C. Day
Nature Reviews Nephrology, 2013, Volume 9, Number 8, Page 478
Arkendra De, Kristen Meier, Rong Tang, Meijuan Li, Thomas Gwise, Shanti Gomatam, and Gene Pennello
Journal of Cardiovascular Translational Research, 2013, Volume 6, Number 4, Page 449
Siri Lillegraven, Nina Paynter, Femke H. M. Prince, Nancy A. Shadick, Espen A. Haavardsholm, Michelle L. Frits, Christine K. Iannaccone, Tore K. Kvien, Michael E. Weinblatt, and Daniel H. Solomon
Arthritis Care & Research, 2013, Volume 65, Number 4, Page 526
David Salat, Anna Penalba, Teresa García-Berrocoso, Mireia Campos-Martorell, Alan Flores, Jorge Pagola, Alejandro Bustamante, Manuel Quintana, Dolors Giralt, Carlos Molina, José Alvarez-Sabín, and Joan Montaner
Cerebrovascular Diseases, 2013, Volume 35, Number 3, Page 220
S. Apostolakis, M. Zubaid, W.A. Rashed, A.A. Alsheikh-Ali, W. Almahmeed, A. Shehab, K. Sulaiman, A. AlQudaimi, N. Asaad, H. Amin, D.A. Lane, and G.Y.H. Lip
International Journal of Cardiology, 2013, Volume 168, Number 2, Page 1644
Carlos Iribarren and Sabee Molloi
Current Cardiovascular Risk Reports, 2013, Volume 7, Number 2, Page 126
H. E. Wang, G. Jain, R. J. Glassock, and D. G. Warnock
Nephrology Dialysis Transplantation, 2013, Volume 28, Number 6, Page 1447
J. R. Dahlen, M. J. Price, H. Parise, and P. A. Gurbel
Thrombosis and Haemostasis, 2012, Volume 109, Number 5, Page 808
Rakesh K. Mishra, Yongmei Li, Ana C. Ricardo, Wei Yang, Martin Keane, Magdalena Cuevas, Robert Christenson, Christopher DeFilippi, Jing Chen, Jiang He, Radhakrishna R. Kallem, Dominic S. Raj, Jeffrey R. Schelling, Jackson Wright, Alan S. Go, and Michael G. Shlipak
The American Journal of Cardiology, 2013, Volume 111, Number 3, Page 432
Julio A. Chirinos, Jan G. Kips, David R. Jacobs, Lyndia Brumback, Daniel A. Duprez, Richard Kronmal, David A. Bluemke, Raymond R. Townsend, Sebastian Vermeersch, and Patrick Segers
Journal of the American College of Cardiology, 2012, Volume 60, Number 21, Page 2170
Davide Capodanno, Carmelita Marcantoni, Margherita Ministeri, Fabio Dipasqua, Luca Zanoli, Stefania Rastelli, Sarah Mangiafico, Maria Sanfilippo, Gaetano Romano, and Corrado Tamburino
International Journal of Cardiology, 2013, Volume 168, Number 1, Page 396
Caroline Courtais, Nils Kuster, Anne-Marie Dupuy, Margit Folschveiller, Riad Jreige, Anne-Sophie Bargnoux, Julie Guiot, Sophie Lefebvre, Jean-Paul Cristol, and Mustapha Sebbane
The American Journal of Emergency Medicine, 2013, Volume 31, Number 1, Page 215
Thomas G. Liman, Vera Zietemann, Silke Wiedmann, Gerhard J. Jungehuelsing, Matthias Endres, Frank A. Wollenweber, Ian Wellwood, Martin Dichgans, and Peter U. Heuschmann
International Journal of Stroke, 2013, Volume 8, Number 6, Page 484
Gregory E. Rice, Sebastian E. Illanes, and Murray D. Mitchell
International Journal of Endocrinology, 2012, Volume 2012, Page 1
Thor Ueland, Asbjørn Svardal, Erik Øie, Erik T. Askevold, Ståle H. Nymoen, Bodil Bjørndal, Christen P. Dahl, Lars Gullestad, Rolf K. Berge, and Pål Aukrust
International Journal of Cardiology, 2013, Volume 167, Number 5, Page 1892
Niku Oksala, Ilkka Seppälä, Jussi Hernesniemi, Leo-Pekka Lyytikäinen, Mika Kähönen, Kari-Matti Mäkelä, Antti Reunanen, Antti Jula, Mika Ala-Korpela, and Terho Lehtimäki
Annals of Medicine, 2013, Volume 45, Number 2, Page 141
Liisa Byberg, Rolf Gedeborg, Thomas Cars, Johan Sundström, Lars Berglund, Lena Kilander, Håkan Melhus, and Karl Michaëlsson
Journal of Bone and Mineral Research, 2012, Volume 27, Number 4, Page 797
A. Ganna, M. Reilly, U. de Faire, N. Pedersen, P. Magnusson, and E. Ingelsson
American Journal of Epidemiology, 2012, Volume 175, Number 7, Page 715
Andreas P. Kalogeropoulos, Vasiliki V. Georgiopoulou, Christopher R. deFilippi, John S. Gottdiener, and Javed Butler
JACC: Cardiovascular Imaging, 2012, Volume 5, Number 2, Page 131
K. T. HayGlass
Clinical & Experimental Allergy, 2011, Volume 41, Number 12, Page 1656
G. Lippi, E. J Favaloro, and M. Plebani
International Journal of Clinical Practice, 2011, Volume 65, Number 12, Page 1221
Daniela Giannessi
Pharmacological Research, 2011, Volume 64, Number 1, Page 11
Maria Carmo P. Nunes, Rodrigo Citton Padilha Reis, Enrico Antônio Colosimo, Antônio Luiz P. Ribeiro, and Manoel O.C. Rocha
International Journal of Cardiology, 2011, Volume 150, Number 2, Page 224
Stanley S. Levinson
Clinical Chemistry and Laboratory Medicine, 2011, Volume 49, Number 7
Adriana Unić, Dunja Rogić, and Gordana Rajsman
Journal of Medical Biochemistry, 2011, Volume 30, Number 3
John P. A. Ioannidis
PROTEOMICS - Clinical Applications, 2011, Volume 5, Number 5-6, Page 241
Karl J. Lackner
Clinical Chemistry and Laboratory Medicine, 2010, Volume 48, Number 12

Comments (0)

Please log in or register to comment.