Skip to content
Licensed Unlicensed Requires Authentication Published by De Gruyter Oldenbourg August 31, 2022

Uncertainty visualization: Fundamentals and recent developments

David Hägele, Christoph Schulz, Cedric Beschle, Hannah Booth, Miriam Butt, Andrea Barth, Oliver Deussen and Daniel Weiskopf


This paper provides a brief overview of uncertainty visualization along with some fundamental considerations on uncertainty propagation and modeling. Starting from the visualization pipeline, we discuss how the different stages along this pipeline can be affected by uncertainty and how they can deal with this and propagate uncertainty information to subsequent processing steps. We illustrate recent advances in the field with a number of examples from a wide range of applications: uncertainty visualization of hierarchical data, multivariate time series, stochastic partial differential equations, and data from linguistic annotation.

Funding source: Deutsche Forschungsgemeinschaft

Award Identifier / Grant number: 251654672—TRR 161

Funding source: Fonds Wetenschappelijk Onderzoek

Award Identifier / Grant number: 12ZL522N

Funding statement: This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation)—Project ID 251654672—TRR 161 (Projects A01, B07, and D02). Hannah Booth is funded by a postdoctoral fellowship from the Research Foundation – Flanders (FWO), Grant no. 12ZL522N.


1. B. Bach, C. Shi, N. Heulot, T. Madhyastha, T. Grabowski, and P. Dragicevic. Time Curves: Folding time to visualize patterns of temporal evolution in data. IEEE Transactions on Visualization and Computer Graphics, 22(1):559–568, 2016.10.1109/TVCG.2015.2467851Search in Google Scholar PubMed

2. F. Barteld, S. Ihden, I. Schröder, and H. Zinsmeister. Annotating descriptively incomplete language phenomena. In Proceedings of LAW VIII - The 8th Linguistic Annotation Workshop, pages 99–104, Dublin, Ireland, August 2014. Association for Computational Linguistics and Dublin City University.10.3115/v1/W14-4915Search in Google Scholar

3. A. Barth and A. Stein. A study of elliptic partial differential equations with jump diffusion coefficients. SIAM/ASA Journal on Uncertainty Quantification, 6(4):1707–1743, 2018.10.1137/17M1148888Search in Google Scholar

4. C. Beck, H. Booth, M. El-Assady, and M. Butt. Representation problems in linguistic annotations: Ambiguity, variation, uncertainty, error and bias. In Proceedings of the 14th Linguistic Annotation Workshop, pages 60–73, Barcelona, Spain, December 2020. Association for Computational Linguistics.Search in Google Scholar

5. S. Belia, F. Fidler, J. Williams, and G. Cumming. Researchers misunderstand confidence intervals and standard error bars. Psychological Methods, 10(4):389–396, 2005.10.1037/1082-989X.10.4.389Search in Google Scholar PubMed

6. E. M. Bender and B. Friedman. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604, 2018.10.1162/tacl_a_00041Search in Google Scholar

7. G.-P. Bonneau, H.-C. Hege, C. R. Johnson, M. M. Oliveira, K. Potter, P. Rheingans, and T. Schultz. Overview and state-of-the-art of uncertainty visualization. In C. D. Hansen, M. Chen, C. R. Johnson, A. E. Kaufman, and H. Hagen, editors, Scientific Visualization, pages 3–27. Springer, 2014.10.1007/978-1-4471-6497-5_1Search in Google Scholar

8. H. Booth, A. Breitbarth, A. Ecay, and M. Farasyn. A Penn-style Treebank of Middle Low German. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 766–775, Marseille, France, May 2020. European Language Resources Association.Search in Google Scholar

9. N. Brich, C. Schulz, J. Peter, W. Klingert, M. Schenk, D. Weiskopf, and M. Krone. Visual analytics of multivariate intensive care time series data. Computer Graphics Forum, 2022. in Google Scholar

10. K. Brodlie, R. S. Allendes Osorio, and A. Lopes. A review of uncertainty in data visualization. In J. Dill, R. A. Earnshaw, D. J. Kasik, J. A. Vince, and P. Chung Wong, editors, Expanding the Frontiers of Visual Analytics and Visualization, pages 81–109. Springer, 2012.10.1007/978-1-4471-2804-5_6Search in Google Scholar

11. A. Caliskan, J. J. Bryson, and A. Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186, April 2017.10.1126/science.aal4230Search in Google Scholar PubMed

12. T. Cassidy, B. McDowell, N. Chambers, and S. Bethard. An annotation framework for dense event ordering. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 501–506, Baltimore, Maryland, June 2014. Association for Computational Linguistics.10.3115/v1/P14-2082Search in Google Scholar

13. T. Chen, Z. Jiang, A. Poliak, K. Sakaguchi, and B. Van Durme. Uncertain natural language inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8772–8779, Online, July 2020. Association for Computational Linguistics.10.18653/v1/2020.acl-main.774Search in Google Scholar

14. E. H. Chi. A taxonomy of visualization techniques using the data state reference model. In Proceedings of the IEEE Symposium on Information Visualization, pages 69–75. IEEE Computer Society, 2000.Search in Google Scholar

15. H. de Smet. Analysing reanalysis. Lingua, 119(11):1728–1755, 2009.10.1016/j.lingua.2009.03.001Search in Google Scholar

16. H. De Smet and M.-A. Markey. The spark or the fuel? on the role of ambiguity in language change. Journal of Historical Syntax, 5(36):1–24, 2021.Search in Google Scholar

17. Carolin Fiedler, Willy Scheibel, Daniel Limberger, Matthias Trapp, and Jürgen Döllner. Survey on user studies on the effectiveness of treemaps. In Proceedings of the 13th International Symposium on Visual Information Communication and Interaction, VINCI’20. Association for Computing Machinery, 2020. Article 2.10.1145/3430036.3430054Search in Google Scholar

18. T. Gebru. Race and gender. In M. D. Dubber, F. Pasquale, and S. Das, editors, The Oxford Handbook of Ethics of AI. Oxford University Press, Oxford, 2020.10.1093/oxfordhb/9780190067397.013.16Search in Google Scholar

19. J. Görtler, C. Schulz, D. Weiskopf, and O Deussen. Bubble treemaps for uncertainty visualization. IEEE Transactions on Visualization and Computer Graphics, 24(1):719–728, 2018.10.1109/TVCG.2017.2743959Search in Google Scholar PubMed

20. T. Grätsch and K.-J. Bathe. A posteriori error estimation techniques in practical finite element analysis. Computers & Structures, 83(4-5):235–265, 2005.10.1016/j.compstruc.2004.08.011Search in Google Scholar

21. M. Greis, H. Schuff, M. Kleiner, N. Henze, and A. Schmidt. Input controls for entering uncertain data: Probability distribution sliders. Proceedings of the ACM on Human-Computer Interaction, 1(EICS), June 2017. Article 3.10.1145/3095805Search in Google Scholar

22. R. B. Haber and D. A. McNabb. Visualization idioms: A conceptual model for visualization systems. In G. M. Nielson, B. D. Shriver, and L. J. Rosenblum, editors, Visualization in Scientific Computing, pages 74–93. IEEE Computer Society Press, 1990.Search in Google Scholar

23. J. E. Härd. Syntax des Mittelniederdeutschen. In Werner Besch, Anne Betten, Oskar Reichmann, and Stefan Sonderegger, editors, Sprachgeschichte: Ein Handbuch zur Geschichte der deutschen Sprache und ihrer Erforschung, volume 2, pages 1456–1463. de Gruyter, 2000.Search in Google Scholar

24. N. Heulot, M. Aupetit, and J-D. Fekete. ProxiLens: Interactive Exploration of High-Dimensional Data using Projections. In M. Aupetit and L. van der Maaten, editors, EuroVis Workshop on Visual Analytics using Multidimensional Projections. The Eurographics Association, 2013.Search in Google Scholar

25. M. Höferlin, B. Höferlin, D. Weiskopf, and G. Heidemann. Uncertainty-aware video visual analytics of tracked moving objects. Journal of Spatial Information Science, 2(1):87–117, 2011.10.5311/JOSIS.2011.2.1Search in Google Scholar

26. J. Hullman, X. Qiao, M. Correll, A. Kale, and M. Kay. In pursuit of error: A survey of uncertainty visualization evaluation. IEEE Transactions on Visualization and Computer Graphics, 25(1):903–913, 2019.10.1109/TVCG.2018.2864889Search in Google Scholar PubMed

27. J. Hullman, P. Resnick, and E. Adar. Hypothetical outcome plots outperform error bars and violin plots for inferences about reliability of variable ordering. PLOS ONE, 10(11):1–25, November 2015.10.1371/journal.pone.0142444Search in Google Scholar PubMed PubMed Central

28. D. Jurafsky and J. H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Speech Recognition, and Computational Linguistics. Prentice-Hall, 2nd edition, 2009.Search in Google Scholar

29. D. Jurgens. Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 556–562, Atlanta, Georgia, June 2013. Association for Computational Linguistics.Search in Google Scholar

30. Svetlana Kiritchenko and Saif Mohammad. Examining gender and race bias in two hundred sentiment analysis systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 43–53, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.10.18653/v1/S18-2005Search in Google Scholar

31. S. Liu, D. Maljovec, B. Wang, P.-T. Bremer, and V. Pascucci. Visualizing high-dimensional data: Advances in the past decade. IEEE Transactions on Visualization and Computer Graphics, 23(3):1249–1268, 2017.10.1109/TVCG.2016.2640960Search in Google Scholar PubMed

32. S. López-Pintado and J. Romo. On the concept of depth for functional data. Journal of the American Statistical Association, 104(486):718–734, 2009.10.1198/jasa.2009.0108Search in Google Scholar

33. A. M. MacEachren, R. E. Roth, J. O’Brien, B. Li, D. Swingley, and M. Gahegan. Visual semiotics amp; uncertainty visualization: An empirical study. IEEE Transactions on Visualization and Computer Graphics, 18(12):2496–2505, 2012.10.1109/TVCG.2012.279Search in Google Scholar PubMed

34. M.-L. Merten and N. Seemann. Analyzing constructional change: Linguistic annotation and sources of uncertainty. In Proceedings of the Sixth International Conference on Technological Ecosystems for Enhancing Multiculturality, TEEM’18, pages 819–825. Association for Computing Machinery, 2018.10.1145/3284179.3284320Search in Google Scholar

35. T. Munz, D. Väth, P. Kuznecov, N. T. Vu, and D. Weiskopf. Visual-interactive neural machine translation. In Proceedings of Graphics Interface 2021, 2021. Article 30.Search in Google Scholar

36. A. Pang, C. M. Wittenbrink, and S. K. Lodha. Approaches to uncertainty visualization. The Visual Computer, 13(8):370–390, 1997.10.1007/s003710050111Search in Google Scholar

37. B. Plank, D. Hovy, and A. Søgaard. Learning part-of-speech taggers with inter-annotator agreement loss. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 742–751, Gothenburg, Sweden, April 2014. Association for Computational Linguistics.10.3115/v1/E14-1078Search in Google Scholar

38. D. Sacha, H. Senaratne, B. Chul Kwon, G. Ellis, and D. A. Keim. The role of uncertainty, awareness, and trust in visual analytics. IEEE Transactions on Visualization and Computer Graphics, 22(1):240–249, 2016.10.1109/TVCG.2015.2467591Search in Google Scholar PubMed

39. P. Schramowski, C. Turan, S. Jentzsch, C. Rothkopf, and K. Kersting. The moral choice machine. Frontiers in Artificial Intelligence, 3:36, 2020.10.3389/frai.2020.00036Search in Google Scholar PubMed PubMed Central

40. H. J. Schulz, S. Hadlak, and H. Schumann. The design space of implicit hierarchy visualization: A survey. IEEE Transactions on Visualization and Computer Graphics, 17(4):393–411, 2011.10.1109/TVCG.2010.79Search in Google Scholar PubMed

41. M. Skeels, B. Lee, G. Smith, and G. Robertson. Revealing uncertainty for information visualization. Information Visualization, 9(1):70–81, 2010.10.1145/1385569.1385637Search in Google Scholar

42. M. Sondag, W. Meulemans, C. Schulz, K. Verbeek, D. Weiskopf, and B. Speckmann. Uncertainty treemaps. In IEEE Pacific Visualization Symposium, PacificVis 2020, pages 111–120, 2020.10.1109/PacificVis48177.2020.7614Search in Google Scholar

43. Yong Wang, Qiaomu Shen, Daniel Archambault, Zhiguang Zhou, Min Zhu, Sixiao Yang, and Huamin Qu. AmbiguityVis: Visualization of ambiguity in graph layouts. IEEE Transactions on Visualization and Computer Graphics, 22(1):359–368, 2016.10.1109/TVCG.2015.2467691Search in Google Scholar PubMed

44. D. Weiskopf. Uncertainty visualization: Concepts, methods, and applications in biological data visualization. Frontiers in Bioinformatics, 2, 2022.10.3389/fbinf.2022.793819Search in Google Scholar

45. J Welch, P Ford, R Teplick, and R Rubsamen. The Massachusetts General Hospital-Marquette Foundation hemodynamic and electrocardiographic database–Comprehensive collection of critical care waveforms. Clinical Monitoring, 7(1):96–97, 1991.Search in Google Scholar

46. Y. Wu, Guo-Xun Yuan, and Kwan-Liu Ma. Visualizing flow of uncertainty through analytical processes. IEEE Transactions on Visualization and Computer Graphics, 18(12):2526–2535, 2012.10.1109/TVCG.2012.285Search in Google Scholar PubMed

Received: 2022-05-17
Revised: 2022-08-12
Accepted: 2022-08-13
Published Online: 2022-08-31
Published in Print: 2022-08-26

© 2022 Walter de Gruyter GmbH, Berlin/Boston

Scroll Up Arrow