Skip to content
Accessible Unlicensed Requires Authentication Published by Oldenbourg Wissenschaftsverlag March 27, 2018

Generating Structured Data by Nontechnical Experts in Research Settings

Andre Breitenfeld, Florian Berger, Ming-Tung Hong, Maximilian Mackeprang and Claudia Müller-Birn
From the journal i-com


Semantic technologies provide meaning to information resources in the form of machine-accessible structured data. Research over the past two decades has commonly focused on tools and interfaces for technical experts, leading to various usability problems regarding users unfamiliar with the underlying technologies – so-called nontechnical experts. Existing approaches to semantic technologies consider mostly consumers of structured data and leave out the creation perspective. In this work, we focus on the usability of creating structured data from textual resources, especially the creation of relations between entities. The research was conducted in collaboration with scholars from the humanities. We review existing research on the usability of semantic technologies and the state of the art of annotation tools to identify shortcomings. Subsequently we use the knowledge gained to propose a new interaction design for the creation of relations between entities to create structured data in the subject-predicate-object form. We implemented our interaction design and conducted a user study which showed that the proposal performed well, making it a contribution to enhance the overall usability in this field. However, this research provides an example of how technically sophisticated technology needs to be “translated” to make it usable for nontechnical experts. We need to extend this perspective in the future by providing more insight into the internal functioning of semantic technologies.

Funding source: Bundesministerium für Bildung und Forschung

Award Identifier / Grant number: 03IO1617

Funding statement: We gratefully acknowledge the financial support of the Elsa-Neumann-Stipend of the state of Berlin, of the German Research Foundation (Deutsche Forschungsgemeinschaft) with the Cluster of Excellence “Image Knowledge Gestaltung: An Interdisciplinary Laboratory” (EXC 1027), and the Federal Ministry of Education and Research of Germany (Bundesministerium für Bildung und Forschung) in the framework of “Ideas to Market” (project number 03IO1617).


We would like to express our very great appreciation to all persons that contributed to the neonion software: Lukas Benedix, Tina Klüwer and Alexa Schlegel. The research presented here was only possible because of their work.


[1] W3C DATA ACTIVITY building the web of data. in Google Scholar

[2] T. Berners-Lee. Linked Data. in Google Scholar

[3] C. L. Borgman. Big Data, Little Data, No Data. MIT Press, 2015.Search in Google Scholar

[4] M. Burghardt. Usability recommendations for annotation tools. In Proceedings of the Sixth Linguistic Annotation Workshop, pages 104–112, Jeju, Republic of Korea, July 2012. Association for Computational Linguistics.Search in Google Scholar

[5] M. Burghardt. Engineering annotation usability – toward usability patterns for linguistic annotation tools, September 2014. URL Pattern wiki: in Google Scholar

[6] J.-P. Cahier and M. Zacklad. Socio-semantic web applications: towards a methodology based on the theory of the communities of action. resource, 8 (400), 2004.Search in Google Scholar

[7] J. J. Carroll, I. Dickinson, C. Dollin, D. Reynolds, A. Seaborne, and K. Wilkinson. Jena: Implementing the semantic web recommendations. In Proc. of the WWW, pages 74–83. ACM, 2004.Search in Google Scholar

[8] P. Ciccarese, M. Ocana, and T. Clark. Open semantic annotation of scientific publications using DOMEO. J. Biomedical Semantics, 3 (S-1): S1, 2012a. URL in Google Scholar

[9] P. Ciccarese, M. Ocana, and T. Clark. Open semantic annotation of scientific publications using DOMEO. Journal of Biomedical Semantics, 3 (Suppl 1): S1, 2012b. ISSN 2041-1480. doi:10.1186/2041-1480-3-S1-S1. URL in Google Scholar

[10] R. Crawford. The Techniques of Creative Thinking: How to Use Your Ideas to Achieve Success. Hawthorn Books, 1954. URL in Google Scholar

[11] A. S. Dadzie, M. Rowe, and D. Petrelli. Hide the stack: Toward usable linked data. In Lecture Notes in Computer Science, pages 93–107, 2011.Search in Google Scholar

[12] J. Damerow, B. E. Peirson, and M. D. Laubichler. Don’t panic! a research system for network-based digital history of science. In Future of Historical Network Research Conference, 2013.Search in Google Scholar

[13] P. Di Maio. Toward global user models for semantic technologies: Emergent perspectives. In Proc. of ASWC, volume 8, pages 141–152, 2008.Search in Google Scholar

[14] O. Gilson, N. Silva, P. W. Grant, and M. Chen. From web data to visualization via ontology mapping. In Computer Graphics Forum, volume 27, pages 959–966, 2008.Search in Google Scholar

[15] E. Goodman, M. Kuniavsky, and A. Moed. Observing the user experience: A practitioner’s guide to user research. IEEE TPC, 56 (3): 260–261, 2013.Search in Google Scholar

[16] M. Grassi, C. Morbidoni, M. Nucci, S. Fonda, and F. Piazza. Pundit: Augmenting web contents with semantics. Literary and Linguistic Computing, 28 (4): 640–659, 2013.10.1093/llc/fqt060Search in Google Scholar

[17] O. Hartig. Provenance information in the web of data, 2009.Search in Google Scholar

[18] S. R. Herring, B. R. Jones, and B. P. Bailey. Idea generation techniques among creative professionals. In System Sciences, 2009. HICSS ’09. 42nd Hawaii International Conference on, pages 1–10, Jan 2009. doi:10.1109/HICSS.2009.241.Search in Google Scholar

[19] T. Hey, S. Tansley, K. M. Tolle, et al.The fourth paradigm: data-intensive scientific discovery, volume 1. Microsoft research Redmond, WA, 2009.Search in Google Scholar

[20] A. Hinze, R. Heese, M. Luczak-Rösch, and A. Paschke. Semantic enrichment by non-experts: usability of manual annotation tools. In ISWC, pages 165–181. Springer, 2012.Search in Google Scholar

[21] A. Hoffmeister, F. Berger, M. Pogorzhelskiy, G. Zhang, C. Zwick, and C. Müller-Birn. Toward cyber-physical research practice based on mixed reality. In M. Burghardt, R. Wimmer, C. Wolff, and C. Womser-Hacker, editors, Mensch und Computer 2017 – Workshopband, Regensburg, 2017. Gesellschaft für Informatik e.V.Search in Google Scholar

[22] J. Huang, R. White, and G. Buscher. User see, user point: gaze and cursor alignment in web search. In Proc. of the SIGCHI Conference, pages 1341–1350. ACM, 2012.Search in Google Scholar

[23] N. Ide and L. Romary. International standard for a linguistic annotation framework. Nat. Lang. Eng., 10 (3–4): 211–225, Sept. 2004. ISSN 1351-3249. URL in Google Scholar

[24] A. Jameson. Usability and the Semantic Web. In ESWC, pages 3. Springer, 2006.Search in Google Scholar

[25] N. Karam, C. Müller-Birn, M. Gleisberg, D. Fichtmüller, R. Tolksdorf, and A. Güntsch. A Terminology Service Supporting Semantic Annotation, Integration, Discovery and Analysis of Interdisciplinary Research Data. Datenbank-Spektrum, 16 (3): 195–205, Nov. 2016. ISSN 1610-1995. URL in Google Scholar

[26] D. R. Karger. The semantic web and end users: What’s wrong and how to fix it. IEEE Internet Computing, 18 (6): 64–70, 2014.10.1109/MIC.2014.124Search in Google Scholar

[27] A. Khalili, S. Auer, and D. Hladky. The rdfa content editor – from wysiwyg to wysiwym. In Proceedings of COMPSAC 2012 – Trustworthy Software Systems for the Digital Society, July 16–20, 2012, Izmir, Turkey, 2012. URL in Google Scholar

[28] A. Kiryakov, B. Popov, I. Terziev, D. Manov, and D. Ognyanoff. Semantic annotation, indexing, and retrieval. Web Semantics: Science, Services and Agents on the World Wide Web, 2 (1): 49–79, 2004. ISSN 1570-8268. URL in Google Scholar

[29] S. Lohmann, S. Negru, F. Haag, and T. Ertl. Visualizing ontologies with vowl. Semantic Web, 7 (4): 399–419, 2016.10.3233/SW-150200Search in Google Scholar

[30] V. Lopez, M. Fernández, E. Motta, and N. Stieler. PowerAqua: Supporting users in querying and exploring the Semantic Web. Semantic Web, 3 (3): 249–265, 2012.Search in Google Scholar

[31] M. Marcińczuk, J. Kocoń, and B. Broda. Inforex – a web-based tool for text corpus management and semantic annotation. In N. C. C. Chair, K. Choukri, T. Declerck, M. U. Doğan, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, and S. Piperidis, editors, Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey, may 2012. European Language Resources Association (ELRA). ISBN 978-2-9517408-7-7.Search in Google Scholar

[32] M. J. McGuffin and R. Balakrishnan. Fitts’ law and expanding targets: Experimental studies and designs for user interfaces. TOCHI, 12 (4): 388–422, 2005.10.1145/1121112.1121115Search in Google Scholar

[33] E. T. Meyer and R. Schroeder. The world wide web of research and access to knowledge. Knowledge Management Research & Practice, 7 (3): 218–233, Sept. 2009. ISSN 1477-8246. URL in Google Scholar

[34] C. Müller-Birn, T. Klüwer, A. Breitenfeld, A. Schlegel, and L. Benedix. Neonion: Combining human and machine intelligence. In Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work & Social Computing, CSCW’15 Companion, pages 223–226, New York, NY, USA, 2015. ACM. ISBN 978-1-4503-2946-0. URL in Google Scholar

[35] J. L. Navarro-Galindo and J. Samos. The FLERSA tool: adding semantics to a web content management system. IJWIS, 8 (1): 73–126, 2012. URL in Google Scholar

[36] M. Neves and U. Leser. A survey on annotation tools for the biomedical literature. Briefings in Bioinformatics, 2012.Search in Google Scholar

[37] J. Nielsen. 10 usability heuristics for user interface design. Fremont: Nielsen Norman Group. [Consult. 20 maio 2014]. Disponível na Internet, 1995.Search in Google Scholar

[38] K. Ohara, T. Berners-Lee, W. Hall, and N. Shadbolt. Use of the semantic web in e-research. World Wide Research: Reshaping the Sciences and Humanities, page 130, 2010.Search in Google Scholar

[39] D. Oldman, M. Doerr, and S. Gradmann. Zen and the art of linked data. In R. S. a. U. Susan Schreibman, editor, A New Companion to Digital Humanities, pages 251–273. John Wiley & Sons, Ltd, 2015. doi:10.1002/9781118680605.ch18.Search in Google Scholar

[40] J. Pratt, P. V. Radulescu, R. M. Guo, and R. A. Abrams. It’s alive! animate motion captures visual attention. Psychological Science, 21 (11): 1724–1730, 2010.10.1177/0956797610387440Search in Google Scholar

[41] C. Preist. A conceptual architecture for semantic web services. In ISWC, pages 395–409. Springer, 2004.Search in Google Scholar

[42] G. Rizzo and R. Troncy. Nerd: Evaluating named entity recognition tools in the web of data. 2011.Search in Google Scholar

[43] A. Sears. Heuristic walkthroughs: Finding the problems without the noise. Int. J. Hum. Comput. Interaction, 9 (3): 213–234, 1997.10.1207/s15327590ijhc0903_2Search in Google Scholar

[44] L. Serafini and A. Tamilin. Drago: Distributed reasoning architecture for the semantic web. In ESWC, pages 361–376. Springer, 2005.Search in Google Scholar

[45] H.  N. Talantikite, D. Aissani, and N. Boudjlida. Semantic annotations for web services discovery and composition. Comput. Stand. Interfaces, 31 (6): 1108–1117, Nov. 2009. ISSN 0920-5489.Search in Google Scholar

[46] V. Uren, P. Cimiano, J. Iria, S. Handschuh, M. Vargas-Vera, E. Motta, and F. Ciravegna. Semantic annotation for knowledge management: Requirements and a survey of the state of the art. Web Semant., 4 (1): 14–28, Jan. 2006. ISSN 1570-8268.10.1016/j.websem.2005.10.002Search in Google Scholar

[47] K. van Deemter and R. Kibble. What is coreference, and what should coreference annotation be? In Proceedings of the Workshop on Coreference and Its Applications, CorefApp ’99, pages 90–96, Stroudsburg, PA, USA, 1999. Association for Computational Linguistics. URL in Google Scholar

[48] D. Vrandečić and M. Krötzsch. Wikidata: A Free Collaborative Knowledgebase. Commun. ACM, 2014.Search in Google Scholar

[49] A. Widlöcher and Y. Mathet. The glozz platform: A corpus annotation and mining tool. In Proc. of the DocEng, pages 171–180. ACM, 2012.Search in Google Scholar

[50] M. Wilkinson and et al.The fair guiding principles for scientific data management and stewardship. Nature Scientific Data, (160018), 2016. URL in Google Scholar

Published Online: 2018-03-27
Published in Print: 2018-04-25

© 2018 Walter de Gruyter GmbH, Berlin/Boston