1 Introduction

In the era of increasing data sets (often summarized by the term “big data”) and social media, seeking and publishing digital (learning) resources has fundamentally changed. Using search engines or digital databases with millions of books and scientific publications means an almost unrestricted access to such resource repositories. Beside typical ways of publishing information, collaboration projects such as Wikipedia, blogs and social networks are popular ways for producing and publishing information which steadily increase the set of available digital resources.

Resource spaces are interrelated sets of resources. If someone wants to explore a specific topic, a resource space could be composed of articles from Wikipeda or blogs, textbooks, lecture slides etc. In order to efficiently work with such a resource space, a possible way is to analyze resources and to (partially) structure them considering semantic relations or logical dependencies.

In this article, we investigate the research question how users can be supported to orientate in and navigate through resource spaces. Based on (meta) information extracted from resources, we implemented two different visualizations that structure and visualize sets of resources considering relations among resources. In Section 2, we review the state of the art in visualizing interrelated information. We then sketch our approach to automatically analyze sets of resources in Section 3. We conducted a lab study in order to evaluate the approach. The results of the evaluation are summarized and discussed in Section 4. In Section 5, we exemplarily discuss how this approach can be used to help learner orientate in sets of learning resources in a web-based learning system for programming. Finally, in Section 6, we present our conclusions and give an outlook on future work.

2 State of the Art

In order to provide an efficient access to a resource space, a common way is to process and visualize its structures. In a meta-analysis, Nesbit and Adesope could show that using manually structured visualizations of learning resources leads to statistically significant improvements of learning achievements of learners [1]. Also, they found that concept maps, which aim at supporting learners comprehend contents and relations by manually creating and visualizing them, are associated with increased knowledge retention. Concept maps have been subject of study since decades [2, 3] and have been used in several approaches for supporting users to organize resource sets. Canas and Novak, for instance, developed the “CmapTool” which enables users to individually and collaboratively create concept maps and, thus, to manually structure resources [4]. CmapTool also provides mechanisms to suggest appropriate learning resources using web-based search queries. Webster is similar to CmapTool in facilitating individual organization of resources for reference and learning purposes by using concept maps [5]. A semi-automatic structuring has been implemented in the tool “Nestor” which visualizes traces of web-based search queries [6]. Users can also manually annotate these traces in order to emphasize relevant web resources. There are other approaches that help users process and visualize weighted search results [7]. Those static visualizations, however, do not support an efficient navigation and orientation through the visualized resource spaces. A resource typically has several dimensions of information (e.g. textual content and meta information such as author’s name, quality, etc.) so that the density of information increases with the amount of resources and users could be faced up with information overloading. Some approaches investigate how structures can be derived using user preferences or recommendations of other users [8]. In most cases, this information is not available from the beginning but needs to be gathered during usage. This limits the usage of automation since most of the approaches require complete data sets even from the beginning.

The severity of analyzing and structuring resources heavily increases with the increasing amount of resources. The efficient identification of relevant information and the dependencies between those can not be realized manually anymore at a certain point of analysis. In the field of text mining, there are several concepts that aim at automatically analyzing huge amounts of documents and structuring those documents into homogeneous groups [9]. However, these approaches mainly focus on identifying and revealing dependencies between resources instead of enabling users to identify and access relevant resources.

In this article, we, therefore, propose an approach that aims at visualizing spaces based on dependencies among resources in order to provide orientation in extensive resource spaces. Furthermore, our approach aims at enabling users to manually navigate to a space. Concerning this, we first introduce pipeline of techniques which identifies and derives a hierarchical structure of dependencies between resources. Each level of this hierarchy is annotated by terms which completely and compactly summarize the underlying subspace. For visualization, we implemented two different visualization tools which we compared in a lab study concerning their qualities to help users orientate in and navigate through resource spaces.

3 Analysis and Visualization of Resource Spaces

In our approach, the starting point of a resource space is an unstructured set of digital resources consisting of different formats (e.g. text documents, HTML source codes, PDF documents, audio or video files) and forms (e.g. text books, lecture slides, etc.). Based on the visualization pipeline of Card and colleagues [10], we propose a chain of techniques which automatizes the analysis and structuring of such unstructured sets of resources. The outcome of this approach is then used to provide a hierarchical visualization considering semantic relations among resources.

3.1 Analysis of Dependency Structures

Given a resource space about “software engineering”, this can be structured into subspaces such as “process model” and “software architecture”. The latter one can, again, be structured into “preliminary design” and “detailed design”. Given a set of resources, there is typically a natural hierarchical structure based on semantic relations within this set. Visualized as a root tree, resources could be represented by leaves. Semantically similar resources can be combined to a sub tree which will then be represented by an inner node of the tree. Those semantically similar inner nodes can then, again, be combined to a parent node. This operation can be repeated until the root of the tree is reached, representing the complete resource space. The resulting hierarchy has the property that each level of the tree has a certain level of detail regarding the contents it represents which accumulates from top to bottom. When a user is requested to navigate through such structures, she could reduce the set of resources which may be of interest for her by descending the tree from the root to underlying sub trees. Here, the inner nodes can give orientation which sub trees are related to the user’s objective. In order to provide appropriate orientation, those nodes need a condensed description of the underlying sub trees. This can be realized by extracting keywords from the underlying set of resources. For that reason, a resource needs to be analyzed and appropriate (i.e. meaningful) keywords need to be extracted from its content. We, therefore, derive the requirement that resources need to be compared semantically based on keywords which need to be extracted automatically.

Before we are able to conduct semantic comparisons of resources, the first step in the visualization pipeline should be to find a data model that makes resources from different sources comparable. Based on this model, we are then able to analyze resources and to identify semantic relations. A common way of semantically representing resources is using n-gram models [11, pp. 192–195]. Textual contents can be separated into word groups (also known as terms) consisting of n words which, then, represent the content of a resource. Since not all extracted terms from a resource are suitable to describe its content, terms need to be weighted in order to identify the most relevant of them. TF-IDF [12, pp. 8–9] is an appropriate method to identify the most relevant terms from text corpora by weighting terms that occur often in many resources lower, and weighting terms that occur rarely in resources higher. A subset of the most relevant terms can then be used to describe the content of a resource in a consolidated way.

After computing resource contents into comparable sets of terms, we are able to generate the second part of the visualization pipeline: generating a structure that considers semantic relations among resources. Using an agglomerative clustering algorithm [11, pp. 500–513], resources can be grouped into clusters considering their semantic similarity [13]. On the next level, similar clusters can then be grouped to new clusters until the top of the hierarchy (represented by a single cluster with embedded sub clusters) is reached. From intersection of related terms, we can, finally, derive a description for each cluster.

Figure 1

Resource space visualization using a rooted tree.

Figure 2

Resource space visualization using a treemap.

4 Evaluation

We conducted a lab study in order to evaluate whether or not the approach proposed in Sec. 3 is suitable for efficiently and effectively supporting navigation through and orientation in resource spaces. For study purposes, we created a resource space related to the topic “software engineering” consisting of 137 resources from a set of digital textbooks (68), articles from Wikipedia (33), lecture manuscripts/slides (19), websites/videos (10), audio podcasts (4), and others (3).

The automatic analysis and structuring of the resources is based on the prototypical implementation as described in Sec. 3.1. The resulting structure was, then, visualized (as proposed in Sec. 3.2) using the JavaScript framework “D3.js”.

5 An Example Application in the Domain of Learning Programming

We prototypically implemented (see Sec. 3) and evaluated (see Sec. 4) two different visualizations which use a hierarchy derived from semantic analysis (see Sec. 3.1) of resource contents. Even though the results are not clear as to whether or not the approach proposed in Sec. 3 supports users to navigate through and orientate in resource spaces, the findings give some reason to believe that this approach might lead to better understanding of the underlying knowledge and, also, might reduce information overload.

Since each session of the study was held over a period of 120 minutes whereof 80 minutes were spent to work on a multiple choice test, the study was not qualified (due to its short-term characters) to measure whether or not the approach is beneficial also in realistic longer-term learning scenarios where learners need to identify and use relevant resources and, based on these resources, acquire knowledge. Here, it is crucial to allow learners to focus on knowledge acquisition. Therefore, it is necessary to provide tools which reduce information overloading.

Based on the observed tendency that experimental condition TreeMap outperformed the experimental condition RootedTree, we decided to implement a prototypical visualization based on a treemap data structure in a learning system in order to evaluate the approach in a long-term experiment. In a research project funded by the German Science Foundation (DFG), we developed a web-based learning system for Java programming called JavaFIT^[7] (formerly known as FIT Java Tutor [17]). JavaFIT integrates several pedagogical strategies to impart knowledge to learners: the system (i) allows learners to directly access and display learning resources (e.g. documents or video tutorials), and (ii) gives learners the opportunity to directly apply their knowledge in tests, quizzes, clozes, and programming tasks. For the latter, we developed an integrated development environment (IDE) which enables users to write, compile, and execute Java programs. A crucial aspect in (ii) is to provide feedback to learners while or after solving a problem. In case of tests, quizzes, and clozes we provide an overall score after solving a problem which indicates how well a learner acquired the underlying knowledge. For programming tasks, learners can request feedback to their programs while solving a given problem. This feedback is based on appropriate sample solutions. We, here, use machine learning techniques [18] in order to identify appropriate (sample) solutions automatically from structured solution spaces consisting of both learner and expert solutions [19]. Furthermore, learners can test their programs with predefined test cases which indicate how well learners’ programs solved a given problem, thus supporting learners to self-regulate their learning processes using the feedback the system gives.

As mentioned in (i), the system allows users to directly display learning resources. Overall, we provide a set of about 100 resources (as of Dec. 2016). In a previous version of the system, we provided a visualization based on a static structure of curriculum units (as it is typically implemented in learning management systems (LMS)) which was manually designed by a domain expert. However, this approach has several drawbacks. The underlying structure of the visualization has to be designed manually which means a lot of effort not only to create but also to maintain (in case of new learning resources) this visualization. Furthermore, the underlying structure does not necessarily reflect users’ needs. It might help learners navigate through a set of resources in a default way, but it does not support learning processes where learners want to determine their way through the space individually. And, finally, a static visualization (as we used in a previous version) does not allow learners to adapt the visualization by filtering or searching resources.

Considering the above mentioned drawbacks, we integrated an adaptive and adaptable visualization (as illustrated in Fig. 3) to the learning system which provides a hierarchical top view on resources similar to typical treemaps. Technically, we use the Javascript library 3D.js which provides several layouts for visualizing hierarchies^[8]. In this case, we use a hierarchical pack layout which uses a treemap as underlying data structure. For structuring a set of resources into a hierarchy of resources, we use a similar approach as described in Sec. 3 where resources are semantically compared based on sets of terms. However, we do not extract these terms automatically (as proposed in Sec. 3.1) but we manually assign them to resources. This is caused by the fact that not every single resource has sufficient textual data which could be analyzed (e.g. video tutorials hosted on YouTube do not necessarily have descriptions nor keywords).

Figure 3

Visualization of a space consisting of learning resources. The visualization uses a hierarchical pack layout based on typical treemaps and can be adapted to learners’ needs by filtering and searching resources.

The visualization is based on semantically similar resource clusters. Due to the relatively low number of resources, we limited the depth of sub-clustering to 1 which means that sub clusters do not have sub clusters. However, with increasing sets of resources this could be changed in order to obtain a fine-granular visualization. The visualization can be adapted to a user’s need using filter tools (e.g. displaying video tutorials only) and provides a zoom which a user can use to zoom into relevant (sub-)clusters. We also implemented a hover effect displaying related terms and meta information when moving a mouse pointer on a cluster or a resource. For awareness and orientation purposes, we use different colors to emphasize those clusters whose related resources a learner already used and whose underlying knowledge she probably acquired. The coloring^[9] is based on a learner model which records her progress and performance related to a learning resource.

6 Conclusion & Future Work

The evaluation results confirms the initially stated problem using resource spaces: the control group WebSearch used 64 resources in 80 minutes. This shows that a user who interacts with (unstructured) resource spaces as they are made accessible by search engines, is faced up with extensive sets of resources which need to be analyzed by the user itself. The results also indicate that using a web-based search engine might be efficient in terms of time spent on the system, but this does probably not necessarily lead to a better understanding of the underlying knowledge. This could be caused by information overloading where a user probably find relevant information but does not actually process and acquire them. Tendentially, the experimental conditions RootedTree and TreeMap better dealt with this issue which resulted in improvements from PreT to PostT.

We applied the approach in a learning system called JavaFIT in order to replace static curriculum structures by an adaptive and adaptable visualization which considers learners’ needs of individually organized learning processes. We are currently conducting a long-term experiment using a publicly available version of the learning system. Everyone who is interested in learning Java programming can register and use the system. We started the public instance in Sep. 2016. With this public instance, we are currently not conducting any controlled studies since its usage in the first months was relatively low, and thus we do not have any reliable results yet.

In upcoming studies, we intend to evaluate the helpfulness of the visualization by conducting a survey among registered users. Thus, the application in the domain of learning programming will provide us with a deeper insight of whether the approach is appropriate to efficiently and effectively support navigation and orientation processes in resource spaces. We are also going to address the problem of manually assigned terms by elaborating the proposed methods in such a way that resources with just a few (meta) information can be analyzed, too.