Open Access Published by De Gruyter September 30, 2016

Usability evaluation of a locomotor therapy device considering different strategies

Sonja Langthaler, Alexander Lassnig, Christian Baumgartner and Jörg Schröttner

Abstract

Usability of medical devices is one of the main determining factors in preventing use errors in treatment and strongly correlates to patient safety and quality of treatment. This thesis demonstrates the usability testing and evaluation of a prototype for locomotor therapy of infants. Therefore, based on the normative requirements of the EN 62366, a concept combined of evaluation procedures and assessing methods was created to enable extensive testing and analysis of the different aspects of usability. On the basis of gathered information weak points were identified and appropriate measures were presented to increase the usability and operating safety of the locomotor prototype. The overall outcome showed an usability value of 77.4% and an evaluation score of 6.99, which can be interpreted as “satisfactory”.

1 Introduction

Usability has become more and more relevance in the development of medical products. Deficits through insufficient usability can lower the operating safety as well as the quality of treatment and therefore increase costs in health care. To increase safety an intuitive and easy handling should be provided [1], [2].

To lower use associated risks and ensure safety, the standard EN 62366 provides, in the context of risk management, methods to implement an usability-orientated-process and verify the usability accordingly [3].

Beside device safety an optimized usability can improve the whole treatment process and thereby raise the quality of treatment. Potential negative effects on the compliance of the patient may be avoided which furthermore can enhance the process as well as the therapeutic success. Hence, the motivation of this work was to deal with different usability methods and assess the usability of a locomotor therapy device.

2 Material and methods

2.1 Proceeding

The usability testing was split into two phases: The first phase covers usability tests with volunteers (n of 13). The test was restricted to software operations and offered a first feedback about usability and general weak points in the software design. Based on this information, improvements of usability could be achieved.

The second phase comprises usability tests with the intended users, involving the whole user interface and application process. This allows the evaluation of usability and definition of selective measures to optimize usability and increase operation safety of the device. The number of test subjects was defined by 5–10 participants, according to recommendations from the literature (cp. [1], [4]). The test was finally performed with four physical therapists for infants.

The usability test in both phases was realised as “think aloud”. The tasks were selected based on the main operation functions and thus involves all frequently-used and safety-related functions. The effectiveness was determined by the task performance and the number and severity of use errors, respectively. Efficiency was evaluated by the handling time. To receive a comprehensive insight of the subjective user satisfaction the inquiry comprised a face to face interview and a questionnaire. The applied questionnaire accords to the “System Usability Scale” (SUS) from Brooke [5], adapted to the medical context as well as the specific problem. Furthermore, a heuristic evaluation based on 10 defined heuristics according to Backhaus [1] was performed.

2.2 Evaluation methods

The analysis of data and evaluation of usability was carried out by three methods – “PROMEDIKS” [6], “UseProb” [2] and defined usability aims, as suggested by the standard EN 62366 [3].

PROMEDIKS from Backhaus [6] represents an evaluation method based on the influence factors “operability”, “functionality” and their deficits. The identification of deficits occurs on a given set of criteria, which were modified for the evaluation of the prototype (see Table 1) and not identified by the median as suggested. Similarly the relevance of processes and operations must be assessed and considered in the evaluation. Therefore, for processes and operations, which can affect the compliance of infants or the safety, a weighting factor from 4 to 5 for the relevance of tasks (TRi) and processes (PRi) was used. A weighting factor from 1 to 3 was considered depending on the frequency of use of tasks and processes. Quantitative operability (QO) and process support (QPS) can be calculated as defined in Equations 1 and 2. The geometric mean of these parameters leads to the usability value (UV), see Equation 3 [6].

(1) QO = ( TR i . ( A UA i ) ) ( TR i . UA max ) .100 %

Table 1:

Modified criteria for identification of operation deficits.

Operation deficit Value Criteria Measure
No deficit 1 <30%, Slight failure No
Potential deficit 2 ≥30%, Slight failure Sensible
<15%, Severe failure
Severe deficit 3 ≥15%, Severe failure Necessary

QO: quantified operability; TRi: relevance of task, UAi: assessment of deficit, UAmax: maximum of usability assessment, A: n + 1, n: number of estimation levels

(2) QPS = ( PR i . PA ai ) ( PR i . PA max ) .100 %

QPS: quantified process support, PRi: relevance of process, PAai: actual functional process support, PAmax: maximum of process support

(3) UV = QO.QPS

UV: Usability Value, QO: quantified operability, QPS: quantified process support

UseProb from Büchel [2] presents a standardized evaluation method for medical devices based on a risk analysis. Usability problems, which were identified through a heuristic evaluation, the usability test and the user inquiry were estimated according to severity, incidence rate as well as the potential harm. The evaluation is based on the most serious problem of each task. The evaluation result can be calculated and the usability can be assessed by the given scale [2].

The defined Usability Aims comprise objective aims and imply dimensions for effectiveness and efficiency. They were derived from main operation functions and consist of several aims for each task [3]. In this stage the aims simply represent handling attributes of the system. To get a result of the usability, the achievement of objectives was used and examined.

3 Results

The use of PROMEDIKS in phase 1 shows an impressive usability value of 85.3% (see Figure 1, UVTP_1). The analysis of the questionnaire leads to a SUS Score of 89.1 which represents an “excellent” user satisfaction [7]. The examination of usability aims in contrast to the results of PROMEDIKS showed a target achievement of 45.5% which is not acceptable. The main goal of phase 1 was to detect general weak points of the software operations and to derive suitable measures. Major measures after testing comprise an increase of type size for better legibility, highlighting and labelling of relevant functions as well as the variation of terms to prevent inconsistencies.

Figure 1: Graphic presentation of the usability value, based on [6].

Figure 1:

Graphic presentation of the usability value, based on [6].

The outcome of phase 2, as shown in Table 2, represents a lower usability value of 77.4%. This lower value results from the process of patient fixation, whereby seven severe operation deficits occurred. Figure 1 depicts the quantified results of the handling process separated from tests including software operations. The operability of the software could be improved up to 96% (see Figure 1), mainly based on the implemented measures from phase 1. The usability aims could also be increased to 68% and lead to an acceptable result. The user satisfaction was slightly reduced from 89.1 to 82.9, which still represents a “good” to “excellent” user assessment [7]. Major points of criticism from the inquiry related to the handling of patient positioning and fixation.

Table 2:

Results of test phase 1 and 2.

Method Phase 1 Phase 2
Usability value (PROMEDIKS) 85.3% 77.4%
Questionnaire 89.1 82.9
Usability aims 45.5% 68.0%
Evaluation result (UseProb) a 6.99

aThe UseProb method was not used in phase 1.

The application of UseProb leads to at least 51 usability problems and 13 items which describe good or excellent solutions. The evaluation result is 6.99. In summary, this leads to an evaluation level of seven which corresponds to a satisfying usability where personal or material injuries are highly improbable [2].

4 Conclusion

On the basis of the results from phase 2 the usability of the locomotor therapy device was assessed as “satisfactory”. In spite of different evaluation strategies the used methods approximately lead to equal results.

Most of the detected weak points and deficiencies slightly affect the usability and can be assessed as “trivial” in regard to the good learnability. The general high user satisfaction proves this conclusion. Furthermore it could be shown that the implemented measures from phase 1 increased the usability of the software operation.

Usability deficiencies, which possibly affect patient safety, mainly concern the procedure of patient positioning and patient fixation. Special training courses as wells as a setting-in period should be considered to decrease the risk for this handling procedure. In addition to that checklists for attachment and fixation could be implemented.

These measures should lead to optimized usability and higher operation safety of the locomotor therapy prototype. The effectiveness of these measures needs to be evaluated in further testing. Therefore a field testing is planned, whereby under the consideration of the intended use of the device a final and valid evaluation should be carried out.

Author’s Statement

Research funding: The author state no funding involved. Conflict of interest: Authors state no conflict of interest. Material and Methods: Informed consent: Informed consent is not applicable. Ethical approval: The conducted research is not related to either human or animal use.

References

[1] Backhaus C. Usability-Engineering in der Medizintechnik. Heidelberg-Dordrecht-London-New York: Springer; 2010. Search in Google Scholar

[2] Büchel D. Entwicklung einer allgemeingültigen Standardprozedur zur Überprüfung der Gebrauchstauglichkeit medizintechnischer Produkte. Eberhard Karls Universität zu Tübingen, Tübingen, Dissertation; 2010. Search in Google Scholar

[3] EN ISO 62366. Medizinprodukte, Anwendung der Gebrauchstauglichkeit auf Medizinprodukte. 2008. Search in Google Scholar

[4] Faulkner N. Beyond the five-user assumtion: Benefits of increased sample sizes in usability testing. Behav Res Meth Ins C. 2003;35:379–83. Search in Google Scholar

[5] Brooke J. A quick and dirty usability scale. Usability Evaluation in Industry: Taylor &amp; Francis Ltd; 1996, ch. 21. p. 189–94. Search in Google Scholar

[6] Backhaus C. Entwicklung einer Methodik zur Analyse und Bewertung der Gebrauchstauglichkeit von Medizintechnik. Technische Universität Berlin, Berlin, Dissertation; 2004. Search in Google Scholar

[7] Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. Journal of Usability Studies. 2009;4:114–23. Search in Google Scholar

Published Online: 2016-9-30
Published in Print: 2016-9-1

©2016 Sonja Langthaler et al., licensee De Gruyter.

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.