Accessible Published by De Gruyter November 23, 2020

Quality benchmarking of smartphone laboratory medicine applications: comparison of laboratory medicine specialists’ and non-laboratory medicine professionals’ evaluation

Snežana Jovičić, Joanna Siodmiak, Marta Duque Alcorta, Maximillian Kittel, Wytze Oosterhuis, Kristin Moberg Aakre, Per Jørgensen, Vladimir Palicka ORCID logo, Marge Kutt, Mikko Anttonen, Mileva Georgieva Velizarova and Jania Marc

Abstract

Objectives

There are many mobile health applications (apps) now available and some that use in some way laboratory medicine data. Among them, patient-oriented are of the lowest content quality. The aim of this study was to compare the opinions of non-laboratory medicine professionals (NLMP) with those of laboratory medicine specialists (LMS) and define the benchmarks for quality assessment of laboratory medicine apps.

Methods

Twenty-five volunteers from six European countries evaluated 16 selected patient-oriented apps. Participants were 20–60 years old, 44% were females, with different educational degrees, and no professional involvement in laboratory medicine. Each participant completed a questionnaire based on the Mobile Application Rating Scale (MARS) and the System Usability Scale, as previously used for rating the app quality by LMS. The responses from the two groups were compared using the Mann-Whitney U test and Spearman correlation.

Results

The median total score of NLMP app evaluation was 2.73 out of 5 (IQR 0.95) compared to 3.78 (IQR 1.05) by the LMS. All scores were statistically significantly lower in the NLMP group (p<0.05), except for the item Information quality (p=0.1631). The suggested benchmarks for a useful appear: increasing awareness of the importance and delivering an understanding of persons’ own laboratory test results; understandable terminology; easy to use; appropriate graphic design, and trustworthy information.

Conclusions

NLMP’ evaluation confirmed the low utility of currently available laboratory medicine apps. A reliable app should contain trustworthy and understandable information. The appearance of an app should be fit for purpose and easy to use.

Introduction

Smartphone health applications (apps) represent a form of the mobile health (mHealth) technologies [1]. The so-called health apps may be used by health professionals as a tool for medical education, at the point of care, as a clinical reference, or are recommended to patients for self-monitoring and active participation in the clinical process [2]. They play a central role in mHealth network making the connection between the end-users and its elements, like wearables and tracking sensors, medical devices, electronic health records, health data aggregators, and various tools (i.e., testing, performance tracking, social networking, analytics, storing data or advertisement networks) [3].

Health apps can also function independently. They are usually downloaded from app stores (iTunes, Google Play, etc.), where they are mostly classified in the “Health and fitness” or “Medical” category. Some are intended to be used by patients on a daily basis, as a journal for health and fitness activities, or as a support for management of chronic diseases, while others may be used by healthcare professionals.

The content of mobile health apps is not regulated and controlled by governments unless they function as part of a medical device. This is regulated by the Food and Drug Administration (FDA) in the USA and with the Medical Device Regulation 2017/745 in the EU. According to these regulations, if a mobile health app is used in medical diagnosis or treatment, i.e., if it qualifies as a medical device, then it requires regulatory FDA review and approval in the USA, and conformity assessment to obtain a CE marking in Europe. If this is not the situation, they are not subjected to regulatory oversight, which is the case with the majority of freely available health-related applications [4], [5].

Many of the mobile health apps involve, in different ways, laboratory medicine data. The scopes of their use, as well as the quality of the related information in this field, are important issues, which are only scarcely analyzed. The evaluation of the quality of these apps was performed by the European Federation of Clinical Chemistry and Laboratory Medicine Working Group on Patient-Focused Laboratory Medicine. The results of this study showed the lowest quality score, as evaluated independently by the two laboratory medicine specialists (LMS), in the category of apps designed for patients that provided recommended values and decision limits of selected analysis with basic information about the causes of their increase or decrease [6]. In order to obtain the best benchmarks for the quality of apps intended for patients, that are dealing with laboratory medicine data, we wanted to include in the evaluation process the target group of users for whom the apps were designed. The aim of this study was to compare users’ (i.e., potential patients’) opinion, that is the non-laboratory medicine professionals (NLMP), with the quality evaluation done by LMS and define the points of reference for quality assessment of apps using laboratory medicine data.

Materials and methods

Study selection

The process of selection of smartphone apps using in any way laboratory medicine data, freely available from the “Google Play” and “App Store”, is described in our previously published work [6]. Out of the 52 apps included in the quality evaluation process, 26 of them were intended for patients. These were apps that were recognized as such, providing recommended values and decision limits for selected parameters in combination with essential information about the causes of their changes, and apps for monitoring the state of user’s health status using a variety of parameters, including glucose and/or cholesterol levels as laboratory data. Out of these 26 patient-oriented apps, we have selected 16 (Table 1) that had a unique content without duplicating other apps. The misbalance between the Android and the iPhone apps is the consequence of the fact that most of the iPhone apps intended for patients requested a fee for being downloaded. Therefore, we could not include them in the study.

Table 1:

Apps included into the evaluation process (n=16).

Name of the app Developer Mobile platform
Recommended values and decision limits of selected analysis with basic information about the causes of increase or decrease designed for patients (n=11)
Blood test guide O’Clock Software Pvt, Ltd

https://play.google.com/store/apps/details?id=com.ocs.bloodtest&hl=en
Android, iPhone
Blood test guide Rola Tech

https://play.google.com/store/apps/details?id=com.blood.testre&hl=en
Android
Blood tests results explained Villov Frisky Apps

https://play.google.com/store/apps/details?id=com.friskyapps.bloodtestresult&hl=en
Android
Blood tests results explained Brain 2016

https://play.google.com/store/apps/details?id=com.brain2016.bloodtest&hl=en
Android
CALIPER app The Hospital for Sick Children, Toronto, Canada

https://play.google.com/store/apps/details?id=ca.sickkids.CaliperMobileApp&hl=en
Android
Lab values free Sammysoft

https://play.google.com/store/apps/details?id=de.sammysoft.labor.free&hl=en
Android
Lab values with interpretation HS Developers

https://play.google.com/store/apps/details?id=hsappdeveloper.labvalues&hl=en
Android
Laboratories Zedney Creative, Jeddah, Saudi Arabia

https://play.google.com/store/apps/details?id=com.zedney.laboratoriesapp&hl=en
Android, iPhone
Normal lab values Nooglesoft, New Delhi, India

https://play.google.com/store/apps/details?id=com.nooglesoft.normal.labvalues&hl=en
Android
Pathology lab dictionary Akshar Clearing Agency

https://play.google.com/store/apps/details?id=aksharclearing.agency.pathLab&hl=en
Android
Quick labref NikaInformatics

https://play.google.com/store/apps/details?id=nika.informatics.quick.LabRef&hl=en
Android
Apps for monitoring the state of user’s health through a wide range of health parameters, including glucose and/or cholesterol as laboratory data (n=5)
Health assistant Mromso & Friends

https://play.google.com/store/apps/details?id=com.wsmrs.hassi&hl=en
Android
Health diary PNN Soft, Kyiv, Ukraine

https://play.google.com/store/apps/details?id=com.pnn.healthdiary&hl=en
Android
My health tracker Walter Gross

https://play.google.com/store/apps/details?id=com.wpg.healthtracker&hl=en
Android
S health Samsung Electronics Co, South Korea

https://play.google.com/store/apps/details?id=com.sec.android.app.shealth&hl=en
Android
Tactio health Tactio Health Group

https://play.google.com/store/apps/details?id=com.tactiohealthgroup.tactiohealth&hl=en
Android

App content analysis

We have recruited 25 volunteers, from Serbia, Poland, Spain, Germany, and the Netherlands, with no professional involvement in laboratory medicine to evaluate selected apps (NLMP). Volunteers were of both gender (14 men, 11 women), of different age (20–60 years), and level of highest education (3 with high school education, 15 with a university degree, and 7 with a Ph.D.). The purpose of the study was explained to all the volunteers and all of them signed an informed consent. Ethical approval was deemed not required since the questionnaires were anonymous, collecting only age, level of education and gender out of the personal information, there was no other intervention, and this was not a medical trial in the sense that the integrity of the body was compromised.

The volunteers were asked to download each of these apps and explore their content and functions. According to their impressions and opinions about the app, each participant completed the questionnaire (Supplemental Table 1) that we designed based on the Mobile Application Rating Scale (MARS) as it had been used for rating the app quality by LMS [6], in order to compare these two ratings between LMS and NLMP. Additional questions based on the System Usability Scale were also used [7], [8], [9].

We have grouped the questions in five areas. The first four were equivalent with the MARS items:

  1. 1)

    Subjective quality – considering if the users would recommend the app to others, how many times do they think they would use the app in the next 12 months, would they pay for it, and the overall star rating, which quantifies the users’ overall satisfaction with the app. These questions are identical with the MARS subjective quality questions.

  2. 2)

    Information quality – considering the understanding the language used in the app, explanation of information with enough details, the understanding of provided information, and personal opinion on the trustworthiness of information found in the app. These questions are not identical, but equivalent to the questions for the MARS information mean score.

  3. 3)

    App functionality – questions whether the app was easy to use, was the graphic design and the overall appearance in concordance with its purpose. These points are not identical, but equivalent with the questions of MARS functionality and aesthetics mean scores.

  4. 4)

    Motivational aspect – questions that assess the possible impact of apps on users’ awareness of the importance of addressing the meaning of their laboratory test results, knowledge/understanding of the meaning of laboratory test results, their attitude towards improving their health, intentions/motivation to address their health issues, and to encourage them to further seek medical help (if it is required). Motivational aspect questions were not part of the MARS questionnaire.

  5. 5)

    App usability – this was evaluated through the questions corresponding with the System Usability Scale in accordance with the ISO 9241 standard – Ergonomics of Human System Interaction [8], [9], that is, would it be recommended to others who might benefit from it, what would be the frequency of the app usage in the next 12 months, was it easy to use and was the graphic design and the overall appearance appropriate with the app purpose.

In the evaluation of apps that allowed interactive usage, with inserting personal laboratory results into the app and tracking their changes (Apps for monitoring the state of user’s health through a wide range of health parameters, including glucose and/or cholesterol as laboratory data, Table 1) we have included the assessment of the apps’ engagement. This included questions about the possibility to insert personal results into the app, to track changes in time, and the motivational aspect of these actions. The possible answers were “yes” or “no”.

Statistical analysis

All the questions were designed according to the Likert scale (i.e., the possible answers were in the form of a 5 point scale ranging from a strong disagreement to a strong agreement with the statement in the question). Since the NLMP’ grades were not normally distributed, we have determined the median grade of all NLMP’ answers for each question regarding one app. From there, scores for every category of questions, and a total score, were calculated as means of median grades of all NLMP for particular questions, to maintain equivalence with MARS scoring, in order to compare them with LMS’ evaluation from the previous study [6]. A general linear model of univariate analysis of variance was used to evaluate the influence of NLMP’ age, gender, and level of education on total scores. For the four categories equivalent to individual MARS sections, we have compared median NLMP’ and LMS’ category scores and total scores of all apps using Mann-Whitney U test. Also, we have evaluated relationships between LMS’ MARS ratings, app store ratings (ratings of “Google Play” and/or “App Store” users), and NLMP’ ratings of individual apps using Spearman rank correlation test. In the correlation testing, besides the above mentioned, we have included NLMP’ overall star rating of apps (question 4, Supplemental Table 1), and their opinion on the trustworthiness of app’s information (question 15, Supplemental Table 1), as well as the NLMP assessment of the usability of apps. We also asked volunteers to select out of all these characteristics those they consider the most important points for an app useful to NLMP. Statistical analysis was performed using Medcalc Software (version 18), and IBM® SPSS® Statistics (version 20).

Results

General linear model univariate analysis of variance showed that total scores of apps did not depend on age, gender, and level of education of NLMP (the F-values describing the influence of these independent variables on total scores of the evaluation of each individual app were not statistically significant with p>0.05). The median total score was 2.73 out of 5 (IQR 0.95), ranging from 1.93 (Health Diary by PNN Soft, Kyiv, Ukraine) to 3.67 (Caliper App from The Hospital for Sick Children, Toronto, Canada). A summary evaluation of the analyzed data is presented in Table 2. The LMS’ scores were in general lower than the NLMP’ scores, with statistically significant differences between median scores for subjective quality (p=0.0311), functionality (p<0.0001), and for total scores (p=0.0001), but with no significant difference for information quality (p=0.1631).

Table 2:

Summary evaluation data for the analyzed apps (n=16).

Participant questionnaire elements NLMP rating, median (IQR) LMS

MARS rating, median (IQR)
p-Valuea
Evaluation of MARS and MARS equivalent items
Subjective quality 2.50 (0.97) 3.25 (1.19) 0.0311
Information quality 3.50 (1.16) 3.66 (1.62) 0.1631
App functionalityb 3.20 (0.67) 4.75 (1.25) <0.0001
Total score 2.73 (0.95) 3.78 (1.05) 0.0001
Additional evaluation
Motivational aspect 2.80 (1.60) / /
Usability 3.00 (1.00) / /
Overall rating of appsc 3.00 (1.00) / /
Trustworthiness of apps’ informationd 3.00 (1.75) / /
App store ratings e (1–5) 4.25 (0.5)

  1. IQR, interquartile range; NLMP, non-laboratory medicine professionals, LMS, laboratory medicine specialists; MARS, Mobile Application Rating Scale. aMann-Whitney U test. bCorresponding MARS Functionality and Aesthetics. cRefers to question 4 of the participants questionnaire. dRefers to question 15 of the participants questionnaire. eTaken from “Google Play” and/or “App Store”.

In order to compare NLMP and LMS evaluation of apps, correlations between them were evaluated using the Spearman rank correlation test. The Spearman’s ρ correlation coefficients between NLMP overall ratings of apps were statistically significant with all the elements of the NMLP questionnaires, the evaluated usability, the total score, as well as with the MARS total score, engagement, and subjective quality. On the other hand, the NLMP’ impression on the trustworthiness on information within the app did not correlated significantly neither with MARS total score, MARS individual scores, nor usability evaluation.

Spearman’s correlation coefficients (ρ) between LMS’ MARS ratings, app store ratings and NLMP’s ratings are presented in Table 3. NLMP’s total score had only borderline significance (p=0.067) with average rating reported by the app store, and with the MARS total score (p=0.064). On the other hand, the total MARS score, as well as MARS engagement, information quality, and subjective quality scores, evaluated by LMS, were all significantly correlated with the usability estimated by NLMP. App store ratings did not correlate significantly with either one element of the NLMP’s quality evaluation criteria or with MARS individual scores.

Table 3:

Spearman’s corelation coefficients (ρ) between LMS’ MARS ratings, app store ratings and NLMP’ ratings.

Participants’ questionnaire
Subjective quality Information quality App functionality Total score Motivation aspect Usability Overall rating Trustworthiness
LMS subjective quality 0.523 a n.s. n.s. n.s. 0.595 a 0.540 a 0.589 a n.s.
LMS information quality n.s. n.s. n.s. n.s. n.s. 0.497 a n.s. n.s.
LMS functionality n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s.
LMS total score n.s. n.s. n.s. n.s. n.s. 0.545 a 0.539 a n.s.
App store rating n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s.

  1. LMS, laboratory medicine specialist; NLMP, non-laboratory medicine professionals; n.s., not significant (p>0.05). ap<0.01.

Not all apps intended for entering personal laboratory results into the app and tracking their changes allowed this interaction, nor could they track the changes in time. In any case, the motivation for interactive usage of apps that allowed it was significantly correlated with the NLMP’s total score (Spearman’s ρ=−0.375, p=0.007).

We also asked volunteers to select out of all analyzed characteristics those they consider the most important for an app useful to NLMP (Figure 1). For each participant it was allowed to choose more than one. More than 50% of NLMP selected delivering knowledge/understanding of the meaning of laboratory test results (74%),easy use of the app (74%), using understandable terminology (58%), trustworthy information within the app (58%), and increasing awareness of the importance of addressing the meaning of laboratory test results (53%).

Figure 1: 
Summary of analyzed app characteristics and percentages of NLMP participants that selected each one of them as important (more than one choice was possible).
App characteristics: 1. increasing awareness of the importance of addressing the meaning of my laboratory test results; 2. delivering knowledge/understanding of the meaning of my laboratory test results; 3. changing my attitude towards improving my health; 4. increasing my motivation to address my health issues; 5. to encourage me to seek help for my condition from a medical doctor; 6. using the terminology understandable to me; 7. detailed enough explanation of information I need; 8. the information provided within the app easy for me to understand; 9. the app is easy to use; 10. graphic design and the overall appearance of the app appropriate and in accordance with the app purpose; 11. trustworthy information within the app.

Figure 1:

Summary of analyzed app characteristics and percentages of NLMP participants that selected each one of them as important (more than one choice was possible).

App characteristics: 1. increasing awareness of the importance of addressing the meaning of my laboratory test results; 2. delivering knowledge/understanding of the meaning of my laboratory test results; 3. changing my attitude towards improving my health; 4. increasing my motivation to address my health issues; 5. to encourage me to seek help for my condition from a medical doctor; 6. using the terminology understandable to me; 7. detailed enough explanation of information I need; 8. the information provided within the app easy for me to understand; 9. the app is easy to use; 10. graphic design and the overall appearance of the app appropriate and in accordance with the app purpose; 11. trustworthy information within the app.

Discussion

In this study, we compared users’ (i.e., NLMP’) opinion with the quality evaluation done by the medical experts (LMS) in order to establish the benchmarks for quality assessment of apps using laboratory medicine data intended for patients. NLMP’ ratings were significantly lower than in the evaluation performed by the LMS, with the exception of usability estimates. This correlated significantly with both app rating (MARS) individual and total scores, indicating that usability connects best the objective elements of app quality with the users’ assessment. Also, NLMP expressed their insecurity in trustworthiness of information within apps.

There was a significant correlation of NLMP and LMS overall general usability (star) rating. On the other hand, there was no significant correlation between the NLMP impression on the trustworthiness of information within the app with neither the LMS’s total quality score nor apps’ usability. This demonstrates the lack of agreement of LMS’ overall quality estimation of the apps and NLMP’ confidence in the information they contain. Also, if we consider the usability as “general quality of the appropriateness to a purpose” [9], and the purpose of these apps should be to help patients to better understand their laboratory results, we would expect that the usability agrees with the trustworthiness of the information that app delivers. However, the median percentage of “don’t know” answers for the direct question about the trustworthiness of information within the apps was remarkably high, 31%, indicating a lack of confidence. This is in agreement with the fact that the affiliation was unknown for 50% of examined apps, that will add to lack of confidence. The other half, except the CALIPER app with the university affiliation, were developed commercially.

The other indicator of information reliability evaluated by NLMP, the information quality score, also did not correlate significantly with the LMS’ total score. This was expected considering that one-third of NLMP couldn’t estimate the trustworthiness of information provided within the app. On the other hand, the information quality evaluated by LMS was significantly correlated with the LMS total MARS score. This was due to the competence of LMS to evaluate the reliability of information provided by the app. However, since the median NLMP’ and LMS’ scores for information quality did not differ significantly, we may assume that the NLMP, in the end, came to the same conclusion on the quality of information within the app probably leaning on their general satisfaction on using the app.

There were no statistically significant correlations between individual LMS items and the corresponding individual NLMP’ scores, nor between the LMS’s and NLMP’s evaluation total scores. This difference in patients’ and professionals’ opinion of the quality of apps was also reported in a study that evaluated smartphone apps for chronic kidney disease (CKD) [10]. The authors explained that the reason for this disagreement is understandable since the patients with CKD and physicians usually disagree on important aspects of their care, as well as that the patients’ perception of app usefulness differs from the professional one.

There are potential difficulties in comprehension of laboratory results by patients, especially when presented in traditional form of laboratory reports. Also, patients and medical professionals have different views on understanding laboratory results. Hence, it is easy to conclude that these issues caused the disagreement in laboratory medicine professionals’ and NLMP’ overall evaluation scores of the analyzed apps [11], [12], [13]. Most of the apps had a rather linear presentation of laboratory values with textual explanations, often written in a language style not adequate for laypersons. Therefore, the NLMP may not have been aware of the quality of the app’s content, even if LMS recognized it as adequate and correct. Within some apps explanations were too long and over-detailed, so the user would likely to lose interest in reading it. In some others, the contents were too scarce, more adequate as a reminder for a professional. This could also explain the significant difference in subjective quality and app functionality between LMS’ and NLMP’ evaluation (Table 2).

The app store ratings did not correlate significantly neither with the NLMP’ evaluation total score nor for the LMS’ scores. This only confirmed their unreliability and doubt that they are often made by persons or services hired by publishers to provide positive ratings and reviews for their apps [14].

The only evaluated characteristic of analyzed apps that correlated significantly with all the individual LMS item scores, as well as with the LMS total score was NLMP’ usability score. It seems that usability, as the sum of effectiveness and efficiency of the system, as well as the users’ satisfaction, connects best the objective elements of app quality evaluated by laboratory professionals, contained within the MARS evaluation system, with the NLMP’ assessment, as indicated in previous studies [9], [10].

When asked to express their opinion about the most important characteristics of an app useful to patients, most of NLMP selected that app should increase awareness of the importance of addressing the meaning of laboratory test results, deliver knowledge and understanding of the meaning of laboratory test results, use understandable terminology, be easy to use and contain trustworthy information. We believe that these, together with the graphic design and the overall appearance in accordance with the app purpose, would be adequate reference points for the quality of laboratory medicine apps (Table 4). We have included also the graphic design as the usability element, although less than 50% of NLMP have marked it as significant, since the importance of usability was demonstrated with its significant correlation with both the professional and laypersons’ evaluation.

Table 4:

Benchmarks for the quality of laboratory medicine apps.

Benchmarks for the quality of laboratory medicine apps
1. To increase awareness of the importance of addressing the meaning of own laboratory test results;
2. To deliver knowledge/understanding of the meaning of own laboratory test results;
3. To use the terminology understandable to users;
4. To be easy to use;
5. To have appropriate graphic design and the overall appearance in accordance with the app purpose, and
6. To deliver trustworthy information.

We have to emphasize that all the volunteers in this study were healthy persons with no known chronic conditions. Considering that the patients with chronic diseases are more motivated to search for information related to their condition, they should be more interested to become familiar with the expected values of their laboratory tests results [11]. On the other hand, there are patients that are not very motivated to know more about their health, i.e., beyond the information they receive from their physician [15]. Healthy persons are more likely to be in the second category. We may consider this, together with the small number of participants, major limitation of this study.

Conclusions

The median total quality score of lay persons’ (NLMP’) evaluation was even lower than the (LMS’) score of the professional evaluation, confirming the poor quality of currently freely available smartphone apps dealing with laboratory medicine data. According to the NLMP’ and LMS’ evaluation of these apps, we have defined the points of reference for quality assessment of these apps. Of particular importance is the trustworthiness of information, with clearly stated and visible references. The adequate style of presenting information, understandable for lay persons follows. Finally, the graphics and the appearance of the app should be fit for purpose and in the function of easy use. These points could be considered as benchmarks for quality evaluation of patient-oriented laboratory medicine apps.


Corresponding author: Snežana Jovičić, Center for Medical Biochemistry, Clinical Center of Serbia, and Department for Medical Biochemistry, Faculty of Pharmacy, University of Belgrade, Koste Todorovića 8, 11000 Belgrade, Serbia, Phone: +381 63 876 89 53, E-mail:

  1. Research funding: None declared.

  2. Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.

  3. Competing interests: Authors state no conflict of interest.

  4. Informed consent: Informed consent was obtained from all individuals included in this study.

  5. Ethical approval: The local Institutional Review Board deemed the study exempt from review.

References

1. Bhavnani, SP, Narula, J, Sengupta, PP. Mobile technology and the digitization of healthcare. Eur Heart J 2016;37:1428–38. https://doi.org/10.1093/eurheartj/ehv770.Search in Google Scholar

2. Aungst, TD, Clauson, KA, Misra, S, Lewis, TL, Husain, I. How to identify, assess and utilize mobile medical applications in clinical practice. Int J Clin Pract 2014;68:155–62. https://doi.org/10.1111/ijcp.12375.Search in Google Scholar

3. mHealth Developer Economics 3rd report – Connectivity in digital health. Research2Guidance; 2018. Available from: .Search in Google Scholar

4. U.S. Food and Drug Administration. Policy for device software functions and mobile medical applications guidance for industry and Food and Drug administration staff; 2019. Available from: [Accessed 16 November 2019].Search in Google Scholar

5. The European Parliament and the Council of the European Union. Regulation (EU) 2017/745 of the European parliament and of the council of 5 april 2017 on medical devices, amending directive 2001/83/EC, regulation (EC) No 178/2002 and regulation (EC) No 1223/2009 and repealing council directives 90/385/EEC and 93/42/EEC. Official Journal of the European Union, 5.5.2017. Available from: [Accessed 16 Nov 2019].Search in Google Scholar

6. Jovičić, S, Siodmiak, J, Watson, ID. Quality evaluation of smartphone applications for laboratory medicine. Clin Chem Lab Med 2018;57:388–97.Search in Google Scholar

7. Stoyanov, SR, Hides, L, Kavanagh, DJ, Zelenko, O, Tjodronegoro, D, Mani, M. Mobile App Rating Scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth Uhealth 2015;3:e27. https://doi.org/10.2196/mhealth.3422.Search in Google Scholar

8. Brook, J. SUS: a retrospective. J Usability Stud 2013;8:29–40.Search in Google Scholar

9. Brook, J. SUS – a quick and dirty usability scale. In: Jordan, PW, Thomas, B, Weerdmeester, BA, McClelland, IL, editors. Usability evaluation in industry. London: Taylor and Francis; 1996:189–94 pp.Search in Google Scholar

10. Singh, K, Diamantidis, SJ, Ramani, S, Bhavsar, NA, Mara, P, Warner, J, et al.. Patients‘ and nephrologists’ evaluation of patient-facing smartphone apps for CKD. Clin J Am Soc Nephrol 2019;14:523–9. https://doi.org/10.2215/cjn.10370818.Search in Google Scholar

11. Wilkie, P. Patient views on understanding laboratory results. Clin Chem Lab Med 2019;57:371–4. https://doi.org/10.1515/cclm-2018-0868.Search in Google Scholar

12. Witteman, HO, Zikmund-Fisher, BJ. Communicating laboratory results to patients and families. Clin Chem Lab Med 2019;57:359–64. https://doi.org/10.1515/cclm-2018-0634.Search in Google Scholar

13. Watson, ID. A patient focused relationship for specialists in laboratory medicine. Clin Chem Lab Med 2019;57:383–7.Search in Google Scholar

14. Fayazi, A, Lee, K, Cavarlee, J, Squicciarini, A. Uncovering crowdsourced manipulation of online reviews. In: SIGIR15: proceedings of the 38th International ACM SIGIR conference on research and development in information retrieval. New York: Association for Computing Machinery 2015:233–42 pp. https://doi.org/10.1145/2766462.2767742.Search in Google Scholar

15. Hibbard, J, Gilburt, H. Supporting people to manage their health. An introduction to patient activation. London: King’s Fund, 2014.Search in Google Scholar

Supplementary Material

The online version of this article offers supplementary material (https://doi.org/10.1515/cclm-2020-0869).

Received: 2020-06-08
Accepted: 2020-10-29
Published Online: 2020-11-23
Published in Print: 2021-03-26

© 2020 Walter de Gruyter GmbH, Berlin/Boston