Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Open Medicine

formerly Central European Journal of Medicine

Editor-in-Chief: Darzynkiewicz, Zbigniew


IMPACT FACTOR 2018: 1.221

CiteScore 2018: 1.01

SCImago Journal Rank (SJR) 2018: 0.329
Source Normalized Impact per Paper (SNIP) 2018: 0.479

ICV 2018: 156.09

Open Access
Online
ISSN
2391-5463
See all formats and pricing
More options …
Volume 12, Issue 1

Issues

Volume 10 (2015)

HIV/STI prevention interventions: A systematic review and meta-analysis

Jason Globerman / Sanjana Mitra / David Gogolishvili / Sergio Rueda / Laura Schoffel / Kira Gangbar / Qiyun Shi / Sean B. Rourke
  • Department of Psychiatry, University of Toronto; Centre for Urban Health Solutions, Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, Canada
  • Other articles by this author:
  • De Gruyter OnlineGoogle Scholar
Published Online: 2017-12-14 | DOI: https://doi.org/10.1515/med-2017-0064

Abstract

Behavioral interventions can prevent the transmission of HIV and sexually transmitted infections. This systematic review and meta-analysis assesses the effectiveness and quality of available evidence of HIV prevention interventions for people living with HIV in high-income settings. Searches were conducted in MEDLINE, EMBASE, PsycINFO, and CDC Compendium of Effective Interventions. Interventions published between January, 1998 and September, 2015 were included. Quality of evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE). Forty-six articles and 63 datasets involving 14,096 individuals met inclusion criteria. Included articles were grouped by intervention type, comparison group and outcome. Few of these had high or moderate quality of evidence and statistically significant effects. One intervention type, group-level health education interventions, were effective in reducing HIV/STI incidence when compared to attention controls. A second intervention type, comprehensive risk counseling and services, was effective in reducing sexual risk behaviors when compared to both active and attention controls. All other intervention types showed no statistically significant effect or had low or very low quality of evidence. Given that the majority of interventions produced low or very low quality of evidence, researchers should commit to rigorous evaluation and high quality reporting of HIV intervention studies.

This article offers supplementary material which is provided at the end of the article.

Keywords: HIV; sexually transmitted infections; prevention; intervention, risk behavior

1 Introduction

HIV is a serious health issue in North America. In the United States alone, over 1.2 million people are living with HIV, many of whom are unaware of their infection [1]. While the estimated incidence of HIV in the United States and Canada has remained stable in recent years [1,2], among people living with HIV, there has been a marked increase in the diagnosis of other sexually transmitted infections (STIs), particularly syphilis and gonorrhea [3].

Studies have demonstrated that many people living with HIV do not consistently practice safer sex, placing themselves and others at risk for HIV or STI infection/co-infection [4]. For instance, the prevalence of unprotected anal intercourse (UAI) among HIV-positive men who have sex with men (MSM) with either an unknown HIV-status or HIV-negative partner was 26% (95% CI 21-30%) [5]. Prevalence of UAI was even higher with HIV-positive partners (30%; 95% CI 25–36%) [5].

While most HIV prevention programs target HIV-negative individuals, targeting sexual risk behaviors in HIV-positive people can prevent the transmission of HIV and other STIs to uninfected individuals. For people living with HIV, these interventions can also prevent co-infections with other STIs and the acquisition of other strains of HIV.

The US Centers for Disease Control and Prevention (CDC) maintains an up-to-date Compendium of Evidence Based Interventions and Best Practices for HIV Prevention. The Compendium identifies “evidence-based behavioral interventions proven to reduce HIV risk” [6], however, it does not quantitatively synthesize data across studies or assess the quality of available evidence [6]. Unlike previously conducted systematic reviews and meta-analyses [4,7,8], the present review stands out in three ways: (1) the use of the CDC classifications of behaviroal HIV interventions; (2) the use of GRADE to summarize the quality of available evidence; and (3) the assessment of outcome measures beyond sexual risk behaviors.

This review aims to assess the effectiveness of behavioral HIV/STI prevention interventions among people living with HIV in high-income settings through quantitative synthesis (meta-analysis) of data from experimental studies (randomized controlled trials and non-randomized trials). It also aims to assess the quality of available evidence using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) tool [9,10]. Only studies conducted in high income countries, as defined by the World Bank [11], were included. Identifying evidence-based HIV prevention interventions from high income settings may help guide decision-makers, including government policymakers, on where best to allocate funding and other resources for program development.

2 Methods

2.1 Protocol and registration

This study has been designed and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) tool [12]. Analytic methods and inclusion criteria were specified and documented in advance and are available in the systematic review protocol (Supplementary material - file 1).

2.2 Eligibility criteria

Included studies addressed interventions to prevent HIV and/or STIs in people living with HIV. Only randomized-controlled trials (RCTs) and non-randomized trials (quasiexperimental studies) were analyzed. Studies were grouped by intervention category, comparison group, and outcome. Effectiveness was assessed by a series of meta-analyses. Quality of evidence was assessed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) tool [9,13,14]. Only studies conducted in high-income countries, as defined by the World Bank, were included [11]. The following STIs (in addition to HIV) were included in this review: syphilis, chlamydia, gonorrhea, trichomoniasis, genital or anal warts, genital herpes, lymphogranuloma venereum (LGV), and hepatitis B and C. Studies addressing biomedical interventions (e.g., pre-exposure prophylaxis, microbicides, and vaccination/immunization) were excluded.

A prevention intervention was defined as a “specific activity (or set of related activities) intended to change the knowledge, attitudes, beliefs, behavior, or practices of individuals and populations, to reduce their health risk. An intervention has a distinct process, outcome objectives, and a protocol outlining the steps for implementation” [15,16].

The US CDC categorization of behavioral interventions was used to classify interventions [15,16,17,18]. (Supplementary material - file 2). To our knowledge, this classification system is the most comprehensive method of categorizing studies by intervention type.

Comparison groups in all identified studies were categorized as ‘attention’ controls (comparing the effectiveness of the intervention with no intervention or with general health information) or ‘active’ controls (comparing the effectiveness of the intervention with another HIV/STI prevention intervention). These two groups were analyzed separately since it is more difficult to detect a statistically significant difference between groups when an intervention is compared to an active control versus an attention control.

Data from included studies were classified into the following outcomes to evaluate their effectiveness (in order of importance): i) change in HIV/STI incidence; ii) change in self-reported or observed risk behavior; iii) change in knowledge, attitudes and beliefs regarding the HIV/STI prevention.

In cases where a study reported multiple measures of the same outcome, only one measure was selected. For example, specific to changes in self-reported or observed risk behavior, the hierarchy used was as follows: unprotected anal intercourse, unprotected vaginal intercourse, unprotected oral intercourse, condom use, multiple partners, and frequency of sexual encounters [19].

This review includes peer-reviewed articles published in English between January 1, 1998 and September 30, 2015.

2.3 Information sources

Databases consulted included: the Cochrane Database of Systematic Reviews, MEDLINE (1996-present); MEDLINE In-Process & Other Non-Indexed Citations, PsycINFO (1806-present); and EMBASE (1980-present). The US CDC’s Compendium of Evidence Based HIV Behavioral Interventions [6] and Effective Interventions [20] were also searched.

Electronic search strategies were developed in consultation with a reference librarian at Robarts Library, University of Toronto. Reference lists of identified systematic reviews and meta-analyses were further searched to locate additional papers.

2.4 Search

The following terms, in various combinations, were searched: prevent*, HIV, sexually transmitted diseases, hepatitis B, hepatitis C, syphilis, gonorrhea, chlamydia, papillomavirus, wart*, condyloma*, genital herpes, trichomon*, lymphogranuloma, LGV (Supplementary material - file 3). Searches were not limited by study designs, publication types, populations, intervention categories, comparison groups or outcome measures.

2.5 Study selection

Titles and abstracts of all references were screened by two independent reviewers using Distiller SR [21]. Inclusion was based on study type, population, intervention, disease, outcome measure, study jurisdiction, publication year and publication language. Full-text versions of all references identified as “include” or “unclear” were retrieved and additional inclusion assessments of those identified as “unclear” completed. Disagreements between reviewers were resolved by consensus.

2.6 Data collection process

The data extraction form was designed and pilot tested using ten randomly selected studies. Data were extracted by one reviewer and checked independently for accuracy by a second reviewer. Discrepancies were resolved through discussion with a third reviewer. Data were processed using DistillerSR [21]. When data were missing or unclear, the authors of the original papers were contacted to obtain further details.

2.7 Data items

The following information was extracted from each included study: study design, objectives, country/city, sample sizes (intervention and control groups), intervention category (Supplementary material - file 2), duration of intervention, comparison group (active vs.attention control), length of follow-up, and outcome measure(s) with corresponding effect sizes.

Ethical approval

The conducted research is not related to either human or animals use.

2.8 Risk of bias in individual studies

Risk of bias assessments were completed for all individual studies using the Cochrane risk of bias tool [22,23]. A judgment of high, low or unclear was assigned for each of the seven criteria for every included study. Non-randomized trials automatically scored “high risk of bias” in at least one domain (“random sequence generation”).

2.9 Summary measures

Effectiveness of interventions was evaluated based on results of meta-analyses conducted for each combination of intervention, comparison group and outcome. Meta-analyses were conducted using Comprehensive Meta-Analysis (CMA) version 2 [24]. Odds ratios were used when the outcome was HIV/STI incidence, and standardized mean differences (SMD) were used when the outcomes were risk behavior and/or knowledge, attitudes, and beliefs.

2.10 Synthesis of results

For HIV/STI incidence, most included studies reported their results in the form of event rates. Pooled odds ratios (OR) and 95% confidence intervals (CI) were calculated for this outcome. We judged that reduction of 25% or more in odds of acquiring HIV infection was an appreciable benefit, and an OR ≤ 0.75 was considered as effective. This is in line with the GRADE handbook suggestion that default threshold for appreciable benefit is relative risk reduction of 25% or more [9]. For the other two outcomes, included studies reported results in ORs, Chi-squared statistics or means/standard deviations. CMA was used to convert different statistics into SMDs. Random effect models were used to calculate pooled SMDs and 95% CIs for these outcomes. Following widely used standards, SMDs of 0.20 were interpreted as small effect sizes, those above 0.50 as medium effect sizes, and those above 0.80 as large effect sizes [25]. The I2 index was used to assess the heterogeneity between studies.

2.11 Risk of bias across studies

Random effect models were selected for meta-analyses under the assumption that true effect sizes varied from study to study, and because definitions and measurement scales for outcome variables were different across studies. Publication bias was examined using funnel plots.

2.12 Assessment of quality of available evidence

Quality of available evidence for each intervention category was assessed using the GRADE tool [9,13,14]. Quality of evidence in this context refers to the extent to which one can be confident that an estimate of effect is correct. GRADE’s approach to rating the quality of evidence begins with the study design and then addresses five reasons to ‘downgrade’ the quality of evidence (risk of bias, imprecision, inconsistency, indirectness, and publication bias) followed by three reasons to ‘upgrade’ the quality of evidence (large effect, dose response, plausible residual confounding) [9,14].

For HIV/STI incidence, summary of findings tables included the number of studies and number of participants, length of follow-up, confidence in effect estimates (quality of evidence) and the best estimates of relative and absolute effect. ORs were used as the measure of relative effect applied to the control group to generate absolute risk [26]. For continuous outcomes (risk behavior and knowledge, attitude and beliefs) pooled results were presented as SMDs [27]. As a final step on GRADEproGDT, quality of evidence was rated as high, moderate, low or very low for all intervention, comparison group and outcome combinations [9,10,14].

3 Results

3.1 Study selection

Figure 1 illustrates study inclusion and exclusion processes. After database searches, duplicate removal, and the review of other sources and reference checks were complete, 25,865 titles and abstracts were reviewed. Initial screening resulted in 544 full text articles being further assessed for eligibility. Of these, 46 studies met inclusion criteria. Some studies contributed more than one data set, resulting in a total of 63 (k=63) datasets for meta-analyses. Datasets were grouped by intervention, comparison group and outcomes resulting in 17 groups (Supplementary material - file 4).

Flow chart of study selection process
Figure 1

Flow chart of study selection process

3.2 Study characteristics

Of the 46 studies included in the meta-analysis, 40 were randomized controlled trials [28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67], while the remaining six were non-randomized trials [68,69,70,71,72,73]. All included studies were conducted in the United States.

The total sample size of included studies was 14,096 (range 25 to 2,135). In addition to being HIV-positive, study participants belonged to a variety of other groups. Twenty-four percent (n=11) of studies were conducted among men who have sex with men (MSM), 17% (n=8) among ethnocultural minorities, and 15% (n=7) among people who use drugs. There were four studies among women, four among older adults, three studies among youth, three among individuals with childhood sexual abuse histories, and two studies among individuals who were unstably housed. One study included prisoners, one study focused on rural populations. Eight studies included general HIV-positive populations with no other characteristics specified.

Intervention follow-ups ranged from two to 25 months. Five studies collected outcome measures on the change in HIV/STI incidence rates, 45 studies collected data on changes in sexual risk behavior, while five studies collected data on changes in HIV knowledge, attitudes, and beliefs. Intervention categories included: individual-level health education; group-level health education; counseling testing and referral services; and comprehensive risk counseling and services. There were two additional categories: combined individual- and group-level interventions, and interventions not classified elsewhere (housing assistance, and spiritual therapy).

3.3 Results of individual studies

Descriptive characteristics of included studies and interventions are presented in Table 1.

Table 1

Descriptive characteristics of 46 behavioural interventions for people living with HIV

3.4 Synthesis of results

Results of pooled effect sizes and quality of evidence of HIV/STI prevention interventions for people living with HIV have been summarized in Table 2.

Table 2

Meta-analysis and Quality of Evidence of HIV Prevention Interventions for People Living with HIV in High-income Settings

Few intervention, comparison group and outcome combinations had high or moderate quality of evidence and statistically significant summary effects (Supplementary material - file 5).

3.5 High and Moderate Quality of Evidence and Statistically Significant Summary Effects

Quality of evidence was high and summary effect was statistically significant although minimal (k=7; SMD=-0.15, 95%CI=-0.25, -0.05; p=0.003; I2=0) for comprehensive risk counseling in reducing sexual risk behavior when compared to active control. Moderate quality of evidence and statistically significant summary effects (k=14; SMD: -0.35, 95%CI=-0.49, -0.20; p=0.000; I2=72) were observed for comprehensive risk counseling in reducing sexual risk behavior when compared to attention controls. Similar results (k=2; OR: 0.26, 95%CI=0.12, 0.56; p=0.001) were found for group level health education interventions aimed at reducing HIV incidence when compared to attention controls (Supplementary material - file 5).

3.6 Moderate Quality of Evidence and Statistically Non-Significant Summary Effects

Moderate quality of evidence and statistically non-significant summary effects were observed for two interventions: individual level health education and housing assistance. Both interventions were found to reduce sexual risk behavior when compared to attention controls (k=5; SMD: -0.08, 95%CI=-0.17, 0.004; p=0.063; I2=0; and k=1; SMD: -0.17, 95%CI=-0.42, 0.09; p=0.208 respectively). Similar results (k=8; SMD: -0.09, 95%CI=-0.20, 0.02; p=0.114; I2=0) were observed for group level health education interventions aimed at reducing sexual risk behavior when compared to active control (Supplementary material - file 5).

3.7 Low and Very Low Quality of Evidence

A statistically significant effect with low quality of evidence was found for individual level health education interventions in reducing sexual risk behavior when compared to active controls (k=2, SMD: -0.36, 95%CI=-0.61, -0.11; p=0.005; I2=0). Similar results were found for group level health education interventions in reducing sexual risk behavior when compared to attention controls (k=10; SMD: -0.55, 95%CI=-0.90, -0.20; p=0.002; I2=85), however quality of evidence was very low. Group level health interventions, when compared to both attention controls (k=1; SMD: 0.58, 95%CI=0.06, 1.10; p=0.030) and active controls (k=3; SMD: 0.27, 95%CI=0.11, 0.44; p=0.001; I2=0), also improved HIV knowledge, attitudes, and beliefs, however quality of evidence was very low and low respectively.

The quality of evidence for all other combinations of interventions, comparison groups and outcomes was either low or very low, with non-significant effects (Supplementary material - file 5).

3.8 Risk of bias within studies

Risk of bias summaries present the assessment in five domains for each study separately (Supplementary material - file 6). Study assessments of risk of bias were used in determining quality of evidence.

Selection bias

11% (n=5) of studies were judged high risk for random sequence generation (not describing a randomized approach to sequence generation), 37% (n=17) were judged low risk (describing a randomized approach to sequence generation), and 52% (n=24) were judged as unclear risk.

For allocation concealment, 7% (n=3) of studies were judged high risk (not using a method to conceal allocation assignment), 28% (n=13) were judged low risk (using a method to conceal allocation with sufficient detail), and 65% (n=30) were judged as unclear risk.

Performance bias

No studies were judged high risk and had no blinding or incomplete blinding of participants and personnel, while 33% (n=15) were judged low risk, taking adequate measures to blind study participants and personnel, and 67% (n=31) were judged unclear risk.

Detection bias

No studies were judged high risk for no or incomplete blinding of outcome assessment. Sixty-three percent (n=29) of studies were judged low risk and took adequate measures to blind outcome assessment, and the remaining 37% (n=17) of studies were judged as unclear risk.

Attrition bias

9% (n=4) of studies were judged high risk and did not report missing outcome data due to attrition or exclusion from the analysis, 78% (n=36) of studies were judged low risk for incomplete outcome data, and 13% (n=6) of studies were judged as unclear risk.

Reporting bias

No studies were judged high risk (not reporting the study’s pre-specified primary outcomes). Four percent (n=2) were judged low risk and reported complete data, while 96% (n=44) were judged as unclear risk because study protocol was not available or it was not possible to judge whether the published study reported all pre-specified expected outcomes.

3.9 Risk of bias across studies

A varying degree of heterogeneity was observed within each intervention, comparison group and outcome combinations. This inconsistency was explored by I-squared statistics (Table 2). On two occasions (group level interventions compared to attention control and comprehensive risk counseling and services compared to attention control) overall quality of evidence has been downgraded because of substantial or considerable heterogeneity (Supplementary material - file 5). Publication bias was examined using funnel plots and was also taken into consideration when rating the quality of evidence (per GRADE methods) [9,74,75]. On two occasions (with individual level interventions compared to attention control and group level interventions compared to attention control) overall quality of evidence has been downgraded because of strongly suspected publication bias (Supplementary material - file 5).

4 Discussion

This review of randomized and non-randomized controlled trials assessed the effectiveness of HIV/STI prevention interventions for people living with HIV in high income settings. Sixty-three datasets from 46 primary studies were grouped by intervention, comparison group, and outcomes resulting in 17 unique combinations which were meta-analyzed and assessed for quality of evidence. Two intervention types reported statistically significant summary effects with high or moderate quality of evidence. These included comprehensive risk counseling and services and group level health education interventions.

High and moderate quality of evidence with a statistically significant summary effect was found for comprehensive risk counseling and services on sexual risk behaviour when compared to both active and attention controls. It can therefore be said with a high or moderate level of confidence that the true effect is likely to be close to the summary effect and that more research on the effects of this intervention on sexual risk behavior would likely not change the findings of the meta-analysis [9]. Although both summary effects were minimal (SMD <0.20), when compared to active controls (SMD = -0.15), the magnitude of effect for attention controls was larger (SMD = -0.35). This trend demonstrates the reduced effects of an intervention when compared to an active control versus an attention control. Similar results were found for group level health education interventions when compared to attention controls. Group level interventions, demonstrated statistically significant summary effects in reducing HIV/STI incidence with moderate quality of evidence

Moderate quality of evidence and statistically non-significant summary effects were found for individual level health education and housing assistance interventions in reducing sexual risk behavior when compared to attention controls. It can be said with moderate level of confidence that more research on these interventions will likely not change the results of this meta-analysis, and therefore these interventions are unlikely to reduce sexual risk behavior of people living with HIV.

Some group-level health education interventions were found to be effective or promising in reducing HIV incidence as well as HIV knowledge, attitudes, and beliefs, however variation in pooled effects and quality of evidence precludes a clear conclusion on the effectiveness of this intervention. Furthermore, while group-level health education interventions compared to attention controls were shown to be effective in reducing HIV/STI incidence they did not show statistically significant effects in reducing sexual risk behaviors. These findings demonstrate a need for further investigation.

The remaining combinations of interventions, comparison groups and outcomes resulted in low or very low quality of evidence and therefore no conclusive interpretation of the summary effects, whether statistically significant or non-significant, could be made. Low or very low quality of evidence stemmed from a variety of issues including: inadequate randomization, inadequate blinding of participants and personnel, limited number of studies, small sample sizes, heterogeneity of pooled effects, and indirect outcome measures [9].

Common characteristics among effective interventions include sessions that are: theory-based, tailored one-on-one interventions, typically grounded in counseling or case management, targeting multiple health concerns (beyond skills building in relation to safe sex), and delivered over a longer period of time (average of five months). These characteristics are similar to effective prevention interventions among people living with HIV identified by two previous reviews [4,7]. While one other review found individual level health education interventions as promising for HIV prevention among people living with HIV [7], the present review suggests that longer, more comprehensive individualized interventions are more likely to be effective in reducing sexual risk behaviors among people living with HIV. Individual level health education interventions are often focused on sexual risk reduction only and are typically shorter in duration (on average one month). In contrast, comprehensive risk counseling and services are tailored to address an individual’s sexual risk behavior in addition to unique life circumstances and health concerns, including mental health, substance use, and physical health. Nonetheless, it is important to acknowledge common barrier to implementation of comprehensive risk counseling and services, including: competing priorities, staff time, and limited financial resources.

Several factors may have contributed to the statistically significant, high or moderate quality interventions. Studies in this meta-analysis were grouped according to the CDC’s classification of behavioral interventions; however, the process of categorization of interventions may involve subjectivity.

Very few studies measured change in incidence of HIV/STIs, the most direct outcome measure indicating effectiveness of HIV/STI prevention interventions. Rather, a majority reported measures related to changes in sexual risk behavior. Such self-reported and indirect measures are subject to social desirability bias [76] and do not necessarily result in changes in HIV/STI incidence [9].

Less than half of the studies included in this review reported using appropriate measures for random sequence generation, reducing selection bias, and producing comparable groups in both intervention and control arms. A majority of studies did not report or were unclear about their reporting of blinding study participants; however, it is important to note it is not possible to ensure blinding of participants and personnel, given the nature of behavioral interventions. Across studies, the majority of other domains were rated ‘unclear’ as study authors failed to explicitly report on risk of bias items, particularly in the domain of reporting bias (96% of all studies). The overall lack of uniform risk of bias reporting practices may have also contributed to a lower quality of evidence. Future trials should aim to improve reporting in several key areas related to risk of bias judgments.

Finally, while this review is conducted among people living with HIV, there is wide diversity within this group (e.g. MSM, heterosexual men, women, older adults, youth and individuals from a variety of ethnic backgrounds). Such diversity makes it difficult to estimate the true effect of an intervention for each group separately, given the aggregation of data for various sub-populations to produce a summary effect. Similarly, multiformity of each intervention type (i.e. content, intensity, duration) and variations of length of follow-up across studies may have an impact on the results of the meta-analysis. Such variation also makes it difficult to generalize what intervention strategies contribute to positive impacts, and in what populations they can be successfully implemented.

To our knowledge, this is the most comprehensive systematic review and meta-analysis available on behavioral prevention interventions for people living with HIV in high income settings. Within a landscape of limited public health funding, the findings of the present review can be used as a tool to support public health decisionmaking by assisting in the prioritization and allocation of funds for HIV prevention strategies. While other systematic reviews have been conducted on this topic, this review stands out in its use of the CDC classifications of behavioral interventions and GRADE to summarize the quality of available evidence, as well as its assessment of outcome measures such as HIV/STI incidence and knowledge, attitudes and beliefs, in addition to commonly reported sexual risk behavioral [4,7,8].

Few published reviews were identified on the topic. A 2006 meta-analysis [4] of 12 studies by Crepaz et al. demonstrated that prevention interventions significantly reduced unprotected sex and acquisition of STIs. A 2014 meta-analysis of 21 studies by Yin et al. [8] demonstrated a short-term impact of interventions on self-reported unprotected anal intercourse, but no conclusions on long-term effects. A 2014 systematic review of 48 studies on the same topic by Crepaz et al. [7] evaluated each study against established criteria for study design, implementation, analysis, and strength of findings to assess risk of bias and intervention effects. Reviewers identified 14 studies with low risk of bias and significant positive intervention effects, while the remaining 34 studies had high risk of bias and non-significant positive intervention effects. While Crepaz and colleagues evaluated studies on an individual basis, rather than evaluating the body of evidence, its findings are similar to that of present review, pointing to a lack of well-designed and rigorously evaluated primary research.

While there are a number of primary studies evaluating the effects of behavioral interventions among people living with HIV, the dearth of high quality primary literature and reviews on the topic make drawing conclusions regarding effective prevention interventions difficult. Future research should focus on designing and evaluating such interventions within a more rigorous framework.

4.1 Limitations

Several limitations may restrict the validity of the present review. The inclusion of peer-reviewed studies published in English only may have contributed to reporting bias. Another limitation is the lack of subgroup or sensitivity analysis, however this was justified as the number of data sets included in each meta-analysis was small given the numerous intervention, comparison group and outcome combinations meta-analyzed separately. Additionally, non-randomized controlled trials were included in our analysis, which may have introduced selection bias and skewed the results. However, this risk has been minimized by downgrading quality of evidence through assessing risk of bias as part of the GRADE process. Finally, while this review was inclusive of all studies conducted in high income countries, only US studies met eligibility criteria for inclusion. Specific regional, racial, economic, and political and health system-related characteristics unique to the US may limit the generalizability of results and warrant caution in interpretation.

5 Conclusions

People living with HIV are at risk of transmission, in addition to contracting different STIs as well as other strains of HIV. Theory-based behavioral interventions provide an opportunity to reduce risk behavior and HIV/STI transmission among this population. Interventions identified as having statistically significant pooled effects with a high or moderate quality of evidence should be considered by clinics, AIDS service organizations, community-based organizations, and public health agencies. Moreover, given the number of interventions with low or very low quality of evidence, researchers should commit to conducting rigorous evaluations and high quality reporting of studies assessing the effectiveness of HIV prevention interventions. Moving evidence-based prevention research for people living with HIV into practice is one critical step in making a greater impact on the HIV epidemic.

References

  • [1]

    Centers for Disease Control and Prevention. Monitoring Selected National HIV Prevention and Care Objectives by Using HIV Surveillance Data - United States and 6 Dependent Areas - 2012. http://www.cdc.gov/hiv/library/reports/surveillance/ Accessed 6 October, 2015. HIV Surveillance Supplemental Report 2014;19(3). 

  • [2]

    Public Health Agency of Canada. HIV and AIDS in Canada: Surveillance Report to December 31, 2013. Minister of Public Works and Goverment Services Canada; 2014. Google Scholar

  • [3]

    Kalichman SC, Pellowski J, Turner C. Prevalence of sexually transmitted co-infections in people living with HIV/AIDS: Systematic review with implications for using HIV treatments for prevention. Sex Transm Infect 2011;87(3):183-90. Google Scholar

  • [4]

    Crepaz N, Lyles CM, Wolitski RJ, Passin WF, Rama SM, Herbst JH, et al. Do prevention interventions reduce HIV risk behaviours among people living with HIV? A meta-analytic review of controlled trials. AIDS 2006;20(2):143-57. Google Scholar

  • [5]

    Crepaz N, Marks G, Liau A, Mullins MM, Aupont LW, Marshall KJ, et al. Prevalence of unprotected anal intercourse among HIV-diagnosed MSM in the United States: A meta-analysis. AIDS 2009;23(13):1617-29. Google Scholar

  • [6]

    Centers for Disease Control. Compendium of Evidence-Based Interventions and Best Practices for HIV Prevention. http://www.cdc.gov/hiv/prevention/research/compendium/rr/index.html Accessed 6 Oct 2015. 2014. 

  • [7]

    Crepaz N, Tungol-Ashmon MV, Higa DH, Vosburgh W, Mullins MM, Barham T, et al. A systematic review of interventions for reducing HIV risk behaviors among people living with HIV in the United States, 1988-2012. AIDS 2014;28(5):13. Google Scholar

  • [8]

    Yin L, Wang N, Vermund SH, Shepherd BE, Ruan Y, Shao Y, et al. Sexual risk reduction for HIV-infected persons: A meta-analytic review of “positive prevention” randomized clinical trials. PLoS ONE 2014;9(9):e107652. CrossrefGoogle Scholar

  • [9]

    Schunemann H, Brozek J, Guyatt G, Oxman A. GRADE handbook for grading quality of evidence and strength of recommendations. Updated October 2013. Available from www.guidelinedevelopment.org/handbook. The GRADE Working Group; 2013. 

  • [10]

    McMaster University. GRADEpro GDT: GRADEpro Guideline Development Tool (Software). Developed by Evidence Prime, Inc. Available from www.gradepro.org. 2015. 

  • [11]

    The World Bank. Country and Lending Groups. http://data.worldbank.org/about/country-classifications/country-and-lending-groups Accessed 6 Oct 2015. 2014. 

  • [12]

    Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med 2009;6(7):e1000097. PubMedCrossrefGoogle Scholar

  • [13]

    Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction - GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 2011;64(4):383-94. Google Scholar

  • [14]

    Balshem H, Helfand M, Schunemann HJ, Oxman AD, Kunz R, Brozek J, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 2011;64(4):401-6. Google Scholar

  • [15]

    Centers for Disease Control and Prevention. Glossary of HIV Prevention Terms. http://www.cdc.gov/hiv/policies/funding/announcements/PS12-1201/pdf/Attachment-I.pdf Accessed 6 Oct 2015. 2012. 

  • [16]

    Washington State Department of Health. Effective HIV Interventions and Strategies (Volume 2) http://www.doh.wa.gov/YouandYourFamily/IllnessandDisease/HIVAIDS/Prevention/Interventions/InterventionTypes.aspx Accessed 6 Oct 2015. 2009. 

  • [17]

    Centers for Disease Control and Prevention. Evaluation Guidance Handbook: Strategies for Implementing the Evaluation Guidance for CDC-Funded HIV Prevention Programs http://www.cdc.gov/hiv/topics/evaluation/health_depts/guidance/strat-handbook/pdf/guidance.pdf Accessed 6 Oct 2015. 2002. 

  • [18]

    South Carolina Department of Health and Environmental Control. Recommended HIV Prevention Interventions. http://www.scdhec.gov/health/disease/stdhiv/docs/pp_CH%204%20Interventions.pdf Accessed 6 Oct 2015. 2011. 

  • [19]

    Public Health Agency of Canada. HIV transmission risk: A summary of the evidence. http://www.catie.ca/sites/default/files/HIV-TRANSMISSION-RISK-EN.pdf Accessed 6 Oct 2015. Public Health Agency of Canada; 2012. 

  • [20]

    Centers for Disease Control and Prevention. Effective Interventions - HIV prevention that Works. www.effectiveinterventions.cdc.gov Accessed 6 Oct 2015. 2014. 

  • [21]

    Evidence Partners. Distiller SR. www.systematic-review.ca (software). Ottawa, Canada; 2014. 

  • [22]

    Guyatt GH, Oxman AD, Vist G, Kunz R, Brozek J, Alonso-Coello P, et al. GRADE guidelines: 4. Rating the quality of evidence - study limitations (risk of bias). J Clin Epidemiol 2011;64(4):407-15. Google Scholar

  • [23]

    Higgins JPT, Altman DG, Sterne JAC. Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. Available from www.cochrane-handbook.org. The Cochrane Collaboration; 2011. 

  • [24]

    Borenstein M, Hedges L, Higgins J, Rothstein H. Comprehensive meta-analysis version 2. 2005. Google Scholar

  • [25]

    Cohen J. Statistical power analysis for the behavioral sciences. Psychology Press; 1988. Google Scholar

  • [26]

    Guyatt GH, Oxman AD, Santesso N, Helfand M, Vist G, Kunz R, et al. GRADE guidelines: 12. Preparing Summary of Findings tables - binary outcomes. J Clin Epidemiol 2013;66(2):158-72. Google Scholar

  • [27]

    Guyatt GH, Thorlund K, Oxman AD, Walter SD, Patrick D, Furukawa TA, et al. GRADE guidelines: 13. Preparing Summary of Findings tables and evidence profiles - continuous outcomes. J Clin Epidemiol 2013;66(2):173-83. Google Scholar

  • [28]

    Coleman CL, Jemmott L, Jemmott JB, Strumpf N, Ratcliffe S. Development of an HIV risk reduction intervention for older seropositive African American men. AIDS Patient Care STDS 2009;23(8):647-55. Google Scholar

  • [29]

    Cosio D, Heckman TG, Anderson T, Heckman BD, Garske J, McCarthy J. Telephone-administered motivational interviewing to reduce risky sexual behavior in HIV-infected rural persons: A pilot randomized clinical trial. Sex Transm Dis 2010;37(3):140-6. Google Scholar

  • [30]

    Illa L, Echenique M, Jean GS, Bustamante-Avellaneda V, Metsch L, Mendez-Mulet L, et al. Project ROADMAP: Reeducating older adults in maintaining AIDS prevention: A secondary intervention for older HIV-positive adults. AIDS Educ Prev 2010;22(2):138-47. Google Scholar

  • [31]

    Jones D, Kashy D, Villar-Loubet O, Weiss S. Enhancing acceptability and use of sexual barrier products among HIV concordant and discordant couples. AIDS and Behavior 2013;17(6):2185-93. Google Scholar

  • [32]

    Kalichman SC, Cherry C, Kalichman MO, Amaral CM, White D, Pope H, et al. Integrated behavioral intervention to improve HIV/AIDS treatment adherence and reduce HIV transmission. Am J Public Health 2011;101(3):531-8. Google Scholar

  • [33]

    Klein CH, Lomonaco CG, Pavlescak R, Card JJ. WiLLOW: Reaching HIV-positive African-American women through a computer-delivered intervention. AIDS and Behavior 2013;17(9):3013-23. Google Scholar

  • [34]

    Lovejoy TI, Heckman TG, Suhr JA, Anderson T, Heckman BD, France CR. Telephone-administered motivational interviewing reduces risky sexual behavior in HIV-positive late middle-age and older adults: A pilot randomized controlled trial. AIDS and Behavior 2011;15(8):1623-34. Google Scholar

  • [35]

    Marhefka SL, Buhi ER, Baldwin J, Chen H, Johnson A, Lynn V, et al. Effectiveness of healthy relationships video-group-A videoconferencing group intervention for women living with HIV: Preliminary findings from a randomized controlled trial. Telemed J E Health 2014 Feb;20(2):128-34. CrossrefPubMedGoogle Scholar

  • [36]

    McKirnan DJ, Tolou-Shams M, Courtenay-Quirk C. The Treatment Advocacy Program: A randomized controlled trial of a peer-led safer sex intervention for HIV-infected men who have sex with men. J Consult Clin Psychol 2010;78(6):952-63. Google Scholar

  • [37]

    Richardson JL, Milam J, McCutchan A, Stoyanoff S, Bolan R, Weiss J, et al. Effect of brief safer-sex counseling by medical providers to HIV-1 seropositive patients: A multi-clinic assessment. AIDS 2004;18(8):1179-86. Google Scholar

  • [38]

    Rosser BR, Hatfield LA, Miner MH, Ghiselli ME, Lee BR, Welles SL. Effects of a behavioral intervention to reduce serodiscordant unsafe sex among HIV positive men who have sex with men: The Positive Connections randomized controlled trial study. J Behav Med 2010;33(2):147-58. Google Scholar

  • [39]

    Safren SA, O’Cleirigh CM, Skeer M, Elsesser SA, Mayer KH. Project enhance: A randomized controlled trial of an individualized HIV prevention intervention for HIV-infected men who have sex with men conducted in a primary care setting. Health Psychol 2013;32(2):171-9. Google Scholar

  • [40]

    Sikkema KJ, Wilson PA, Hansen NB, Kochman A, Neufeld S, Ghebremichael MS, et al. Effects of a coping intervention on transmission risk behavior among people living with HIV/AIDS and a history of childhood sexual abuse. J Acquir Immune Defic Syndr 2008;47(4):506-13. Google Scholar

  • [41]

    Sikkema KJ, Abler L, Hansen NB, Wilson PA, Drabkin AS, Kochman A, et al. Positive choices: Outcomes of a brief risk reduction intervention for newly HIV-diagnosed men who have sex with men. AIDS and Behavior 2014;18(9):1808-19. Google Scholar

  • [42]

    Teti M, Bowleg L, Cole R, Lloyd L, Rubinstein S, Spencer S, et al. A mixed methods evaluation of the effect of the protect and respect intervention on the condom use and disclosure practices of women living with HIV/AIDS. AIDS and Behavior 2010 Jun;14(3):567-79. CrossrefPubMedGoogle Scholar

  • [43]

    El-Bassel N, Jemmott JB, Landis JR, Pequegnat W, Wingood GM, Wyatt GE, et al. National Institute of Mental Health Multisite Eban HIV/STD Prevention Intervention for African American HIV Serodiscordant Couples: A cluster randomized trial. Arch Intern Med 2010;170(17):1594-601. Google Scholar

  • [44]

    Gilbert P, Ciccarone D, Gansky SA, Bangsberg DR, Clanon K, McPhee SJ, et al. Interactive “Video Doctor” counseling reduce: drug and sexual risk behaviors among HIV-positive patients in diverse outpatient settings. PLoS ONE 2008;3(4):e1988. CrossrefGoogle Scholar

  • [45]

    Healthy Living Project Team. Effects of a behavioral intervention to reduce risk of transmission among people living with HIV: the healthy living project randomized controlled study. J Acquir Immune Defic Syndr 2007;44(2):213-21. Google Scholar

  • [46]

    Kalichman SC, Rompa D, Cage M, DiFonzo K, Simpson D, Austin J, et al. Effectiveness of an intervention to reduce HIV transmission risks in HIV-positive people. American Journal of Preventive Medicine 2001;21(2):84-92. Google Scholar

  • [47]

    Kurth AE, Spielberg F, Cleland CM, Lambdin B, Bangsberg DR, Frick PA, et al. Computerized counseling reduces HIV-1 Viral load and sexual transmission risk: Findings from a randomized controlled trial. J Acquir Immune Defic Syndr 2014;65(5):611-20. CrossrefPubMedGoogle Scholar

  • [48]

    Margolin A, Avants SK, Warburton LA, Hawkins KA, Shi J. A randomized clinical trial of a manual-guided risk reduction intervention for HIV-positive injection drug users. Health Psychol 2003;22(2):223-8. Google Scholar

  • [49]

    Metsch LR, Pereyra M, Messinger S, Del RC, Strathdee SA, Anderson-Mahoney P, et al. HIV transmission risk behaviors among HIV-infected persons who are successfully linked to care. Clinical Infectious Diseases 2008;47(4):577-84. Google Scholar

  • [50]

    Myers JJ, Shade SB, Rose CD, Koester K, Maiorana A, Malitz FE, et al. Interventions delivered in clinical settings are effective in reducing risk of HIV transmission among people living with HIV: Results from the Health Resources and Services Administration (HRSA)’s special projects of national significance initiative. AIDS and Behavior 2010;14(3):483-92. Google Scholar

  • [51]

    Petry NM, Weinstock J, Alessi SM, Lewis MW, Dieckhaus K. Group-based randomized trial of contingencies for health and abstinence in HIV patients. J Consult Clin Psychol 2010;78(1):89-97. Google Scholar

  • [52]

    Sikkema KJ, Hansen NB, Kochman A, Santos J, Watt MH, Wilson PA, et al. The development and feasibility of a brief risk reduction intervention for newly HIV-diagnosed men who have sex with men. J Community Psychol 2011;39(6):717-32. Google Scholar

  • [53]

    Velasquez MM, von Sternberg K, Johnson DH, Green C, Carbonari JP, Parsons JT. Reducing sexual risk behaviors and alcohol use among HIV-positive men who have sex with men: A randomized clinical trial. J Consult Clin Psychol 2009;77(4):657-67. Google Scholar

  • [54]

    Wingood GM, DiClemente RJ, Mikhail I, Lang DL, McCree DH, Davies SL, et al. A randomized controlled trial to reduce HIV transmission risk behaviors and sexually transmitted diseases among women living with HIV: The WiLLOW Program. J Acquir Immune Defic Syndr 2004;37(Suppl 2):S58-S67. Google Scholar

  • [55]

    Wolitski RJ, Gomez CA, Parsons JT. Effects of a peer-led behavioral intervention to reduce HIV transmission and promote serostatus disclosure among HIV-seropositive gay and bisexual men. AIDS 2005;19(Suppl 1):S99-S109. CrossrefGoogle Scholar

  • [56]

    Wolitski RJ, Kidder DP, Pals SL, Royal S, Aidala A, Stall R, et al. Randomized trial of the effects of housing assistance on the health and risk behaviors of homeless and unstably housed people living with HIV. AIDS and Behavior 2010;14(3):493-503. Google Scholar

  • [57]

    Fogarty LA, Heilig CM, Armstrong K, Cabral R, Galavotti C, Gielen AC, et al. Long-term effectiveness of a peer-based intervention to promote condom and contraceptive use among HIV-positive and at-risk women. Public Health Reports 2001;116(Suppl 1):103-19. CrossrefGoogle Scholar

  • [58]

    Naar-King S, Wright K, Parsons JT, Frey M, Templin T, Lam P, et al. Healthy choices: Motivational enhancement therapy for health risk behaviors in HIV-positive youth. AIDS Educ Prev 2006;18(1):1-11. Google Scholar

  • [59]

    Purcell DW, Latka MH, Metsch LR, Latkin CA, Gomez CA, Mizuno Y, et al. Results from a randomized controlled trial of a peer-mentoring intervention to reduce HIV transmission and increase access to care and adherence to HIV medications among HIV-seropositive injection drug users. J Acquir Immune Defic Syndr 2007;46(Suppl. 2):S35-S47. Google Scholar

  • [60]

    Rotheram-Borus MJ, Swendeman D, Comulada WS, Weiss RE, Lee M, Lightfoot M. Prevention for substance-using HIV-positive young people: telephone and in-person delivery. J Acquir Immune Defic Syndr 2004;37(Suppl 2):S68-S77. Google Scholar

  • [61]

    Sorensen JL, Dilley J, London J, Okin RL, Delucchi KL, Phibbs CS. Case management for substance abusers with HIV/AIDS: A randomized clinical trial. Am J Drug Alcohol Abuse 2003;29(1):133-50. Google Scholar

  • [62]

    Wyatt GE, Longshore D, Chin D, Carmona JV, Loeb TB, Myers HF, et al. The efficacy of an integrated risk reduction intervention for HIV-positive women with child sexual abuse histories. AIDS and Behavior 2004;8(4):453-62. Google Scholar

  • [63]

    Lovejoy TI, Heckman TG, Sikkema KJ, Hansen NB, Kochman A. Changes in sexual behavior of HIV-infected older adults enrolled in a clinical trial of standalone group psychotherapies targeting depression. AIDS and Behavior 2015;19(1):1-8. Google Scholar

  • [64]

    Williams JK, Wyatt GE, Rivkin I, Ramamurthi HC, Li X, Liu H. Risk reduction for HIV-positive African American and Latino men with histories of childhood sexual abuse. Arch Sex Behav 2008;37(5):763-72. Google Scholar

  • [65]

    Golin CE, Earp JA, Grodensky CA, Patel SN, Suchindran C, Parikh M, et al. Longitudinal effects of SafeTalk, a motivational interviewing-based program to improve safer sex practices among people living with HIV/AIDS. AIDS and Behavior 2012;16(5):1182-91. Google Scholar

  • [66]

    Rose CD, Courtenay-Quirk C, Knight K, Shade SB, Vittinghoff E, Gomez C, et al. HIV intervention for providers study: A randomized controlled trial of a clinician-delivered HIV risk-reduction intervention for HIV-positive people. J Acquir Immune Defic Syndr 2010;55(5):572-81. Google Scholar

  • [67]

    Mausbach BT, Semple SJ, Strathdee SA, Zians J, Patterson TL. Efficacy of a behavioral intervention for increasing safer sex behaviors in HIV-positive MSM methamphetamine users: Results from the EDGE study. Drug and Alcohol Dependence 2007;87(2-3):249-57. PubMedCrossrefGoogle Scholar

  • [68]

    Grinstead O, Zack B, Faigeles B. Reducing postrelease risk behavior among HIV seropositive prison inmates: The health promotion program. AIDS Educ Prev 2001;13(2):109-19. Google Scholar

  • [69]

    Lapinski MK, Randall LM, Peterson M, Peterson A, Klein KA. Prevention options for positives: The effects of a health communication intervention for men who have sex with men living with HIV/AIDS. Health Commun 2009;24(6):562-71. Google Scholar

  • [70]

    Lewis JR, Boyle DP, Lewis LS, Evans M. Reducing AIDS and substance abuse risk factors among homeless, HIV-infected, drug-using persons. Res Soc Work Pract 2000;10(1):15-33. Google Scholar

  • [71]

    Margolin A, Schuman-Olivier Z, Beitel M, Arnold RM, Fulwiler CE, Avants SK. A preliminary study of spiritual self-schema (3-S+) therapy for reducing impulsivity in HIV-positive drug users. J Clin Psychol 2007;63(10):979-99. Google Scholar

  • [72]

    Schwarcz SK, Chen YH, Murphy JL, Paul JP, Skinta MD, Scheer S, et al. A randomized control trial of personalized cognitive counseling to reduce sexual risk among HIV-infected men who have sex with men. AIDS Care 2013;25(1):1-10. Google Scholar

  • [73]

    Rotheram-Borus MJ, Lee MB, Murphy DA, Futterman D, Duan N, Birnbaum JM, et al. Efficacy of a preventive intervention for youths living with HIV. Am J Public Health 2001;91(3):400-5. Google Scholar

  • [74]

    Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, et al. GRADE guidelines: 7. Rating the quality of evidence - inconsistency. J Clin Epidemiol 2011;64(12):1294-302. Google Scholar

  • [75]

    Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, et al. GRADE guidelines: 5. Rating the quality of evidence - publication bias. J Clin Epidemiol 2011;64(12):1277-82. Google Scholar

  • [76]

    Fisher R. Social desirability bias and the validity of indirect questioning. J Cons Res 1993;20:303-15. CrossrefGoogle Scholar

Footnotes

    About the article

    Received: 2016-12-08

    Accepted: 2017-10-27

    Published Online: 2017-12-14


    Conflict of interest: Authors state no conflict of interest. The authors received no specific funding for this work.


    Citation Information: Open Medicine, Volume 12, Issue 1, Pages 450–467, ISSN (Online) 2391-5463, DOI: https://doi.org/10.1515/med-2017-0064.

    Export Citation

    © 2017 Jason Globerman et al.. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. BY-NC-ND 4.0

    Supplementary Article Materials

    Citing Articles

    Here you can find all Crossref-listed publications in which this article is cited. If you would like to receive automatic email messages as soon as this article is cited in other publications, simply activate the “Citation Alert” on the top of this page.

    [1]
    Aven Sidhu, Rohan Kakkar, and Osamah Alenezi
    Reviews on Recent Clinical Trials, 2019, Volume 14, Number 1, Page 61
    [2]
    Tam Nguyen, Bach Tran, Mercerdes Fleming, Manh Pham, Long Nguyen, Huong Le, Anh Nguyen, Huong Le, Thang Nguyen, Van Hoang, Xuan Le, Quan Vuong, Manh Ho, Van Dam, Thu Vuong, Ha Do, Vu Nguyen, Huong Nguyen, Huyen Do, Phuong Doan, Hai Nguyen, Carl Latkin, Cyrus Ho, and Roger Ho
    International Journal of Environmental Research and Public Health, 2018, Volume 15, Number 11, Page 2567
    [3]
    Rui Zhang, Ling Chen, Ya deng Cui, and Ge Li
    SAHARA-J: Journal of Social Aspects of HIV/AIDS, 2018, Volume 15, Number 1, Page 31

    Comments (0)

    Please log in or register to comment.
    Log in