Development of an Internet Intervention to Address
Behaviors Associated with Skin Cancer Risk among Young Adults
Carolyn Heckman a, Susan Darlowa, Teja Munshia, Carolyn Carusob, Lee Ritterbandb,c, Stephanie Raivitcha, Linda Fleishera, Sharon Manned
Purpose: Skin cancer is the most common cancer in the US, and its incidence is increasing. The major risk factor for skin cancer is exposure to ultraviolet radiation (UV). Young adults tend to expose themselves to large amounts of UV and engage in minimal skin protection, which increases their skin cancer risk. Interventions are needed to address risk behaviors among young adults that may lead to skin cancer. The nternet offers a cost-effective way to widely disseminate efficacious interventions. The current paper describes the development of an online skin cancer risk reduction intervention (UV4.me) for young adults.
Procedures: The iterative development process for UV4.me followed best-practice guidelines and included the following activities: individual interviews, focus groups, content development by the expert team, acceptability testing, cognitive interviewing for questionnaires, quality control testing, usability testing, and a pilot randomized controlled trial. Participant acceptability and usability feedback was assessed.
Principal Results: The development process produced an evidence-informed intervention that is individually-tailored, interactive, and multimedia in nature based on the Integrative Model of Behavior Prediction, a model for internet interventions, and other best-practice recommendations, expert input, as well as user acceptability and usability feedback gathered before, during, and after development.
Major Conclusions: Development of an acceptable intervention intended to have a significant public health impact requires a relatively large investment in time, money, expertise, and ongoing user input. Lessons learned and recommendations are discussed. The comprehensive process used may help prepare others interested in creating similar behavioral health interventions.
Development of an Internet Intervention to Address
>Behaviors Associated with Skin Cancer Risk among Young Adults
Skin cancer is the most common cancer, with nearly five million cases treated annually in the US; its incidence has been increasing in recent years (Donaldson & Coldiron, 2011; Gordon, 2013; Nikolaou & Stratigos, 2014; Tuong, Cheng, & Armstrong, 2012; USDHHS, 2014). Most skin cancers are non-melanomas, which have been estimated to be increasing by 2.6% in incidence per year (1999-2006) (Donaldson & Coldiron, 2011), and melanomas have been increasing at a rate of 1.5-6% per year (2002-2011) (USDHHS, 2014). Known risk factors for skin cancer include personal or family history of skin cancer, fair skin, and ultraviolet radiation (UV) from the sun and/or indoor tanning (Goldberg et al., 2007; Lazovich et al., 2010; Markovic et al., 2007; Nikolaou & Stratigos, 2014; Psaty, Scope, Halpern, & Marghoob, 2010; Qureshi, Zhang, & Han, 2011; Siskind, Aitken, Green, & Martin, 2002; Vishvakarman & Wong, 2003). It is common for young adults (e.g., aged 18-25 years) to expose themselves to large amounts of UV without proper skin protection (e.g., wearing adequate sunscreen) (Buller et al., 2011; Coups, Manne, & Heckman, 2008; Heckman, Coups, & Manne, 2008; Stanton, Janda, Baade, & Anderson, 2004). For example, US adolescents have had the lowest skin protection rates of all age groups (Stanton et al., 2004), with only 39% applying sunscreen when going outdoors in the summer (Cokkinides et al., 2006). US adolescents also engage in increased exposure to natural and artificial UV as they move into adulthood (MacNeal & Dinulos, 2007). Invasive skin cancer is the second most diagnosed cancer among young adults (Bleyer & Barr, 2009). For these reasons, it is important to have interventions that are effective in addressing skin cancer risk behaviors among young adults. Indeed, the Surgeon General has recently published a call to action to prevent skin cancer (USDHHS, 2014).
Numerous skin cancer prevention interventions have been developed. Systematic reviews and meta-analyses have found some of these interventions to be efficacious for specific populations (Horsham, Sendall, et al., 2014; Lin, Eder, & Weinmann, 2011; Rodrigues, Sniehotta, & Araujo-Soares, 2013; Saraiya, Glanz, et al., 2004; Williams, Grogan, et al., 2013). However, many of these interventions have two key limitations. First, many have emphasized only education and awareness, which tend to result in limited changes in actual behavior, particularly changes that are sustained over time (Aarestrup et al., 2014; Hart & Demarco, 2008; Horsham et al., 2014; Keeney et al., 2009; Roberts & Black, 2009). However, one type of intervention that has shown promise in modifying behaviors associated with skin cancer development is those that focus on the negative effects on appearance of UV exposure and lack of protection (Williams, Grogan, Clark-Carter, & Buckley, 2013). For example, a series of studies by Mahler and colleagues found that taking and showing young adults photos of their existing facial UV damage resulted in decreased UV exposure and increased skin protection (Gibbons, Gerrard, Lane, Mahler, & Kulik, 2005; Mahler, Kulik, Gerrard, & Gibbons, 2013; Mahler et al., 2005). Another limitation of many skin cancer prevention interventions is that they are often delivered in-person, which can be difficult to scale up for wider dissemination. Internet interventions, alternatively, can be cost-effective and disseminated widely. Since 97% of US young adults aged 18-29 years use the internet (Pew_Research_Internet_Project, 2014), internet-based interventions may have considerable potential to appeal to young adults and provide a greater likelihood of being utilized than other types of interventions. Although a couple of studies have reported on the development and/or testing of online skin cancer prevention interventions (Bowen et al., 2012; Kimlin & Parisi, 2001), these have focused on children or melanoma patients and their family members. To our knowledge, this is the first paper describing the development or formative assessment of an online intervention to address risk behaviors among at-risk young adults that may lead to skin cancer.
Internet interventions more extensively informed by a theoretical framework tend to be more efficacious in modifying health behaviors than those that use theory less extensively (Webb, Joseph, Yardley, & Michie, 2010). The theoretical framework for the current intervention was adapted from the Integrative Model of Behavior Prediction (IM) (Fishbein, Hennessy, Yzer, & Douglas, 2003). The IM includes background variables such as demographics; cognitive variables such as beliefs, attitudes, norms, and self-efficacy; intentions; and behavior. We also emphasized appearance concerns, which is a major factor associated with tanning behavior among young adults (Cafri et al., 2006; Danoff-Burg & Mosher, 2006; Hillhouse, Turrisi, Stapleton, & Robinson, 2008). The intervention is targeted to young adults, individually-tailored based on IM variables, and interactive utilizing multiple media formats, characteristics that have been shown to be components of effective health behavior interventions (Bewick et al., 2008; Bock, Graham, Whiteley, & Stoddard, 2008; Davies, Spence, Vandelanotte, Caperchione, & Mummery, 2012; Griffiths, Lindenmeyer, Powell, Lowe, & Thorogood, 2006; Lustria et al., 2013; van den Berg, Schoones, & Vliet Vlieland, 2007).
The purpose of the current paper is to describe the systematic and comprehensive process of development of UV4.me, an online intervention to address risk behaviors that may lead to skin cancer among young adults. This description provides 1) an example of the application of best-practice guidelines, 2) our acceptability and usability findings, as well as 3) more specific lessons learned for others contemplating or preparing for internet health behavior intervention development.
The development of the web intervention and preparation for a randomized controlled trial (RCT) involved a comprehensive process including several mixed-method steps incorporating input from stakeholders (i.e., multi-disciplinary experts and the target population) over a three-year period. The process was guided by intervention development and assessment guidelines for behavioral therapy, web-based interventions, and health communications programs including health literacy best-practices (Barak, Klein, & Proudfoot, 2009; Danaher & Seeley, 2009; Fleisher et al., 2014; National_Cancer_Institute, 2002; Ritterband & Tate, 2009; Ritterband, Thorndike, Cox, Kovatchev, & Gonder-Frederick, 2009). In preparation for future efficacy and effectiveness trials, the phases of intervention development and assessment included intervention planning, content and website development, initial assessment and revision, and pilot testing. Prior web health intervention development projects have used various aspects or combinations of these development steps (Berry et al., 2010; Fergus et al., 2014; Michie et al., 2012; Moore et al., 2013; Pachankis, Lelutiu-Weinberger, Golub, & Parsons, 2013; Riiser, Londal, Ommundsen, Sundar, & Helseth, 2013; Villegas et al., 2014; Wolpin et al., 2014), but few have reported on all of them.
- Methods and Results
The current project was developed by a multidisciplinary team, which included individuals with expertise in skin cancer prevention, young adults, internet interventions, psychology, qualitative and quantitative research methods, health literacy/communication, computer programming, psychometrics, and instructional design. The four developmental phases of this mixed-methods project are illustrated in Table 1. The methods and results from each step are described below. This project was approved and monitored by a cancer center Institutional Review Board, and informed consent was obtained from research participants.
- Phase 1. Intervention Planning
Individual interviews . We recruited a convenience sample of 25 young adults participating in an in-person skin cancer prevention randomized controlled intervention trial from a university campus in Philadelphia (average age = 23 years [range from 20 to 25 years], 64% white, 72% female). The purpose of these individual interviews was to have participants rate proposed topics and web features on a 5-point Likert-type scale (See Table 2) and offer suggestions and feedback prior to online development and to support an application for funding. Participants attended an in-person session in which cognitive interviews were conducted. Participants were asked to “think aloud” (Willis, 2005) as they responded to the draft online baseline survey and reviewed a tailored skin cancer prevention pamphlet from a prior successful intervention (S. Manne et al., 2010). The tailored color pamphlet focused on constructs previously found to predict cancer prevention behaviors such as perceived risk, salience and coherence (the perception that a behavior is consistent with beliefs about how to protect health), benefits and barriers, social influence (e.g., doctor’s recommendations), self-efficacy, and intentions. Additionally, images under consideration for inclusion in the intervention that focused on appearance concerns (e.g., UV damage and age-progression photos) were presented. Participants were asked a series of questions about what they liked, did not like, and what suggestions they had about the survey, pamphlet, and study, for example, about games to incorporate into the website, potential incentives, recruitment, and strategies to minimize attrition. Responses were summarized by the first author. Feedback at this stage was positive (See Table 2). This feedback helped inform decisions about recruitment/enrollment, surveys, web intervention topics and features, and incentives. For example, we decided to include an avatar activity, and a decision was made against the use of Twitter or text-messaging due to their low ratings by participants. Although we were interested in using Facebook, we decided it would not be feasible for the current project given time and budget constraints but would be considered for future projects.
Focus Groups. Once project funding was secured, a convenience sample of young adults recruited from the sponsoring institution and the local community participated in two sets of focus groups of 4-8 people each, to assist in shaping the content of the web program. As material differed across the groups, some individuals participated more than once. The first set of focus groups (n = 4) met twice and reviewed potential images and provided suggestions for general web program topics and specific activities that were identified during the individual interviews. Preliminary content was drafted and some images chosen in preparation for the second set of focus groups, which were held with groups of 6-8 young adults during four meetings. This set of focus groups reviewed other images and provided more specific suggestions about how to target and tailor material for young adults. The groups reviewed initial mock-ups of the intervention introduction, and a number of key areas of focus of the intervention, including why people tan, why people should not tan, indoor tanning, skin damage, skin cancer, and decreasing tanning. The groups also provided feedback and suggestions for the theme of the website and how to engage young adults with a skin cancer prevention website, in response to a series of interview questions. Feedback and suggestions about the interactive avatar activity that would demonstrate the effects of UV exposure and protection on an animated character were also gathered (e.g., what should it look like, should it be individually tailored, how should it work, how should it be integrated into the website). Qualitative feedback was grouped into themes by a research staff member and summarized for the research team. The focus group findings helped us select and refine the initial theme, topics, content, images, and activities to be used in the web program as well as plans for how to make the web program engaging to young adults. The findings also helped to identify areas of content that participants might have difficulty understanding (e.g., complicated figures).
2.2. Phase 2. Content and Website Development
Synthesis of Material into Website Content. Based on a synthesis of the individual interviews, focus groups, our expertise, and the literature (Barak et al., 2009; Ritterband & Tate, 2009; Ritterband et al., 2009), the multi-disciplinary team collaborated to create the design, modules, and other activities to be included in the web program (See Table 3). The intervention was intended to be interactive, tailored, utilize multiple media formats, and maximize participant engagement while minimizing burden. Twelve modules were created, each with content related to a specific topic (e.g., sunscreen, indoor tanning, skin cancer; See Figure 1) determined to be important in terms of risk or protective behaviors and their correlates. In addition, several other more general website sections (e.g., avatar, MyStuff – a printable summary of tailored goals and recommendations) were developed. Each module was expected to take about 10 minutes to review, and it was assumed that many participants would not view all the modules, and in fact, may only use the website once, so an attempt was made to have each module stand alone and be as focused as possible on encouraging behavior change.
Tailoring algorithms were created to direct participants to focus on certain modules first based on their responses to a few initial questions (e.g., the indoor tanning module was recommended if participants said they tanned indoors). Throughout the web program, participants were asked questions and were provided with tailored feedback (e.g., “Do you know people who tan? If so, how likely are they to affect your choice to tan or not?”). On some pages, additional information, such as definitions, was available by clicking on or hovering over an icon, image, or text. Once the content was finalized, many images were purchased from online stock photo websites, and online searches were conducted to identify some of the included videos (n=7) and comics (n=7) related to the topic areas (permission was obtained to use identified videos and comics). All video authors granted permission for use for free, and all comic authors charged a fee ranging from $25 to $45. Several photos were also taken in-house of female and male young adults with varied skin types ranging from fair to dark, and then two software programs were used to create a library of photos simulating current UV damage (Mirror – www.canfieldsci.com) and UV damage with age progression (https://www.aprilage.com/ageme) for tailoring purposes (Figures 3 and 4). In addition, the research team created a series of brief videos of a young physician who had tanned indoors and survived melanoma.
A number of interactive elements were created to increase engagement in the web program. For example, at the end of each module was a goal-setting section in which participants could choose to set a pre-specified goal for the next two weeks or not (e.g., “For the next two weeks, I will not use a tanning bed, booth, or sunlamp.”). Another section of the website contained printable versions of all of the participant’s goals from each module and links to other online skin cancer prevention resources (called “MyStuff”). The web program also included a personalized avatar (See below). An instructional designer reviewed the material and activities and helped redesign some activities to make them more user-friendly, interactive, and to increase the potential for understanding and retention (Hilgart, Ritterband, Thorndike, & Kinzie, 2012).
Avatar Development. We worked with an existing web-based avatar system (doppelme.com) and incorporated it into the web program homepage (See Figure 1). A number of relevant animated items were added to allow for further personalization of the avatar by the target population and for the topic of focus (e.g., sunscreen with different SPFs, hats with different types of brims, different types of sunglasses). Based on responses to a few questions in the introductory section of the web program, a participant’s avatar was tailored to his/her gender and phenotype (e.g., skin type, eye color, hair color). When participants completed a module, additional items were made available for use with their avatar (e.g., sunscreen, hat, tanning bed). Depending on whether participants chose to use protective or non-protective items on their avatar, the avatar would become red or not, and participants would see a tailored message (“Ouch, you have a sunburn! …”).
Participants from the acceptability, usability, and pilot testing provided feedback on the avatar (Table 4). Users spent an average of 1.5 to 3 minutes exploring the avatar. Feedback from acceptability and usability testing showed that users enjoyed the avatar customization activity and expressed the desire to complete more modules in order to unlock additional features. Changes made as a result of feedback included the user’s eye, hair, and skin color being provided as the default upon beginning with the avatar and clarifying some of the associated text. Software “bugs” were identified during acceptability and usability testing, such as the avatar failing to “burn” at the appropriate times, and were resolved prior to pilot testing.
Health Literacy Evaluation. To ensure that the content of the website was understandable to a broad audience, health literacy experts (LF and SR) systematically evaluated all website text using software and their health literacy expertise. The health literacy experts were the director of the office of health communications and health disparities (LF) and the director of the resource and education center (SR) at a comprehensive cancer center. A software program, Health Literacy Advisor (Health Literacy Innovations, LLC), was used to evaluate and identify complex terms, complex health terms, polysyllabic words (i.e., words with more than three syllables), and long sentences (i.e., sentences with 15 or more words). A readability score was also calculated by the program using the Precise SMOG (“Simple Measure of Gobbledygook”) Index formula. This formula assesses and provides a reading grade level, which research indicates is more reliable than other measures for health-related materials (Wang, Miller, Schmitt, & Wen, 2013). The findings from the health literacy software were evaluated by the health literacy expert and the research team, and the text was simplified to improve readability and comprehension. One of the health literacy experts then reviewed the revised text and suggested any final edits. Other components of the website were also evaluated that are known to contribute to overall readability, such as the way content and images are designed, displayed, and organized, as well as ease of navigation (e.g., font color, size, and style). Suggestions were provided based on health literacy best practice for creating easy-to-use websites (Redish, 2007; U.S. Department of Health and Human Services, 2010).
2.3. Phase 3. Preliminary Assessment and Revision
Cognitive Interviewing. A smaller team of content, questionnaire, and health literacy experts (SR and a staff member who had been trained by her) developed the initial questionnaire using existing items/scales from the literature and created some new ones. The “assessment team” revised the initial questionnaire in order to make it more simple, readable, personal, consistent, and less ambiguous. We then pre-tested the questionnaire using cognitive interviewing (Willis, 2005). A convenience sample of 20 young adults recruited at the sponsoring institution and from the local community completed the questionnaire and provided immediate oral feedback in-person. See Table 5 for demographic characteristics. Cognitive interviewing was conducted in-person in two ways. Half of the participants (n = 10) responded to all the items and provided general feedback afterwards. The other half of participants (n = 10) were asked to use “think aloud” procedures as they read and responded to a subsample of items that were developed or revised significantly by the team and then responded to specific probes about these items. Based on the cognitive interviewing, items were removed, moved, or added, and again made simpler, more specific, personal, and consistent, as well as less ambiguous. Additional detail about the methods and results of the cognitive interviewing including the final measures will be provided in a separate paper.
Acceptability Testing. Acceptability testing was modeled after that used by the US National Cancer Institute for an internet-based program for breast cancer patients (Atkinson et al., 2007). Acceptability testing was conducted to assess and increase the suitability of content for the intended audience (i.e., young adults at moderate to high risk of developing skin cancer) including attractiveness, comprehension, appropriateness, and persuasion. One research staff member with experience in qualitative data collection administered a structured interview, while a second staff member with similar experience video recorded the in-person interview and took notes. Two separate rounds of acceptability testing of the initial web program were conducted with a convenience sample of young adults recruited at the sponsoring institution (but unknown to research staff) and from the local community. The first round (n = 15) was conducted after all the basic content and format was in place, and the second round (n = 11) was conducted after the interactive activities (e.g., avatar, damage and aging photos, goal-setting) were in place. The second round was divided into participants who were asked to focus on specific interactive activities (n = 6) and those (n = 5) who were asked to explore the website freely. The latter individuals viewed an average of four modules and spent 35 minutes exploring the website. Participants in each round provided both positive and negative feedback about the overall web program and about specific content and activities. Participants rated items created by the team regarding likeability, usefulness, and personal applicability on a scale from 0-10, with 10 being the best (See Table 6). In both rounds, some participants thought that the program was geared toward adolescent, female, fair-skinned tanners. They also provided additional feedback about the homepage, information, format, icons, length, images, videos, quizzes, games, goal-setting, the avatar, modules, and functionality. In the second round, some participants provided additional feedback on the enrollment process, estimated how long they might spend on the website, and the content in MyStuff.
Usability Testing (n = 12). In usability testing, users are asked to perform typical tasks with a product, or simply explore it freely, while their behaviors are observed and recorded to identify design flaws that cause user errors or difficulties (Bastien, 2009). For testing of the UV4.me intervention, one staff member administered an in-person structured interview (USDOHHS, usability.gov), while a second staff member video recorded the interview and took notes. Two separate rounds of usability testing were conducted with a convenience sample of young adults recruited at the sponsoring institution and from the local community. In one round of testing (n = 6), we assigned users to complete certain tasks (e.g., use the avatar, view a video, use the goal-setting activity) and monitored how easily they were able to complete them and what questions and problems they had. Users spent an average of 2-3.5 minutes on each task. Usability was also assessed with the System Usability Scale (Brooke, 1986), a ten-item robust measure designed to assess perceived effectiveness as well as efficiency of and satisfaction with a system. The website, as tested in the first round of usability testing, received a score of 86.7, with a standard deviation of 13.5. A score above 68 indicates above-average usability (Brooke, 1986). Users estimated that, if using the system on their own (at home and without being monitored), they would spend an average of 40 minutes using the website, with a range of 15 minutes to 2 hours. When asked how much they liked the website on a scale of 1 to 10, the website was rated with an average score of 7.5 (SD = 2.4).
In the second round of usability testing (n = 6), users were asked to explore the site from start to finish (i.e., from completing the screener to completing the final follow-up assessment). The purpose of this testing was to ensure that the programmed automation (e.g., email notifications, tailored content) was working properly. Though the purpose of this testing was not to review specific content and activities within the modules, users were offered the opportunity to explore this content if interested and as time permitted.
Quality Control Testing. Throughout the development process and particularly at the end prior to the pilot trial, the project manager and programmers at BeHealth Solutions, Inc. (the company that programmed the website, assessments, and data management system) conducted quality control testing in order to ensure that all intervention and data collection and management features were functioning properly. This consisted of testing the auto-enrollment process (eligibility, online consent, randomization), assessments, intervention including tailoring and the avatar, email reminders, data capture, and data reports. Testing was conducted by reviewing the content of the entire site as well as creating a number of mock subject profiles and taking them through a variety of scenarios to see if they worked properly (e.g., completing the screening form as an ineligible subject to see if he/she was deemed ineligible and received appropriate notifications).
2.4 Phase 4. Pilot testing
After completion of usability testing, a pilot trial was initiated using all of the planned procedures to determine whether the automated enrollment, website, and data management programs were working properly, finalize our data management plans, and establish recruitment and eligibility rates for the main RCT (Lackey & Wingate, 1986), which tested the effects of the interventions on self-reported UV exposure and protection behaviors at 3-weeks and 12-weeks after baseline. Participants were recruited from an existing internet research panel by Survey Sampling International (SSI). SSI panelists were exposed to brief web banner ads about the study from which they could click to link to UV4.me. Eligibility criteria included being 18-25 years old, at moderate to high risk of developing skin cancer based on the Brief Risk Assessment Tool (Glanz et al., 2003), and without a personal history of skin cancer. Using the automated enrollment procedures, 222 people were screened, 96 were eligible, 53 then provided informed consent and submitted completed baseline surveys that had been cognitively tested, and 46 submitted follow-up surveys after three weeks.
Participants who completed the baseline survey were randomized to the experimental UV4.me intervention program, the existing Skin Cancer Foundation website, and to an assessment-only condition. Out of the subjects who were randomized to UV4.me and completed the follow-up survey (n = 15), thirteen (93%) accessed the intervention modules. We estimated, based on log-in and module completion times, that subjects spent on average about 90 minutes using the website (standard deviation of 105 minutes, with a range of 0 to 344 minutes) and logged into the website about eight separate times (standard deviation of 9.6 times, with a range of 0 to 35 times). Among those who accessed the modules, perceptions were positive overall, with ratings ranging from 4.4 (on a scale of 1 to 5) to 4.7 (SD = 0.5-0.7). Specifically, participants reported satisfaction with the website [M(SD) = 4.5 (0.5]. The highest rated item was that the website was useful [M(SD) = 4.7 (0.5)], followed by easy [M(SD) = 4.6 (0.5)], convenient [M(SD) = 4.5 (0.7)],and helpful [M(SD) = 4.4 (0.7)]. These participants reported that the website increased their knowledge about skin protection [M(SD) = 4.5 (0.7)], their confidence to protect their skin [M(SD) = 4.5 (0.7)], and their knowledge about skin cancer [M(SD) = 4.4 (0.8)]. Participants also reported that they would be likely to recommend the website to others [M(SD) = 4.5 (0.7)].
Though no problems were reported with the website itself, about half of the subjects who were provided with access to the website reported that not having time to visit the site was at least “a little problem,” with two subjects indicating that this was “a big problem.” Open-ended feedback indicated that the information presented in the modules was most helpful, as well as the short length and interactivity of the modules, and that a few subjects found the avatar to be the least helpful part of the website and/or did not understand its purpose. One user noted that it would be helpful for the website to be available in other languages, such as Spanish. As a result of the pilot, we refined the study screener and assessment items, clarified instructions on some of the introductory pages, and updated some of our data management procedures.
This manuscript documented the process used for development and preliminary assessment of a tailored internet intervention to address skin cancer risk behaviors among young adults. The process for the current project involved four phases: intervention planning, content and website development, initial assessment and revision, and pilot testing. To summarize, the overall lessons and recommendations are as follows: 1) gather input from multi-disciplinary experts and the intended population during the entire development process; 2) use mixed methods to provide the most complete and useful information; 3) include the planned population and procedures in formative and pilot testing; 4) plan and manage the time, money, and expertise needed for the project carefully prior to and throughout the project; and finally, 5) be willing and able to revise plans, procedures, the website, and potentially the budget and timeline as the project progresses. The challenges of conducting this type of work are, in part, the time, money, and expertise required to conduct such a project. A formal cost analysis was not conducted, but such analysis could be useful. However, it is important to note that the costs of initially developing an intervention would be considered “sunk” costs, and what may be more pertinent to consider for the future are the costs of eventual dissemination and sustainability, which we have not yet assessed. Additionally, it can be difficult to obtain feedback and responses from participants that apply to the full-fledged intervention using prototype materials; however, developing the full-fledged intervention prior to obtaining some feedback would most certainly require later modification to meet user needs. More specific practical lessons from each of the phases of the process that we hope will be useful to other internet intervention developers and researchers are discussed below.
- Intervention planning
Individual Interviews: For efficiency’s sake and in preparation for future funding applications, it can be helpful to “piggy-back” qualitative interviews onto an ongoing trial. A structured interview incorporating both qualitative and quantitative feedback allows one to gather both general and specific data both systematically and efficiently. We recommend including some sample materials for participants to react to while also being open to creation of new activities.
Focus Groups: Young adults expressed interested in providing feedback and serving as research participants, but they are somewhat transient across time due to changes in school, work, and living situation. Thus, flexibility when contacting them and scheduling group meetings is required. Other methods such as online focus groups could be considered.
- Content and website development: This was the most labor-intensive aspect of the development process. Our goal was to make the material and activities as engaging and appealing as possible to our target population. Translating conceptual ideas for interactive activities and processes into feasibly programmable ones was challenging at times. However, during this time, we learned to prioritize what material, activities, and processes would be included in the program. Content was developed and edited extensively and repeatedly over time. The development of a specific timeline, prioritization of content and activities, and an open and ongoing discussion between content, intervention, instructional design, health literacy, and programming experts can facilitate efficient use of time and avoid wasted or redundant effort. The timeline should include simultaneous (rather than sequential) work tasks and be somewhat flexible with extra time built in if possible in order to utilize time efficiently and accommodate unexpected setbacks and technical difficulties.
Avatar Development: Although they liked the avatar and said they were motivated to access more items for use with their avatar, users tended to like other website features better and used the avatar for only a few minutes during acceptability and usability testing. Despite our efforts, some users continued to say that they perceived the avatar activity and appearance to be appropriate for adolescents rather than young adults and that they didn’t see the point or wouldn’t use it themselves but that others might. Unfortunately, we did not have the funds to make additional, more extensive changes to the avatar. Our experience with the avatar might be an example of the incongruence between what people think or say they want, like, will use, and will actually use and benefit from, that sometimes occurs when developing and assessing behavioral health interventions. Thus, pre-testing important intervention components is critical to the ultimate success of intervention efforts.
Health Literacy Evaluation: It is well-known that scientists do not tend to write at the level of the average citizen (Eichler, Wieser, & Brugger, 2009). Recommended best-practice strategies to create understandable reading materials, interventions, and web interventions exist (Redish, 2007; U.S. Department of Health and Human Services, 2010). These include simplifying words, phrases, and sentences and attending to the way content and images are designed, displayed, and organized, as well as the ease of navigation. Our initial website content was revised so that it would be more understandable to a broad audience of young adults.
- Preliminary assessment and revision
Cognitive Interviewing: Unless measures are developed with the assistance of assessment experts and cognitively tested with the target audience, they may not be reliable and valid. Cognitive testing improved the format and instructions for scales and made items simpler, more personal and consistent, as well as less ambiguous.
Acceptability Testing: Acceptability testing assisted us in making the web program content, format, media, activities, and functionality more appropriate and acceptable to the target population. Participants provided feedback about who they thought the website was designed for so that we could revise the site to make it more applicable to our target population. This helped us identify which sections and videos were too long and complex and which did not provide suitable information. It also became clear that the young adults did not want to engage in a somewhat lengthy tutorial before using the website and that some aspects of the navigation were confusing despite our instructions. Thus, we attempted to make navigation as similar to other commonly-used websites and as intuitive as possible.
Usability Testing: Usability testing allowed us to identify and correct specific problems, such as email notifications not firing properly. Overall, usability testing revealed bugs with the website and misunderstandings on the part of the users, such as with the avatar and with the goal-setting process. This prompted us to correct these issues, clarify instructions, and attempt to make activities more intuitive. Some “usability” testing can be conducted by staff themselves, but the use of potential participants provides invaluable information for future intervention programs and research trials.
Quality Control Testing: Although necessary, this type of testing has rarely been described in the literature, perhaps because it is often conducted by programmers rather than investigators or interventionists. However, it is useful for as many members of the team and “pilot” participants as possible to participate in this type of testing both by reviewing content and programming if possible and creating mock subjects to test various participation scenarios. This testing identified several technical glitches that were able to be remedied by the programmers prior to actual participant enrollment.
- Pilot testing: Conducting a pilot test helped us accomplish the following: identify additional problems in a small sample before full release of the intervention trial in a larger RCT; develop, refine, and practice our data management procedures; assess eligibility, enrollment, and follow-up rates in order to finalize the larger RCT recruitment plan and sample size. The pilot was helpful in facilitating the activities of both study participants and research staff for the main RCT. Pilot trials should be conducted as similarly as possible to the main trial in order to be best prepared for the larger RCT. Otherwise, the main trial is likely to identify additional problems in a larger sample that could have been avoided or minimized.
The strengths of this project are the systematic development of an internet intervention using multiple methods and input from both experts and young adults. Though these efforts were extensive, we believe that they are necessary to develop an efficacious web intervention that an intended population will actually use. However, other projects may be able to include fewer rounds of assessment or numbers of individuals assessed. The key is to reach saturation so that additional feedback will be redundant. Once this occurs, further feedback is unnecessary. One limitation is that we used a convenience sample of young adults rather than an online panel for much of the formative steps so that we could work with them in person rather than online or over the telephone. Use of identical recruitment strategies and populations for the formative, pilot, and randomized controlled phases would be preferable. Another limitation is the small sample size for some of the formative steps. Although within the range of recommendations for formative assessment and qualitative research in order to result in saturation (Bastien, 2009; Willis, 2005), a few more participants in some of the phases may have provided additional useful information.
After the pilot trial, a fully-powered RCT was initiated in the spring of 2014 to compare the experimental UV4.me intervention program to the existing Skin Cancer Foundation website and to an assessment only condition. Reports on the RCT are forthcoming. A dissemination trial assessing various strategies to engage young adults in the intervention is also in the works.
The preceding report serves as an exemplar of the process of developing a health behavioral internet intervention. It is important to publish detailed descriptions of intervention development procedures in order to help prepare other researchers for the development process and so that reports of intervention trials can be interpreted properly (Barretto, Bingham, Goh, & Shope, 2011). Readers might also want to refer to alternative recent comprehensive approaches to web intervention development including the use of Intervention Mapping (Bartholomew, Parcel, Kok, Gottlieb, & Fernandez, 2011), instructional design (Hilgart et al., 2012), the PRECEDE model (Kattelmann et al., 2014), the process map of Elwyn and colleagues (2011), and the approaches of Barretto and colleagues (2011) and Chee and colleagues (Chee, Lee, Chee, & Im, 2014).
This work was funded by R01CA154928 (CH), T32CA009035 (SD), and P30CA006927 (Cancer Center grant). We thank the following for their assistance with this project: the staff of BeHealth Solutions, Inc. including Mary Grove, Peter Braswell, Gabe Heath, and Michelle Hilgart for creating the online assessment and data management system and interactive activities, Jennifer Burns and the Resource and Education Center at Fox Chase Cancer Center for their assistance with user testing, Clifford Perlis for his early input on the study and intervention design, Denisa Hoxha for her assistance with the identification and collection of multimedia materials, Helene Conway for her assistance with manuscript preparation, as well as the young adults who provided their feedback on the intervention.
Dr. Ritterband is an equity holder of BeHealth Solutions, Inc, which developed the data management system and helped develop the intervention described in this paper. Dr. Ritterband’s conflict of interest (COI) is being managed by a COI committee at the University of Virginia, in accordance with their respective conflict of interest policies.
Table 1. The Developmental Phases of the UV4.me Project
|1. Intervention Planning||Individual Interviews||25||Nov and Dec 2009|
|Obtaining grant funding||NA||Feb 2010 – Mar 2011|
|Focus Groups||14||July – Dec 2011|
|2. Content and website development||Synthesis of Material||NA||Jan-July 2012|
|Avatar Development||NA||Jan-Oct 2013|
|Health Literacy Evaluation||NA||April 2012 and April 2013|
|3. Preliminary assessmentand revision||Cognitive Interviewing||20||Aug – Dec 2012|
|Acceptability Testing||26||Aug – Nov 2012 andJuly – Sept 2013|
|Usability Testing||12||Dec 2013 – Mar 2014|
|Quality Control Testing||NA||Oct 2013 – Mar 2014|
|4. Pilot Testing||Pilot Testing||53||Mar and April 2014|
Table 2. Phase 1: Intervention Planning – Ratings from Individual Interviews
|Mean Ratings (n = 25)|
|Symptoms of skin cancer||4.84|
|Sunburns, Skin self-exams||4.80|
|How UV damages skin||4.76|
|Images of twins, one aged||5.00|
|Incentives ($20 per hour)||4.94|
|Images of skin damage||4.88|
Table 3. Phase 2: Outline of Content and Features of the Web Intervention
|Web Modules||Web Topics||Sample Interactive and Tailored Features||Sample IM Constructs Addressed|
|Introduction||-Purpose, features-How to use the website-Tailoring questions||-NA|
|Why do people tan?||-Change in tanning trends over time-What do you know about tanning?-Who tans in your circle?||-Personalized responses (e.g., feedback to responses regarding “what do you know about tanning” quiz items)||-Norms-Knowledge|
|Indoor tanning||-The truth about indoor tanning-Risky behaviors||-Personalized responses (e.g., feedback to responses regarding indoor tanning true/false quiz items)||-Knowledge-Behaviors|
|To tan or not to tan||-Why people tan-Reasons for tanning-Tanorexia||-Personalized responses (e.g., feedback regarding user’s own personal reasons for tanning)-Tanorexia quiz||-Self-efficacy-Intentions-Behaviors|
|UV and health||-UV and its effects on the skin (e.g., vitamin D, acne) and immunity-Sunburns-Skin type and sunburn risk-Skin damage||-Skin type and sunburn risk-UV damage photo of similar individual (See Figure 2)||-UV related Beliefs|
|UV and looks||-What is UV?-Celebrities and tanning-Photo-aging- Feeling good about yourself without tanning||– Age progression photos of similar individual (See Figure 3)||-Attitudes toward appearance|
|Skin damage||-Skin cancer risk factors-Melanoma rates around the world-New cases of skin cancer-Impact of global warming on UV levels||-Personal risk calculator-Personalized feedback on melanoma quiz||-Risk perception-Knowledge|
|Skin cancer||-Skin cancer types/prevalence-Videos about skin cancer-Skin cancer risk factors-Treatment-Skin cancer and darker skin||-Personal risk calculator||-Risk perception|
|Sunscreen||-Sun Protection Factor (SPF)-How to use sunscreen-Reasons for not using sunscreen||-Sunscreen SPF and reapplication calculator-Personalized responses (e.g., feedback regarding which sunscreen is best for user’s skin)||-Self-efficacy-Norms-Intentions-Behaviors|
|Clothes||-Protective clothes-Reasons for not wearing protective clothes||-Personalized responses (e.g., feedback regarding user’s habit of wearing protective clothes)||-Self-efficacy-Intentions-Behaviors|
|Sunless tanning||-Reasons people sunless tan-Have you tried sunless tanners before?-Better sunless tanning||-Personalized feedback and responses to sunless tanning quiz and reasons for not using sunless tanner||-Self-efficacy-Norms-Intentions-Knowledge-Behaviors|
|Shade||-Staying in the shade-Reasons for not being in the shade||-Personalized responses (e.g., feedback regarding user’s personal reasons for avoiding shade)||-Self-efficacy-Intentions-Behaviors|
|Skin exams||-Checking for skin cancer-ABC’s of skin cancer-Reasons for not doing skin exams||-Personalized responses (e.g., feedback regarding user’s personal reasons for not doing skin exams)||-Self-efficacy-Intentions-Behaviors|
Table 4. Phase 2: Avatar Feedback
|TYPE OF FEEDBACK||SAMPLE QUOTE(S)||N|
|Positive||Like the number of options it provides to customize/personalize the avatar||17|
|I liked the fact that some items are locked. This will encourage people to go through the website more to unlock them.||7|
|It seems fun||6|
|(Miscellaneous) Good. Nice. Liked it. Good visual representation. Made sense. Laid out well.||6|
|People my age may like it||3|
|The avatar draws the attention first||3|
|The avatar was one of the things I liked best||3|
|(What would you do if you saw your avatar start to burn?) I don’t know. I guess it’s kind of a game. I would put on a hat, sunscreen, maybe move it to a shadier place.||3|
|Interactive. I’m interested in seeing how to interact with it. I want to make him look nice.||3|
|Overall – exciting||2|
|Interesting twist to care for something||2|
|I like the explanation on what needs to change with the avatar so it does not get burnt||1|
|Negative||Not sure why certain things are locked. Not sure how to use unlocked items.||11|
|Younger people will be more motivated to unlock things for their avatar, not people of my age||5|
|I feel as though the avatar is a little pointless||5|
|The avatar was least helpful at understanding skin cancer risks and prevention||4|
|Wouldn’t take the time to use it||4|
|(Miscellaneous – Concept) Initially was not sure what the avatar was about. There should be more to do with the avatar. The avatar is just something extra to do on the website. What is the picture doing here? (without a guide)||4|
|Avatar was one of the things I liked the least||3|
|(Participant was unable to complete the task due to technical difficulties. E.g., Tried to save changes and was logged off.)||3|
|(What would you do if you saw your avatar start to burn? Was confused by the meaning of the question.) Would try to figure out why the avatar was burning. Perhaps the avatar burned based on my responses to questions.||3|
|Why doesn’t it have hair?||3|
|o (Miscellaneous – Appearance) The font explaining the avatar doesn’t stand out. Add the word ‘Home’ with the homepage icon at the top of the screen. Cannot change colors for sunglasses and hats.||3|
|Is there any information, or is it just the avatar?||2|
|Maybe a moving (animated/talking) avatar would be more interesting||2|
|I am not into internet personification||2|
|o Arrows on either side of the items (when there are no more options on the next page for that particular category) do not make sense; remove them.||2|
|o It’s basic with some customization; not many options for skin tone. Participant suggested adding more options, not just summer clothes.||2|
|There is no sunscreen option to put on the avatar. (Participant didn’t notice the sunscreen option.)||1|
|o It reminds me of a Facebook app in which you have to create an avatar||2|
|Can you change the skin color?||2|
|o (Noticed certain options were locked.) I guess you would have to complete modules to unlock options.||2|
|Have avatar available first, then the modules||1|
|I want the avatar to look like me||1|
|o (Participant never used arrows to view more options for the avatar)||1|
Table 5. Phase 3: Demographics of User Testing (Cognitive Interviewing, Acceptability and Usability Testing) (N = 47) a
|Age in years||22.3 (2.3)|
|Female Sex||34 (72.3)|
|African-American or Black||2 (4.3)|
|Asian American||1 (2.1)|
|Very fair||7 (14.9)|
a Some users who completed cognitive interviewing (n = 20) also completed acceptability (n = 26) or usability testing (n = 12)
Table 6. Phase 3 – Acceptability Ratings
|Mean (SD) for Round 1 (n = 15)(0 = worst, 10 = best)||Mean (SD) for Round 2 (n = 11)(0 = worst, 10 = best)|
|Likability||7.5 (1.0)||7.7 (1.8)|
|Usefulness||8.5 (1.3)||8.7 (1.5)|
|Personal applicability||8.2 (1.6)||7.8 (2.3)|
|Features liked best||Videos; age progression photos; interactive sections; informational pop-up boxes; educational information – especially that associated with skin exams||Educational information; interactive sections; tanning trends timeline; videos from physician with personal skin cancer history; SPF calculator; age progression photos|
|Features liked least||Format (e.g., appearance, large blocks of text)||Tutorial; avatar; length of some videos|
Figure 1. UV4.me Home Page
Figure 2. Tailored Photos Showing Simulated Current UV Damage
Figure 3. Tailored Age Profession Photos Showing Simulated UV Damage
Aarestrup, C., Bonnesen, C. T., Thygesen, L. C., Krarup, A. F., Waagstein, A. B., Jensen, P. D., & Bentzen, J. (2014). The effect of a school-based intervention on sunbed use in Danish pupils at continuation schools: a cluster-randomized controlled trial. Journal of Adolescent Health, 54(2), 214-220. doi: 10.1016/j.jadohealth.2013.08.011
Atkinson, N. L., Massett, H. A., Mylks, C., Hanna, B., Deering, M. J., & Hesse, B. W. (2007). User-centered research on breast cancer patient needs and preferences of an Internet-based clinical trial matching system. J Med Internet Res, 9(2), e13. doi: v9i2e13 [pii]
Barak, A., Klein, B., & Proudfoot, J. G. (2009). Defining Internet-supported therapeutic interventions. Annals of Behavioral Medicine, 38, 4-17. doi: 10.1007/s12160-009-9130-7
Barretto, A. I., Bingham, C. R., Goh, K. N., & Shope, J. T. (2011). Developing a web-based health promotion intervention: a case study from a brief motivational alcohol program. Health Promot Pract, 12(2), 193-201. doi: 10.1177/1524839909353740
Bartholomew, L. K., Parcel, G. S., Kok, G., Gottlieb, N. H., & Fernandez, M. E. (2011). Planning Health Promotion Programs: An Intervention Mapping Approach. San Francisco, CA: John Wiley & Sons, Inc.
Bastien, J. M. (2009). Usability testing: a review of some methodological and technical aspects of the method. Int J Med Inform. doi: S1386-5056(08)00209-8 [pii]
Berry, D. L., Halpenny, B., Wolpin, S., Davison, B. J., Ellis, W. J., Lober, W. B., . . . Wulff, J. (2010). Development and evaluation of the personal patient profile-prostate (P3P), a Web-based decision support system for men newly diagnosed with localized prostate cancer. J Med Internet Res, 12(4), e67. doi: 10.2196/jmir.1576
Bewick, B. M., Trusler, K., Barkham, M., Hill, A. J., Cahill, J., & Mulhern, B. (2008). The effectiveness of web-based interventions designed to decrease alcohol consumption–a systematic review. Preventive Medicine, 47(1), 17-26. doi: 10.1016/j.ypmed.2008.01.005
Bleyer, A., & Barr, R. (2009). Introduction–impact of malignant diseases on young adults II. Seminars in Oncology, 36(5), 380. doi: 10.1053/j.seminoncol.2009.07.002
Bock, B. C., Graham, A. L., Whiteley, J. A., & Stoddard, J. L. (2008). A review of web-assisted tobacco interventions (WATIs). J Med Internet Res, 10(5), e39. doi: 10.2196/jmir.989
Bowen, D. J., Hay, J. L., Mayer, J., Kuniyuki, A., Meischke, H., Harris, J., . . . Burke, W. (2012). Predictors of recruited melanoma families into a behavioral intervention project. Contemp Clin Trials, 33(1), 85-92. doi: 10.1016/j.cct.2011.09.016
Brooke, J. (1986). SUS – A quick and dirty usability scale. Digital Equipment Co Ltd, Reading, United Kingdom.
Buller, D. B., Cokkinides, V., Hall, H. I., Hartman, A. M., Saraiya, M., Miller, E., . . . Glanz, K. (2011). Prevalence of sunburn, sun protection, and indoor tanning behaviors among Americans: Review from national surveys and case studies of 3 states. Journal of the American Academy of Dermatology, 65(5 Suppl 1), S114-123. doi: S0190-9622(11)00608-6 [pii]
Cafri, G., Thompson, J. K., Roehrig, M., van den Berg, P., Jacobsen, P. B., & Stark, S. (2006). An investigation of appearance motives for tanning: The development and evaluation of the Physical Appearance Reasons For Tanning Scale (PARTS) and its relation to sunbathing and indoor tanning intentions. Body Image, 3(3), 199-209. doi: 10.1016/j.bodyim.2006.05.002
Chee, W., Lee, Y., Chee, E., & Im, E. O. (2014). Practical Guidelines for Development of Web-Based Interventions. Comput Inform Nurs. doi: 10.1097/cin.0000000000000088
Cokkinides, V., Weinstock, M., Glanz, K., Albano, J., Ward, E., & Thun, M. (2006). Trends in sunburns, sun protection practices, and attitudes toward sun exposure protection and tanning among US adolescents, 1998-2004. Pediatrics, 118(3), 853-864. doi: 118/3/853 [pii]
Coups, E. J., Manne, S. L., & Heckman, C. J. (2008). Multiple skin cancer risk behaviors in the U.S. population. American Journal of Preventive Medicine, 34(2), 87-93. doi: S0749-3797(07)00655-1 [pii]
Danaher, B. G., & Seeley, J. R. (2009). Methodological Issues in Research on Web-Based Behavioral Interventions. Annals of Behavioral Medicine. doi: 10.1007/s12160-009-9129-0
Danoff-Burg, S., & Mosher, C. E. (2006). Predictors of tanning salon use: behavioral alternatives for enhancing appearance, relaxing and socializing. J Health Psychol, 11(3), 511-518.
Davies, C. A., Spence, J. C., Vandelanotte, C., Caperchione, C. M., & Mummery, W. K. (2012). Meta-analysis of internet-delivered interventions to increase physical activity levels. Int J Behav Nutr Phys Act, 9, 52. doi: 10.1186/1479-5868-9-52
DiIorio, C., Escoffery, C., Yeager, K. A., McCarty, F., Henry, T. R., Koganti, A., . . . Price, P. (2009). WebEase: development of a Web-based epilepsy self-management intervention. Preventing chronic disease, 6(1), A28.
Donaldson, M. R., & Coldiron, B. M. (2011). No end in sight: the skin cancer epidemic continues. Seminars in Cutaneous Medicine and Surgery, 30(1), 3-5. doi: 10.1016/j.sder.2011.01.002
Eichler, K., Wieser, S., & Brugger, U. (2009). The costs of limited health literacy: a systematic review. Int J Public Health, 54(5), 313-324. doi: 10.1007/s00038-009-0058-2
Eng, T. R., Gustafson, D. H., Henderson, J., Jimison, H., & Patrick, K. (1999). Introduction to evaluation of interactive health communication applications. Science Panel on Interactive Communication and Health. American Journal of Preventive Medicine, 16(1), 10-15.
Escoffery, C., McCormick, L., & Bateman, K. (2004). Development and process evaluation of a web-based smoking cessation program for college smokers: innovative tool for education. Patient Education and Counseling, 53(2), 217-225. doi: 10.1016/s0738-3991(03)00163-0
Evangelista, L. S., Stromberg, A., Westlake, C., Ter-Galstanyan, A., Anderson, N., & Dracup, K. (2006). Developing a Web-based education and counseling program for heart failure patients. Progress in Cardiovascular Nursing, 21(4), 196-201.
Fergus, K. D., McLeod, D., Carter, W., Warner, E., Gardner, S. L., Granek, L., & Cullen, K. I. (2014). Development and pilot testing of an online intervention to support young couples’ coping and adjustment to breast cancer. European Journal of Cancer Care (English Language Edition), 23(4), 481-492. doi: 10.1111/ecc.12162
Fishbein, M., Hennessy, M., Yzer, M., & Douglas, J. (2003). Can we explain why some people do and some people do not act on their intentions? Psychol Health Med, 8(1), 3-18. doi: 10.1080/1354850021000059223
Fleisher, L., Buzaglo, J., Collins, M., Millard, J., Miller, S. M., Egleston, B. L., . . . Meropol, N. J. (2008). Using health communication best practices to develop a web-based provider-patient communication aid: the CONNECT study. Patient Education and Counseling, 71(3), 378-387. doi: 10.1016/j.pec.2008.02.017
Fleisher, L., Ruggieri, D. G., Miller, S. M., Manne, S., Albrecht, T., Buzaglo, J., . . . Meropol, N. J. (2014). Application of best practice approaches for designing decision support tools: the preparatory education about clinical trials (PRE-ACT) study. Patient Education and Counseling, 96(1), 63-71. doi: 10.1016/j.pec.2014.04.009
Ghahari, S., Packer, T. L., & Passmore, A. E. (2009). Development, standardisation and pilot testing of an online fatigue self-management program. Disability and Rehabilitation, 31(21), 1762-1772. doi: 10.1080/09638280902751956
Gibbons, F. X., Gerrard, M., Lane, D. J., Mahler, H. I., & Kulik, J. A. (2005). Using UV photography to reduce use of tanning booths: a test of cognitive mediation. Health Psychology, 24(4), 358-363.
Glanz, K., Schoenfeld, E., Weinstock, M. A., Layi, G., Kidd, J., & Shigaki, D. M. (2003). Development and reliability of a brief skin cancer risk assessment tool. Cancer Detection and Prevention, 27(4), 311-315. doi: S0361090X03000941 [pii]
Goldberg, M. S., Doucette, J. T., Lim, H. W., Spencer, J., Carucci, J. A., & Rigel, D. S. (2007). Risk factors for presumptive melanoma in skin cancer screening: American Academy of Dermatology National Melanoma/Skin Cancer Screening Program experience 2001-2005. Journal of the American Academy of Dermatology, 57(1), 60-66. doi: 10.1016/j.jaad.2007.02.010
Gordon, R. (2013). Skin cancer: an overview of epidemiology and risk factors. Seminars in Oncology Nursing, 29(3), 160-169. doi: 10.1016/j.soncn.2013.06.002
Griffiths, F., Lindenmeyer, A., Powell, J., Lowe, P., & Thorogood, M. (2006). Why are health care interventions delivered over the internet? A systematic review of the published literature. Journal of Medical Internet Research, 8(2), e10. doi: 10.2196/jmir.8.2.e10
Hallett, J., Maycock, B., Kypri, K., Howat, P., & McManus, A. (2009). Development of a Web-based alcohol intervention for university students: processes and challenges. Drug Alcohol Rev, 28(1), 31-39. doi: 10.1111/j.1465-3362.2008.00008.x
Hart, K. M., & Demarco, R. F. (2008). Primary prevention of skin cancer in children and adolescents: a review of the literature. Journal of Pediatric Oncology Nursing, 25(2), 67-78. doi: 10.1177/1043454208314499
Heckman, C. J., Coups, E. J., & Manne, S. L. (2008). Prevalence and correlates of indoor tanning among US adults. Journal of the American Academy of Dermatology, 58(5), 769-780. doi: S0190-9622(08)00132-1 [pii]
Hilgart, M. M., Ritterband, L. M., Thorndike, F. P., & Kinzie, M. B. (2012). Using instructional design process to improve design and development of Internet interventions. J Med Internet Res, 14(3), e89. doi: 10.2196/jmir.1890
Hillhouse, J., Turrisi, R., Stapleton, J., & Robinson, J. (2008). A randomized controlled trial of an appearance-focused intervention to prevent skin cancer. Cancer, 113(11), 3257-3266. doi: 10.1002/cncr.23922
Horsham, C., Auster, J., Sendall, M. C., Stoneham, M., Youl, P., Crane, P., . . . Kimlin, M. (2014). Interventions to decrease skin cancer risk in outdoor workers: update to a 2007 systematic review. BMC Res Notes, 7, 10. doi: 10.1186/1756-0500-7-10
Kattelmann, K. K., White, A. A., Greene, G. W., Byrd-Bredbenner, C., Hoerr, S. L., Horacek, T. M., . . . Morrell, J. S. (2014). Development of Young Adults Eating and Active for Health (YEAH) internet-based intervention via a community-based participatory research model. J Nutr Educ Behav, 46(2), S10-25. doi: 10.1016/j.jneb.2013.11.006
Keeney, S., McKenna, H., Fleming, P., & McIlfatrick, S. (2009). Attitudes, knowledge and behaviours with regard to skin cancer: a literature review. Eur J Oncol Nurs, 13(1), 29-35. doi: S1462-3889(08)00148-8 [pii]
Kimlin, M., & Parisi, A. (2001). Usage of real-time ultraviolet radiation data to modify the daily erythemal exposure of primary schoolchildren. Photodermatology, Photoimmunology and Photomedicine, 17(3), 130-135.
Kukafka, R., Lussier, Y. A., Patel, V. L., & Cimino, J. J. (2001). Developing tailored theory-based educational content for WEB applications: illustrations from the MI-HEART project. Studies in Health Technology and Informatics, 84(Pt 2), 1474-1478.
Lackey, N. R., & Wingate, A. L. (1986). The pilot study: one key to research success. Kansas Nurse, 61(11), 6-7.
Lazovich, D., Vogel, R. I., Berwick, M., Weinstock, M., Anderson, K. E., & Warshaw, E. M. (2010). Indoor tanning and risk of melanoma: A case-control study in a highly exposed population. Cancer Epidemiology, Biomarkers and Prevention, 19(6), 1557-1568.
Long, A. C., & Palermo, T. M. (2009). Brief report: Web-based management of adolescent chronic pain: development and usability testing of an online family cognitive behavioral therapy program. Journal of Pediatric Psychology, 34(5), 511-516. doi: 10.1093/jpepsy/jsn082
Lustria, M. L., Noar, S. M., Cortese, J., Van Stee, S. K., Glueckauf, R. L., & Lee, J. (2013). A meta-analysis of web-delivered tailored health behavior change interventions. J Health Commun, 18(9), 1039-1069. doi: 10.1080/10810730.2013.768727
MacNeal, R. J., & Dinulos, J. G. (2007). Update on sun protection and tanning in children. Current Opinion in Pediatrics, 19(4), 425-429. doi: 10.1097/MOP.0b013e3282294936
Mahler, H. I., Kulik, J. A., Gerrard, M., & Gibbons, F. X. (2013). Effects of photoaging information and UV photo on sun protection intentions and behaviours: a cross-regional comparison. Psychol Health, 28(9), 1009-1031. doi: 10.1080/08870446.2013.777966
Mahler, H. I., Kulik, J. A., Harrell, J., Correa, A., Gibbons, F. X., & Gerrard, M. (2005). Effects of UV photographs, photoaging information, and use of sunless tanning lotion on sun protection behaviors. Archives of Dermatology, 141(3), 373-380. doi: 141/3/373 [pii]
Manne, S., Jacobsen, P. B., Ming, M. E., Winkel, G., Dessureault, S., & Lessin, S. R. (2010). Tailored versus generic interventions for skin cancer risk reduction for family members of melanoma patients. Health Psychology, 29(6), 583-593. doi: 10.1037/a0021387
Markovic, S. N., Erickson, L. A., Rao, R. D., Weenig, R. H., Pockaj, B. A., Bardia, A., . . . Creagan, E. T. (2007). Malignant melanoma in the 21st century, part 1: epidemiology, risk factors, screening, prevention, and diagnosis. Mayo Clinic Proceedings, 82(3), 364-380. doi: 10.4065/82.3.364
Michie, S., Brown, J., Geraghty, A. W., Miller, S., Yardley, L., Gardner, B., . . . West, R. (2012). Development of StopAdvisor: A theory-based interactive internet-based smoking cessation intervention. Transl Behav Med, 2(3), 263-275. doi: 10.1007/s13142-012-0135-6
Moore, S. K., Guarino, H., Acosta, M. C., Aronson, I. D., Marsch, L. A., Rosenblum, A., . . . Turk, D. C. (2013). Patients as collaborators: using focus groups and feedback sessions to develop an interactive, web-based self-management intervention for chronic pain. Pain Med, 14(11), 1730-1740. doi: 10.1111/pme.12200
National_Cancer_Institute. (2002). Making Health Communication Programs Work: USDHHS.
Nikolaou, V., & Stratigos, A. J. (2014). Emerging trends in the epidemiology of melanoma. British Journal of Dermatology, 170(1), 11-19. doi: 10.1111/bjd.12492
Onken, L. S., Blaine, J. D., & Battjes, R. J. (1997). Behavioral therapy research:A conceptualization of a process. In S. W. Henngler & R. Amentos (Eds.), Innovative approaches from difficult-to-treat populations (pp. 477-485). Washington, DC: American Psychiatric Press.
Pachankis, J. E., Lelutiu-Weinberger, C., Golub, S. A., & Parsons, J. T. (2013). Developing an online health intervention for young gay and bisexual men. AIDS Behav, 17(9), 2986-2998. doi: 10.1007/s10461-013-0499-8
Pew_Research_Internet_Project. (2014). Health Fact Sheet. Retrieved November 17, 2014, from http://www.pewinternet.org/fact-sheets/health-fact-sheet/
Psaty, E. L., Scope, A., Halpern, A. C., & Marghoob, A. A. (2010). Defining the patient at high risk for melanoma. International Journal of Dermatology, 49(4), 362-376. doi: 10.1111/j.1365-4632.2010.04381.x
Qureshi, A. A., Zhang, M., & Han, J. (2011). Heterogeneity in host risk factors for incident melanoma and non-melanoma skin cancer in a cohort of US women. Journal of Epidemiology, 21(3), 197-203.
Redish, J. (2007). Letting Go of the Words: Writing Web Content that Works (Interactive Technologies). San Francisco, CA: Elsevier Inc.
Riiser, K., Londal, K., Ommundsen, Y., Sundar, T., & Helseth, S. (2013). Development and usability testing of an internet intervention to increase physical activity in overweight adolescents. JMIR Res Protoc, 2(1), e7. doi: 10.2196/resprot.2410
Ritterband, L. M., & Tate, D. F. (2009). The science of internet interventions. Annals of Behavioral Medicine. doi: 10.1007/s12160-009-9132-5
Ritterband, L. M., Thorndike, F. P., Cox, D. J., Kovatchev, B. P., & Gonder-Frederick, L. A. (2009). A behavior change model for internet interventions. Annals of Behavioral Medicine. doi: 10.1007/s12160-009-9133-4
Roberts, D. C., & Black, D. (2009). Comparison of interventions to reduce sun exposure. Behavioral Medicine, 35(2), 67-76. doi: 82K094431402K667 [pii]
Siskind, V., Aitken, J., Green, A., & Martin, N. (2002). Sun exposure and interaction with family history in risk of melanoma, Queensland, Australia. International Journal of Cancer, 97(1), 90-95.
Stanton, W. R., Janda, M., Baade, P. D., & Anderson, P. (2004). Primary prevention of skin cancer: a review of sun protection in Australia and internationally. Health Promot Int, 19(3), 369-378. doi: 10.1093/heapro/dah310
Tuong, W., Cheng, L. S., & Armstrong, A. W. (2012). Melanoma: epidemiology, diagnosis, treatment, and outcomes. Dermatologic Clinics, 30(1), 113-124, ix. doi: 10.1016/j.det.2011.08.006
U.S. Department of Health and Human Services, O. o. D. P. a. H. P. (2010). Health Literacy Online: A guide to writing and designing easy-to-use health Web sites.
USDHHS. (2014). The Surgeon General’s Call to Action to Prevent Skin Cancer. Washington, DC.
van den Berg, M. H., Schoones, J. W., & Vliet Vlieland, T. P. (2007). Internet-based physical activity interventions: a systematic review of the literature. J Med Internet Res, 9(3), e26. doi: 10.2196/jmir.9.3.e26
Villegas, N., Santisteban, D., Cianelli, R., Ferrer, L., Ambrosia, T., Peragallo, N., & Lara, L. (2014). The development, feasibility and acceptability of an Internet-based STI-HIV prevention intervention for young Chilean women. International Nursing Review, 61(1), 55-63. doi: 10.1111/inr.12080
Vishvakarman, D., & Wong, J. C. (2003). Description of the use of a risk estimation model to assess the increased risk of non-melanoma skin cancer among outdoor workers in Central Queensland, Australia. Photodermatology, Photoimmunology and Photomedicine, 19(2), 81-88.
Wang, L. W., Miller, M. J., Schmitt, M. R., & Wen, F. K. (2013). Assessing readability formula differences with written health information materials: application, results, and recommendations. Res Social Adm Pharm, 9(5), 503-516. doi: 10.1016/j.sapharm.2012.05.009
Webb, T. L., Joseph, J., Yardley, L., & Michie, S. (2010). Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res, 12(1), e4. doi: v12i1e4 [pii]
Williams, A. L., Grogan, S., Clark-Carter, D., & Buckley, E. (2013). Appearance-based interventions to reduce ultraviolet exposure and/or increase sun protection intentions and behaviours: a systematic review and meta-analyses. Br J Health Psychol, 18(1), 182-217. doi: 10.1111/j.2044-8287.2012.02089.x
Willis, G. B. (Ed.). (2005). Cognitive interviewing: a tool for improving questionnaire design. Thousand Oaks, CA: Sage Publications, Inc.
Wolpin, S. E., Halpenny, B., Whitman, G., McReynolds, J., Stewart, M., Lober, W. B., & Berry, D. L. (2014). Development and usability testing of a web-based cancer symptom and quality-of-life support intervention. Health Informatics J. doi: 10.1177/1460458213495744
aFox Chase Cancer Center, 333 Cottman Avenue, Philadelphia, PA 19111. Carolyn.Heckman@FCCC.edu; Susan.Darlow@fccc.edu; Teja.Munshi@fccc.edu; Stephanie.Raivitch@fccc.edu; Linda.Fleisher@fccc.edu; Elizabeth.Handorf@fccc.edu
bBeHealth Solutions, LLC, 375 Greenbrier Drive, Charlottesville, VA 22901
cUniversity of Virginia Health System, 1215 Lee Street, Charlottesvile, VA 22908
dRutgers Cancer Institute of New Jersey, 195 Little Albany Street, New Brunswick, NJ 08901-1914 firstname.lastname@example.org