Emergency Manager Perceptions of the Effectiveness and Limitations of Mass Notification Systems: AMixedMethodStudy

Disasters can move quickly. Effective communication is a critical resource that can significantly enhance public safety. A mass notification system (MNS) uses text messaging to inform constituents of crisis, provide recommendations, connect to resources, and has the advantage of speed. Limited research has been conducted on the variables that influence the effectiveness, utilization, and perceptions of MNS. The extant study employs a multi-method approach to advance the scholarly knowledge onMNS. All emergencymanagers in a state were surveyed on issues of MNS enrollment, utilization, and brand. A subgroup of emergency managers were then interviewed to provide depth to the survey findings. Key findings indicate wide variability in MNS usage, little relationship between population size and enrollment, and a high perceived importance of MNS as a communicationmodality. Policy implications and recommendations are offered.


Introduction
In a time of crisis or disaster, information is a critical commodity. Decision-making is driven by information. Shelter in place or evacuate, direction of evacuation, urgency of action; all represent pre-crisis decisions predicated on quality information. Post-crisis information can similarly mean the difference between life and death-acquiring water, fuel, and food, access to shelter, health-care locations, recovery facilities, and similar, are all outcomes which require information. Even in non-crisis situations, information helps to drive decision-making (Haddow and Haddow 2008;Kapucu 2008).
There are many modalities for crisis period information distribution. Traditional media such as radio and television can provide detailed perspectives but presuppose electricity (at both distribution and reception points), attention of the public, and timely information. Newspapers can provide the highest level of detail but are often obviated due to the time lag between publication and acquisition, in conjunction with delivery requirements. Internet news sources are a tremendous source of timely information but require both power and signal supply (at both distribution and reception points), and the reliability of information may be questionable.
Mass notification systems (MNS) utilize cell phones and Short Message Service (SMS) or text messaging. This approach represents an important step in crisis communications (Leiva 2014) since many people have cell phones with them at all times and there is a high level of redundancy in signal distribution. A brief piece of information, texted to a large portion of the population, can provide preliminary alerts or key pieces of information (Madden 2015). Cell phones are not the perfect information distribution mechanism-they are vulnerable to spam, hacking, and false attribution. Bristow (2004) suggested that minimizing the consequences of a disaster is a direct artifact of effective communication between first responders and the community. While there is limited scholarly literature on the effectiveness of mass notification systems, the available studies indicate that an MNS can offer significant benefits to a community. Case studies testing an MNS demonstrate their importance and potential for mitigating the consequences of disasters (Edwards, Cuthbertson, and Peterson 2011;Kapucu 2008;Martin and Rice 2012).
Anecdotally, utilizing an emergency alert system can make a major difference in disasters. The recent California wildfires represent a key example. When wildfires struck the Sonoma and Napa county region in December of 2017, some local emergency managers decided not to use the emergency alert systems available to them (St. John 2018). They referenced a concern about inducing panic behavior and a lack of connectivity. Lake County officials made a different decision-they issued an emergency alert, broadcast through cell phones, ordering a mandatory evacuation. There were zero fatalities in that county and 33 deaths in Sonoma County (Davis and Somashekar 2017). Although Sonoma County is more populated and was hit harder by the fire, the lack of evacuation notices placed residents in a position where decisions were made with limited information.
Unfortunately the lessons of the Sonoma and Napa wildfires were not enacted by the following year. During November of 2018, the Camp Fire became the world's costliest disaster of the year (Rice 2019). A fast moving fire, exacerbated by dry conditions and 50 mile-per-hour winds, the Camp Fire quickly set upon Paradise, California. Within just a few hours, the city of Paradise was destroyed and the fire quickly moved on to other cities. Some emergency alerts were sent but distribution was inconsistent. Approximately 25% of Paradise residents had opted in to the emergency alert (CodeRed) system, however, many received no alerts. Some citizens received a warning, some received a more serious evacuation notice, and a large portion of residents received no contact at all. Only half of the cell phones in the area received any kind of alert at any time. Frustratingly, a number of citizens received alerts after the fire had destroyed their homes. Text messaging on mass notification systems are often first-in last-out, resulting in impracticable delays in text delivery. Seventeen cell phone towers were destroyed by the fire and 66 were damaged, impeding alert distribution.
The MNS is not a panacea of information distribution. There are important limitations and regulations, dictated by Federal Emergency Management Agency (FEMA) and the Federal Communications Commission (FCC), which define how and when notifications can be sent (Levia 2014). Additionally, satellites and towers must be in working order, phone carriers must have agreed to communicate messages (company participation is voluntary), and municipalities must have an agreement in place with FEMA. Limitations associated with communicating across cell phones via text messages also exist (brevity of messages, presumption of power, and recipient responsiveness). Effective decision-making is predicated on knowledge management (von Lubitz, Beakley, and Patricelli 2008). Decisions made under rapidly changing conditions are especially challenging. Boyd's (1976) Observe/Orient/Decide/Act (OODA) Loop-Observe, Orient, Decide, Act-provides a framework through which decision-making can be better understood. The intersection of knowledge management and decision-making is especially important as emergency managers make communication choices in alerts. An emergency alert system can be used mistakenly. This failure has the short term effect of causing undue alarm, and potentially panic, coupled with the long term effect of reducing trust and responsiveness to emergency alerts. The Hawaii ballistic missile crisis of 2018 is an example of an emergency alert failure (Fisher 2018;Wang 2018). During late 2017, North Korea tested several ballistic missiles and claimed their devices could reach Hawaii. During an internal drill, an employee misunderstood the nature of the drill, believed that the hypothetical missile attack was a real attack, and sent an emergency alert stating: "Ballistic missile threat inbound to Hawaii. Seek immediate shelter. This is not a drill." The message briefly caused panic as news media interrupted broadcasts to distribute the message. Local and state leaders quickly informed media of the mistake, however, 38 min elapsed before a correction emergency alert was sent. The Federal Communications Commission had to approve a revisionist alert as federal guidelines precluded a state agency from issuing a correction.
Given the potential role that mass notification systems can play in disasters and periods of crises, a greater understanding of their role, efficacy, and utility is needed.
The following sections review the available literature on MNS, describe a set of research methods and analysis on MNS enrollment, review qualitative perspectives of emergency managers on MNS, and posit conclusions and policy recommendations.

Literature Review
Effective communication between officials and citizens prior to, during, and after crisis situations can increase public safety and expedite response and recovery. MNS are intended to connect the public at large with emergency management personnel, providing a vehicle for relaying critical information. The effectiveness of an MNS is predicated on a number of variables, including contact information possessed by emergency management systems, speed with which information is provided, utility of the information, and the urgency of the communications (Haddow and Haddow 2008).
The proliferation of social media has produced an important network for reaching persons in threat of a crisis. Jurisdictions typically rely on cell phone text systems and traditional media (i.e. local television and radio) to communicate alerts. The omnipresence of social media platforms (such as Facebook, Instagram, Twitter, and others) provide a communication vehicle that may surpass traditional alert networks. Carvin et al. (2010) and Martin and Rice (2012) stress the role of social media. They contend that the sheer scope of social media and the pervasive, always-on modality of social media platforms, can reach far more people than crisis communications via traditional media (such as radio, television, and newspapers). Social media is not risk-free. The rapid dissemination of news must be contrasted with the possible dissemination of false-news. Malware and hacking also represent risks. Incorporating change-authentication and attribution systems on mass notification systems, with tagging such as block chain accounting, would likely further enhance the security of messaging systems. Pechta, Brandenburg, and Seeger (2010) support the utility of social media as a communication device in crisis situations. They advocate a multi-channel modality of crisis communications where there is a feedback system. This approach changes constituents from recipients of information to participants in the information distribution process. There is a critical flaw, however, in this approach. Citizens may not exercise the same judgment and discretion as emergency management professionals. Although there may be scenarios, such as a tornado, where eye witnesses can provide better information than a remote expert, in pre-disaster communication casting the widest possible net has many benefits but also risks. A panicky citizen Tweet (e.g. "Evacuate Now or Die"), if re-Tweeted on high profile platforms, can produce a dangerous and unnecessary flight.
There have been several studies of emergency notifications systems at colleges and universities. The findings of these studies generally note the importance of complete contact databases and the rapidity with which alerts are broadcast (Elsass, McKenna, and Schildkraut 2016;Madden 2015). Colleges and university campuses are not comparable to broader communities for a number of reasons. Most importantly, a college or university system has contact information for all members of their community by necessity. Students, faculty, and staff are beholden to a registration and/or payroll system which collects contact information as a matter of course. While a small percentage of the contact information is inaccurate or outdated, the vast majority of college and university community members are reachable. That is not necessarily true for a city, town, county, or even state where only a fraction of residents have provided their contact information to emergency authorities. Those local government institutions which do have contact information (i.e. city, county, or state tax collectors, schools, utilities) may be reluctant to share that information absent a judicial directive.
In his study of Florida's response to a prolific Hurricane season (in 2004 Florida was hit with four successive hurricanes), Kapucu (2008) found that effective disaster planning, communication between government officials and first responders, and community alert strategies were critical. Kapucu notes that citizens often do not listen to alerts. There are many possible explanations for this behavior: citizens may assume the alerts apply to others, that their government is mistaken, that the risk is exaggerated, or they experience alert fatigue. As noted in the federal by the Select Bipartisan Committee to Investigate the Preparation for and Response to Hurricane Katrina (2006) thousands of people stayed in New Orleans despite dire warnings regarding Hurricane Katrina. Sometimes the decision to ignore warnings is purposive (plans to "ride out the storm"). For some citizens, as evinced in the studies that followed Hurricane Katrina, many citizens lack the capacity or means to leave their homes without government agencies providing, and effectively advertising, transportation and temporary residence modalities (Bipartisan Committee Report 2006). Crisis fatigue in a place frequently wracked with weather related situations may also impede exhortations from government officials.
As with many technology choices, it is entirely possible that there is no single best approach. Malizia et al. (2010) reviewed emergency notification system protocols and attempted to define an ontology for mass notification, revolving around technology and need. They noted a high degree of variability in the various technologies available, with some concentrating on landline phones, some on cell phones, some SMS/text centric, some fax, and some based on Internet communiques (such as emails or alerts). The authors contend that matching community needs with technology access and sources is an important step in selecting the right notification system. They conclude that there is no single "right" system for citizen oriented disaster communications. Instead, jurisdictions must go through a complex calculus evaluating cost, effectiveness, range, technology access, and communication needs. The mode of communication represents one element influencing effectiveness. Another important element in the communication process is the decision-making of the communicators. Emergency Managers are frequently responsible for coordinating alerts and emergency communications. Key decision points in the mass notification process include: presensitization warnings, word choices, urgency evident in the message, frequency and timing of communiques, and integration with other media sources. This decisionmaking process is complex and influenced by an array of factors.
The deficits in the existing literature on MNS indicate several important areas of research and raise key research questions. The utility and effectiveness of MNS, the variables influencing MNS implementation, and emergency manager perceptions of MNS warrant study. One of the key gaps in the literature is the perceptions of emergency managers. This is important as those persons play a critical role in the utilization of these systems and they should have high knowledge levels regarding their potential. Research questions addressing these issues include: How are persons enrolled in a Mass Notification System? How does a jurisdiction select a Mass Notification Systems?
How are messages sent via Mass Notification Systems?
What is the role of the Emergency Manager in defining and distributing notifications?
The following description of methods and population addresses these topics.

Research Methodology
The role and efficacy of MNS warrants research, particularly concerning enrollment and MNS utilization. The following paragraphs describe a research methodology for collecting and analyzing relevant data. A mixed methods approach was employed to facilitate triangulation of findings (Yin 2014). In this approach, alignment between quantitative and qualitative findings enhances outcome validity. Additionally, each data set provides insight into the other and can facilitate interpretation.

Population
The Virginia Department of Emergency Management (VDEM), in response to a request by researchers, provided contact information for 129 emergency coordinators across Virginia. Most jurisdictions were county level although some were city level. The Commonwealth of Virginia has 95 counties and 34 independent cities which are considered county level equivalents for census and population purposes. Each jurisdiction (county or city) provides an emergency coordinator's name and contact information to the state level agency, VDEM. This contact information was provided to researchers. This population database represents a census rather than a sample. A census is a depiction of all possible participants whereas a sample is a sub-group of a population. Based on the size (n = 129) and the availability of full contact information, researchers elected to use a census population approach since a sampling approach would reduce validity and reliability.

Data Collection-Survey
Emergency coordinators were contacted via email, sent from one of the researchers, with a request and invitation to participate in research on regional usage of MNS. A link to a survey was included in the email. A second round of email and survey links was sent to non-respondents. All data were downloaded to a spreadsheet database system and then analyzed.
Prior to solicitation of participation, all elements of the study were approved by the Institutional Review Board of the University which houses the authors. This step insured ethical research procedures and appropriate usage and security of data.
The survey posed a series of questions on MNS type, accounts, utilization, and registration. The following questions were posed: -What is the name of your Mass Notification System? -How many registered accounts are in your MNS? -Is your MNS "opt-in"? -How many accounts did your system add in FY2017? -How many public-facing alerts were sent in FY2017? -How many accounts did your system lose or shed in the last measurable period?
These questions were selected due to simplicity, ease of answering, and relevance to the research questions. Usable responses were received from 40 participants, representing a 31% response rate.

Participant Demographics
Following administration of the survey, jurisdiction demographics were collected. Table 1 describes demographics of both responding and non-responding jurisdictions.
When comparing the demographics of the Responding Jurisdictions to the Non-Responding Jurisdictions, they are similar on most variables except Population and Households. The Population mean for the Responding Jurisdictions is notably higher. This is because a concerted effort was made by researchers to obtain information from the largest, and therefore most influential, jurisdictions in the state. For example, there are four Non-Responding Jurisdictions with populations that exceed 100,000 (the range for those four jurisdictions is 105,210-183,218). The Responding Jurisdictions include thirteen jurisdictions with over 100,000 residents, including four with populations between 200 and 299,000, three between 300 and 399,000, two between 400 and 499,000, and one with over 1.1 million residents. Thus the sample is skewed in terms of population. This is evident in the very large standard deviations (204,210 for Responding Jurisdictions vs. 30,641 for Non-Responding Jurisdictions). While there is a clear disparity between the Responding and Non-Responding Jurisdictions in terms of population, the focus of this study is on those agencies with MNS rather than a between groups comparison.

Data Collection-Qualitative
Interviews were conducted with several emergency managers via phone following survey administration. These participants represent a non-systematic sub sample of survey completions, best described as a convenience sample. A total of five interviews were conducted with emergency manager that participated in the survey. A survey question was posed to respondents asking if they would be willing to participate in a follow-up interview. All who responded affirmatively were contacted. Two of the seven affirmative responses subsequently declined or refused to participate, leaving a total of five interview participants. The qualitative portion of the data collection was designed to generate data which supplemented survey findings and provided insight into the decisionmaking of emergency managers regarding MNS issues. Key constructs addressed during the semi-structured interviews included: MNS system used, the MNS selection process, utilization decision-making, enrollment process, obstacles to enrollment, and current MNS needs.
The interviews followed a semi-structured method with a list of constructs to be tapped, retaining flexibility to move among constructs as the participant provided information. Notes were taken during the interview. Immediately after the interview, these brief notes were converted to more comprehensive notes including details. Qualitative findings were then analyzed to identify themes and patterns. Responses to each construct were then reviewed and patterns were identified and tracked. Outlier responses (those provided in very limited numbers) were treated as far less representative than responses echoed across participants. These consistent responses were designated as themes and represent the qualitative findings.
The interviews were not recorded in order to guarantee confidentiality and increase honesty in responding. Since questions were posed on budgets, supervisor decision-making, and the authority of the participant, researchers were concerned that honesty in responding would be diminished if interviews were recorded. Prior to conducting interviews, conversations with former emergency managers were conducted to validate the constructs. Those conversations raised concerns about recording the interviews. Researchers recognize this is not the ideal data collection approach but the small population and concerns over honesty in responding led to this approach.

Data Analysis Plan
Survey data were collected in a spreadsheet program then transferred to an SPSS database. In addition to the survey data, census data were collected and assigned Emergency Manager Perceptions of Mass Notification Systems to each jurisdiction. These data included: population, income, households, poverty rate, percent disabled, education, and population density information. Descriptive analyses were conducted, followed by correlations and basic inferential tests, including ANOVA (Analysis of Variance). Interview data were analyzed for themes and patterns across respondents.
Qualitative interview data were analyzed from a positivist perspective (Miles, Huberman, and Saldana 2019). Notes taken during the interviews and interview summaries, written after the interviews were completed, provided the data. A positivist approach to qualitative data analysis is predicated on identification of recurring data points (Lin 1998). Those data points that occur most frequently are treated as most significant while those that occur infrequently are viewed as least significant, with the possibility for exceptions.

Results
To address research questions on Mass Notification System enrollment, utility, and usage, survey data were collected from emergency coordinators across Virginia. Forty coordinators (out of 129 in the entire population) returned completed surveys (for a response rate of 31%). Additionally, a series of interviews were conducted with emergency managers to provide insight into survey findings and key project constructs.
Survey results indicated a wide range of participants in terms of populations served. Seventeen respondents represented populations between 5000 and 50,000, ten served populations between 50,000 and 100,000, seven represented populations between 100,000 and 250,000, five served populations between 250,000 and 500,000, and one served a population over one million. The median household income for responding jurisdictions was approximately $62,000 with a standard deviation of $24,000.
The survey posed several questions about the nature of the mass notification system utilized in the responding district. A significant majority of responding districts (33 of 40, or 82.5%) are opt-in. The remaining seven (17.5%) are not opt-in. The brand of mass notification system utilized was requested. Half of participating districts (N = 20, 50%) use the trademarked brand Everbridge. CodeRed was the second most commonly utilized brand (N = 10, 25%). The remaining districts reported using a different brand (N = 10, 25%).
Participants were asked how often they send notifications. Most jurisdictions (80%) have sent less than 10 alerts in the past year. Two jurisdictions sent between 100 and 500 and three sent between 1000 and 3000. One jurisdiction reported sending 5600 notifications in one year, which averages to over 15 notifications sent per day. The jurisdictions sending hundreds of notifications each year reported that the vast majority of notifications were traffic related. Traffic notifications informed users of accidents, construction, weather issues, or other delays. The granularity of data for certain types of communication are quite high resulting in difficulties of non-homogeneous characterization.
A series of correlations were conducted between key variables. Correlations (bivariate) were run between Population, Income per Capita, Poverty Rate, and Registered Accounts in MNS. As demonstrated in Table 2, Population, Income per Capita, and Poverty Rate are all significantly correlated. Registered Accounts in MNS is not significantly correlated, suggesting that large jurisdictions do not necessarily have more registered MNS accounts.
Analysis of Variance (ANOVA) tests were conducted between Population and Registered MNS Accounts. The F score was 5.232 (at 33 degrees of freedom), with a significance score of 0.096. The outcome of the ANOVA was not statistically significant indicating that jurisdiction population does not predict registered account numbers.
The lack of statistical significance in the ANOVA is likely an artifact of several small jurisdictions which claim large numbers of MNS accounts. The finding that population size does not predict MNS accounts is interesting and potentially important. This indicates that some jurisdictions have been successful in garnering account registrations while others, including several large jurisdictions, have not been successful. There are large jurisdictions, encompassing several hundred thousand residents, with low numbers of accounts registered for the MNS. If a significant event were to occur, the population of these jurisdictions would not be well informed via the MNS. Semi-structured interviews were conducted with a non-systematic subsample of survey participants. These interviews drew on findings from the survey analysis and addressed issues of MNS utilization, selection, effectiveness, obstacles, and needs. Several themes emerged from the interviews. The emergency managers that were interviewed overwhelmingly felt that MNS represented an important and potentially critical modality of communication with the public. During a time of crisis, particularly a rapidly evolving scenario (such as a weather emergency), MNS could significantly enhance public safety. Participants also indicated that MNS should be used infrequently to avoid development of notification fatigue. When told that some jurisdictions used MNS hundreds, even thousands of times per year, participants were surprised and noted that overuse could lead to ignoring important messages.
The emergency managers interviewed consistently expressed concern about enrollment issues. While they were disappointed that a small portion of their constituency voluntarily enrolled for MNS communications, they felt greater recruitment and education would enhance enrollment. The obstacle to these recruiting methods revolved around funding. Unfortunately, none of the interview participants played a significant role in the selection of the MNS for their jurisdiction, noting this was a jurisdictional bidding and contract process. Several expressed a desire for more flexibility in issuing notifications. Some participants suggested that collaborating with public safety agencies, particularly fire, police, and emergency medical services, could extend the capacity of the MNS.
When asked about the decision-making process of which issues rise to the level of warranting MNS consideration, variation existed among interview participants. Some used pre-defined protocols that list some of the issues that can be considered for MNS. Other participants described a subjective process which weighed potential risk and consequences with concerns about unnecessary communication and alarm. No participant expressed regret or concern about the decision-making process utilized by their jurisdiction.

Conclusions
The rapidity and urgency of crisis situations necessitates effective communication modalities. A MNS uses SMS text messages sent to cell phones to provide critical information to constituents who have enrolled in the MNS. While federal agencies such as the National Weather Service can send weather alerts to all cell-phones in a region, an MNS requires enrollment. While the literature on crisis communication is extensive, the research on the effectiveness and limitations of MNS is nascent. Studies on discrete communities such as college and university campuses have demonstrated that an MNS is a viable communication modality (Elsass et al. 2016;Madden 2015). Research conducted in jurisdictions like cities and towns is less defined-while an MNS can be a highly effective communication tool, enrollment and messaging utilization play an important role (Bristow 2004;Leiva 2014). Critical points in the MNS process are emergency manager decision-making, citizen recruitment and enrollment, and messaging utilization.
Recent examples of California wildfires underscore the importance of timing and decision-making. In some communities, MNS warnings provided important information to residents who were able to evacuate prior to extremely dangerous wildfires (Davis and Somascheker 2017). In other communities, warnings were either not issued or came too late to matter (St. John 2018). Although a raging and rapidly moving wildfire is an extreme example, the urgency and involuntariness of crisis demonstrates the importance of communication between decision makers and their constituents.
Multiple gaps exist in the literature regarding MNS. The role and decisionmaking process of emergency managers, the persons generally responsible for utilization of the MNS, is poorly understood. The variables influencing MNS type and selection are also understudied. Exogenous variables such as population and demographics have not been considered relative to MNS usage.
The extant study attempted to address some of these deficits using a mixed method approach. A survey of all emergency managers (relying on a census rather than a sample) was conducted. Questions were posed addressing MNS brand, usage, and enrollment issues. Subsequent to the survey, a convenience sample of follow-up interviews were conducted to provide depth on the survey results.

Key Findings
Research results were derived through several modalities-a survey of emergency managers, interviews with a small number of survey participants, and collection of census and economic data to highlight jurisdictional differences. Several interesting survey findings emerged. The majority of responding jurisdictions send a very small number of messages to those enrolled (less than ten per year). Some jurisdictions send a much higher number-this has implications for message fatigue and continued attention to messages. Population size and economic variables seemed to play little role in MNS enrollment and messaging. Enrollment is a critical issue for MNS and decision makers need to further explore enrollment recruitment. The most frequently selected MNS brand is Everbridge, followed by CodeRed.
Semi-structured interviews with emergency managers echoed some of the survey findings. Enrollment in MNS represents a critical area of concern. Strong views were evinced regarding the potential for MNS in crisis situations. Overuse was a major concern and managers were very surprised that some responding jurisdictions use their MNS on a daily basis. Recommendations for enhancing enrollment via education and communication, and collaboration with other public safety agencies, were offered by participants.
The decision-making process employed by emergency managers assessing when to send alerts, and the information those alerts should include, is important and interesting. Interviews indicated there is variation in the process and weighing of decision points. See an application of a conceptual decision-making framework illustrated as the OODA Loop (Boyd:1976).

Policy Implications
A mass notification system which reaches a very small percentage of residents is not an effective tool for crisis managers. Findings from the extant research indicate that a small percentage of residents in participating jurisdictions have opted into the MNS. Enhancing the reach of an MNS should be a high priority of emergency managers and their organizations. It is not clear what the "right" number of registrations for an MNS should be. A jurisdiction that works hard at securing registrations might earn as much as 20-30% of residents' contact information. A jurisdiction that simply provides a portal on a city or county website could have as few as 1 or 2% of residents registered. This begs the question of how hard should jurisdictions work to secure registrations on their MNS.
Jurisdictions must make careful decisions regarding the person(s) who have access to mass communication networks. Their qualifications and judgment should be assessed and approved pre hoc. The risk of political influence exists, especially in highly charged situations. Transparency is also important in this context. Similarly, the messages need to go to the right people in a secure fashion. Proper authentication of enrollment criteria may allow for improved protection against communication data hacking vulnerabilities.
Elliot Aronson, the famed social psychologist, in a review of the literature on motivation and human action states that fear of an outcome is not enough to motivate people to action (2008). Instead, people need to be motivated and a credible path needs to be provided which people can follow as a way to resolve the potential crisis. This has direct implications for recruiting persons to join a mass notification system. Aronson states: To convince people to prepare for disasters and change behavior they are happy with, therefore, we must puncture their rationalizations. We must make them feel vulnerable to the disaster and competent in preparing for it coping with it. Inducing fear is an excellent way to induce people to behave in a rational manner in the face of disaster, but if and only if that fear is accompanied by recommendations for action that are concrete, effective, and doable (Aronson 2008: 869).
While fear may be an effective motivator, that does not make it the best motivator. Fear can increase momentum or freeze action, which makes it an important variable in the emergency preparedness process.
In contrast to Aronson's approach, Neocleous (2007Neocleous ( , 2008 discusses the role of government responsibility in protecting people. Neocleous also presents a compelling argument for the risks of government over-reach, suggesting these always end in failures (2008). Rather than predicating motivation on fear, Neocleous suggests motivation can be predicated on such concepts as equity.
From a policy and recruiting perspective, utilizing social good as a motivator, in conjunction with a full awareness of risks and benefits, may motivate people to participate in a mass notification system. Portraying residents as participants in safety process, where someone who receives notifications can then contact friends and family and encourage them to take action, could serve as an example of the kind of behavior agency decision makers wish to encourage. When jurisdictions decide to place a priority on securing registrations for MNS, residents will be safer and better informed.

Limitations
This study faced a number of methodological limitations that reduce internal validity and limit the scope of generalizability. The response rate of 31% was not as high as the authors desired. It is possible that findings were derived from outliers and non-representative jurisdictions. Population and demographic analysis suggest otherwise but, absent a significantly higher response rate, inferences to nonrespondents should be made with caution. All data were collected in a single state. While the authors believe that the findings are reasonable representative of the conditions and population of that state, generalizing findings to other states should be done with caution. The survey was conducted following a census model but the qualitative data collection, via interviews, was a convenience sample of only a small number of survey respondents. The qualitative findings are interesting but, given the small N, are not generalizable.

Future Research
The research was conducted without funding and relied on the willingness of local emergency managers to participate. Funding would facilitate greater data collection modalities and attempts. Multi-stage surveys, more interviews, and case studies of jurisdictions serving varying populations (i.e. large urban, suburban, rural settings) would significantly inform the scholarly literature. Data collection with citizens, drawn from populations of those enrolled in MNS and those not enrolled, could inform decision makers as they attempt to expand the lists of persons connected to local MNS. More sophisticated analysis of the notifications sent and their resonance with intended populations could inform emergency managers regarding topics like messaging fatigue and efficacy.
The decision-making process of emergency managers, particularly in moments of extreme stress and duress, also warrants further study. Test and retest scenario-driven exercises that assess the decision making process will likely improve stressed decision making and warrant evaluation. Application of a conceptual framework on decision-making, such as Boyd's OODA Loop (1976) could provide insight into the key decision points and the variables that influence those steps.
A mass communication system, as with any high priority technology, represents a viable target for hacking. Cybersecurity research and policy consideration represents a needed element in the crisis communication field.