Qualities before Quantities: A Framework to Develop Dynamic Assessment of the Nonprofit Sector

1 Why Does the Nonprofit Sector Need an Index?

An index is an aggregation of metrics that assess quantitatively a state or phenomenon that cannot be measured directly. For example, the Legatum Prosperity Index assesses global prosperity by compiling national indicators such as economic quality, business environment, governance, personal freedom, social capital, safety and security, education, health, and natural environment. These data then inform policy development (The Legatum Institute 2017).

The purpose of an index is threefold. First, it portrays the current state of a phenomenon. Second, it illuminates cycles and trends over time, making it possible to assess its likely future direction. Third, it guides effective policymaking (Stock and Watson 1989). The process of index development includes selecting what is to be measured and identifying the correct blend of leading (predictive), lagging (confirmatory), and coincidental (real-time) measures. Other process components include determining the methodology for collecting and interpreting the data, and comparing those with data from prior years to assess trends and predict future trajectories (Stock and Watson 1989).

While the private and public sectors assess their health with measures such as financial statements, the Dow Jones Average, and Gross Domestic Product, the nonprofit sector currently has no definitive index to measure its vitality. This is problematic because without a commonly accepted measurement framework trackable over time, it is difficult to recognize trends and develop effective policy interventions. Lack of sector-wide data also makes it challenging to focus collective attention, organize coordinated action, and secure resources to address systemic problems.

Yet measures exist at the macro (societal) and organizational levels that can be used to develop such an index. Examples at the macro level include community wellbeing indicators (e. g. Warner and Kern 2013; Kim 2016) that aggregate measures such as health outcomes, social cohesion, land and water quality, and economic opportunity. Healthy Communities initiatives (Norris 2014) and the National Conference on Citizenship’s Civic Health Index (National Conference on Citizenship 2017) measure civic health, such as the Greater Seattle Civic Health Index (Seattle CityClub 2018) and Texas Civic Health Index (Jennings and Bhandari 2018). Similarly, universities in San Diego (Deitrick et al. 2016), North Central Florida (Jones et al. 2016), and elsewhere have researched and issued reports on the vitality of their communities’ nonprofit sectors.

Regional initiatives like these provide useful information for local policymaking and action. However, their different methodologies and data categories make national and global comparison very difficult. Further, these measures are not necessarily applicable at the organizational level. If they were, nonprofit organizations could use these same metrics to develop programmatic goals and assess organizational impact and effectiveness to populate community indicators.

Instead, nonprofit organizations “ … find themselves devoting scarce resources to juggling multiple, and sometimes irreconcilable, performance and accountability expectations” (Ebrahim 2010, 629). It not surprising, then, that focus regularly defaults to financial measures since financial sustainability is problematic for many organizations. Examples of common financial metrics include solvency (e. g. financial ratios) and liquidity (e. g. cash flow, short-term investments and debt), with more progressive organizations adding measures of financial flexibility such as reputation, creditworthiness, and donor engagement (Zietlow 2010). However, financial metrics overlook non-market measures of well-being, such as health and social cohesion

2 Problems of Measurement

Many widely accepted metrics have been developed for the private sector. Why, then, is measurement more difficult in the nonprofit sector? Here I discuss four reasons: intangible resources, interdependencies across time and scale, the need for collaboration among people with divergent viewpoints, and path dependence.

3 Design Questions and Guiding Principles

An index can help develop this interdependent, multi-level understanding. Selecting the wrong metrics, however, can damage organizations and communities by promoting resource investment in activities that do not lead to envisioned impacts and leave clients without services (Cutter et al. 2008). Before choosing measures, it is vital to reflect on the purpose and goals of the measurement, working with stakeholders to identify desired qualitative attributes before diving into quantification (Meadows 1998). This deliberation should consider tradeoffs that choices entail, because selecting what and how to measure signals what we value and simultaneously constitutes and perpetuates values (Meadows 1998; Bowles 1991; Graeber 2001).

To develop an index framework that addresses the measurement issues identified above, this paper asks two fundamental questions: 1) how can a nonprofit health index reflect the full span of the nonprofit sector’s value creation performance, and 2) how can interaction effects be measured and accounted for across levels? Complexity science and axiology (philosophy of values) serve as the conceptual pillars of a proposed framework. A complexity approach is essential because human systems are unpredictable and interdependent, e. g. small perturbations can have effects elsewhere in the system that cause serious unintended consequences (Carlson and Doyle 2002). Metrics seek to impose control and order on a system, but control is not always possible in complex environments (Cameron and Abouchar 1991).

Thus, rather than seeking to reduce complexity, metrics and analytical methods must allow for self-organization and emergence. Emergent phenomena arise spontaneously from the interactions of individual agents (Rickles, Hawe, and Shiell 2007). Enhancing agency fosters emergence and increases the generative potential of a system (Bandura 2006). Because human systems are driven and affected by agentic choices and purposive goal setting (teleology) it is imperative to articulate the values that underlie choices and goal selection. Index development must provide a way to make explicit what people value (e. g. freedom, inclusion, equity), identify what they want to strive for (e. g. mission, goals, objectives), acknowledge and consider tradeoffs, and account for consequences of their choices and their impact on others (Mayer, van Daalen, and Bots 2013).

4 Capabilities as a Design Framework

An index based on a capabilities approach is a way to operationalize these conceptual pillars. The premise of capabilities is that the development of individual capacities in people leads to development (emergence) of organizational and civic capacity (Sen 2005; Nussbaum 2001). A capabilities approach starts with the question, “What is each person able to do and to be?” (Nussbaum 2011, 18), with the goal of increasing personal freedom and agency.

A capabilities approach aligns with three elements of resilient complex systems: diversity, reciprocity, and emergence. Diversity is heterogeneity characterized by noticeable variation of social and cultural identities among people co-existing in the various contexts (Stevens Institute for Technology 2006). More diversity increases a system’s stability and capacity to generate new options (Folke 2006; MaCann 2000). Reciprocity (mutual exchange of beneficial acts) promotes trust, sustained exchange, and ongoing alignment toward shared goals under dynamic conditions (Bruni 2008). This exchange process is recursive: the outputs of the system (trust, shared goals and values) loop back through a feedback process to become new physical and social inputs for subsequent cycles (Tidball & Weinstein 2011). Together, the qualities of reciprocity and diversity promote a system’s capacity for emergence, fostering its resilience (the sustained ability to recover from perturbations). These qualities enhance the system’s evolvability, the capacity to expand degrees of freedom by leveraging randomness. This improves the system’s ability to fit with a greater number of possible future environments (Nature 2018).

From a values perspective, a capabilities approach supports freedom, dignity, and the emergence of community flourishing because it promotes continuous expansion of individual and collective degrees of freedom (Nussbaum 2011). Capabilities exist and can be measured at the organizational and societal level (Kato, Ashley, and Weaver 2017), providing an architecture that accommodates emergence through mutual interaction effects among people, organizations, networks, and communities. In organizations, capabilities are often discussed in terms of value creation capacity, particularly absorptive (Todorova and Durisin 2007), adaptive (March 1991), and generative (Castillo and Trinh Forthcoming). These activities are generally called capacity building in the nonprofit sector (Castillo 2016; Fowler 2004).

5 Social Accounting as a Way to Measure Capabilities

It is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.

Maslow 1966

If the purpose of accounting is to provide information about relevant events, communicate about and promote control of organizational assets, and produce data to support decision making (Sorter 1969; Gray 2006), then financial accounting neglects essential aspects of the nonprofit sector’s value creation inputs and outputs (e. g. moral purpose, cooperation, mutual benefit). Accountability—the process of being held responsible for one’s choices and actions—is another important objective of accounting. However, accountability consists of vertical (legalistic) and horizontal (relational) dimensions (Moncrieffe 2011). Relational elements include equitable representation of interests, transparency, and participative deliberation. Because conventional accounting focuses solely on financial resources, it addresses only the vertical aspect of accountability (demonstrating legal compliance). It does not attend to relational accountability. A nonprofit index must therefore address both vertical and horizontal accountability to reflect its relational and moral dimensions (Vosselman 2016).

How can capabilities (e. g. agency, value creation potential) address these gaps? And how can they be measured and accounted for to provide a starting point for an index? Social accounting emerged in the 1970s as a way to attend to values and relational accountability, recognizing both social and economic dimensions and impacts of organizations. Social accounting illuminates the value that an organization adds to its stakeholders through its production of outputs, outcomes, and externalities (Land 1996). Social accounting represents both financial and non-financial resources, sometimes ascribing financial values to non-monetized outcomes such as volunteer labor through market comparisons (Mook, Richmond, and Quarter 2003). Social entrepreneurs use various forms of social accounting to show how they produce social return on investment (Nicholls 2009). In a nonprofit context, this value added approach conveys that wealth is produced as a result of many different agents working together cooperatively for public benefit (Burchell, Clubb, and Hopwood 1985; Mook 2014).

Social accounting provides a fuller picture of an organization’s resource inputs and outputs and the effect these have on various stakeholders, e. g. customers, vendors, shareholders, employees, communities where they do business, etc. (Mook and Machokoto 2017). One challenge, however, is comparability. In conventional financial accounting, Generally Accepted Accounting Principles (GAAP) make comparing balance sheets of various organizations rather straightforward. The non-standardized aspect of social accounting makes comparison more difficult.

6 Developing an Index

Operationalizing capabilities as multiple forms of capital (e. g. social, intellectual, political, reputational) is a way to account for both tangible and intangible resources and value creation. A starting point for creating a national index is to gather existing indicators and sort them into a capitals framework. Organizations such as Independent Sector, GuideStar, National Council on Nonprofits, National Center for Charitable Statistics, Urban Institute, Nonprofit Finance Fund and others conduct ongoing research and share their findings regularly. Much of these data can be categorized as capitals. Building from those and other reports, Table 1 provides examples of possible indicators for each type of capital at the individual, organizational, community, and national levels.

At the community level, regional wellbeing indicator initiatives can be operationalized as multiple capitals (Kee 2017). Similarly, Anand, Hunter, Carter, Dowding, Guala, and Van Hees (2009) describe a variety of measures for capabilities at the individual level. Kato, Ashley, and Weaver (2017) have identified measures used at the organizational and community levels. It would be fairly straightforward to translate these various capabilities measures at each level into a multiple capitals framework.

At the organizational level, adopting a capabilities approach is a way for organizations to align their operations, capacity building, program evaluation, strategic planning, and stakeholder reporting. To begin, nonprofits can create measures for each type of capital in a way that is meaningful to them. For example, tracking the number of people who follow the organization on social media could be an indicator of its social capital. This connects a strategic objective (increasing community engagement) to a key performance indicator (Kanter and Paine 2012) that could be used in an organization’s annual reporting as an indicator of its social capital.

Scholars have developed a variety of measurement methods, including surveys and network analysis (Lakon, Godette, and Hipp 2008), for many of the capitals. Measures exist for social capital (Van Der Gaag and Webber 2008), intellectual capital (Petty and Guthrie 2000; Chen, Zhu, and Xie 2004; Liebowitz and Suen 2000), reputational capital (Wartick 2002; Cravens, Goad Oliver, and Ramamoorti 2003), political capital (Casey 2008), rule of law (Skaaning 2010; Santos 2012); and spatial capital (Marcus 2010). Adopting such measures and framing nonprofits’ work as capabilities development (i. e. multiple capitals) would provide a strong narrative to explain their value creation process and impact. It also provides powerful new language grounded in sound logic (complexity theory) to discuss their work with policymakers, donors, volunteers, and other stakeholders.

At the individual level, measures can assess capabilities developed among people served by the organization. For example, surveys can assess to what extent program participants increased their connections to other people in the community (social capital), knowledge and skills (intellectual capital), and voting and civic participation (political capital). With thoughtful design, these measures could serve as an evaluation framework to assess program effectiveness (e. g. administering pre- and post-program surveys). Instead of an organization being pulled to attend to multiple accountabilities, framing program objectives as the development of capabilities (multiple capitals) aligns evaluation with the organization’s social accounting reporting. In a similar vein, annual evaluations and professional development plans for employees can be translated into a multiple capitals framework (e. g. increasing their skills and knowledge; improving workplace relationships; increasing their level of engagement). Aggregating this data provides an assessment of the organization’s human capital for its social accounting reporting.

6.1 A Case Example

To illustrate what this might look like in practice, let us consider the community transformation described in Emery and Flora’s 2006 research on rural development and how this could be depicted using Integrated Reporting. That case study begins by describing how community leaders used their social capital to recruit participants for an eight-month leadership development program. Using human, financial, and intellectual capital to offer the course, it trained three cohorts of people. Besides increasing students’ knowledge and leadership skills, the evaluation found development of cultural capital (changing norms that increasingly valued community involvement) and political capital as “ … people outside the traditional leadership structure were finding voice in community affairs.”

This in turn affected structural aspects of power as more youth and diverse stakeholders became involved. For example, the community’s 2004 election saw multiple candidates run for each position, whereas before it had been difficult to recruit even one person to run. “Without changes in traditional leadership structure and actors, the community could not have mobilized citizens to support changes” (Emery and Flora 2006, 25). It also attracted new investment of financial capital from utility companies and growth in planned giving bequests through the program’s partnership with the Nebraska Community Foundation. In the business community, the program’s local entrepreneurship team worked with youth, businesses, and the regional economic development council to grow its entrepreneurship portfolio to over 100 businesses and attract new manufacturing infrastructure. Figure 1 portrays some of the value creation transformations using an Integrated Reporting diagram.

Figure 1:

An integrated report for Emery & Flora’s Spiraling Up Emery and Flora (2006).

Some readers may notice this diagram resembles a logic model, a tool used in the nonprofit sector to plan and evaluate programs and illustrate a theory of change. This is one of the major benefits of Integrated Reporting—it promotes integrated thinking (the alignment of strategy, resource inputs, and value creation processes to show how these combine to produce outputs, outcomes, and impact) in a way that links strategy to reporting.

Another benefit is that IR captures externalities (impacts the organization makes on the external environment). Nonprofits’ externalities are generally positive, e. g. the public benefit they create through mission fulfillment. The IR model makes this public goods production very visible. Similarly, negative externalities such as pollution (if applicable) should also be portrayed to demonstrate transparency and accountability (Muller, Mendelsohn, and Nordhaus 2011). Adopting Integrated Reporting in the nonprofit sector is thus a way to normalize transparent reporting of a firm’s impacts on society. This can raise public expectations for similar transparency in the private sector.

Why is it desirable to adopt a multiple capitals approach to metrics across multiple levels? First, as discussed previously, value creation happens through interactions across multiple levels and through indirect effects (Kenny 2018). Outputs at one level become new inputs at other levels. Understanding the health of each level avoids reductionism. Second, the sector in many ways has become siloed by sub-sectors (e. g. environmental, human services, arts and culture). This fragmentation inhibits collaborative work on systemic challenges like poverty and homelessness. By seeing the entire landscape of value creation and how these subsectors weave together across levels, organizations can coordinate more effectively on shared large-scale goals like community flourishing. A third reason is the balancing of power differentials. Scarcity of financial capital magnifies the power of funders (Choi 2017). Illuminating the entire spectrum of resources that stakeholders bring to the table can help balance those power differentials. A capabilities (multiple capitals) approach may also reduce reporting burdens by unifying strategy and accounting into an accountability framework that is usable in multiple contexts. Finally, a multiple capitals approach can illuminate fundamental aspects of the nonprofit sector’s value creation currently overlooked by conventional accounting.

Adopting this capitals framework across multiple scales (individuals, organizations, networks, regions, states, and nationally) aligns stakeholders’ different visions of what success looks like—capabilities development. This multiple capitals approach also provides a cohesive and compelling policy narrative. Figure 2 illustrates how a capabilities approach, using multiple capitals as measures, can scale across levels (individual, organizational, collaboration, community, nation), perhaps connecting to global initiatives such as the United Nations’ Sustainable Development Goals (SDGs).

Figure 2:

Fractal architecture. Integrated reporting and its multiple capitals framework can be used at any level (e. g., individual, organization, community, national). Investment in capabilities at lower levels expands degrees of freedom for both that level and above, producing emergent macro properties. For example, Mortenson (1997) empirically demonstrated how investment in higher education produces individual benefits for the learner as well as macro-level benefits such as a stronger tax base, prepared workforce, higher levels of parent involvement in schools, and higher levels of civic participation.

The flexibility of this capabilities architecture acknowledges and accommodates the importance of context. Best practices in Integrated Reporting require a section called Materiality to discuss potential risks (e. g. financial, environmental, social) to performance and goal attainment and how the organization plans to address and manage that risk (Eccles, Krzus, and Ribot 2015). In that section, nonprofit organizations would discuss their particular demographics, financial and governance regulations, community needs, and other contextually unique aspects of their operations to provide a complete picture of their operating environment. The best Integrated Reports use a blend of qualitative descriptions, quantitative measures, and data visualization to depict “ … the relationship between a company’s environmental, social, and governance performance and its profitability and value” (Eccles, Krzus, and Ribot 2015, 115). While metrics are not yet standardized, Integrated Reporting offers a conceptual framework to align collective problem framing and solution creation.

7 Accounting for Interaction Effects across Levels

The previous sections described how capabilities can be operationalized as multiple forms of capital and measured at the individual, organizational, and societal levels. The case example illustrated in a simplistic way how value is created across multiple levels over time by converting resource inputs into outputs, outcomes, and impact. It also visualized how some of these externalities cycle back to become new inputs for various levels.

Figure 3 illustrates development as the dynamic flow of capabilities across multiple levels through interaction effects over time. This figure is inspired by Gottlieb’s notion of probabilistic epigenesis, a model he developed over many decades of empirical work studying growth and development in birds. His research suggests that development is not a linear process, but rather the qualitative reorganization of structures arising from interactions across multiple levels, e. g. genes, neural activity, behavior, and higher levels of cognitive function (Valsiner 2007). Understanding the interplay between these various levels requires both qualitative and quantitative analysis (Valsiner 2007; Shrestha and Arnett 2008).

Figure 3:

Development over time through multi-level interaction effects.

Ecology similarly examines interdependent processes and relationships across multiple levels, e. g. organisms, populations, communities, food webs, and ecosystems over multiple time scales (Woodward, Perkins, and Brown 2010). To deal with such complexity, ecologists first seek qualitative understanding, e. g. pattern recognition. The nonprofit sector must similarly learn to identify patterns across multiple levels, such as the development and transformation of capabilities across scales.

In the natural sciences, computational science methods like agent-based modeling (ABM) have become important tools to understand such complex behavior. ABM has shown great analytical capacity to reveal underlying dynamics of complex adaptive systems. ABM conducts experiments in silico, revealing patterns that develop through repeated interactions among individual agents over time. These patterns sometimes include self-organization and emergence, leading to insights about generative mechanisms underlying those phenomena (Macal and North 2010).

Because ABM has the computational capacity to handle vast amounts of data, it is increasingly embraced as a policy tool in fields such as epidemiology (Auchincloss and Diez Roux 2008), land use planning (Matthews et al. 2007), and economics (Farmer and Foley 2009). Empirical data ideally informs model development (Janssen and Ostrom 2006) through a participatory process that includes affected stakeholders (Vennix 1999). In the context of a nonprofit sector index, ABM can accommodate non-standardized data from the individual, organization, and community levels to populate computer simulations. Experimenting with different parameters (e. g. differences in operating contexts, policies, types and quantities of capabilities) can increase understanding of synergies, tradeoffs, and policy options.

8 Policy Implications and Concluding Thoughts

This paper has outlined a conceptual framework to guide development of a multi-level nonprofit sector index using a capabilities approach. It described multiple capitals as a way to operationalize and align capabilities across multiple levels. While these levels are not yet quantitatively comparable, this framework creates an architecture that can accommodate emergence and interaction effects across multiple levels and time scales. Thus, it avoids reductionism and loss of synergy through decoupling.

The final section of this paper presented two examples from the natural sciences to show how synergies, tradeoffs, and interaction effects across levels can be investigated and modeled. Computational science offers a promising path to generate new insights as to how capitals (capabilities) produce synergistic effects, what factors enable and constrain that production, and how various contexts impact production and recirculation. From a practical standpoint, rather than adopting simplistic measures (e. g. efficiency ratios, indirect cost rates) that can degrade long-term capacity, agent-based modeling can help philanthropists and policymakers create investments and policies that develop capabilities that lead to increasing returns (Busenitz and Barney 1997).

Admittedly this paradigm shift to social accounting will take time, just as it took time to evolve and standardize financial reporting. What might we learn from another historic paradigm transition? Six hundred years ago, people “knew” that matter existed in two forms, liquid and solid. As scientists began to suspect that a third form of matter existed (gases), they did not start by insisting on quantification. Rather, they carefully and repeatedly observed dynamic interactions and their effects. Over time, they began to recognize patterns, eventually enabling them to understand the underlying mechanisms and principles (e. g. how mass, volume, pressure, and temperature affect molecular interactions). By developing sufficient qualitative understanding through experimentation and analysis, the ability to measure ultimately followed (Gaston, Subramanian, and Zauscher 2006). As policymakers, philanthropists, nonprofit professionals, volunteers, and scholars develop a new understanding of intangible resources and complexity dynamics, the capacity to develop a quantified index will similarly emerge.