Survey Design Best Practices

In our efforts to address complex issues like poverty, hunger, and job creation, it's crucial to measure the effectiveness of our programs. Whether you're part of a nonprofit organization, an impact investor, or an ecosystem builder, understanding the true impact of your interventions is key.

In this article, we'll dive into seven essential tips for designing and implementing surveys that genuinely reflect the outcomes of your interventions.

You can understand these tips well, if you first understand following real-world use case and follow examples at the end of each TIPs section.

Real-world Use Case

In today's rapidly evolving job market, the challenge of reskilling and upskilling has become increasingly critical. Workers across various industries face the pressing need to adapt to new technologies and changing job requirements, creating a demand for effective educational programs tailored to these emerging skills. Throughout this article, we will use this problem statement of reskilling and upskilling as a foundation to explore each of the seven tips, providing specific examples to illustrate how these strategies can effectively address the challenges in the context of modern workforce development.

TIP 1: Strategizing Survey Design Methodology

When it comes to measuring the impact of interventions, particularly those aimed at poverty alleviation through job creation, the design of your survey is a critical factor. It’s not enough to merely measure the increase in household income; a deeper understanding of how this income is being utilized is essential. Let’s delve into this with a detailed example.

Imagine your organization has launched a program in a rural community to create jobs by setting up a local handicrafts center. The direct impact, presumably, is an increase in household income for those employed at the center. However, to truly gauge the effectiveness of this intervention, your survey needs to explore beyond these surface-level metrics. It should investigate how this additional income is impacting the family's living conditions and quality of life.

For instance, your survey could include questions like:

• “Has the increased income led to better nutritional food choices for your family?”
• “Are you able to afford healthcare services more easily now?”
• “Is there an increase in educational opportunities or resources for your children?”

These questions help you understand whether the job creation is translating into tangible improvements in the lives of the community members.

For organizations that might not have the resources to hire a consultant for tailored survey design, tools like the Poverty Probability Index (PPI®) are extremely beneficial. The PPI, with its straightforward yet revealing questions, can shed light on various aspects of household welfare. For instance, a question about the roofing material of a house is not just about the physical structure, but also a proxy indicator of the family’s economic status. Similarly, asking about the number of children attending school can provide insights into the family's priorities and the impact of increased income on education.

To continue with our example, if a family previously living in a house with a thatched roof now lives in a house with a metal roof, this could indicate an improvement in their economic status, likely linked to the job opportunities created. Additionally, if more children from these households are now attending school, it suggests a positive shift in prioritizing education, possibly due to the stabilizing effect of the new income source.

By carefully crafting your survey questions to capture these nuanced changes, you can obtain a comprehensive understanding of how your intervention is influencing the community. This approach not only measures the immediate economic impact but also uncovers the broader, long-term benefits in enhancing the overall living conditions of the beneficiaries.

Context: Survey for an online course platform offering reskilling programs.

‍Example Questions:

• "How has this course impacted your current job performance?"
• "Are you applying new skills learned in this course to your work?"
• "What changes in your career trajectory do you attribute to this course?"
• (Focuses on the practical application and career impact of the reskilling program.)

TIP 2: Best Practices in Survey Design for Global Application

Expanding a successful intervention across multiple geographies is a complex task, particularly when it comes to ensuring the effectiveness of your survey tools. A strategic approach involves starting small, learning, and then scaling – an approach that proves vital in maintaining the relevance and effectiveness of your data collection.

Consider, for example, an organization that has developed a successful microfinance program in India and is now looking to implement it in other South Asian countries. The initial step would be to conduct a pilot survey in one region of India, where the program has been most effective. The survey might focus on various aspects such as the impact of microfinance on local businesses, changes in household income, and improvements in living standards.

After collecting and analyzing this initial data, the organization might observe certain patterns or receive feedback that could lead to refinements in the survey. For instance, they might realize that questions regarding the type of business (agriculture, retail, services, etc.) are too broad and need to be more specific to capture the diverse economic activities in the region.

When considering expansion to a neighboring country like Bangladesh, the organization should first assess similarities and differences in key factors such as economic conditions, cultural norms, and language. While both countries might have similarities in economic structure and some cultural aspects, there might be significant differences in other areas, such as specific challenges faced by local entrepreneurs.

The organization could then adapt the survey to fit the new context. This might involve translating the survey into the local language, altering questions to reflect local economic activities, or even modifying the mode of survey delivery considering local infrastructure and technology access.

For instance, in areas with limited internet connectivity, relying on digital surveys might not be effective, and face-to-face interviews could be more appropriate. Similarly, questions about the impact of microfinance might need to be adjusted to reflect the unique business environments in different regions.

This step-by-step, region-by-region approach allows the organization to apply the learnings from one context to another effectively. By tailoring the survey to each new region, the organization can ensure that the data collected is accurate and relevant, thereby providing a solid foundation for assessing the impact of their microfinance program in diverse geographic settings.

In summary, when applying learnings to other geographies, it's crucial to respect and adapt to the unique characteristics of each new area. This ensures not only the relevance of your data collection but also enhances the likelihood of replicating the success of your interventions in new environments.

Context: Expanding the online course platform to different countries.

‍Example Questions:

• "Are the course materials relevant to your regional job market?"
• "Do you have access to necessary technology to complete this course in your region?"
• "What local industry skills would you like to see more courses on?"
• (Tailors the course offerings and materials to different regional job markets and technological accessibility.)

TIP 3: Leveraging Observational Data in Survey Design Best Practices"

Integrating observational data collection into your surveys can be a game-changer, especially when working in environments where respondents might be overwhelmed by lengthy questionnaires. This approach not only streamlines the data collection process but also enriches the accuracy and authenticity of the information gathered.

To illustrate this, let's consider a practical scenario. Imagine an organization working on a housing improvement program in a rural community. The primary goal is to assess the living conditions before and after the intervention. Instead of relying solely on respondent-driven answers, the organization's field workers can make direct observations during their visits.

For example, one of the survey questions might be about the type of material used in the construction of the house, specifically the roofing. In a traditional survey approach, this would be a direct question posed to the household. However, by observing the roofing material firsthand, the field workers can immediately note whether it's thatch, tile, metal, or another material. This direct observation provides accurate data without requiring the respondent to articulate an answer, which is particularly useful in situations where respondents might have limited literacy or are not comfortable with formal survey methods.

Another example could be assessing access to clean water. Instead of asking the household if they have access to clean water, which could lead to subjective interpretations of what constitutes 'clean water', the field workers could observe the water source directly. They could note the type of water source (well, tap, river, etc.), its condition, and even take samples for testing if necessary.

Observational data collection also has the advantage of capturing nuances that might be lost in translation during a verbal survey. For instance, the condition of household assets, the cleanliness of the living environment, or the general well-being of the residents can be more accurately gauged through direct observation.

Furthermore, this approach can lead to more honest and less biased data. Sometimes, respondents might provide answers they think the surveyors want to hear, or they might understate or overstate certain aspects due to social desirability bias. Observation cuts through these potential distortions, offering a more objective view of the situation.

In summary, integrating observational methods into your data collection strategy can significantly enhance the quality of your data. It reduces the burden on respondents, improves the accuracy of the information collected, and provides a richer, more nuanced understanding of the on-ground realities. This method is particularly valuable in contexts where direct interaction and observation can reveal more than just verbal or written responses.

Context: Observing the usage patterns of the online platform and course engagement in various regions.

‍Example Observations:

• Monitoring the time spent on different course modules.
• Noting the completion rates of courses by region.
• Observing the types of courses most frequently enrolled in different areas.
• (Provides insights into user engagement and preferences, enhancing course relevance.)

TIP 4: Incorporating Survey Scoring Systems: Advanced Survey Design Techniques

Implementing a scoring system for survey responses is an effective way to quantify and track progress over time, particularly when dealing with diverse data sets. This approach simplifies the process of monitoring improvements and reporting findings in a standardized manner, which is crucial for comparing results across different communities or countries.

Continuing with our example of the housing improvement program in a rural community, let's explore how scoring can be applied to different aspects of the survey, building upon the observational data collection method discussed earlier.

Consider a scenario where, in addition to observing the type of roofing material, field workers also gather information about educational access for children in the household. They find that in the first year of the survey, most families are able to send only two children to school. Based on this, they assign a score of 1 point for every two children attending school. In the subsequent year, after various educational initiatives have been implemented, they observe an increase, with most families now sending four children to school. This progress is quantified by assigning a higher score of 2 points for four children attending school.

This scoring system provides a clear, quantifiable measure of the program's impact on education. It allows the organization to easily track improvements in educational access over time. Moreover, when this data is aggregated across different regions, it offers a comparative view of the impact in various communities. For instance, if another region shows a slower rate of improvement, scoring could help in identifying this trend and prompt further investigation into the underlying causes.

The same scoring approach can be applied to other areas of the survey. For example, improvements in water quality or access to healthcare could be scored in a similar way, with points assigned based on observable criteria established by the organization. This could include parameters like the proximity of a clean water source or the frequency of healthcare visits.

The advantage of a scoring system is that it translates qualitative observations into quantitative data, which can be more easily analyzed and communicated to stakeholders. It provides a clear, numerical representation of the impact, making it easier to track progress, identify areas needing more attention, and demonstrate the effectiveness of the intervention to funders or other interested parties.

In summary, a scoring mechanism is a valuable tool in outcome measurement, as it offers a standardized way to track improvements, compare results across different settings, and communicate impact effectively. When combined with observational data collection, it creates a robust framework for understanding and reporting the progress of community development initiatives.

Context: Using a scoring system to evaluate the effectiveness of courses in enhancing skills.

‍Example Scoring:

• Assign points based on skill proficiency before and after the course.
• Score career advancement opportunities utilized post-course completion.
• Rate learner satisfaction and skill application in the workplace.
• (Quantifies the impact of reskilling on career advancement and skill development.)

TIP 5: Analyzing Correlations: Mastering Survey Design Methodology

Analyzing correlations in survey results is a powerful tool for uncovering deeper insights and understanding the true impact of interventions. It moves beyond mere data aggregation, helping organizations to explore causality and address root causes of issues more effectively.

Let's continue with our rural community example, where a housing improvement program and educational initiatives have been implemented. In addition to these, suppose the organization also introduced a program to boost household income through local craft businesses. Now, the organization hypothesizes that this increase in household income will lead to more nutritious food consumption among these families.

To test this hypothesis, the organization correlates the data on increased income with changes in dietary patterns. They collect data on household income and simultaneously survey the types of food regularly consumed by the families. This could include questions on the frequency of consuming fruits, vegetables, protein-rich foods, and other indicators of a nutritious diet.

If the analysis reveals a positive correlation between increased income and improved nutrition, it validates the hypothesis. However, if the correlation is weak or non-existent, it prompts a deeper investigation. This could uncover barriers to accessing nutritious food, such as lack of availability in local markets, high prices of healthy food options, or a lack of awareness about nutritional needs.

For example, the organization might find that despite higher incomes, families are still consuming a diet low in nutrients. This could indicate that while they can afford better food, it's not available in their local market. In response, the organization could work on introducing local vegetable gardening projects or partnering with suppliers to bring a variety of nutritious foods to the community.

This approach of correlating results to find causality is crucial. It helps organizations move beyond surface-level assumptions and address the underlying factors affecting their target communities. By understanding the 'why' behind the data, interventions can be tailored more effectively to meet the actual needs and challenges faced by the community.

In summary, correlating different sets of data allows organizations to draw meaningful conclusions about the impact of their interventions. This not only helps in refining current programs for greater effectiveness but also provides valuable insights for future planning and implementation of similar initiatives in other communities.

Context: Correlating course completion with career progression.

‍Example Correlation Analysis:

• Analyzing the relationship between course completion and salary increments.
• Examining the correlation between specific skill acquisition and job promotions.
• Assessing the impact of course completion on job retention rates.
• (Determines the direct impact of reskilling courses on career advancement.)

TIP 6: Outcome Star: A Pioneering Approach in Survey Design Best Practices

Outcome Stars are a transformative approach to measuring and supporting positive change and wellbeing in various sectors, including Adult Care and Domestic Violence. This innovative tool is structured around a series of questions, each rated on a 'Journey of Change' scale. It's designed to track the progress of individuals or groups over time, providing crucial insights for crafting more effective interventions.

To illustrate the application of Outcome Stars, let's consider an organization working in the domain of domestic violence prevention and support. This organization could implement the Outcome Star to track the progress of individuals who have experienced domestic violence, with the aim of measuring improvements in their safety, wellbeing, and independence.

The Outcome Star for this context might include dimensions such as personal safety, access to legal support, emotional health, and social connections. Each dimension would have a scale – for example, from 1 (indicating severe challenges in that area) to 5 (indicating significant improvement or resolution of the challenges). As participants in the program engage with various support services and interventions, their progress in each dimension is regularly assessed and rated on the scale.

For instance, a participant might start with a low rating in personal safety due to ongoing threats or unsafe living conditions. Over time, as they access shelter services and legal support through the program, their situation improves, reflected in a higher rating on the scale. This progress is visually represented on the Outcome Star, providing both the participant and the support workers with a clear picture of the journey so far and areas that still need attention.

This method is particularly effective because it emphasizes the journey of change, rather than just the endpoint. It acknowledges that progress in complex areas like domestic violence recovery can be non-linear and multifaceted. Regularly updating the Outcome Star allows for adjustments in the support plan as the individual's needs and circumstances evolve.

Moreover, when used across a group of participants, Outcome Stars can aggregate data to show trends and common challenges. This collective insight is invaluable for organizations in evaluating the effectiveness of their programs and identifying areas for improvement or additional resource allocation.

In summary, the Outcome Star is a powerful tool for tracking and supporting progress in complex social issues. Its structured yet flexible approach provides a comprehensive view of an individual's journey, enabling more tailored and effective support. For organizations, it offers a clear framework for measuring impact and refining interventions to achieve the best possible outcomes for their beneficiaries.

Context: Implementing the Outcome Star to track individual learner progress.

‍Example Dimensions:

• Tracking changes in skill proficiency over time.
• Monitoring improvements in job performance.
• Assessing career progression stages post-course completion.
• (Evaluates the holistic impact of reskilling on an individual's career path.)

TIP 7: Enhancing Survey Design through Investor Collaboration

The role of investors and accelerators in the realm of social impact and development projects has significantly evolved. They are no longer passive participants who merely finance projects and await results. Instead, their involvement has deepened, playing a pivotal role in shaping the success of interventions.

To contextualize this evolving role, consider a scenario where a social impact fund invests in a series of community development projects, such as our earlier example of a housing improvement program in a rural area. In this evolved role, investors do more than provide funding; they actively participate in defining the outcomes, metrics, and the design of surveys that measure the effectiveness of these interventions.

For example, an investor might collaborate with the organization running the housing program to develop a comprehensive impact measurement framework. This could involve determining key performance indicators (KPIs) such as the number of improved houses, the quality of life enhancements, and the long-term sustainability of these improvements. They might also bring in technological resources to establish a digital platform for collecting and analyzing survey data, making the process more efficient and scalable.

Moreover, these investors could facilitate access to expert consultants who specialize in impact measurement, ensuring that the surveys are not only well-designed but also culturally sensitive and contextually relevant. They might also propose innovative data collection methods, like using mobile technology to reach remote areas or integrating GIS mapping to visually represent the impact.

This collaborative approach creates a synergy where the investors' expertise and resources complement the on-ground knowledge and capabilities of the implementing organization. It leads to a more nuanced understanding of the community's needs and the intervention's impact, ultimately enhancing the effectiveness of the project.

Furthermore, by being directly involved in shaping the outcomes and measurement strategies, investors gain a deeper insight into the impact of their investments. This knowledge can guide future funding decisions, leading to more targeted and impactful investments.

TIP 7: Investors' Contribution

Context: Investor involvement in the development and expansion of reskilling courses

.Example Contributions:

• Providing funds for course development in emerging tech fields.
• Offering insights into labor market trends to guide course topics.
• Supporting technology upgrades for more interactive and engaging course delivery.
• (Enhances the course platform's relevance, accessibility, and alignment with current job market demands.)

In summary, the active involvement of investors and accelerators in defining and measuring the outcomes of social impact projects represents a significant shift towards a more collaborative and results-oriented approach. This paradigm not only augments the effectiveness of the interventions but also fosters a culture of accountability and continuous learning among all stakeholders involved.