AVOIDING A BLANK STARE

Using cognitive pretesting to improve the WEAI

As a researcher, how often have you wondered whether survey respondents understand your survey questions? Ensuring that survey questions generate the intended information is especially tricky when asking questions on topics that don’t have standardized modules. When a carefully crafted question is met by a blank stare, giggle, or unexpected response, it can be challenging to determine what went wrong and how to fix it. 

Enter cognitive pretesting, which involves conducting a survey while simultaneously collecting observational or verbal information about the responses (Willis, 2005).  The purpose is to systematically identify and analyze sources of response error in surveys and to use that information to improve the quality and accuracy of survey instruments (Johnson, 2013).  In practice, cognitive testing may include following a questionnaire module with a series of standardized questions about how well the respondent understood the module. Questions could cover terms that might have been ambiguous or complex, whether the respondent found the question difficult or thought others would, or whether the respondent understood the question as intended, by asking him or her to repeat it back in their own words. This process allows researchers to ensure the validity of their questions and is especially important with new or revised instruments and when surveys are implemented across cultures or languages (Johnson, 2013). IFPRI researchers recently put this tool to use on a revised and updated version of the Women’s Empowerment in Agriculture Index (WEAI 2.0). [1]

Following collection of the baseline WEAI surveys in 19 USAID Feed the Future countries, a 2013 learning event among partners and practitioners helped reveal the survey’s most challenging components.  The most problematic modules across all countries included time use, autonomy in production, and group membership.  In addition, they felt the survey was too long. Partners decided to address these issues by developing another version of the questionnaire that was shorter and that clarified the problematic sections for respondents while ensuring cross-national comparability.  IFPRI conducted cognitive pretesting of the WEAI 2.0 in Bangladesh and Uganda in 2014.

Following cognitive testing guidelines, we used a diverse sample of at least 10-15 households per language group. We interviewed 120 respondents in Uganda (from four major language groups) and 70 in Bangladesh (with one major language).

Time frame and recall issues

Through standardized probing, we investigated whether the timeframe specified in a question was the same as the timeframe referenced in the response.  In Uganda, the most glaring recall issue came from the question, “Did you yourself participate in [ACTIVITY] in the past 12 months (that is, during the last [one/two] cropping seasons)?”  When asked how much time respondents included in their response, answers ranged anywhere from 3-12 months; 35% of respondents either could not come up with the recall period used or referred to a timeframe other than 12 months.  Modification of the question to, “Did you yourself participate in [ACTIVITY] in the past 12 months (that is, during the last [one/two] cropping seasons), from [PRESENT MONTH] last year to [PRESENT MONTH] this year?”, resulted in much less recall error with only 6% of respondents stating periods of less than one year.

Photo credit: Chiara Kovarik

Cognitive interviews can also be helpful in testing different recall periods.  Time use data were initially collected using a 24-hour recall module, which was cited as problematic because it took substantial time to administer, did not capture seasonality, and may not be indicative of a usual day.  We therefore tested an alternative one-week recall.  The findings show that in Uganda, roughly 59% of respondents in both rounds said their activities change from day to day and 16.1% of these respondents cited seasonality as the reason. In Bangladesh, half of respondents in the first pretest and two-thirds of respondents in the second pretest said their schedule changed from day to day.  These findings illustrate the need to develop time use modules that address daily variations.

Difficult questions

Another way to identify cognitive problems is to ask the respondent both whether they thought a question was difficult and whether they thought others would find it difficult.  Inconsistencies between these answers can reveal comprehension problems. This was apparent in testing questions regarding autonomy in production using vignettes, which in the WEAI 2.0 were short stories about farmers engaged in various agricultural production activities.  In Uganda, 7-14% of respondents said they found the questions difficult, but 29-60% thought others would have difficulty.  In Bangladesh, few respondents noted these questions as difficult to answer themselves but 29-39% said they thought others would have difficulty.The examples here highlight the valuable insights gained from cognitive pretesting.  While all surveys face resource constraints, conducting cognitive testing as part of a standard pretest can ultimately save time for respondents, enumerators, and analysts while reducing response error. It is especially useful for new instruments or ones that will be implemented across diverse settings.

Works Cited:

Willis, Gordon B. Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage, 2005.

Johnson, Kiersten. Cognitive Pretesting of Cross-nationally Comparable Survey Instruments in a Developing Country Context Seminar. International Food Policy Research Institute. Washington, DC. 9 May 2014.