A recent paper, based on a “survey of participants in a large-scale business plan competition experiment, [in Nigeria] in which winners received an average of US$50,000 each, is used to elicit beliefs about what the outcomes would have been in the alternative treatment status.”
So what exactly was done? Business owners were basically asked what would have happened to their business had an alternative business investment process taken place, as opposed to the one that took place under the competition outcome. “Winners in the treatment group are asked subjective expectations questions about what would have happened to their business should they have lost, and non‐winners in the control group asked similar questions about what would have happened should they have won.”
“Ex ante one can think of several possibilities as to the likely accuracy of the counterfactuals”:
- “…business owners are not systematically wrong about the impact of the program, so that the average treatment impact estimated using the counterfactuals should be similar to the experimental treatment effect. One potential reason to think this is that in applying for the competition the business owners had spent four days learning how to develop a business plan… outlining how they would use the grant to develop their business. The control group [competition losers] have therefore all had to previously make projections and plans for business growth based on what would happen if they won, so that we are asking about a counterfactual they have spent time thinking about.”
- ”…behavioral factors lead to systematic biases in how individuals think of these counterfactuals. For example, the treatment group may wish to attribute their success to their own hard work and talent rather than to winning the program, in which case they would underestimate the program effect. Conversely they may fail to take account of the progress they would have made anyway, attributing all their growth to the program and overstating the effect. The control group might want to make themselves feel better about missing out on the program by understating its impact (...not winning does not matter that much). Conversely they may want to make themselves feel better about their current level of business success by overstating the impact of the program (saying to themselves I may be small today, but it is only because I did not win and if I had that grant I would be very successful).”
The actual results show that business owners “do not provide accurate counterfactuals” even in this case where competition awards (and thus intervention or shock) was very large.
- The authors found that “both the control and treatment groups systematically overestimate how important winning the program would be for firm growth…
- “…the control group thinks they would grow more had they won than the treatment group actually grew”
- “…the treatment group thinks they would grow less had they lost than the control group actually grew”
Or in other words: losers overestimate benefits of winning, winners overestimate the adverse impact from losing... and no one is capable of correctly analysing own counterfactuals.
Full paper is available here: McKenzie, David J., Can Business Owners Form Accurate Counterfactuals? Eliciting Treatment and Control Beliefs About Their Outcomes in the Alternative Treatment Status (May 10, 2016, World Bank Policy Research Working Paper No. 7668: http://ssrn.com/abstract=2779364)
