III.C: METHODOLOGY

Test programs that then may be tried on a grander scale.

Other Information:

Although costly and time-consuming, RCTs are considered helpful in rigorously establishing impact of relatively mature open government initiatives. There is a difference of opinion, however, as to the applicability of this approach to evaluating government. For this approach to work, experiments have to be set up ahead of time with a control group and a group that is part of the program. There are ethical issues involved, as providing government services to one part of the population and withholding them from another is tricky. One possibility that has been part of our federal "experiment" is to have individual states or localities test programs that then may be tried on a grander scale. Michael Greenstone has written about the need for these kinds of experiments. In "Toward a Culture of Persistent Regulatory Experimentation," Greenstone points out some of the problems with randomized controlled trials and evaluating government programs in general. "Doing What Works," the project of the Center for American Progress, also calls for experimenting at the state and local level and adapting what works to the national level... Quasi-experimental techniques can also be useful in evaluating outcomes while non-experimental methods can be a valuable starting point for new initiatives. Recent developments and new tools such as big data should also factor into how open government evaluations are designed. And finally, as discussed in Section II, qualitative assessments are gaining renewed attention as ways of assessing open and participatory government initiatives. There are recent attempts to make such assessments more rigorous and also to mix qualitative and quantitative assessments, as advocated by Khargram and Thomas. These two argue for a new "Platinum" standard that incorporates RCTs with qualitative evidence to be used in assessing public administration and third party governance.

Stakeholder(s):

• Michael Greenstone


• Coalition for Evidence-Based Policy: The Coalition for Evidence-Based Policy has a website devoted to "Top Tier Evidence" that promotes the use of randomized controlled studies. They rely on a recent National Academy of Sciences report that includes "Recommendation on Criteria for Establishing Strong Evidence of Effectiveness." The Coalition suggests, though, that the possibility of selecting a randomized intervention and control groups is itself a significant problem. The agency contemplating the intervention must be able to make assignments to those who will receive the intervention as well as to those who won't, or who will receive a different intervention.


• National Academy of Sciences

Indicator(s):

Indicator:1

Content