8.3: Workflow
[Develop] open workflow Other Information:
Given that academic science is a largely public institution funded by public money, it is surprising that there is so little
transparency and accountability for the research process. Beyond the published reports, science operates as a "trust me" model
that would be seen as laughably quaint for ensuring responsibility and accountability in state or corporate governance. In
some areas of science, however, it is understood that transparency in the scientific workflow underlies credibility and accuracy.
Stakeholder(s):
- National Institutes of Health: For example, clinicaltrials.gov is a National Institutes of Health-sponsored study registry for clinical trials.
- International Committee of Medical Journal Editors: In 2005, the International Committee of Medical Journal Editors started requiring authors to register their randomized controlled
trials prior to data collection as a condition for publication.
- Companies: Companies sponsoring trials have an obvious financial conflict of interest for the outcome of the research. A registry makes
it more difficult to hide undesired outcomes. Indeed, using registry data, Mathieu, Boutron, Moher, Altman, and Ravaud (2009)
found that 31% of adequately registered trials showed discrepancies between the registered and published outcomes. For those
in which the nature of the discrepancies could be assessed, 82% of them favored reporting statistically significant results.
- Scientists: Of course, money is not the only source of conflict of interest. Scientists are invested in their research outcomes via their
interests, beliefs, ego, and reputation. Some outcomes may be more desirable than others—particularly when personal beliefs
or prior claims are at stake. Those desires may translate into design, analysis, and reporting decisions that systematically
bias the accuracy of what is reported, even without realizing that it is occurring (Kunda, 1990; Mullen, Bauman, & Skitka,
2003).
- Laboratories: Public documentation of a laboratory's research process makes these practices easier to detect and could reduce the likelihood
that they will occur at all (Bourne, 2010). Further, registration of studies prior to their completion solves one aspect of
the file-drawer effect—knowing what research was done even if it does not get published (Schooler, 2011).
- Researchers: An obvious concern about transparency of workflow is that researchers are not interested in most of the details of what goes
on in other laboratories. Indeed, though advocating this strongly, we do not expect that we would routinely look at the details
of other laboratory operations. However, there are occasions for which access would be useful. For example, when we are inspired
by another researcher's work and aim to adapt it for our research purposes, we often need more detail than is provided in
the summary reports. Access to the materials and workflow will be very useful in those cases.
- Data.gov: Further, although we do not care to look at the public data about U.S. government expenditures ourselves (http://www.data.gov/),
we are pleased with the transparency and the fact that someone can look. Indeed, much as investigative journalism provides
accountability for government practice, with open workflow, new contributors to science might emerge who evaluate the knowledge
accumulation process rather than produce it and are valued as such.
- Story Tellers: Finally, using a registry in an open workflow can clarify whether a finding resulted from a confirmatory test of a strong
a priori prediction or was a discovery in the course of conducting the research. The current default practice is to tell a
good story by reporting findings as if the research had been planned that way (Bem, 2003). However, even if we intend to disclose
confirmation versus discovery, our recollection of the project purpose may not be the same as the project purpose when it
began. People reconstruct the past through the lens of their present (Schacter, 2001). People are more likely to presume that
what they know now was how they conceived it at the beginning (Christensen-Szalanski & Willham, 1991; Fischoff, 1977; Fischoff
& Beyth, 1975). Without a registry for accountability, findings may be genuinely and confidently espoused as confirmatory
tests of prior predictions when they are written for publication. However, discoveries are more likely to leverage chance
than are confirmatory tests. What appears to be "what we learned" could be "what chance told us." The point of making a registry
available is not to have a priori hypotheses for all projects and findings; it is to clarify when there was one and when there
was not. When it is a discovery, acknowledge it as a discovery.
- Statisticians: As Tukey (1977) summarized in support of discovery: "Once upon a time statisticians only explored. Then they learned . . .
to confirm a few things exactly, each under very specific circumstances. As they emphasized exact confirmation, their techniques
inevitably became less flexible. The connection of the most used techniques with past insights was weakened. Anything to which
a confirmatory procedure was not explicitly attached was decried as 'mere descriptive statistics,' no matter how much we had
learned from it. (p. vii)" Discovery is critical for science because learning occurs by having assumptions violated. Strong
narratives focusing on what was learned are useful communication devices, and simple disclosures of how it was learned are
useful accuracy devices.
Indicator(s):
|
