5.3: Collection Evaluation and Customer Satisfaction

Develop better methods for evaluating collection and measuring satisfaction of customer needs.

Other Information:

We must have an evaluation system that allows us to assess how well we are doing to satisfy our customers and fill intelligence gaps, and that helps us make informed decisions about difficult tradeoffs between collection platforms and future acquisition capabilities. This evaluative process must also help us accurately determine whether deficiencies are due to collection activities and capabilities themselves or to shortfalls in processing and exploitation of the information collected. To do so, we need a set of complementary evaluation programs that provide micro-level data on satisfaction of specific requirements as well as macro-level data on performance across the collection disciplines; information on the performance of individual collection systems on both an absolute and relative basis, but also assessments of how we are doing as a Community to address critical needs. In other words, we need both broad-based studies that cut across issues and collection disciplines, and in-depth studies of single issues and individual collection disciplines. Finally, we need longitudinal studies that allow for trend analysis as well as narrowly focused studies that provide valuable "lessons learned" in collecting against specific targets or issues. Implementing Actions: • Establish a steering group under the chairmanship of the ADCI/AP to provide oversight of a Community-wide and multifaceted evaluation program. The steering group will oversee/monitor the efforts described below. ADCI/AP will provide an annual update to the DCI, DDCI/CM, and the NIPB on the overall evaluation effort. • Advance the use of existing agency collection evaluation methodologies on the relative performance of collection systems and platforms over time. Expand to include other organizations where appropriate. • Review/build upon the recent effort of the ADCI/AP and ADCI/C in developing a joint annual report on the state of the Intelligence Community. This review should cut across agencies and be based on a consistent, repeatable methodology. It should provide an in-depth evaluation of how well we have performed in meeting customer needs and filling critical intelligence gaps on a series of key issues. It should take into account competing requirements and identify shortfalls in the collection, processing, and exploitation cycle. It should also provide actionable recommendations to overcome identified shortfalls. • Appoint blue ribbon panels—ideally a mix of inside and outside experts—to prepare ad hoc "lessons learned" studies on event-driven issues or topics of critical concern to our customers. • Establish a cross-agency working group to explore types of electronic feedback/evaluation mechanisms in use or planned as part of collection management systems currently under development (e.g., audit trails, site visit measures, popup screens, and mandatory versus voluntary evaluation menus). Cross-fertilize with activities underway in analytic organizations as part of production reengineering efforts. Evaluate commercial methods and software that might provide easy and consistent statistics across the Community on customer usage and satisfaction and on demand-to-response ratios for standing and ad hoc intelligence requirements.

Indicator(s):

Indicator:1

Content