Updated on: Feb 11, 2008
Early in 2005, NASA quietly closed down an $11.3 million, state-of-the-art program designed to collect critical data on air travel safety. In four years of operation, the program compiled data from nearly 30,000 interviews with pilots, but then NASA abruptly buried their responses for two years, seemingly more concerned with protecting the airline industry from negative public opinion than protecting the public itself.
Media and Congressional pressure finally forced the agency to release the data in December 2007, but not before NASA officials deleted several components essential to scientific analysis.
A new plan to make flying safer
The data was produced by an ambitious NASA project known as the National Aviation Operations Monitoring System (NAOMS), which was initiated in 1998 to develop a new system for surveying pilots as a means to identify areas for improvement in air travel safety.1
Through phone interviews with pilots, NAOMS aimed to capture elusive statistics on the rate of "risk-elevating events"—events that in isolation do not cause a serious accident, but when added together serve as indicators of overall risk.2 Examples include miscommunications between pilots and air traffic controllers, disturbances caused by passengers, bird strikes, or aircraft flying too close to one another.3
Between 2001 and the end of 2004, NAOMS researchers conducted 24,000 interviews of commercial airline pilots and another 5,000 interviews of general aviation pilots.4 Plans were in the works to extend the survey to air-traffic controllers, aircraft mechanics, and other aviation professionals when NASA leadership suddenly canceled the survey in early 2005.5
Survey responses buried
After NASA terminated NAOMS, the existing data sat unused for two years until an anonymous agency insider informed The Associated Press (AP) of the sequestered survey results.6 The AP filed a Freedom of Information Act (FOIA) request for the buried data, only to be denied by NASA because, in the agency's own words, the data "could materially affect the public confidence in, and the commercial welfare of, the air carriers and general aviation companies."7 Public outrage over the implication that NASA was catering to the financial interests of the aviation industry over its commitment to protecting public safety prompted the U.S. House of Representatives Committee on Science and Technology to launch an immediate investigation into the matter.8
At the Committee hearing on October 31, 2007, NASA Administrator Michael Griffin dismissed the controversial statement from the FOIA rejection as a "regrettable" error in wording and asserted NASA's first priority is to ensure public safety.9 He insisted the principal reason for rejecting the AP's FOIA request was to protect the survey respondents, as the raw data contained confidential commercial information and "information that could compromise the anonymity of individual pilots."10
Under pressure from the Committee, Griffin agreed to release the NAOMS data "as soon as possible,"11 but maintained that the agency would first remove specific references that could be used to deduce a respondent's identity—such as airline names or flight numbers—from the data.12
Essential data removed from release
According to Stanford professor in Humanities and Social Sciences Jon Krosnick, the lead consultant in developing and implementing the NAOMS survey methodology, NASA removed so much information that a proper analysis of that data is now impossible.13
To accurately analyze the survey data, Krosnick says, researchers must first correct for double counting—the possibility that a single risk-elevating event was reported by two or more witnesses.14 For instance, if a pilot and co-pilot were both interviewed during the same time period, they might both report the same bird strike.
The NAOMS survey was designed to collect all of the information necessary to make this correction, but several necessary components were redacted from the public release, 15 including: whether the pilot was recalling information from the past 30 or 90 days;16 the day and month of each pilot interview; the size of the fleet for each pilot's air carrier; and the specific make and model of the aircraft flown.17
Without this information, the scope and quality of any analysis of this data to identify trends in aviation safety risks—which was the original purpose of the NAOMS survey18—are seriously limited.
NASA washes its hands of the project
Despite strong support from the NAOMS team19 and the urging of Congress,20 Administrator Griffin appears to have no intention of reactivating the project. As one reporter put it, Griffin "[hasn't] said anything good about it from the get-go."21 Griffin has claimed the survey data cannot be validated since it was not properly peer-reviewed,22 that the project was poorly organized,23 and has even likened the data collected through NAOMS surveys as mere "hangar talk".24
These negative statements belie six years of work by the NAOMS team. Dr. Krosnick testified before the House Committee on Science and Technology that the NAOMS methodology was thoroughly peer-reviewed. Not only had the NAOMS team "held dozens of meetings, workshops, and consultations around the country with aviation experts, interested parties, and social scientists to describe the project's methodology and get reviews, comments, and suggestions," he said, but the project had also received approval from the White House Office of Management and Budget, which reviews all federal survey projects to ensure they are optimally designed.25
Robert Dodd, the principal investigator for NAOMS until the project's termination in 2005, weighed in at the hearing as well: "The NAOMS team made an extraordinary effort to clean and validate the data collected through the survey. The resulting data is of good quality and ready for meaningful analysis."26
The fact that NASA spent over $11 million dollars on the program over six years seems to support Dodd and Krosnick, as does the high response rate and enthusiasm from the pilots who participated. As Rep. Brad Miller (D-NC) said, "If 80 percent of the pilots they ask agree to sit still for a half-hour survey, voluntarily, my conclusion is the pilots had something they wanted others to know about."27 But until NASA leadership allows full access to and analysis of the data, no one will know what those pilots had to say.
1. Battelle Memorial Institute. 2007. NAOMS Reference Report: Concepts, Methods, and Development Roadmap. 30 November. Executive Summary. Accessed 11 February 2008.
2. Krosnick, Jon. 2007a. Statement on the National Aviation Operations Monitoring Service. Written Testimony for hearing entitled, "Aviation Safety: Can NASA Do More to Protect the Public?" Committee on Science and Technology, U.S. House of Representatives. 30 October. Accessed 11 February 2008.
3. Battelle 2007. Appendix 11.
4. Griffin, Michael. 2007. Written Testimony for hearing entitled, "Aviation Safety: Can NASA Do More to Protect the Public?" Committee on Science and Technology, U.S. House of Representatives.. 31 October. Accessed 11 February 2008.
5. Krosnick 2007a. See chart on page 4.
6. Beamish, Rita. 2007. "NASA Sits on Air Safety Survey." Associated Press. 22 October. Emphasis in original.
7. Beamish 2007.
8. "NASA's response in effect seems to be saying that it sees its job as putting the commercial interests of the aviation industry above the public's right to aviation safety information." Mark Udall, Chairman. 2007. Opening Statement for hearing entitled, "Aviation Safety: Can NASA Do More to Protect the Public?" Committee on Science and Technology, U.S. House of Representatives. 31 October. Accessed 11 February 2008.
9. Griffin 2007.
10. Griffin 2007.
11. Udall, Mark. 2007. Opening Statement for hearing entitled, "Aviation Safety: Can NASA Do More to Protect the Public?" Committee on Science and Technology, U.S. House of Representatives. 31 October. Accessed 11 February 2008.
12. Gordon, Bart et al. 2007. Committee on Science and Technology, U.S. House of Representatives. Letter to Dr. Michael Griffin, NASA Administrator. 14 November. Accessed 11 February 2008.13. Krosnick, Jon. 2008. Interview with author. 4 February.
14. Krosnick 2007a.
15. Krosnick 2008. Interview.
16. Battelle 2007. Appendix 11.
17. Note that the make and model of aircraft were "generalized" by grouping them into simple size categories: widebody, large, medium, and small. For example, a pilot who had reported an event on an Airbus 300 and another pilot who had reported an event on a Boeing 777 would both be identified only as having flown "widebody" craft. For a full explanation of the redactions, see: NASA. 2007. NAOMS Response Redaction Summary. 31 December. Accessed 11 February 2008.
18. Krosnick 2007a.
19. Krosnick 2007a; Dodd, Robert. 2007. Written Testimony for hearing entitled, "Aviation Safety: Can NASA Do More to Protect the Public?" Committee on Science and Technology, U.S. House of Representatives. 30 October. Accessed 11 February 2008.
20. Chairman Mark Udall (D-CO) has said, "There appears to be a great deal of merit to the NAOMS approach" (see Udall 2007). Bart Gordon (D-TN) and Brad Miller (D-NC), have both been quoted by The Washington Post in favor of "pushing NASA farther" (see "NASA Offers Airline Safety Data" by Matthew Wald. The New York Times. 1 January 2008). Accessed 11 February 2008.
21. See question from CNN report Miles O'Brien in: NASA. 2007. Media Briefing: "Release of Aviation Safety Data" [Transcript]. Moderated by J.D. Harrington, NASA Public Affairs. 31 December. Accessed 11 February 2008.
22. "Unfortunately, none of the research conducted in the NAOMS project, including the survey methodology, has been peer-reviewed to date. Accordingly, any product of the NAOMS project, including the survey methodology, the data, and any analysis of that data, should no be viewed or considered at this stage as having been validated." Griffin 2007. Note that a similar caveat appears on the homepage for the NAOMS project. Accessed 11 February 2008.
23. Michael Griffin, as quoted in NASA. 2007. Media Briefing: "Release of Aviation Safety Data" [Transcript]. Moderated by J.D. Harrington, NASA Public Affairs. 31 December. Accessed 11 February 2008.
24. "NASA was conducting these surveys to try to obtain what many folks would characterize as 'hangar talk.'" Michael Griffin as quoted in NASA. 2007. Media Briefing: "Release of Aviation Safety Data" [Transcript]. Moderated by J.D. Harrington, NASA Public Affairs. 31 December. Accessed 11 February 2008.
25. Krosnick, Jon. 2007b. Oral Testimony for hearing entitled, "Aviation Safety: Can NASA Do More to Protect the Public?" Committee on Science and Technology, U.S. House of Representatives. 30 October. See a webca st of the hearing online. Accessed 11 February 2008.
26. Dodd, Robert. 2007. Written Testimonyfor hearing entitled, "Aviation Safety: Can NASA Do More to Protect the Public?" Committee on Science and Technology, U.S. House of Representatives. 30 October. Accessed 11 February 2008.
27. Chairman Brad Miller, as quoted in: "NASA Offers Airline Safety Data" by Matthew Wald. The New York Times. 1 January 2008. Accessed 11 February 2008.