Over the past two weeks, Patrick Boyle of the American Meat Institute (AMI) has asserted that, according to USDA testing data, the incidence of E. coli O157:H7 in ground beef has decreased by 45% since 2000.[1] Mr. Boyle has also asserted that “According to the Centers for Disease Control and Prevention, E. coli O157:H7 infections in people have decreased by 44 percent since 2000.” Presumably, Mr. Boyle is basing these statistics on data from USDA’s E. coli O157:H7 Verification Testing Program and CDC’s FoodNet surveillance. If so, these statements are inappropriate and misleading.
Mr. Boyle’s first misuse of statistics was to inappropriately use USDA’s E. coli O157:H7 testing data. USDA’s E. coli O157:H7 microbiological testing program is strictly regulatory and was not statistically designed to estimate the prevalence of E. coli O157:H7 in raw ground beef. Different establishments are sampled each year. Further, the methods used to select establishments and to conduct the microbial testing have changed over the years. Several sources, including USDA itself, have noted the limitations of the data obtained from USDA’s Verification Testing Programs.[2,3,4,5,6,7]
With this in mind, let’s really look at Mr. Boyle’s statistic on the incidence of E. coli O157:H7 in raw ground beef. In 2000, USDA conducted microbial tests that showed a certain level of contamination at specific USDA-inspected meat and poultry plants. In 2008, USDA conducted microbial tests at a different group of USDA-inspected meat and poultry plants across the country. Indeed, if you compare the numbers from these two years, there appears to be a 45% reduction. But remember, the data is coming from two different groups of plants! This would be the equivalent of saying that, because Joe weighed 300 lbs in 2000 and Bill weighed 150 lbs in 2008, Americans have experienced a 50% weight loss. No researcher would draw such a conclusion! Clearly, it is inappropriate to use USDA’s E. coli testing data to make year-to-year comparisons, as Mr. Boyle has done.
Mr. Boyle further compounded this error with a second misuse of statistics when he selectively compared 2000 and 2008 data and ignored recent trends that may not reflect positively on the industry’s performance. Let’s use Mr. Boyle’s approach and assume that it is appropriate to use USDA’s E. coli O157:H7 data to make year-to-year comparisons (again, it is not!). A comparison of USDA’s 2008 data to recent years (2003 – 2007) would show an increase in E. coli O157:H7 in raw ground beef.[8]
Similarly, Mr. Boyle selectively compared CDC’s FoodNet 2008 and 2000 data, concluding that E. coli O157:H7 infections have decreased by 44%. Again, this statement is misleading and ignores recent trends. In actuality, CDC does not compare individual years of FoodNet data (i.e. 2008 vs. 2000). Rather, CDC compares the data for a given year to a composite of the 1996-1998 FoodNet data and to a composite of the preceding three years, which in this case would be 2005-2007 FoodNet data. This is done to account for changes in the number of FoodNet sites and changes in the size of the population. It is true that, when comparing the 2008 FoodNet data to the 1996-1998 composite, E. coli O157:H7 infections have decreased 25%. [9,10] However, when comparing the 2008 FoodNet data to the 2005-2007 composite, E. coli O157:H7 infections have not changed significantly.
Mr. Boyle’s recent assertions represent an inappropriate use of data and did not provide the American public with a complete and clear picture of recent trends in E. coli O157:H7. In fairness, Mr. Boyle is not alone in making this error. In fact, USDA’s own Website states that that data from these testing programs provide information about the trends in pathogen presence in meat and poultry products over several years. But as stated above, the data should not be used for these purposes. Even so, using Mr. Boyle’s and USDA’s logic, it could be said that, despite initial declines, the incidence of E. coli O157:H7 in raw ground beef has not changed in recent years and may even be increasing. However, due to significant data limitations, even this statement would be misleading. The reality is that, based on the data collected, no one can reliably conclude that the incidence of E. coli O157:H7 in raw ground beef has gone up, gone down, or stayed the same.
Mr. Boyle has asserted that the “meat industry has made great strides in improving the safety of our products.” It is true that AMI and USDA have spent huge amounts of money to improve food safety over the past 15 years. For this, they should be applauded. Unfortunately, it is also true that, despite these efforts, E. coli O157:H7 remains a problem in raw ground beef. Misusing statistics to paint a “rosy picture” of the meat and poultry industry is not helpful. Instead, we need to openly face this problem and work together to find solutions.
APPENDIX
Progress Report on Salmonella Testing of Raw Meat and Poultry Products, 1998-2002, USDA Food Safety Inspection Service
The prevalence data reported here for Salmonella in raw meat and poultry products have certain limitations that restrict the range of valid statistical inferences. The PR/HACCP verification testing program is strictly regulatory in nature and was designed to track establishment performance rather than to estimate nationwide prevalence of Salmonella in products. Because the program is not statistically designed, different establishments may be sampled from year to year, confounding rigorous trend analyses. Furthermore, it is important to note that the prevalence estimates computed from the FSIS’s pre-HACCP baseline studies and surveys were nationally representative because they were weighted on the basis of the production volume of the sampled establishments. In contrast, the PR/HACCP Salmonella prevalences from the regulatory testing program reported here represent unweighted test results from sampled establishments.
Food Safety and Inspection Service Oversight of Production Process and Recall at ConAgra Plant (Establishment 969), USDA Office of Inspector General Great Plains Region Audit Report, Report No. 24601-2-KC, September 2003
FSIS’ E. coli O157:H7 testing program cannot be used to measure the effectiveness of HACCP on either a company or a nationwide basis. The sampling program, as designed, does not provide scientific, risk-based data to measure the extent of an existing hazard. The data that isproduced does not reflect industry performance because
a) plants like ConAgra, that performed their own E. coli O157:H7testing on carcasses were exempt from sampling of ground beef,
b) sampling plans do not take into account all relevant plant operational or processing factors, and
c) samples taken at the plants that are selected are not always representative of the lot of production or final product.”
Microbiological Testing Programs for Ready-to-Eat (RTE) Meat and Poultry Products, USDA Food Safety Inspection Service
The data presented here are from regulatory testing programs that change from year to year and even within years and thus any comparisons should be made with caution. The findings are referred to as prevalence data in that they are presented in terms of the percentage of regulatory samples that are found to be positive. These programs have not been designed to test for statistically significant change from one year to the next. The aggregate data do, however, provide an overall indication of trends….. None of the RTE sampling programs, either before or after December 2000, were designed to provide statistical estimates of national product prevalence. These programs do, however, provide an indicator of whether pathogen prevalence is increasing or decreasing across different product categories.
Progress Report on Salmonella Testing of Raw Meat and Poultry Products, 1998-2008
www.fsis.usda.gov/Science/Progress_Report_Salmonella_Testing/index.asp
Restructuring how Salmonella sets are scheduled means that comparison of results from 2006 onwards to previous years will be less meaningful in terms of trends. Similarly, the changes to the verification program will prevent valid comparisons of testing results over time (e.g., quarter-to-quarter or year-to-year trends).
Microbiological Results of Raw Ground Beef Products Analyzed for
Escherichia coli O157:H7, Summarized by Calendar Year www.fsis.usda.gov/Science/Ecoli_O157_Summary_Tables/index.asp
Beginning with CY 2008, annual microbiological sample results will be posted according to the date the sample was collected. Prior to CY 2008, yearly posting of microbiological data results was based upon the sample analysis completion date. For this reason, data from CY 2008 can not be directly compared to CY 2007 and prior years. In addition to the change in date criterion, target sampling that incorporates production volume and results history was introduced as well as incorporating a change in the laboratory testing method.
The FSIS Microbiological Testing Program for Ready-to-Eat (RTE) Meat and Poultry Products, 1990-2008
www.fsis.usda.gov/Science/Micro_Testing_RTE/index.asp#trends
Throughout the history of the FSIS microbiological RTE regulatory sampling program, the individual projects have been continuously evolving in response to public health concerns. The data reported here are from these individual sampling projects that have been implemented under the overall RTE program. This testing program has not been designed to test for statistically significant changes from one year to the next. The aggregate data do, however, provide an overall indication of trends…. As mentioned above, the FSIS does not view the results of regulatory testing as estimates of national product prevalence.
References:
- NY Times (http://www.nytimes.com/2009/10/07/opinion/l07meat.html?_r=1), Larry King Live (10/12/09)
- FoodSafety and Inspection Service Oversight of Production Process andRecall at ConAgra Plant (Establishment 969), USDA Office of Inspector
General Great Plains Region Audit Report, Report No. 24601-2-KC,
September 2003
- Microbiological Results of Raw Ground Beef Products Analyzed for Escherichia coli O157:H7, Summarized by Calendar Year www.fsis.usda.gov/Science/Ecoli_O157_Summary_Tables/index.asp
- Progress Report on Salmonella Testing of Raw Meat and Poultry Products, 1998-2002, USDA Food Safety Inspection Service
- Microbiological Testing Programs for Ready-to-Eat (RTE) Meat and Poultry Products, USDA Food Safety Inspection Service
- ProgressReport on Salmonella Testing of Raw Meat and Poultry Products, 1998-2008 www.fsis.usda.gov/Science/Progress_Report_Salmonella_Testing/index.asp
- The FSIS Microbiological Testing Program for Ready-to-Eat (RTE) Meat and Poultry Products, 1990-2008 www.fsis.usda.gov/Science/Micro_Testing_RTE/index.asp#trends
- http://www.fsis.usda.gov/PDF/RGB_Testing_Figure1.pdf
- http://www.cdc.gov/foodnet/factsandfigures/trends.html
- The confidence interval for this estimate is (8%, 39%).