First, there is the Dept. of Defense's quarterly report to Congress about the status in Iraq - Measuring Security and Stability in Iraq. The most recent report is dated November 2006. The report includes 9 trend charts and 1 trend table. Some of these charts are variants of trend charts we have already seen from our look at the Brookings Iraq Index , Michael O'Hanlon's testimony, GlobalSecurity and the Iraq Casualty Coalition . Others, however, are new and complementary to the charts we have seen so far. For example, there is a single chart combining three separate trends (average daily casualties for Iraqi Civilians, Iraqi Security Forces and Coalition Forces) and there is another chart showing "Ethno-Sectarian Incidents and Executions".
I have not yet had the chance yet to study the most recent Dept of Defense Quarterly Report in depth. My initial scan indicates that it includes mention of dozens of other important factors whose trend patterns might help us better understand the situation on the ground, but for which no trend data or trend graphics are presented. Among these missing metrics or what might also be called "invisible indicators" or "faceless factors" are many new metrics that are not included even in our list of more than 100 metrics that we have already mentioned in this series of posts. We have it on our to do list to see if in the next couple of weeks we can bubble up these missing metrics to add to our growing list.
Second, there is a powerful, provocative and quite critical report by Anthony Cordesman analyzing the Dept of Defense's Quarterly Measuring Security and Stability in Iraq report that the DoD issued in May 2006: The Quarterly Report on “Measuring Stability and Security in Iraq:” Fact, Fallacy, and an Overall Grade of “F”
Here's a brief quote from the introduction [emphasis added]:
The media can provide some of this picture, as can outside experts and scholars, but only the US government has the resources and access to information that provide a comprehensive overview of the situation. The quarterly report to Congress issued by the Department of Defense, “Measuring Stability and Security in Iraq,” is supposed to be a key document to achieve this goal.This entire report is well worth reading. It focuses on the reliability and validity of the trend data we are getting - a topic that we have not yet touched on in this report but which is of course highly relevant to everything we are looking at. It would be extremely valuable to repeat this kind of evaluation of the quality of data sources with the latest DoD quarterly report, with the weekly State Dept report, with the Brookings Iraq Index and with ALL the factors that we rate as important and relevant to understanding the situation on the ground in Iraq.
Like the State Department weekly status report on Iraq, however, it is deeply flawed. It does more than simply spin the situation to provide false assurances. It makes basic analytical and statistical mistakes, fails to define key terms, provides undefined and unverifiable survey information, and deals with key issues by omission.
In my view Anthony Cordesman brings a level of rigor to this particular report that is lacking from the other sources we have so far examined and his approach would be well worth emulating going forward.
In the meantime, just because there are questions about the quality of many of the individual data factors, that doesn't mean we should stop doing what we've been doing. This does not mean that we should stop measuring and or that we should stop regularly updating our trend tables and trend charts or that we should stop searching for missing metrics and adding them to our list.
In fact, I think you will find that it is true that by being able examine many trend factors in a time efficient manner provides powerful checks and balances that can help us discover which factors may be most likely to suffer from quality problems.
If you know of other sources of vital Iraq trend data that we are missing, please let us know.
No comments:
Post a Comment