Language and the 18 benchmarks

Here's a link containing the language of the 18 "benchmarks" included in the recent Iraq Supplemental Bill HR 2206

Of the 18, most are not readily amenable to quantification. The short list below are 5 of the more quantifiable targets contained in these benchmarks.

More on this in future posts. For now, the supplemental bill appears to fall afoul of the same language difficulties we have been noting in various articles from around the web - namely mushy metrics, imprecise or missing quantification. If this is the document that is going to guide our evaluation of how things are really going in Iraq, we are in deep trouble as it appears to be structured in ways that are likely to defy exactly that process.

3) equitable distribution of hydrocarbon resources

9) Providing three trained and ready Iraqi brigades to support Baghdad operations.

13) Reducing the level of sectarian violence in Iraq and eliminating militia control of local security.

14) Establishing all of the planned joint security stations in neighborhoods across Baghdad.

15) Increasing the number of Iraqi security forces units capable of operating independently.

17) Allocating and spending $10 billion in Iraqi revenues for reconstruction projects, including delivery of essential services, on an equitable basis.

However, these 5 are amenable to quantification and we could begin now (rather than waiting till September) to make these specifics, to measure them regularly, and to report on them in a way that will be easily accessible to all.

When you look at the full text of benchmarks and at the required reporting standards and timing, it's noteworthy that there are no explicit benchmarks for what the United States is required to do in this time frame. Clearly the situation in Iraq will be dependent both on what the Iraqi government does and what the United States does, but this particular bill places all the responsibility on Iraqi shoulders.

Postponing the Moment of Truth

Check out this excellent Postponing the Moment of Truth cartoon from cartoonstock.com.

This is a perfect match to the imprecise approach, the mushy metrics, the startling lack of numerical quantification, and the complete absence of visualization as exemplified by (but certainly not limited to) Stolberg's See You In September report in the New York Times as highlighted in the previous post.

Revisiting "See You In September"

I want to return to the Sheryl Gay Stolberg article

See You in September, Whatever That Means

published in the New York Times on May 13th, 2007. There are many more important clues we can discover hiding the text. I did not have time to unravel them all during the first Charting Progress in Iraq post on this subject.

Following along the lines of the Forecast on Iraq post we'll once again de-construct and highlight some of the language and see what else that reveals. For fun, I have color coded the quoted text as follows.

Orange - vague metrics (e.g. progress, confidence, winning, losing)

Purple - vague quantification (e.g. up, down, better, more)

Green - specific metrics (e.g. sectarian murders, monthly bomb rate). Unfortunately, none of these have any specific numbers associated with them. Some have vague quantification and others have non at all.

• "decisive progress"
• "looking better"
• “Sectarian murders are down”
• "car bombings ... are up"
• "deaths of American soldiers are up"
• "some in Washington are grasping for a more complete and accurate way to quantify progress"
• “No one knows how to define progress in such a mixed-up situation”
• “We’re having trouble measuring it."
• [Iraq Index includes] "the monthly car-bomb rate"
• [Iraq Index includes] "how many foreign nationals are kidnapped"
• [Iraq Index includes] "how many Iraqis have electricity"
• [Iraq Index includes] "how many Iraqis have ... Internet access"
• [the Iraq index] "is long on numbers and short on analysis"
• "some signs of hope"
• "more grounds for worry than for confidence"
• [Rep Kingston (R-Georgia) wants to] "winnow down the indicators to a manageable number -- say, fewer than a dozen"
• [Rep Kingston (R-Georgia) wants to create] "a standard bi-partisan metric"
• [with such a metric in place] "then you could say who’s winning and losing”
• [Micheal O'Hanlon says] "metrics are grist for a fact based debate but history shows it is dangerous to rely on too few of them"
• [Micheal O'Hanlon says] "metrics were used in Vietnam and we had the wrong ones"
• [the Vietnam metrics] "did net harm to the debate"
• [Micheal O'Hanlon says] "we can’t be exactly precise about which indicators are the conclusive ones”
• "such an index would be politically unpalatable to the White House, which does not want to back itself into a corner by agreeing to someone else’s standard for progress"
• "the White House says the only progress report that counts is the one from Gen. David H. Petraeus, the new top commander in Iraq, and Ryan Crocker, the new ambassador"
• "is the current strategy for waging war going well or not?"
• [Stephen Biddle] did fault the White House for not being more open with the public about its own idea of what constitutes progress
• “By being unbelievably vague about everything,” he said, “they’re making it very hard for congressmen and senators to go to their constituents and say, ‘Look, here’s why things are going better than you might imagine.’ ”
• "Some say measuring progress is simple: you will know it when you see it."
• [Metrics suggested by Senator Robert Bennett (R-Utah)] I want to see life starting to come back,” “I want to see people in markets. I want to see couples strolling down the street, folks sitting at outdoor cafes.”
• [Metrics suggested by Senator Susan Collins (R-Maine) ] “a significant reduction in violence and attacks accompanied by a transfer of more and more authority to the Iraqi forces.”
• [Senator Collins] “the difficult question is going to be if the analysis is mixed, and I suspect it may well be.
• [Senator Lamar Alexander (R-Tennessee) says] “I don’t see any way for us to maintain a long-term presence in Iraq without more bipartisan support.”
• [Rep. Kingston] "hopes to come up with some useful way of figuring out ...whether [to] remain supportive for a little while longer"
  
• [Rep. Kingston has] " heard three years of nearly happy-talk in testimony"
• “We always seem to be about to be around this elusive corner, but we never get there.”

Bottom line:
The language we use to talk about the important aspects of our lives makes a difference. When we use language about an evolving situation this way (vague metrics coupled with imprecise quantification, and lack of any clear trend quantification and visualization), we are going to have a tough time making sense of what's actually happening, what it means, and what our best next step might be.

If we continue along this path of mushy metrics, vague quantification, and numerous instances of what appears to be learned helplessness, come September, we won't be in any better state to evaluate the situation than we are now

I have some more thoughts on some of the key quotes from above for a future post. There is a way forward that addresses Michael O'Hanlon's concerns and answers Rep. Kingston's call for assistance.

Washington Post. Forecast on Iraq

Today's Washington Post featured this story with quotes from Lt. Gen. Douglas E. Lute, the new White House War coordinator (previously know as War Czar):

Nominee to Coordinate War Offers Grim Forecast on Iraq - washingtonpost.com

Here are some quotes relating to assessment of current trends and the direction we are heading. Bold emphasis has been added.

Lute said:

Iraqi factions have shown so far very little progress ... we're not likely to see much difference in the security situation" a year from now.

A CIA expert on radical Islam was quoted by Senator Bayh saying:

our presence in Iraq is creating more members of al-Qaeda than we are killing in Iraq.

The Post noted that the recently passed supplemental funding legislation

requires progress reports from the administration in July and September.

Here are some trend related quotes.

While insisting that incremental progress is being made, administration officials acknowledge that the benchmarks will be all but impossible to meet by the time progress reports are due to Congress.

"If the test is peace and reconciliation and flowers blooming by September, that will be hard to meet," said a senior administration official.

At the least, the administration hopes to be able to demonstrate movement in the right direction.

There has been little progress in four key political areas

At his hearing yesterday, Lute acknowledged that during a review of U.S. policy on Iraq last winter, he had privately expressed skepticism of Bush's "surge" proposal unless matched with more robust efforts by the Iraqis and other U.S. agencies.

"Where are we today?" he [Lute] asked. "Not where any of us would like."

Notice anything here?

Or rather do you notice anything missing here?

There's nary a single number and all the key factors and indicators referenced with regard to progress are not the kind that can normally be quantified. There's also no sense of any of these indicators changing step by step over time.

Let's list them so you can see what I mean.

Here are the metrics discussed in the article

1. "progress"
2. "the security situation"
3. number of members of al-Qaeda created
4. number of members of al-Qaeda killed in Iraq
5. "peace"
6. "reconciliation"
7. "flowers blooming"
8. "movement in the right direction"
9. more robust efforts by the Iraqis
10. more robust efforts by other US agencies
11. progress in four key political areas"
    

• Revisit constitution to better balance power sharing
  
• Oil revenue sharing
• provincial elections
• de-Baathifcation

And equally challenging to actually being able to use these trends to help steer our course forward is the mushy nature of the current "values" of these important indicators:

1. very little
2. not much difference
3. incremental
4. hard to meet
5. movement
6. little
7. more

Contrast the lack of quantification and therefore the lack of clarity with the numerical metrics we discussed in our series of posts yesterday such as:

• peak megawatt hours
• millions of barrels of oil per day
• Iraqi prison population
  

The one bit of good news mentioned is that a progress report from the administration is due in July so at least we won't have to wait for September for that. How much we learn in July will depend an awful lot on whether the progress is report is based on hard metrics with clear quantification or on the kind of mushy, mostly unquantifiable indicators we find the WaPo report.