"It's still too early" or is it?

Here's a link to a June 13th DoD news article announcing the quarterly report to Congress on Iraq DefenseLink News Article: Quarterly Iraq Report Cites Progress, Challenges

It leads off with [emphasis added]:

It’s still too early to assess the impact of the new strategy in Iraq, but more progress is expected as additional troops come on line to boost security in Baghdad, according to the latest quarterly report to Congress, released today.

The executive summary of the quarterly report itself includes:

It is too soon to assess results.

Overall, it is too early to assess the impact of the new approach.

Although it is premature to judge whether FAQ is creating the conditions necessary for political reconciliation, ...

Just to make sure we didn't miss the point, the next day, June 14th, a new article headlined Too Early to Discern Trends in Iraq appeared and opened with paragraph 1

It is too early to discern trends out of the U.S. troop “surge” as part of the Baghdad security plan, defense officials said here today.

and continued in paragraph 3 saying

“It is too early to assess the impact of the new way forward,” a senior defense official said.

This is of a piece with the widespread and repeated propagation of the we have to wait till September meme that seems intended to deaden our ability to judge the facts so far for ourselves.

All that's needed to assess, to judge, to discern the impact of what has happened up until the present moment is to look at the freshest possible data on all the important factors. If we each available ourselves of the weekly sources of such data such as the Brookings Report, the State Dept. Report and the Baghdad Embassy report, the we can then have a fruitful discussion about what each of us thinks the data means and what we feel might be the best course forward.

On the other hand, if we listen to the siren call of the "too early to tell" and "wait till September" crowd, we will forfeit any chance we have of adjusting our behavior to increase our chances of success as we move forward.

Iraq Transition Assistance Office - Weekly Status Repot - May 29, 2007

The Iraq Transition Assistance Office (part of the US Embassy in Baghdad) is now publishing a Weekly Status Report. Here's a link to the May 29th, 2007 report.

This is a continuation of the excellent work Iraq Reconstruction Management Office under a new name. This weekly report rates high on both graphical interest and on providing fresh data with an excellent score for near real time reporting.

Check out the important indicators on page 23. While the tabular format is not for everyone, there are some important indicators in this table that are worth paying attention to.

Dept of State - Iraq Weekly Status Report June 13 2007

Here's the link to the latest Dept of State Iraq Weekly Status Report (June 13, 2007). This report has some much fresher (more recent) charts on oil and electricity than the quarterly DoD report. This report successfully achieves an excellent grade from delivering its data in near real time fashion.

For those interested in keeping up with what's happening on the ground in Iraq, this weekly report is one of the essential pieces to pay attention to.

Ten Trend Charts from the DoD Quarterly Report

The Dept of Defense has just published its mandated quarterly report to Congress on the situation in Iraq - Measuring Stability and Security in Iraq, June 2007 (51 pages). As they state in the Executive Summary, the "report includes specific performance indicators and measures of progress toward political, economic, and security stability in Iraq"

Included as part of the report are 10 static trend graphs which I have reproduced in this post. You can click on each for a full size image. If you would like all 10 JPG images so you can look at full size with your favorite slide viewer, you can download this zip file.

Most of the important performance indicators mentioned in the report did not rate their own static trend graph chart. As we did with the UN document and our scan of recent news articles, we plan to comb through the document and come up with a list of indicators that complement the ones we have already highlighted so as to get a sense of the breadth of important indicators that are already on our radar screen.

Chart 01 - Oil Production - too short a time span. Also missing data for May 2007.


Chart 02 Electricity - Missing data for April and May 2007.



Chart 03 - Sectarian Murders and Incidents . Missing data from May 2007. Weekly data might be important to look at.


Chart 04 - Weapons Caches Found - This is looking promising. Access to weekly and daily data could be even more revealing of ongoing trends. Missing data for May 2007.



Chart 05 - Average Weekly Attacks - Missing last 5 weeks of data. This is the source for the charts that appeared in WaPo and NYT as reported earlier today.

Chart 06 - Average Daily Casualties - Missing last 5 weeks. Bar chart format with three trends makes reading trends for Iraqi Security Forces and Coalition more difficult.


Chart 07 - Hotline Tips - Interesting new performance metric. Looks positive. Missing past month of data. Weekly or daily data reporting could be even more revealing.


Chart 08 - Confidence in Iraqi Government to Improve the Situation. This looks surprisingly to me for its stability over time. I will need to read the text about this in detail. Missing most recent month's data.


Chart 09 - Assessed Capabilities of Police - Difficult to interpret trends when only two data points per item. Stacked bars make interpretation more difficult and time consuming. This is an interesting new metric and it could benefit from higher resolution, near real time reporting.


Chart 10 - Ministry of Defense Forces Assessed Capabilities. This chart also presents some important data but suffers from same failings as previous chart - only 2 data points per indicator, stacked bars are hard to read.


Summary. 10 Static charts covering a range of important indicators is more than almost anyone else except Michael O'Hanlon produces when it comes to highlighting the important trends evolving in Iraq.

So that's definite plus. Someone who spends a few minutes can examine all 10 of these (especially if they can scroll in slide show mode on full size images using this zip file).

On the minus side, the missing last month or last 5 weeks of data is a serious drawback. Many of the charts can be improved on and made more readily useful to ordinary viewers by applying some trend chart best practices principles. These include

• use of full time range
• use of near real time reporting for the most recent interval
  
• selection of best resolution or choice of several different reporting intervals (day, week, month)
• displaying each important indicator separately with appropriate Y axis scale
• having sufficient data points for trends to be discernible - 2 is way too few
  

And worst of all, most of the important indicators have no charts at all and are only discussed in text or in tables showing the current values.

The indicators in this report are important, but this method of presentation falls short of ideal and short of what is needed and what indeed is possible. With the high number of factors discussed in this report, the only method that will actually make this kind of trend data usable to its intended primary audience (Congress) and to the secondary audiences including the media and ordinary interested citizens would be to deliver this data via a 21st century Trend Visualization Appliance.

What's the Best Reporting Interval?

Why Wait for a Quarterly Report? Yesterday's DoD report to Congress is a mandated quarterly report. The executive summary begins with:

This report to Congress, Measuring Stability and Security in Iraq, is submitted pursuant to Section 9010 of the Department of Defense Appropriations Act 2007, Public Law 109-289. The report includes specific performance indicators and measures of progress toward political, economic, and security stability in Iraq, as directed in that legislation. [emphasis added]

My view: This report can do double duty. Quarterly reports are all well and good. They present us with specific performance indicators and measures of progress. They offer an opportunity to take a step back and look at the evolving patterns.

AND between reports, they can be even more helpful on a daily, weekly and monthly basis.

Just as we don't have to wait till July or September to find out how well the "surge" strategy is working, we don't have to hold our breath between quarterly reports. We can make double duty of the infrastructure already in place to produce these reports to create a mechanism that lets us keep track of the most important indicators and measures of progress as we go along.

Here's How.

1. Start where we are at. The quarterly report defines the currently understood list of the most important performance indicators and measures of progress.
   
2. High Frequency Measurement. For each of these important indicators we establish a measurement frequency that is as rapid as is practical at this time.
   
3. Near Real Time Reporting. For each of these important indicators we also establish a near time reporting interval that is as close as is practical to the time of the most recent measurement sample
4. 21st Century Visualization. We then make the full set of high resolution trend data for all the most important performance indicators and measures of progress available. We make it available in a continuously updated form in near real time using a 21st century Trend Visualization Appliance. This makes it possible and manageable to observe the trends of a large number of factors within the time and resource constraints of the intended audiences.
   
5. Improve Frequency. For any important factor where it proves impractical today to measure at the frequency that we think best, we set in motion a process to speed up that frequency as quickly as we can bring it on line.
6. Improve Freshness. For any important factor where it proves impractical today to deliver the most recent measurements in suitable near real time fashion that would meet our needs for fresh data, we set in motion a process to reduce the delay as quickly as we can.
7. Identify and Add Missing Indicators. Finally, as soon as we notice some new and important indicator that is vital to understanding progress in Iraq, we add it to the list of indicators to be included in the Quarterly report AND to be delivered to our Trend Visualization Appliance at high resolution and in near real-time for our consideration between these quarterly reports.

Shorter Version:
If there is a performance indicator that is important enough to be reported on quarterly, then it is even more important to make the underlying trends for that indicator available interactively and efficiently, at high resolution, and in near real time for all interested parties.

The current arrangement (infrequent reports, delayed reporting, lower frequency data) does not appear conducive to the most effective oversight by congress, the media or the general public. We can and must do better if we want to make sense of what is happening in Iraq.

Near Real Time Reporting

For most important situations in life where trend visualization is helpful, we are lucky that real time reporting is not required. However, collecting the data and reporting it within a reasonable time from the end of measurement is important and can deeply impact the value and usefulness of what we have collected.

An Unacceptable Delay. We noted earlier today in the post on the New York Times Iraq Progress Graphic how the underlying data from the DoD quarterly report to Congress only covered up through May 4th. So in effect we were looking at data that wasn't produced in a report till 5 and 1/2 weeks after the end of the most recently reported period.

In my opinion, this represents an unacceptable delay and that delay severely impacts the ability of any audience to gauge what the data means and to decide what to do next.

If the underlying data for attacks and for casualties were collected at the recommended rate of once per day, a near real time report on June 13th would be best if it covered everything that happened up through one, two or three days prior to June 13th.

Fresh Data. The combination of high frequency measurement and near real time reporting means that the data we are looking at is fresh - a vital quality in any rapidly changing situation. There are numerous examples in other aspects of our lives where near real time reporting and high resolution reporting are simply the norm (the stock market being a prime example).

There are sometimes situations that delay and interfere with our ability to roll up the most recent results of a particular indicator. In some cases, these are historical and can be overcome and minimized if we decide to put our minds to it and invest in the needed infrastructure to make it happen. In other cases, there are time lags that are mostly out of our control. The best we can do in these cases may be to put our energy towards devising ways to minimize the delay.

Bottom line: In a trend chart that is going to help us understand what is truly going on, timeliness is very important. When we get our measurement rate set appropriately and we make every effort to minimize the delay between collection and reporting, the trend work that we do will have the maximum value to all those who are interested.

Frequency of Reporting

In the previous post, we brought up the concept of frequency of reporting on a given metric. We haven't talked much about this aspect of trend analysis so far, despite the fact that it can be quite important. And it interacts with another important topic we brought up in some previous posts: near real time reporting.

We'll take a look briefly at frequency in this post, and give a brief look at near real time reporting in the next.

Error on the Side of Too Often.
Each important factor that you wish to measure will prove most useful in the end if you make a conscious choice to consider its frequency of measurement separately on the merits. My general operational rule is wherever possible to err on the side of measuring too often rather than not often enough.

Measurement vs. Reporting Frequency. When you measure at a relatively high frequency, it's possible later to report on the data at either the high collection frequency, or at a reduced frequency in order to smooth out the trend line pattern. On the other hand, if you don't measure frequently enough, you may miss out on short-lived quantum shifts in behavior that will be lost when reporting at longer intervals.

It's unfortunately not uncommon for vital indicators that are already being collected to have their frequency set so that measurements are too far apart.

Measuring Daily is Often a Good Starting Point. Let's take a specific example: the percentage of eligible children who attend primary school. Ideally, in my opinion, a metric such as this would be best if it were measured daily. There will be times when you will want to look at the day to day trend and see if any disruptive multi-day even has shown up. You may also want to understand any patterns that might be dependent on the day of the week. There will be other times when week by week reporting will be just right and still others when month by month will give the clearest view. Before you actually have the data, you cannot really know which reporting interval will prove most revealing.

My recommendations for the important Iraq metrics of the kind we have been scraping from the recent news reporting is that we make every effort to measure them with a frequency of once per day wherever that is possible.

When is Higher Frequency Helpful? There are some indicators where it would be advantageous to measure them even more frequently than once a day. For example, electricity delivery in Baghdad is a factor that I would like to measure minute by minute during the day and to do this both for the city as a whole as well as disaggregating these results by districts in the city and/or by individual power stations. Unless I get a very high resolution frequency, I won't be able to understand the frequency of or length or breadth of districts impacted by each outage. Such diagnostic capability is likely vital to figuring out where the est opportunities are for achieving higher levels of service and moving closer to fulfilling the desired demand.

Obstacles to High Frequency. On the other hand, there will be metrics such as the percentage of Iraqi members of Parliament who are living abroad where the obstacles to measuring may mean that we only get a read out on this indicator once a month. It might be nice to see how this varies day to day, but prove to be logistically impractical to get it.

Adjusting as you go along. There is no perfect frequency and there can likely be differences of opinion as to which frequency is best. The good news is that frequency can be adjusted as needed as we go along. The really important part is identifying all the important metrics and beginning the process to measure and record their values as they change over time.

Scraping Metrics - Part III: UN Report

As we mentioned in an earlier post, UN Secretary General Ban Ki-moon has recently published a quarterly report detailing the situation in Iraq from the UN's point of view.

This post continues with the idea that the news around us give us ideas that help answer the following questions:

• What are the most important indicators?
• What we must measure to form a fuller picture how things are going in Iraq?

When tackling a task like this, the idea is to be on the lookout for important factors that we haven't yet included in our growing composite list.

The Secretary General's report is a rich source for these kinds of ideas. Some of the measurable indicators that come up are named directly. In other cases, we use an idea expressed in the report as a springboard for creating a specific, measurable entity that would at least partially shed some light on how well things were going in that general area.

Thousands of Words and Nary a Picture. The report itself does not include a single trend graph which is a disappointment as the UN has a unique perspective and likely has access to key data that may not be available elsewhere. The report relies on the spoken word for conveying information about trends and therefore falls short of what its potential might have been.

Here then is the list of important UN indicators. Many of these can be profitably subject to disaggregation into Shia, Sunni, Kurd, or by location in Iraq.

1. Number of kidnappings
   
2. Oil revenues - how are these being divided today, prior to the passage of new oil revenue sharing law (by region, by ethnic category, by different government agencies)
   
3. Millions of dollars of new loans secured
4. Number of newly displaced Iraqis inside Iraq
5. Number of newly displace Iraqis who have fled to neighboring countries
6. Millions of dollars of financial assistance provided to internally displaced Iraqis
7. Millions of dollars of financial assistance provide to Iraqis who have fled to other countries
8. Number of Iraqis registered to vote
9. Number of law enforcement personnel charged with serious human rights violations
10. Number of attacks on journalists or media persons
11. Number of casualties (wounded) registered in hospitals
12. Number of families losing their breadwinner to death or incapacitating injury
    
13. Number of family members losing their breadwinner
14. Percentage of eligible children attending school (primary, secondary) (male, female)
    
15. Number of students attending college (male, female)
16. Number of college graduates
17. Number of advanced degrees awarded
18. Number of doctors completing their training
19. Number of nurses completing their training
20. Millions of dollars of Iraqi assistance to displaced persons
21. Percentage of population with safe water supplies
22. Percentage of displace persons having access to the public
23. Percentage of malnourished children
24. Additional contributions to the United Nations Development Group Iraq Trust Fund
25. Number of attacks in the International Zone (Green Zone)
26. Number of incoming rounds of mortars, rockets in the Green Zone
    
27. Number of deaths and injuries in the Green Zone from attacks
28. Number of attacks on strategic targets - e.g. bridges, major thoroughfares, checkpoints near the Green Zone, Parliament building
29. Number of attacks on infrastructure for electricity, water, oil, sewerage
30. Number of neighborhoods in Baghdad with new security outputs
31. Number of attacks on new security outposts
32. Percentage of buildings in Green Zone with hardened overhead protection capable of withstanding attack from large scale ordnance such as 107-mm and 122-mm rockets.

Imagine how much better we would understand what is actually happening in Iraq if we had a trend history and regular, moderately frequent, near real time updates for each of these factors inspired by the Secretary General's report.

Metrics inspired by the news part II - Prison Metrics

Here are some additional metrics inspired by recent news from Iraq.

This batch is based on the seed idea that what is happening in the prisons in Iraq today can shed important light that would otherwise not be visible.

1. Current prison population under Coalition control (male, female)
   
2. Current prison population under Iraqi patrol (male, female)
3. Number in prison who have not yet been charged
   
4. Number in prison who have been charged
5. Number in prison currently under trial
6. Number convicted
7. Number found not guilty and released
8. Average time from arrest to being charged
   
9. Average time from arrest to trial completion
10. Percentage of prison population who are considered insurgents
11. Prison population (Shia, Sunni, Kurd, other)
12. Number arrested per day
13. Number freed per day
14. Number in prison who are under the age of 18
15. Prison population as a percentage of maximum rated capacity
16. Number of deaths while in prison
17. Number of suicides of inmates
18. Percentage of prisoners in solitary confinement
19. Prison population disaggregated by prison location

I am not an expert on prisons, but it would seem that if we could track the indicators above, we could begin to get a sense of how this aspect of life in Iraq today was evolving. Ideally, those who are expert in understanding prisons can add their favorite and most important metrics to the mix if I have not included them.

Bridging Rep. Kingston and Michael O'Hanlon's Concerns

In the two previous posts, we have spoken to the concerns of Rep Kingston and Michael O'Hanlon as captured in the Sheryl Stolberg New York Times See You in September ... article.

To be successful in understanding what is happening in Iraq, we need to listen to both their voices.

A Sufficient Number of Factors. We need to pay heed to O'Hanlon and make sure we have looked at enough different factors.

Manageable Metrics. We have to hear Rep. Kingston's concern and worry about being overwhelmed with too much data or rendered ineffective by not having enough time by making whatever data we report on "Manageable" within the constraints of the time and resources that are available to us.

Eating our Cake. It is possible to meet both of these goals. We are not doing it today, but we are surrounded daily with visible proofs that equally difficult trending situations (see the St. Louis Fed Reserve FRED application as an example of what's possible) can be brought within our grasp and control by the judicious application of 21st century Trend Visualization methods.

Along the Way. Paying attention to our view of the best practices and principles of trend graphic excellence will provide the third ingredient that helps make our efforts in this arena more effective and more readily sharable with others. For a discussion of these, check out the Trend Visualization Principles displayed near the top of the right column of this blog or follow this link to trend principles for a review of some of our other thoughts on this subject.

Dangerous to have too few metrics or the wrong metrics

In a previous post: Revisiting "See You In September" I noted:

There is a way forward that addresses Michael O'Hanlon's concerns

The relevant quotes from O'Hanlon were:

• "metrics are grist for a fact based debate but history shows it is dangerous to rely on too few of them"
• "metrics were used in Vietnam and we had the wrong ones"
• [the Vietnam metrics] "did net harm to the debate"
• "we can’t be exactly precise about which indicators are the conclusive ones”

Here are a few observations related to the above.

Dangerous to Rely on Too Few of Them. I agree entirely. For Iraq, we need to be able to look at at least 50-100 of the most important factors efficiently, all within a relatively small period of time. For expert use, a much higher number of metrics must be available for review in an efficient and effective manner. Michael O'Hanlon's weekly Iraq Index is one of the best examples out there that pays attention to the principle of not relying on too few metrics.

The Metrics were the Wrong Ones. This is a solvable challenge this time around. First, follow the principle above and make sure that we look at ALL of the important factors. How do we decide? We ask all the experts in all the different areas (security, economy, health, ...) to name what they consider to be the most important indicators. Then we capture and archive and report on all the factors that have been suggested, even if only once. Maybe there will be some "wrong ones" in the mix, but when we have the full set of data to work with, the chances of being led astray will be markedly reduced.

Can’t be exactly precise about which indicators are the conclusive ones. We don't have to be precise. Different people will have different opinions. If we capture all of them, record a history of each trend, and make this data available in Static Graphical format, AND as machine readable data suitable for input into a Trend Visualization application, then we can begin to have good and constructive discussions about what different trends mean and what we should do about them.

A "Manageable" Number of Indicators

In a previous post: Revisiting "See You In September" based on the Sheryl Stolberg New York Times article I noted:

There is a way forward that ... answers Rep. Kingston's call for assistance.

According to Stolberg, "some in Washington are grasping for a more complete and accurate way to quantify progress." “No one knows how to define progress in such a mixed-up situation.” Others say “we’re having trouble measuring it.

Rep Kingston (R-Georgia) wanted to "winnow down the indicators to a manageable number -- say, fewer than a dozen." He wanted to create "a standard bi-partisan metric." With such a metric in place, "then you could say who’s winning and losing.”

The key words here are: "manageable number" and the key idea is that the medium we chose to publish these numbers matters to a huge degree. The thesis of this post is that what constitutes "manageable" depends on how we chose to present the data.

Using Unquantified Indicators. If we don't get down and get numeric with the metrics we are interested in, then the manageable number doesn't matter.

Talking about Trends. If all we do is talk about numeric trend data, then maybe the manageable number is only 1 or 2 and even with that small a number of items, most listeners won't end up with a good sense of of how those two indicators are changing over time. Speech is simply not a good way to deal with understanding trend data. And because so few data points for each trend can be presented in this fashion, there is also tremendous opportunity for slanting and spinning the meaning.

Writing about Trends. If we chose to write about it, the manageable number may be 3 to 5 metrics. With written trend information, most readers will not get a full sense or understanding of how the situation is evolving. The written word and written number is not an efficient or an effective way of handling life's most important trends. As is the case with talking about trends, writing about trends normally means that very few data points are actually given for any of the important indicators. Sometimes no value is given at all, only some form of imprecise quantification. Sometimes, only a single value is provided. Sometimes only a pair of values or a single value and a comparison to a previous value. [Take a look at today's articles reporting on the DoD quarterly report to Congress for crystal clear examples of what I am talking about here.]

Documenting Trends with Printed Tables. If we decide to put the information in a printed table of data, then the manageable number may grow to 7-10 factors. This is a step forward. However, only a handful of our citizens will be comfortable with getting their trend information in this fashion and making sense of it. And for most of those included in that handful, they will rarely actually have the time to study the table in sufficient detail to actually get a sense of the trends. For those few with the skill, the interest and the substantial time required, this format will begin to allow some understanding of the trends at work. However, for most of the population, the table format will be close to unusable.

A perfect example of the difficulty of both the written word and printed tables is today's Producer Price Indexes report from the Bureau of Labor Statistics.

Static Trend Charts. If we create a set of static trend charts for each selected metric, then the manageable number can easily jump up to 10 or more and if these trend charts follow best practices, most ordinary citizens will be able to make good sense of them and gain an understanding of how the situation is progressing. Of course, if such charts violate the best practices, the utility of this approach drops off dramatically.

Interactive Trend Charts. If we have access to the trend data via a human engineered interactive trend viewing station, then it is possible for ordinary mortals to deal with dozens and dozens of factors and indicators and gain a deeper gestalt understanding of the situation, all in a reasonable amount of time. With really good interactive tools, examining and learning from hundreds of metrics is within reach.

What constitutes a Manageable Number of Metrics? It depends on the approach we take for representing the trend data.

Since the number of factors we musts look at to gain full insight and understanding into the situation on the ground in Iraq is so high, unless we begin to move towards interactive charting, our chances of success look quite slim indeed.

Right now, almost all the official and unofficial reporting of quantified results is weighted heavily towards talking, writing, and printed tables of data. Those who produce static charts are relatively few and even though we can handle maybe as many as 10 well constructed static charts at a time, the norm is to have only 1 or two as is the case with today's WaPo and NYT articles we posted about earlier. All the rest of reporting will be written, and most of the indicators will not have any numeric value associated with them all.

The Sheryl Stolberg article on the surge also included this telling quote:

"Some say measuring progress is simple: you will know it when you see it."

I agree. This is really the answer to Rep. Kingston's request. We'll know it when we see it, but we will only see it if we show it. And we will only show it if we make the effort to move beyond the current default mode of talking about it and writing about it.

For whatever trend data we gather to be useful, we need to make sure that the manageable number of indicators is actually large enough for us to be able to use that data to deal with the problems that confront us. My own estimate of how many this might be for Iraq is at a minimum in the range of 50 t0 100 of the most important factors for general use and discussion with a much higher number available for expert use on specific focused issues.

NY Times charts based on DoD report

Here's the New York Times graphic that accompanied their story this morning on the DoD report to Congress.

Their stacked bar chart is similar to the unstacked bar chart from the Washington post.

Geographic charts are popular these days but I find the pair of charts in this example difficult to interpret. There are some trends going on with respect to readiness to assume security responsibility, but one has to spend quite a bit of time to figure out what they are from these maps. The eye has to dart back and forth between the two maps to make a before and after comparison. If I am reading these correctly, it looks like some serious backsliding has occurred since Nov 2006. And of course, having only a pair of data points for each trend makes any interpretations or extrapolations risky.

The sectarian incident chart at the top right is useful. I would prefer to see a longer time frame and data that is more up to the minute (e.g. at least through the end of May 2007).

The first chart showing the daily casualties also falls short in only reporting data up through May 4th, 2007. While real time reporting is rarely needed in this kind of situation, important factors such as casualties must be reported in near real time for them to be truly useful.

Metric Ideas culled from the news

Here is a sampling of some important metric ideas culled from news stories over the past few days. The news stories themselves may not have actually called out the specific metrics listed. Rather they reported on some aspect that might have bearing to how well or poorly things were going on the ground in Iraq.

For example, I came across an article mentioning the idea that many Iraqi parliament members live abroad and do not physically show up for sessions. That led me to craft the first 4 metrics that might give us some sense at least of Parliament activity even if we cannot easily or directly measure their progress towards oil-revenue sharing, de-Baathification and so on.

1. Number of members of Iraqi parliament currently living aboard
2. Number of hours of parliament sessions per day
3. Number of parliament members attending each day
4. Number of parliament votes each day
5. Number of sectarian death squad killings
6. Number of sectarian attacks
   
7. Number of bodies in the street in cities like Baghdad
8. Number of attacks in al-Anbar province
9. Violence in Diyala Provinces (deaths, injuries, attacks)
   
10. Violence in Ninevah Provinces (deaths, injuries, attacks)
11. Number of suicide bombings
12. Number of suicide attacks
    
13. Number of Baghdad neighborhoods protected by local militias - Shia, Sunni, mixed
14. Number of Baghdad neighborhoods protected by coalition, Iraqi military, Iraqi police
15. Number of Baghdad neighborhoods not protected
16. Percentage of Iraqis in favor of partition of the country by Shia, Sunni, Kurd, other
    
17. Number and % of violent deaths come from people just being shot down
18. Number and % of violent deaths violent deaths from US military activities
19. Number and % of deaths from bombing Iraqi cities (deaths from aerial strikes)
20. US troop level in Baghdad
21. Baghdad morgue data on bodies received per day
22. # of attacks on Sunni Arabs by the predominantly Shiite government security forces
23. number of Iraqis displaced from their homes per month
24. Number of militias currently known
    
25. Percentage of those militias that have been disarmed
    
26. Current militia troop levels
27. Inflation rate
    
28. Oil production rate in Millions of Barrels per day
    
29. Degree to which demand for electricity outstrips supply

For some of these, we have already suggested additional metrics based on disaggregation (e.g. by reporting on totals as well as breaking down the results by ethnic groups). Other metrics listed above are also aggregates and could similarly and beneficially be disaggregated for additional insights.

In my view, all these are important and yet even though we have listed 29 indicators, this is likely just scratching the surface.

If we wish to make sense of what's going on in Iraq, we really need to be able to literally SEE the trends in everyone of these. We're struggling right now since this is just not happening.

Measuring Stability and Security In Iraq: June 2007 DoD quarterly report to Congress

Here's the link to the just published Dept of Defense quarterly report to Congress on the situation in Iraq.

It contains many interesting trend charts which we will be discussing in future posts..

Unabated Violence - washingtonpost.com

Here's an important and easy to read graphic from this morning's Washington Post showing the impact of the surge so far on the number of reported violent attacks on civilians, Iraqi security forces and coalition forces.

Unabated Violence - washingtonpost.com

My preference for data like this would be to make the three separate trends individually viewable. The patterns for civilians and Iraqi forces are easy to miss because the the much reduced scale when all data is presented on the same chart.

See the accompanying WaPo article- No Drop in Iraq Violence Seen Since Troop Buildup for further details.

UN report on surge progress

Juan Cole reports today in Informed Comment on the UN Secretary General Ban Ki-moon's quarterly report on Iraq.

Several things to note:

a) substantial use of important metrics with numeric quantification (see snippets below). There is also a sense that some effort was invested in sorting out which factors and indicators were the important ones relative to humanitarian concerns. Contrast this to the lack of precision or quantification with the 18 benchmarks, the See you in September article, etc.

b) much wide perspective and more holistic view on what's important to pay attention to in Iraq (beyond the 18 benchmarks).

33. ... related to a humanitarian response. For every death reported in the news, six family members on average are left without a breadwinner. The rising number of displaced persons is also a cause for concern. UNHCR estimates that displacement has continued at an undiminished pace and over 800,000 Iraqis have been internally displaced since the Samarra mosque bombing in February 2006, while 30,000 to 50,000 flee to neighbouring countries each month.

34. The violence is also having a major impact on Iraqi children and their ability to attend school. ... estimated that 17 per cent of primary school-age children were not attending school in 2005 and 2006. This translates into approximately 765,000 children, of whom 61 per cent were girls, even before the recent escalation in the numbers of refugees and internally displaced persons. Dropout rates are also increasingly outstripping school participation. Only 34 per cent of girls and 43 per cent of boys of secondary school age were attending secondary school in 2005 and 2006. . .

...

46. ... increased threat of indirect fire into the International Zone. ... The International Zone experienced 17 attacks in March, 30 in April and 39 by 22 May alone. Since 19 February, indirect fire attacks have reportedly resulted in the deaths of 26 people in the International Zone. The security situation has been further compounded by the increase in car bombs in the vicinity of entry checkpoints to the International Zone. Armed groups operating in Baghdad have demonstrated their ability to strike at well-protected, strategic targets, such as the suicide bombing inside the Parliament building on 12 April. . .

...

60. Iraq’s political and social fabric continued to come under considerable strain during the reporting period as a result of ongoing political, sectarian and criminal violence. Despite the efforts of the Iraqi and multinational security forces to stem violence, progress was slower than had been hoped when security initiatives were launched at the start of 2007. This has been demonstrated by continued attacks on the civilian population, physical infrastructure and political institutions such as the Council of Representatives. . .

This is a small step in the right direction in my opinion.

Language and the 18 benchmarks

Here's a link containing the language of the 18 "benchmarks" included in the recent Iraq Supplemental Bill HR 2206

Of the 18, most are not readily amenable to quantification. The short list below are 5 of the more quantifiable targets contained in these benchmarks.

More on this in future posts. For now, the supplemental bill appears to fall afoul of the same language difficulties we have been noting in various articles from around the web - namely mushy metrics, imprecise or missing quantification. If this is the document that is going to guide our evaluation of how things are really going in Iraq, we are in deep trouble as it appears to be structured in ways that are likely to defy exactly that process.

3) equitable distribution of hydrocarbon resources

9) Providing three trained and ready Iraqi brigades to support Baghdad operations.

13) Reducing the level of sectarian violence in Iraq and eliminating militia control of local security.

14) Establishing all of the planned joint security stations in neighborhoods across Baghdad.

15) Increasing the number of Iraqi security forces units capable of operating independently.

17) Allocating and spending $10 billion in Iraqi revenues for reconstruction projects, including delivery of essential services, on an equitable basis.

However, these 5 are amenable to quantification and we could begin now (rather than waiting till September) to make these specifics, to measure them regularly, and to report on them in a way that will be easily accessible to all.

When you look at the full text of benchmarks and at the required reporting standards and timing, it's noteworthy that there are no explicit benchmarks for what the United States is required to do in this time frame. Clearly the situation in Iraq will be dependent both on what the Iraqi government does and what the United States does, but this particular bill places all the responsibility on Iraqi shoulders.

GAO May 15 Rebuilding Iraq Report

Here's a link to the recent General Accounting Office Rebuilding Iraq Report - May 15, 1007.

Once chart I found enlightening and eye opening was their oil chart showing history covering the range of 1970 to 2005. [Click for larger image]

I also found the enemy initiated attack chart easily digestible although as I have commented before, I am not a huge fan of stacked bar charts unless there is also an easy way to look at the individual indicators.

Postponing the Moment of Truth

Check out this excellent Postponing the Moment of Truth cartoon from cartoonstock.com.

This is a perfect match to the imprecise approach, the mushy metrics, the startling lack of numerical quantification, and the complete absence of visualization as exemplified by (but certainly not limited to) Stolberg's See You In September report in the New York Times as highlighted in the previous post.

Revisiting "See You In September"

I want to return to the Sheryl Gay Stolberg article

See You in September, Whatever That Means

published in the New York Times on May 13th, 2007. There are many more important clues we can discover hiding the text. I did not have time to unravel them all during the first Charting Progress in Iraq post on this subject.

Following along the lines of the Forecast on Iraq post we'll once again de-construct and highlight some of the language and see what else that reveals. For fun, I have color coded the quoted text as follows.

Orange - vague metrics (e.g. progress, confidence, winning, losing)

Purple - vague quantification (e.g. up, down, better, more)

Green - specific metrics (e.g. sectarian murders, monthly bomb rate). Unfortunately, none of these have any specific numbers associated with them. Some have vague quantification and others have non at all.

• "decisive progress"
• "looking better"
• “Sectarian murders are down”
• "car bombings ... are up"
• "deaths of American soldiers are up"
• "some in Washington are grasping for a more complete and accurate way to quantify progress"
• “No one knows how to define progress in such a mixed-up situation”
• “We’re having trouble measuring it."
• [Iraq Index includes] "the monthly car-bomb rate"
• [Iraq Index includes] "how many foreign nationals are kidnapped"
• [Iraq Index includes] "how many Iraqis have electricity"
• [Iraq Index includes] "how many Iraqis have ... Internet access"
• [the Iraq index] "is long on numbers and short on analysis"
• "some signs of hope"
• "more grounds for worry than for confidence"
• [Rep Kingston (R-Georgia) wants to] "winnow down the indicators to a manageable number -- say, fewer than a dozen"
• [Rep Kingston (R-Georgia) wants to create] "a standard bi-partisan metric"
• [with such a metric in place] "then you could say who’s winning and losing”
• [Micheal O'Hanlon says] "metrics are grist for a fact based debate but history shows it is dangerous to rely on too few of them"
• [Micheal O'Hanlon says] "metrics were used in Vietnam and we had the wrong ones"
• [the Vietnam metrics] "did net harm to the debate"
• [Micheal O'Hanlon says] "we can’t be exactly precise about which indicators are the conclusive ones”
• "such an index would be politically unpalatable to the White House, which does not want to back itself into a corner by agreeing to someone else’s standard for progress"
• "the White House says the only progress report that counts is the one from Gen. David H. Petraeus, the new top commander in Iraq, and Ryan Crocker, the new ambassador"
• "is the current strategy for waging war going well or not?"
• [Stephen Biddle] did fault the White House for not being more open with the public about its own idea of what constitutes progress
• “By being unbelievably vague about everything,” he said, “they’re making it very hard for congressmen and senators to go to their constituents and say, ‘Look, here’s why things are going better than you might imagine.’ ”
• "Some say measuring progress is simple: you will know it when you see it."
• [Metrics suggested by Senator Robert Bennett (R-Utah)] I want to see life starting to come back,” “I want to see people in markets. I want to see couples strolling down the street, folks sitting at outdoor cafes.”
• [Metrics suggested by Senator Susan Collins (R-Maine) ] “a significant reduction in violence and attacks accompanied by a transfer of more and more authority to the Iraqi forces.”
• [Senator Collins] “the difficult question is going to be if the analysis is mixed, and I suspect it may well be.
• [Senator Lamar Alexander (R-Tennessee) says] “I don’t see any way for us to maintain a long-term presence in Iraq without more bipartisan support.”
• [Rep. Kingston] "hopes to come up with some useful way of figuring out ...whether [to] remain supportive for a little while longer"
  
• [Rep. Kingston has] " heard three years of nearly happy-talk in testimony"
• “We always seem to be about to be around this elusive corner, but we never get there.”

Bottom line:
The language we use to talk about the important aspects of our lives makes a difference. When we use language about an evolving situation this way (vague metrics coupled with imprecise quantification, and lack of any clear trend quantification and visualization), we are going to have a tough time making sense of what's actually happening, what it means, and what our best next step might be.

If we continue along this path of mushy metrics, vague quantification, and numerous instances of what appears to be learned helplessness, come September, we won't be in any better state to evaluate the situation than we are now

I have some more thoughts on some of the key quotes from above for a future post. There is a way forward that addresses Michael O'Hanlon's concerns and answers Rep. Kingston's call for assistance.

Old problems plague new security plan for Iraq

Here's an interesting post from Fort Wayne's News Sentinel:

KRT Wire 06/08/2007 Old problems plague new security plan for Iraq

The much more precise and quantified language describing the situation on the ground in Iraq in this Editorial by Nancy A. Youssef of McClatchy Newspapers stands in stark contrast to the fuzzy, imprecise, unquantified example we gave in the recent Washington Post Forecast on Iraq story.

Best to check these two articles out for yourself to see what I mean.

In the meantime, here are some actual numeric indicators from the article that will give you a flavor of what I am talking abut.

Number of bodies found on Baghdad's Streets
Troop levels numbers in Baghdad for US, Iraqi forces & Police
Number of neighborhoods under control
People killed in explosion
Number of car bombs

Having actual key measurable indicators mentioned by name and having actual data values for how these have changed over time is strikingly more informative and useful.

Is it good enough? Not nearly. It's a perfect example one interpretation of Von Clauswitz' "the better is the enemy of the good", but that's a subject for another post.

Hat tip to Juan Cole over at Informed Consent.

And kudos to Nancy Youssef for raising the bar.

Applying Trend Analysis Principles with Brilliant Effect

Buying a camera at a fair price is not particularly earth-shaking activity, especially compared to trying to figure out whether our actions in Iraq are helping or hurting.

Nevertheless, here is a simple and striking example of the power of applying the essential principles of trend analysis to the task.

Pricescan.com did a brilliant job on this in my opinion.

The red trend line represents the high price, the blue the average price, and the green the low price. Each tick mark on the X axis represents one week's time and the chart covers the period from 9/11/2006 through 6/4/2007

What did pricescan.com do right

1. they identified the most important factors (in this case the minimum, average, maximum price for the Canon PowerShot AS710 IS camera.
   
2. they collected the raw data each week of the price of this item from a range of sources
3. rather than showing the raw data, they created calculated values of High, Average, Low price for each period
4. They selected a reasonable reporting rate that fit with the data (1 week in this case) and the resulting trend lines are relatively smooth while still giving us enough data points to have confidence in our understanding of the trends we see developing. A total of approximately 38 data points are plotted for each factor.
5. their Y axis scaling fits the data and makes it possible to see the trends in all three indicators that they displayed. The scale values are also easy to read.
6. they are reporting their data in near real time. The most recent data point is only 5 days ago.
   
7. They showed the entire time period ( I am guessing from product introduction right up through last Monday)

And of course, they visualized this data for us rather than telling us about it or showing us a table of 100+ numbers.

Armed with this easy to read chart, anyone interested in purchasing the Canon PowerShot AS710 camera in the next week or so will know whether the offering price is within a reasonable range and can then get on to evaluating the other factors that might impact their buying decision.

This example stands in stark contrast to the most approaches to other important areas of our lives as exemplified in yesterday's Iraq Forecast post. There are no mushy metrics or imprecise quantification or flowery metaphors. There's no postponing the moment of truth cop-outs saying we don't know exactly what the future will bring. Of course we don't know exactly. We never do.

They don't say: "we can't really tell you whether the price of this item is going to become more affordable by September".

There's just the relevant facts that help deal with the specific situation.

My two cents: the better we learn how to apply this kind of approach to the most important problems in our lives, the quicker we will begin to gain control of what's happening.

Washington Post. Forecast on Iraq

Today's Washington Post featured this story with quotes from Lt. Gen. Douglas E. Lute, the new White House War coordinator (previously know as War Czar):

Nominee to Coordinate War Offers Grim Forecast on Iraq - washingtonpost.com

Here are some quotes relating to assessment of current trends and the direction we are heading. Bold emphasis has been added.

Lute said:

Iraqi factions have shown so far very little progress ... we're not likely to see much difference in the security situation" a year from now.

A CIA expert on radical Islam was quoted by Senator Bayh saying:

our presence in Iraq is creating more members of al-Qaeda than we are killing in Iraq.

The Post noted that the recently passed supplemental funding legislation

requires progress reports from the administration in July and September.

Here are some trend related quotes.

While insisting that incremental progress is being made, administration officials acknowledge that the benchmarks will be all but impossible to meet by the time progress reports are due to Congress.

"If the test is peace and reconciliation and flowers blooming by September, that will be hard to meet," said a senior administration official.

At the least, the administration hopes to be able to demonstrate movement in the right direction.

There has been little progress in four key political areas

At his hearing yesterday, Lute acknowledged that during a review of U.S. policy on Iraq last winter, he had privately expressed skepticism of Bush's "surge" proposal unless matched with more robust efforts by the Iraqis and other U.S. agencies.

"Where are we today?" he [Lute] asked. "Not where any of us would like."

Notice anything here?

Or rather do you notice anything missing here?

There's nary a single number and all the key factors and indicators referenced with regard to progress are not the kind that can normally be quantified. There's also no sense of any of these indicators changing step by step over time.

Let's list them so you can see what I mean.

Here are the metrics discussed in the article

1. "progress"
2. "the security situation"
3. number of members of al-Qaeda created
4. number of members of al-Qaeda killed in Iraq
5. "peace"
6. "reconciliation"
7. "flowers blooming"
8. "movement in the right direction"
9. more robust efforts by the Iraqis
10. more robust efforts by other US agencies
11. progress in four key political areas"
    

• Revisit constitution to better balance power sharing
  
• Oil revenue sharing
• provincial elections
• de-Baathifcation

And equally challenging to actually being able to use these trends to help steer our course forward is the mushy nature of the current "values" of these important indicators:

1. very little
2. not much difference
3. incremental
4. hard to meet
5. movement
6. little
7. more

Contrast the lack of quantification and therefore the lack of clarity with the numerical metrics we discussed in our series of posts yesterday such as:

• peak megawatt hours
• millions of barrels of oil per day
• Iraqi prison population
  

The one bit of good news mentioned is that a progress report from the administration is due in July so at least we won't have to wait for September for that. How much we learn in July will depend an awful lot on whether the progress is report is based on hard metrics with clear quantification or on the kind of mushy, mostly unquantifiable indicators we find the WaPo report.

What direction are we moving?

From Chart of the Day's - www.chartoftheday.com: "Quote of the Day

The greatest thing in this world is not so much where we are, but in what direction we are moving.' - Oliver Wendell Holmes"

And the best way to have a good sense of what direction we are moving is to examine (in near real time and with high resolution) the trend data for the most important factors and indicators at work in whatever situation we find ourselves.

What could be simpler than that?

Why Wait Till September in Iraq

We Don't Have To Wait. There's a dangerous meme circulating that says we have to wait till September before we can learn whether the "surge" in Iraq is working. This, in my opinion is simply not true.

Many dimensions of the current situation in Iraq can and are being measured and reported on in near real time. The work of Michael O'Hanlon on the weekly Iraq Index at the Brookings Institution is one excellent proof that it is possible to know were we stand literally week by week and even day by day.

Digestible Trend Data. One principal area where we are not as good as we need to be today is in making sure that the trend data that exists is readily available in a form that is digestible by the intended audience. This intended audience includes the Congress, the main stream news media, the blogosphere, and especially ordinary citizens. Some really important data such as the example of Iraqi prison population that we showed in a post earlier today may only appear in tables. My guess is that most people are not willing or able to get what they need directly from the Iraq Index or similar reports. Those who are willing and able may not have nearly enough time. We can and must make it easier so that more citizens can be included in the process.

Assembling All Key Factors in One Place. A second principal impediment is that that the data is dispersed among many sources. We need to pull it all together so that each of us then becomes able to make a reasoned evaluation of how we are doing right now (without waiting till September) . My efforts at assembling some key trend graphs in a series of posts today and in the summary zip file is a small demonstration of what is possible.

Identifying and Acquiring Any Missing but Still Important Factors. A third thing holding us back from being able to evaluate for ourselves how things are going right now is that some key factors are not being reported in easily findable places.

Twenty First Century Visualization Tools. The fourth factor limiting our possibilities today is that most trend data is presented statically. This severely limits the number of factors that an ordinary person is going to be willing and able to look at to only a handful. On the other hand, with a modern Trend Visualization Viewing Station on the order of what's already available for stock market data from the likes of Big Charts or Prophet Net means that ordinary folk can look at and understand dozens and dozens of factors in a very short space of time.

We Don't Have to Wait. All these difficulties that are blocking us can be remedied without waiting till September. Our failure to remedy them over the past four years meant that it took that long for it to sink in that what we were doing wasn't really working. It's easy to posture that everything is ok when there is no hard data to prove you wrong.

In fact, if we don't begin to remedy them now, we will be in the same boat come September. We will find that we still do not have our arms around the entire situation. The facts we need will still not have been assembled and reported in an easily digestible way. And it will still be way too easy to claim that things are going well, or that we still don't know enough to tell.

Anyone interested in helping blast our way through these obstacles to understanding where we are in near real time, please let me know so we can join forces and see if we can make a difference.

The Data is Out There. Let's go get it and assemble it and make it easy to use and then share it with all interested parties.

Full List of Today's Iraq Trend Charts Contained in Composite Zip File

Here's a screen shot showing the list of JPG and GIF files contained in the composite zip file showing all the Iraq trends that we posted about today.

The first five are from the Inspector General's report (001 to 005).

The next five are from the State Department report. (101-105)

The Deja Vu (#200) slide is from the State Dept Dec 204 report via the Matthew Yglesias post .

slides 201 -209 are from the Brookings report.



slides 301-305 were generated with TLViz using tabular data from the Brookings report.

You can find the composite zip file here.

Slide Show Viewing of Today's Trends

We've covered quite a bit of ground today by reviewing several important documents that have included graphical representations of the trends at work in Iraq today.

We looked at:

1. The Special Inspector General for Iraq Reconstruction's April 2007 Report to Congress
   
2. The Dept. of State's Iraq Weekly Status Report for May 30th, 2007
3. A Matthew Yglesias blog post on Electricity goals from Dept of State's December 2004 Iraq Weekly Status and compared it to the most recent data on the same electricity factor from the most recent report.
   
4. The most recent Iraq Index from Michael O'Hanlon at Brookings Institution
5. Additional troop strength charts created from Tabular data in the Iraq Index report

One of the things I have noticed about blogspot blogging is that while it is easy to upload graphics and embed them in a blog post, it is difficult/impossible to retain useful control over size or placement. The maximum blog spot size usually doesn't do justice to the trend charts that I wish to present. That means that to look at a whole series of slides in a single one of my posts, the reader has to click on each one, and then click to close the window and return to the post. This is time consuming and not particularly pleasant in my mind.

down the road, I will be exploring what other blog posting capabilities are available that will let me present the data laid out the way I want it. In the meantime, I have assembled all 26 jpg and gif charts into a single zip file that I posted here in my data download area.

Once you unzip to a directory of your choice, you can enjoy a full size slide show with your favorite slide viewer - for example, Microsoft Office Picture Manager or IRFANVIEW.

I'd like to hear back whether you think this represents a useful, time saving approach for you. I am interested in learning whether you find having all the trend graphs assembled in one place makes reading them and understanding them more rapid than when looking at the individual reports. And finally, I would enjoy discovering how much value you think is added by combining charts from a wide range of resources and pulling them together in one place.

Turning the Tables

Some important data in the Brookings' Iraq Index only appears in tables (e.g. the prison population data we mentioned in the previous post). Troop strength for US forces and for other Coalition forces may be interest now because of the "surge" in in US Forces that is now reaching its stated goals. However, the troop strength trend data in the most recent report is tabular.

So we turned the tables and transformed it into a small CSV file and created some new charts (see below) using the TLViz interactive trend visualization tool.

If you would like to take a look at the CSV file yourself, you can download it from Troop Strength Table. This table includes three original trends from the Brookings' table (US Troop Strength, Other Coalition Troop Strength, and Total Coalition Force Troop Strength). To that, we have added two calculated trends. The first shows Other Coalition Troop Strength as percentage of Total Troop Strength. The second shows the ratio of US Troop Strength to a nominal value of 135,000.

If you haven't already tried out the TLViz tool that runs on Microsoft Windows as a way for looking at two dimensional tables of trend data, here's another chance. Get started by downloading the base kit. Once you have installed the base kit, you can grab the latest version of TLViz.

If you want to learn more about TLViz, check out some of our earlier posts on the subject starting in chronological order with the ones from early January.


This first chart shows a sharp downtrend in other Coalition forces dropping in half from 25,000 to 12,000 since January 2005.

This chart shows US Troop strength from the beginning of the war. While the uptick since January is sharp, the current level of US troops is not far from the middle of the range it has been operating in for the past 3 years.

This is a chart using the calculated value showing the Other Coalition forces as a percentage of total troop strength. There has been a sharp decline over the past 3 years from a maximum of 17 percent down to the current value of approximately 7.5%.

This is another calculated chart showing how US troop strength has fluctuated around the 135,000 mark for the past few years. The current peak is lower than the two previous peaks.

The final chart shows total coalition forces in Iraq over the past 4 years. The current value of 160,000 looks like it is right in the middle of the operational range for this factor since the beginning of the war.