Are we there yet?

How well does the current set of Iraq trend charts & tables serve our needs?

Answer: I think we are a little better off than we were 10 days ago when I began this series of posts on Iraq, but we certainly are not where we need to be. We can look at the charts that are currently available from multiple sources much more quickly now than we could before by skimming these two pdf files: Combine-Iraq-Trends and vizualizing-trends-ohanlon-testimony.pdf. But each of these documents has it's own and weaknesses as discussed below. And there are a growing number of important missing metrics that as yet have no publicly available trend chart or trend table.

What are the weaknesses of the Combine-Iraq-Trends document?

Answer: This combined report consisting of trend charts drawn from 4 different sources covers a lot of important ground and shares the trend behavior of many factors that appear quite important. On the downside, the big weaknesses of this set of trend charts are numerous:

1. There is a wide variation in formats - This reduces overall understandability and slows down the process of viewing the entire set of charts. Each chart must be examined individually with careful checks to make sure that the range of the X and Y axes are are clear and to locate the legend information that tells you what kind of factor you are looking at. This is in clear violation of the Edward Tufte's idea of small multiples that we discussed in a previous post. Applying small multiple thinking to a set of trend charts means that for the viewer, once understanding is reached regarding how a single chart in a series is laid out and what graphing conventions have been used, succeeding charts can be grasped with much reduced time and effort.
   
2. Time frames of the individual charts are widely different. Some cover the period from the start of the War in 2003 right up to the present moment. Others, cover much shorter periods. For example, the IED chart covers only from January 2006-June 2006. Some charts show monthly data, others show weekly data. This is another violation of the small multiple idea and the consequence for the viewer is that it is more difficult and more time-consuming to make any comparisons between different factors.
   
3. Stale Data: Violation of the "Near Real Time" Principle In many charts such as the IED chart mentioned above, the most recent sample is many, many months ago. If something is rated as an important factor, then it won't do us any good unless we measure and report on it often and that we make the most recent sample data available as quickly as possible. A 7 month old data point of a key trend factor can substantially frustrate our ability to understand what's going on, to determine whether interventions we have taken are working, or to decide what to do next.
4. Overly short time frames. Charts such as the State Dept's Crude Oil Production chart cover an amazingly short time frame - in this case from October 30th, 2006 to January 7th, 2007. While this chart is both interesting, useful, and clear, the viewer is seriously short-changed by not being able to look at what has happened to this key trend factor for the entire period since the war began.
   
5. Unavailability of the Underlying Data. In many cases with these charts, the underlying trend data is unavailable for further review or analysis. This means that the viewer of the chart is sharply impeded from discovering different angles to examine the data (e.g. by combining several factors together in a single chart, or calculating new previously invisible factors as a combination of an original set of factors.) If the original analyst who selected the chart for presentation did not create one chart for each important view, then without the data, these new views are simply unattainable. And of course, it's never possible for the original analyst to provide all the views.
   
6. Data that is provided is Not Readily Reusable (RR). In the relatively small number of cases where the raw trend data is provided, it is painful and time-consuming to transform these data into an RR format. This slows down the effort of understanding and because of the time involved, only a small percentage of potentially interested parties ever actually make these transformations.
7. Missing Metrics, Quality of Metrics. Many of these sources mention other metrics specifically (presumably because they are considered important) but do not provide trend tables or charts. And all metrics, those that are charted and those that are not, must be evaluated against the quality criteria outlined by Anthony Cordesman as noted in

The Quarterly Report on “Measuring Stability and Security in Iraq:” Fact, Fallacy, and an Overall Grade of “F”

What are the weaknesses of the Combine-Iraq-Trends document?

Answer:

1. Stale Data: Violation of the "Near Real Time" Principle - the most recent data point in all of these charts is November 2006 so the trend series displayed is not as up to date (on January 10th when this data was presented) as one might wish.
   
2. Overly long time frame between samples. Way too few samples. Having a total four samples, one year apart creates a trend table that is quite readable on an 8 by 11 sheet of paper. The price is dramatic loss of vital detail and can lead to seriously misleading conclusions.
3. Data that was originally provided was Not Readily Reusable (RR). We were able to work our way around this barrier and create trend charts from the data, but it was costly in time and will serve as an impediment to further analysis of the data for most people most of the time.
4. Missing Metrics, As we noted in an earlier post, there were many Iraq trend factors that Michael O'Hanlon considered important enough to include is his verbal testimony and in the logic of thinking through what those factors meant, but which did not appear in the attached table of 30 factors. Our view is that if a metric is important enough to mention and use for logical argument, it is imperative that the details of how that factor varies over time are provided for further review and analysis by interested parties.
   
5. Quality of Metrics. As for the case of evaluating the Combine-Iraq-Trends document, all metrics, those that are in the table (and therefore charted) and those that are in the text must be evaluated against the quality criteria outlined by Anthony Cordesman as noted in:

   The Quarterly Report on “Measuring Stability and Security in Iraq:” Fact, Fallacy, and an Overall Grade of “F”How
   

What next steps could help things along and help us move closer to our goal of helping both ordinary citizens and decision makers gain a better understanding of the trends at work on the ground in Iraq?

Answer: We've made some progress, but in many ways we have only just begun. Below, I outline some steps I am planning to take with the goal of taking this up to the next level. If you can assist in any way, I would be pleased to hear from you.

1. Continue to harvest and grow our list of missing metrics and invisible indicators, for example by close examination of Dept of Defense quarterly report or the many writings of Anthony Cordesman on the subject.
2. Create a single composite list of all the factors for which we do not yet have trend data or trend charts but which we think would further aid our understanding if they were to be made available.
3. Make inquiries to the original sources to see if the raw trend data for the key factors that have been identified can be made available for further use, ideally in a readily-reusable form to begin with.
   

Tapping the Power of Iraq Readily Reusable Data

In the previous post, we showed how the tabular data from Michael O'Hanlon's testimony to the Senate Foreign Relations Committee could be converted to a Readily Reusable (RR) format as a CSV file ( ohanlon-key-factors.csv ) and how the RR format then made it relatively straightforward and inexpensive to actually look at the visualization of all thirty of the reported trends, one by one as shown in vizualizing-trends-ohanlon-testimony.pdf.

We have found that when using RR data and TLViz, the time savings we achieve make it possible to look at hundreds of trends, one after the other, in a very short time so as to gain a gestalt sense of all the reported factors at work. Typically, moving from tabular form to RR form gives a productivity saving factor of at least 10 to 1. Frequently, the productivity increase is 50 to 1 or more.

After having reviewed each of the 30 reported trend factors from Michael O'Hanlon's testimony, it became apparent that there were some other interesting trends hiding amongst the original data that could be computed with simple calculations from the original data. This is readily achievable by opening the RR csv file with a tool such as Microsoft Excel.

For example, with Excel we could create a new column of data and combine US troop strength and troop strength of other non-US coalition forces and then calculate the percentage of coalitions forces that were non-US. Similarly, we could use the original data to calculate the percentage of US troops that had been killed in that month by IEDs. We could also take factors such as the cumulative number of refugees and convert them into year over year trend data.

Below, we show you the three new trend charts that bring some previously invisible trends into the light of day where we can all see them. (please click on trend graphic for full size image.)




Bottom line. Once the trend data is in RR format, all sorts of new possibilities of understanding open up that can help us better understand what is really going on.

Visualizing the O'Hanlon Trend Data

Once we had the the data from Michael O'Hanlon's Senate Foreign Relations Committee Testimony (Jan 10, 2007) in readily reusable (RR) form, we opened that csv file with TLViz and were able to generate a set of graphs in a matter of minutes showing the individual trends in visual form including some that combined several of the metrics together.

You can see a pdf slide show of the entire set of charts we created in full screen size by clicking on: vizualizing-trends-ohanlon-testimony.pdf

Using Edward Tufte's idea of small multiples, we have combined 4 trends at a time into the pictures below (please click for full size) that show a sample of the charts from the full report.

The first 4 show US troops killed by IED, Iraq Civilian Fatalities, Multiple Fatality Bombings, and Estimated Strength of the Insurgency.

The next 4 trends show Estimated Strength of Shia Militias, Daily Average of Inter-ethnic attacks, Estimated Number of Foreign Fighters, and Number of Daily Attacks by Insurgents or Militias.



The next four small multiples show Iraqi Internally Displace, Iraq Refugees, Iraqi Physicians Murdered or Kidnapped, and Iraqi Physicians who have fled Iraq.


In this next set of trends, the first chart shows the total number of Iraqi Physicians who have murdered, kidnapped, or who have left Iraq. This chart was created by combining two of the originally reported trends. The second chart also combines two original trends, this time comparing the number of Iraqi forces who are technically proficient to the number who are politically dependable. The final two charts in the small multiple set show the Percentage of Household Fuel needs that are being met and the average hours per day of electricity in Baghdad.


This final small multiple set shows the Unemployment Rate, the Per Capita GDP, the Number of Trained Judges, and the Number of Telephone Subscribers.


If you click on these charts and then press F11, you should have a full screen readable version to examine. Of course, if you want to look at each chart by itself in full screen size with easy back and forth navigation, the vizualizing-trends-ohanlon-testimony.pdf is the place to go.

One of the advantages of these charts to the previous set of combined charts that we published is that they all cover the same time period and that makes understanding what they mean just that much easier. Once you understand what the X axis timeline means in one chart, you don't have to recompute that with each chart you examine.

Once again, we suggest that you download this RR data yourself and play with it to see what else you can find, what other patterns are hiding in the data. You can download by right clicking on ohanlon-key-factors.csv. We also suggest that this could be a fine opportunity for you to test out the ways in which TLViz simplifies and speeds up trend analysis work once you have RR data. You can learn more about TLViz from our earlier posts this month and you can download a copy of the full kit for Windows PC's from TrendsThatMatter's Download Page

Iraq Trend Data: O'Hanlon's Testimony to the Foreign Relations Committee

Michael O'Hanlon who is the lead author of the Brookings Iraq Index testified last week before the Senate Foreign Relations Committee. Here's the 4 page text of his January 10th testimony.

The last two pages show a relatively easy to read and understand printed table. The table lists, all in one place, what I assume to be what O'Hanlon thinks are the 30 key factors on the ground in Iraq and how they have varied year by year from November 2003 to November 2006.

Here's a version of that table that I have edited for even better visibility and to keep it from covering two pages. (please click on image for full size view):

The four data columns of the table show the trends for those factors with the data values being captured for the month of November in each of the past 4 years. From a standpoint of printed table readability, limiting the number of trend samples to 4 makes this table much more suited for direct human consumption at the cost of reducing overall trend data quality compared to a set of trend samples for a given factor that showed how that factor changes every month.

While most people are not comfortable with reading tables, this one is actually readable and understandable with the application of a modest amount of effort. Please let us know what you think?

This table as text is of course not a readily-reusable (RR) file. However, it was relatively straightforward for us to convert it into RR format and you can download your own copy by right clicking on ohanlon-key-factors.csv. This RR file is ready for examination with Microsoft Excel or even better with the T4 & Friends tools TLViz or CSVPNG.

You will note that the layout of the data has been inverted with the data rows now representing the 4 monthly sample reading time periods, and the 30 data columns representing the 30 key factors which is the orientation required for TLViz and CSVPNG. This makes for a very wide page, but of course, when using TLViz or CSVPNG to examine the trend data, we do not actually need to read it.

In our next post, we will show you why this conversion from printed table to RR format is so helpful in the way it allows the tabular data to come alive in visual form with very little effort by using a tool such as TLViz.

Creating Readily Reusable Data from Online Tables of Trends in Iraq: a quick example

Here's a quick example of how relatively inaccessible tabular data from a web site can be converted into a readily reusable format.

Starting Point: Tabular data from GlobalSecurity

Step 1. We went to Global Security's Iraq Casualties page and found a table that included one column of monthly data for US Named Dead and second column for US Wounded. The bottom of that page also includes one chart for each of these factors that are worth examining. Note that these charts were not included in the ones we selected from Global Security in our earlier post as we already had charts on US dead and wounded that we had selected from the Iraq Casualty Coalition that we felt were more revealing. We cut and pasted the table from Global Security into Microsoft Excel and played with the data to create two clean columns of CSV data and a time stamp column.

Step 2. We went to Global Security's Boots on the Ground page and cut and pasted the two columns of data showing troops in Iraq and and troops "in theatre" to create a CSV data file which now had 4 data columns.

Step 3. We now had the data in a readily-reusable (RR) form as 4 data column CSV file. We took advantage of this to compute some new data columns derived from the original four columns, namely:

1. the monthly ratio of wounded to dead
2. the monthly change in troop level in Iraq
3. the monthly change in troop level in Iraq as a percent of the previous month's level
4. the three month trailing moving average of fatalities
5. the six month trailing moving average of fatalities
6. the 9 month trailing moving average of fatalities
7. the monthly total of wounded and dead combined

The payoff:

You can take a look at the resulting RR file we have created which now has 11 data columns by downloading it from Iraq-RR-Example.csv. Check it out with Microsoft Excel, or even better this would be a great opportunity to try out TLViz (the TimeLine Visualizer) or the CSVPNG utility - both of which are expressly designed to work with RR data in this format.

You can learn more about these powerful tools in our previous posts: here and here and here and here for TLViz and here for CSVPNG and you can download the latest versions of these powerful utilities from the TrendsThatMatter Download page.

The first payoff of an RR approach is that when tabular data is converted to RR format, it puts you back in the driver seat. You are not dependent on only seeing the charts that some other analyst selected. You can look at every factor. You are not restricted to just the time period that someone else selected. You can zoom in on a period you are most interested in.

A second payoff is that you are not restricted to looking at only the combinations of data (if any) that someone else selected. You can also combine factors together with multiple trends on a single graph. Interactive tools like TLViz make it an order of magnitude easier to create different combinations of key factors and help reveal patterns hiding in the data.

A thrid payoff is that you are not restricted to only looking at the raw data. You can add moving averages and look at the moving average by itself or create a trend graph that shows both the initial raw data and the moving average together as shown in the example below:


A fourth payoff is that you can now create new computed or derived values such as the 7 new metrics we added on to our starting point of four Global Security factors. This facility to compute or derive new trend data by calculation involving the current set of factors is extremely important at revealing the full value that the data might hold.

The final payoff is that as you gather data from other sources, you can turn this into RR format and then add new columns to extend the your existing RR file.

The example we showed in this post is pretty elementary, starting as it does with only a handful of factors. As we have seen from our work with the Iraq war data so far, there are literally dozens of important factors. If we could put them all into a single RR data set, our ability to understand what is going on would be markedly enhanced and many opportunities for discovery would open up.

We are still a long way from that goal. In our next post, we will give you a little more complex example of RR data for Iraq that involves substantially more of the relevant factors.

In the meantime, I suggest you download and take a look at the Iraq-RR-Example.csv and see what else you can discover from the 11 reported factors that we have assembled for you.