Full List of Today's Iraq Trend Charts Contained in Composite Zip File

Here's a screen shot showing the list of JPG and GIF files contained in the composite zip file showing all the Iraq trends that we posted about today.

The first five are from the Inspector General's report (001 to 005).

The next five are from the State Department report. (101-105)

The Deja Vu (#200) slide is from the State Dept Dec 204 report via the Matthew Yglesias post .

slides 201 -209 are from the Brookings report.



slides 301-305 were generated with TLViz using tabular data from the Brookings report.

You can find the composite zip file here.

Slide Show Viewing of Today's Trends

We've covered quite a bit of ground today by reviewing several important documents that have included graphical representations of the trends at work in Iraq today.

We looked at:

1. The Special Inspector General for Iraq Reconstruction's April 2007 Report to Congress
   
2. The Dept. of State's Iraq Weekly Status Report for May 30th, 2007
3. A Matthew Yglesias blog post on Electricity goals from Dept of State's December 2004 Iraq Weekly Status and compared it to the most recent data on the same electricity factor from the most recent report.
   
4. The most recent Iraq Index from Michael O'Hanlon at Brookings Institution
5. Additional troop strength charts created from Tabular data in the Iraq Index report

One of the things I have noticed about blogspot blogging is that while it is easy to upload graphics and embed them in a blog post, it is difficult/impossible to retain useful control over size or placement. The maximum blog spot size usually doesn't do justice to the trend charts that I wish to present. That means that to look at a whole series of slides in a single one of my posts, the reader has to click on each one, and then click to close the window and return to the post. This is time consuming and not particularly pleasant in my mind.

down the road, I will be exploring what other blog posting capabilities are available that will let me present the data laid out the way I want it. In the meantime, I have assembled all 26 jpg and gif charts into a single zip file that I posted here in my data download area.

Once you unzip to a directory of your choice, you can enjoy a full size slide show with your favorite slide viewer - for example, Microsoft Office Picture Manager or IRFANVIEW.

I'd like to hear back whether you think this represents a useful, time saving approach for you. I am interested in learning whether you find having all the trend graphs assembled in one place makes reading them and understanding them more rapid than when looking at the individual reports. And finally, I would enjoy discovering how much value you think is added by combining charts from a wide range of resources and pulling them together in one place.

Looking behind the headline number.

Here's a good example of the power that acrues the minute that we begin to look behind the headline number and see what other factors might bear on the meaning of what we are seeing.

The Big Picture Watching Trading Volume Fade . . .

In this case, insteaed of just looking at the S&P 500, Barry Ritholtz points us to a Michael Kahn Barron's article where he takes a look at the volume as well and discovers some interesting patterns.

What other factors might be worth adding into the mix and giving us even more depth of understanding of the basic workings of this important headline financial indicator?

For example, it would be interesting to see what percentage of the S & P volume was NEW short sales and what percentage of the volulme was Covering Purchases for previous short sales.

It might also be worthwhile to look at the average number of shares per trade which might give a clue into what percentage of the volulme was due to large financial instituions and what part was coming from the "little guy."

Oil Production: Iraq Weekly Status Report - March 21, 2007

Here's the latest Iraq Oil production chart from the Iraq Weekly Status Report, March 21, 2007
showing production at or close to the Oil Ministry Goal for 2007 and holding quite steady .for the past 7 weeks after a slowdown period at the beginning of January.

To completely understand the meaning of current production and this year's goal, it would be helpful if the full range of production data since the beginning of the war were also available in addition to the zoomed in most recent view that is presented.

With that in mind, I assembled a mini time series history (each about 3 months apart) of this chart as it appears in weekly reports stretching back to 2005. Note that the next to the last chart overlaps the charts before and after to more clearly show the January 2007 dip.

Ideally, all this data would be on a single chart and much more readily digestible.

More BBC trend charts for Iraq

BBC NEWS | In Depth | Iraq violence, in figures

The BBC link above is the source for these two easy to read charts -- the first showing the varying levels of US troop strength and the second showing the US Military Death Toll.

Good points of the BBC presentation:

1) Full range of Data. Both charts go back to the beginning of the conflict in 2003 and are up to date as of Feb 2007.

2) Combining Data from Multiple Sources. The BBC has combined data sources obtaining the troop strength information from the Brookings Institution and fatality figures from the Iraq Coalition Casualty Count. The charts are presented on the page so the single Time Axis can be used for both for more ready comparison.

3) Detailed Interactive Event Description Integrated with the Trend Data.
The BBC original interactive chart has the option to click on the event arrows to get a brief snapshot of some of the important events and see how these fit into the overall picture and relate to the underlying trends.

Combined Listing of Iraq Factors

Here's the combined list of 142 factors we have identified so far based on this series of recent posts. I have left off the links to the various sources. You can find them these links in the previous posts.

Which ones do you think are most vital to pay attention to on a regular, near-real-time basis in order to better understand what's happening on the ground?
How do you rank each on a scale of 1 to 10?
How often do you think each factor must be measured and reported?
Are there any ones you think can be dropped?

From the Brookings Iraq Index Report

1. Multiple Fatality Bombings – May 03 – Jan 07
2. Killed and Wounded in Multiple Fatality Bombings – May 03 – Jan 07
3. Average Weekly Attacks by Time Period – 01 Jan 2004 – 11 Aug 2005
4. Improvised Explosive Devices (IED’s) Detonated and Disarmed - Jan 2006 –June 2006
5. US Troops Wounded in Action March - March 2003 – Jan 2007
6. Iraqi Military and Police Killed Monthly – Apr 03 – Jan 07
7. Iraqi Deaths from All Violet Causes – Jan 06- Oct 06
8. Number of Daily Attacks by Insurgents and Militias - June 03 – Nov 06
9. Enemy-Initiated Attacks Against the Coalition and its Partners May 2003 –Aug 2006

U.S Dept. of State's Iraq Weekly Status Report (30 pages, January 10, 2007).

1. Crude oil production for the past 10 weeks.
2. Crude oil export since February 2006
3. Electricity production compared to electricity demand since January 2004.
4. The strength of the local Iraqi currency compared to the dollar
   

GlobalSecurity.org

1. US Ground troops in Iraq from March 2004 through December 2006
2. Average daily casualties - Iraqi and Coalition from 01 April 2004 through 12 May 2006
3. Average weekly attacks by time period 01 April 2004 through 12 May 2006

5 trend graphs drawn from Iraq Coalition Casualties

1. US Soldiers Killed with moving avg & event timing - March 2003 - December 30, 2006
2. US Soldiers Wounded with moving avg - Sept 6, 2003 - Dec 29, 2006
3. IED Fatalities per month - July 2003 - January 2007
4. Iraqi Police/Military fatalities - January 2005 - December 2006
5. Iraqi Civilian fatalities - January 2005 - December 2006

Composite Readily Reusable (RR) example based on Tabular data from GlobalSecurity.org

1. the monthly ratio of wounded to dead
2. the monthly change in troop level in Iraq
3. the monthly change in troop level in Iraq as a percent of the previous month's level
4. the three month trailing moving average of fatalities
5. the six month trailing moving average of fatalities
6. the 9 month trailing moving average of fatalities
7. the monthly total of wounded and dead combined

from Michael O'Hanlon's testimony. 30 factors

1. US Troops in Iraq (thousands)
2. Other Foreign Troops in Iraq (thousands)
3. US Troops Killed
4. US Troops Killed by IEDs
5. US Troops Wounded
6. Iraqi Army/Police Fatalities
7. Iraqi Civilian Fatalities
8. Multiple Fatality Bombings (for the month in question)
9. Estimated Strength of Insurgency
10. Estimated Strength of Shia Militias
11. Daily Average of Inter-ethnic Attacks
12. Estimated Number of Foreign Fighters
13. Number of Daily Attacks by Insurgents/Militias
14. Attacks on Oil/Gas Assets
15. Iraqi Internally Displaced Since April 2003
16. Iraqi Refugees Since April 2003 (Total)
17. Iraqi Physicians Murdered or Kidnapped
18. Iraqi Physicians who have Fled Iraq
19. Iraqi Security Forces who are Technically Proficient
20. Iraqi Security Forces who are Politically Dependable
21. Oil Production (Millions of Barrels/Day prewar 2.5)
22. Percent of Household Fuel Needs Available
23. Electricity Production (in megawatts prewar 4000)
24. Avg Hours/Day of Power Baghdad (prewar 20)
25. Unemployment Rate (%)
26. Per Capita GDP (Real Dollars prewar $900)
27. Number of Trained Judges
28. Telephone Subscribers (prewar 800000)
29. Independent Media Companies (prewar 0)
30. Iraqi Optimism (% who think things going in right direction)

New indicators calculated from combinations of O'Hanlon's metrics

1. total number of Iraqi Physicians who have murdered, kidnapped, or who have left Iraq.
2. percent of coalition troops that are not US troops
3. Percent of US troops killed by IEDs
4. Year over year increase in Iraqi refugees

Additional factors (missing metrics) contained in the text of O'Hanlon's testimony

1. Good year/Bad Year (e.g. subjective on scale of 1 to 10)
   
2. Sense of Progress (e.g. subjective on scale of 1 to 10)
3. Al-Maliki Favorability Rating
4. Al-Maliki Favorability Rating for all Sunni Arabs
5. Rate of Shia attacks on Sunnis
6. Rate of Sunni attacks on Shia
7. Iraqi state subsidies for consumer goods
8. The number of children being immunized.
9. Percentage who feel safe en route to school
10. Rate of private sector investment
11. Slide toward chaos (e.g. subjective on scale of 1 to 10)
12. Percent who think Economy is Poor
13. Percent who think security environment is poor
14. Rate of Jobs Creation
15. Degree of Iraqi Leader consensus on sharing oil
16. American support for the operation
17. Number of coalition forces needed to provide security.
18. Degree of Overwork of our Soldiers and Marines
19. Rate of “Success of military commanders in putting Iraqis to work with their commander emergency response program funds”
20. Degree of Support Iraqi Leaders enjoy outside their own sectarian group.
21. Degree to which Iraqi Leaders are working across sectarian lines.
22. The Probability that a Given Intervention will succeed (e.g the most recent set of changes in strategy and tactics)
    
23. Degree of Pessimism
    
24. Iraqis displaced per month
25. Degree of Ethnic Cleansing

55 missing metrics/invisible indicators/faceless factors from Brookings Iraq Index

1. Percentage of US troop fatalities caused by car bombs (page 3 table)
2. Iraqi civilians killed by US troops (page 9 table)
3. Police deaths in Baghdad (page 12 table)
4. Civilian deaths in Baghdad (page 12 table)
5. Kellogg, Brown and Root Employees in Iraq (table page 14)
6. Combat arms troop strength (table page 14)
7. Combat support troop strength (table page 14)
8. Combat service support troop strength (table page 14)
9. Dept of Defense Civilians in Iraq (table page 14)
10. Total Civilian Personnel in Iraq & Kuwait (table page 14)
11. Iraqis Kidnapped Nationwide (table page 15)
12. Iraqis Kidnapped in Baghdad (table page 15)
13. Iraqi prison population in US Custody (table page 16)
14. Iraqi prison population in Iraqi Custody (table page 16)
15. Active duty troop strength in Iraq (table page 20)
16. Reserve & National Guard troop strength in Iraq (table page 20)
17. Total US Army authorized strength - Activity duty (table page 20)
18. Total US Army authorized strength - National Guard (table page 20)
19. Total US Army authorized strength - Reserve (table page 20)
20. Total US Army authorized strength - Combined Active, National Guard & Reserve (table page 20)
21. Percentage of IEDs found and disarmed (table page 23)
22. Number of daily Iraqi Patrols (chart page 23)
23. Number of daily US Patrols (chart page 23)
24. Number of daily Joint Iraqi-US Patrols (chart page 23)
25. Number of total daily patrols (chart page 23)
26. New passports issued (table page 24)
27. Percent of professional class that has left since 2003 (table page 24)
28. Iraqi refugees heading to Syria per day (table page 24)
29. Iraqi refugees heading to Jordan per day (table page 24)
30. US Air Strikes (Iraq only) (table page 26)
31. C-130 Sorties flown (Iraq and Afghanistan combined) (table page 26)
32. Diesel production (table page 29)
33. Kerosene production (table page 29)
34. Gasoline/Benzene production (table page 29)
35. LPG production (table page 29)
36. Non oil related GDP (table page 33)
37. Total cost of US operations (table page 37)
38. Cost of US defense activities (table page 37)
39. Cost of Veteran's programs (table page 37)
40. Inflation rate (table page 39)
41. Cost of imported gasoline (table page 39)
42. Domestic price of gasoline (table page 39)
43. Relative amount of car traffic (table page 40)
44. Typical waiting time for gasoline (table page 40)
45. Number of registered cars (table page 40)
46. Internet subscribers (table page 41)
47. Tons of mail sent by Iraqis (table page 41)
48. People with potable water (table page 42)
49. People with sewerage system coverage (table page 42)
50. Water treatment capacity (table page 42)
51. Annual Iraqi Medical School Graduates (table page 42)
52. Primary school enrollment percentage (table page 43)
53. High School enrollment percentage (table page 43)
54. Percentage of Iraqis who approve of attacks on US led forces (table page 44)
55. Percentage of Iraqis who strongly oppose presence of Coalition troops (table page 51)

Does reading through this entire list inspire your creativity to identify other missing metrics or other new calculations that might be made by combining this original list?

Are we there yet?

How well does the current set of Iraq trend charts & tables serve our needs?

Answer: I think we are a little better off than we were 10 days ago when I began this series of posts on Iraq, but we certainly are not where we need to be. We can look at the charts that are currently available from multiple sources much more quickly now than we could before by skimming these two pdf files: Combine-Iraq-Trends and vizualizing-trends-ohanlon-testimony.pdf. But each of these documents has it's own and weaknesses as discussed below. And there are a growing number of important missing metrics that as yet have no publicly available trend chart or trend table.

What are the weaknesses of the Combine-Iraq-Trends document?

Answer: This combined report consisting of trend charts drawn from 4 different sources covers a lot of important ground and shares the trend behavior of many factors that appear quite important. On the downside, the big weaknesses of this set of trend charts are numerous:

1. There is a wide variation in formats - This reduces overall understandability and slows down the process of viewing the entire set of charts. Each chart must be examined individually with careful checks to make sure that the range of the X and Y axes are are clear and to locate the legend information that tells you what kind of factor you are looking at. This is in clear violation of the Edward Tufte's idea of small multiples that we discussed in a previous post. Applying small multiple thinking to a set of trend charts means that for the viewer, once understanding is reached regarding how a single chart in a series is laid out and what graphing conventions have been used, succeeding charts can be grasped with much reduced time and effort.
   
2. Time frames of the individual charts are widely different. Some cover the period from the start of the War in 2003 right up to the present moment. Others, cover much shorter periods. For example, the IED chart covers only from January 2006-June 2006. Some charts show monthly data, others show weekly data. This is another violation of the small multiple idea and the consequence for the viewer is that it is more difficult and more time-consuming to make any comparisons between different factors.
   
3. Stale Data: Violation of the "Near Real Time" Principle In many charts such as the IED chart mentioned above, the most recent sample is many, many months ago. If something is rated as an important factor, then it won't do us any good unless we measure and report on it often and that we make the most recent sample data available as quickly as possible. A 7 month old data point of a key trend factor can substantially frustrate our ability to understand what's going on, to determine whether interventions we have taken are working, or to decide what to do next.
4. Overly short time frames. Charts such as the State Dept's Crude Oil Production chart cover an amazingly short time frame - in this case from October 30th, 2006 to January 7th, 2007. While this chart is both interesting, useful, and clear, the viewer is seriously short-changed by not being able to look at what has happened to this key trend factor for the entire period since the war began.
   
5. Unavailability of the Underlying Data. In many cases with these charts, the underlying trend data is unavailable for further review or analysis. This means that the viewer of the chart is sharply impeded from discovering different angles to examine the data (e.g. by combining several factors together in a single chart, or calculating new previously invisible factors as a combination of an original set of factors.) If the original analyst who selected the chart for presentation did not create one chart for each important view, then without the data, these new views are simply unattainable. And of course, it's never possible for the original analyst to provide all the views.
   
6. Data that is provided is Not Readily Reusable (RR). In the relatively small number of cases where the raw trend data is provided, it is painful and time-consuming to transform these data into an RR format. This slows down the effort of understanding and because of the time involved, only a small percentage of potentially interested parties ever actually make these transformations.
7. Missing Metrics, Quality of Metrics. Many of these sources mention other metrics specifically (presumably because they are considered important) but do not provide trend tables or charts. And all metrics, those that are charted and those that are not, must be evaluated against the quality criteria outlined by Anthony Cordesman as noted in

The Quarterly Report on “Measuring Stability and Security in Iraq:” Fact, Fallacy, and an Overall Grade of “F”

What are the weaknesses of the Combine-Iraq-Trends document?

Answer:

1. Stale Data: Violation of the "Near Real Time" Principle - the most recent data point in all of these charts is November 2006 so the trend series displayed is not as up to date (on January 10th when this data was presented) as one might wish.
   
2. Overly long time frame between samples. Way too few samples. Having a total four samples, one year apart creates a trend table that is quite readable on an 8 by 11 sheet of paper. The price is dramatic loss of vital detail and can lead to seriously misleading conclusions.
3. Data that was originally provided was Not Readily Reusable (RR). We were able to work our way around this barrier and create trend charts from the data, but it was costly in time and will serve as an impediment to further analysis of the data for most people most of the time.
4. Missing Metrics, As we noted in an earlier post, there were many Iraq trend factors that Michael O'Hanlon considered important enough to include is his verbal testimony and in the logic of thinking through what those factors meant, but which did not appear in the attached table of 30 factors. Our view is that if a metric is important enough to mention and use for logical argument, it is imperative that the details of how that factor varies over time are provided for further review and analysis by interested parties.
   
5. Quality of Metrics. As for the case of evaluating the Combine-Iraq-Trends document, all metrics, those that are in the table (and therefore charted) and those that are in the text must be evaluated against the quality criteria outlined by Anthony Cordesman as noted in:

   The Quarterly Report on “Measuring Stability and Security in Iraq:” Fact, Fallacy, and an Overall Grade of “F”How
   

What next steps could help things along and help us move closer to our goal of helping both ordinary citizens and decision makers gain a better understanding of the trends at work on the ground in Iraq?

Answer: We've made some progress, but in many ways we have only just begun. Below, I outline some steps I am planning to take with the goal of taking this up to the next level. If you can assist in any way, I would be pleased to hear from you.

1. Continue to harvest and grow our list of missing metrics and invisible indicators, for example by close examination of Dept of Defense quarterly report or the many writings of Anthony Cordesman on the subject.
2. Create a single composite list of all the factors for which we do not yet have trend data or trend charts but which we think would further aid our understanding if they were to be made available.
3. Make inquiries to the original sources to see if the raw trend data for the key factors that have been identified can be made available for further use, ideally in a readily-reusable form to begin with.
   

Consolidating the Iraq Trend Charts from Multiple Sources: A step toward ready-viewability

In the past few days, we've posted a total of 21 charts showing trends that matter on the ground in Iraq. These were culled and selected from 4 separate sources and posted in 4 separate posts. You can scroll down the January 2007 posts from January 17th back to January 13th and find all 21 of these charts in relatively close proximity to each other. Our theory is that by bringing them all to one relatively compact place, we make it easier for those who are interested in learning more about what is happening to have access to the combined analysis and reporting talent from the four sources.

What can we do next? We think this approach makes it easier, but it still is not easy. Due to the current limitations of this blog, the graphs that appear in the blog itself are relatively small in size and to get a full grasp of the message of each trend it is necessary to click on each trend graph to get a full size image. This is pretty unwieldy and involves a lot of extra back and forth clicks to navigate to a good sized view of each trend.

Here's a possible approach to make this a bit easier and less time consuming and require far fewer clicks and allow much better navigation back and forth between the different trend charts. What we've done is to take all 21 trend graphs to create a single down-loadable PDF document that will let you easily view all trends up to the full size your screen while navigating back and forth to the chart you want using the full navigational capabilities of Adobe Acrobat (e.g. page down, page up).

Here's the link - Combine-Iraq-Trends Please let us know what you think of this step towards making these important charts rank higher on a ready-viewability scale.

Did you find the navigation easier?
Did having easy access to all trends in full size improve understanding for you?
What other suggestions do you have to make this even easier?

The Iraq Situation and Readily-Reusable Data

One of the difficulties I have encountered so far is that the data I would really like to examine first hand is not readily reusable and in some cases is completely unavailable in any form.

The first timeline collaboration principle that you will find on the right hand column of our blog template is:

1. Share the data series with the chart. Make sure it is readily reausable

To the best of my knowledge, not one of the 4 sources we have consulted so far (Brookings Iraq Index, The State Dept's Iraq Weekly Status, GlobalSeccurity, or the Iraq Casualty Coalition) has made the data that they used for their own analysis and charting both available and readily reusable as a spreadsheet or CSV file.

Admittedly some of these sources have shared their trend data series, but the method they selected means that they are not easy or ready for reuse without investing a lot of time and effort.

For example,

• Brookings Iraq Index does include some trend data tables with important metrics and frequently labels the data values on their charts. Not all of the interesting trend data that appears in the tables is translated into a chart in the report. For example, see the table on page 5 of the January 8th report that shows details of the trends in cause of death for US Troops.
  
• The Iraq Weekly Status Report from the State Department does not include any trend data at all beyond what they show in their charts. And the time period covered by the charts they have selected is rather short in most cases. For example the crude oil production chart only covers the last 10 weeks and other charts cover less than a year. Only the electricity chart stretches back to the beginning of 2004
• Global Security includes a lot of detailed trend data in tabular form in addition to and complementing the charts they provide. The range of the data presented tabularly tends to cover the entire period from the start of the war until the present.
  
• The Iraq Casualty Coalition also provides substantial amounts of tabular trend data in addition to their charts and their data covers the entire period from the beginning of the war.

Here's the problem.
Most people (both decision makers and ordinary citizens) are not particularly good or particularly thrilled by the idea of trying to make sense of trend data contained in a printed or online table.

My experience indicates that almost all adults (I would estimate the number at 90% or more) can make good sense of trend data presented in line graphs or bar graphs). On the other hand, I estimate that fewer than 10% of the people have the skills, interest or time to extract basic information from a trend table like the one on page 5 of the Brookings Iraq Index, or the many tables you can find at GlobalSecurity or at the Iraq Casualty Coalition.

What impact does this lack -- this lack of data / lack of readily reusable data / lack of easily viewable trend data for all key factors -- have on our ability to understand what's going on?

The way this plays out is that for 99% of the population, they will get their understanding of the trends from their most trusted source.

For a small percentage of people (I am guessing under 5% of the population), that trusted source might be one of the ones we have mentioned so far in this series on Iraq Trends or a similar detailed view of the situation that provides some depth and breadth and doesn't rely on just one or two metrics.

For this small group, if their source is Brookings, and they spend the time wading through the 30 page report, they will have a good chance of making good sense of the charts that Brookings selected. Similarly if their source is the State Department and if they wade through the Weekly Status report they will get a picture of the trends that the State Department selected but not the ones that Brookings was paying attention to. If their source is GlobalSecurity or the Iraq Casualty Coalition, they will get a good grasp of the charts presented at those sites.

Even in this small group that goes and gets their trend understanding from one of these useful and detailed sources (or other similar venues) a small percentage will dig any further into the tabular data that is available. The reason: primarily lack of time and secondarily lack of experience possibly combined with self conceived notions that reading tables is hard.

My reading is that few in this small group will consult and combine the trends from multiple sources like those mentioned above to get a broader perspective. Of course, if you have been reading this series of posts, we have done some of that lifting work for you and made it easier (still not easy in my opinion) to become more widely informed of the composite set of trends that are considered important drawn from a mix of analysts who may have vastly different axes to grind.

For a good portion of the rest of the population beyond this small group, the trusted source for trend data will turn out to be television and other mainstream media. Based on what I have seen from the MSM, this group of individuals will likely be exposed to quite limited trend information in any form and will often end up only being exposed to non-specific, non-numerical statements about the situation on the ground.


What to do? What to do?
Glad you asked. The answer for me is straightforward: ready reusability.

Ready reusability for the Iraq Trend data is easy to say and hard (time consuming, error prone) to do right now. In the next couple of posts, I will give you an example of what I mean by transforming some of the tabular data from expert sources into a readaily reusable form. Once the data is ready to go, I will provide some examples of why I think this is so important.

One More Source of Iraq Trend Graphics - Iraq Coalition Casualties

So far, in our previous posts on Iraq we have pulled together a set of 16 different How Are Things Going in Iraq Charts from three different sources. We have:

• 9 trends from the Brookings Iraq Index
• 4 trends from the Department of State Weekly Iraq Status Report
• 3 trends from Global Security.org

Complementing these are another 5 trend graphs drawn from Iraq Coalition Casualties and these cover some topics not previously addressed in the earlier charts (e.g. IED fatalities). they Also present some important information again but in a different format thereby bringing out additional details and nuances not readily visible earlier - for example adding notations for the timing of important events to the chart showing US Soldiers killed.

The 5 charts show:

1. US Soliders Killed with moving avg & event timing - March 2003 - December 30, 2006
   
2. US Soldiers Wounded with moving avg - Sept 6, 2003 - Dec 29, 2006
3. IED Fatalities per month - July 2003 - January 2007
4. Iraqi Police/Military fatalities - January 2005 - December 2006
5. Iraqi Civilian fatalities - January 2005 - December 2006

This brings us to a total of 21 charts. I think you will see that by arranging these together in a somewhat more convenient and time-saving format than the separate originals, your understanding of what's happening in Iraq has been boosted. I think you will also see that being able to look at all 21 one of these in a relatively short period of time by scrolling down this blog allows you to develop more of a gestalt understanding of how all the factors interact and impact on each other.

• How many charts is enough?
• Are we there yet?
  
• Do we already have everything we need to understand the situation?
• Is it relatively easy for decision makers or interested citizens to get rapid access to all of the most important factors?
• Do we now have the full deck of cards that will allow us to make wise choices?
• Have we arranged things so that automatic monitoring and reporting of key factors enables us to determine quickly whether the actions we take are having the desired effect as we go forward?

I think not. My two cents is that we are not there yet and in the next few posts, I will aim at making some further improvements that make these important trends more accessible to both decision makers and ordinary citizens. I will also lay out a plan for even further improvements that could be available to us in the future if we begin to take appropriate decisive action now.

CSVPNG – A Handy Utility for Readily-Reusable (R-R) Data

The T4 & Friends project at HP we mentioned in earlier posts was at its foundation a project that converted trend data from its original complex format into Readily-Reusable (R-R) format as CSV files. We discovered that any trend data could be so converted and that in most cases the transformation was straightforward, relatively painless and often quite rapid.

As a result of the T4 project, more complex data from a variety of sources was converted to R-R format. Those with interests in this data were then able to enjoy the possibility of looking at it with Excel, loading it into their favorite database or in many cases, analyzing and reporting on what the data meant using TLViz.

We sometimes find it convenient to use the metaphor that

1. data is collected ‘upstream’,
   
2. it is saved in R-R format in historical ‘reservoirs’ for possible future use, and
   
3. it is fed, as needed into ‘downstream’ tools for analysis, reporting, collaboration, …

Since no single such downstream tool does everything you might want, the availability of growing reservoirs of R-R data in CSV format systematically increased the possible value of building new tools or incrementally improving existing tools to take advantage and convert the potential value into real value.

CSVPNG (CSV file to Portable Network Graph Utility) is one such development that has blossomed in the world of ready-reusability. Developed in 2003 by Pat Moran of Hewlett Packard, CSVPNG’s initial purpose was to live up to its name – that is to convert CSV trend data automatically into a set of PNG graphics. These were then all embedded in a single output HTML page. Just this one feature alone added considerable value by automating the output of any sets of standard charts that you might always want to examine.

CSVPNG has then proceeded over the past 4 years to transform itself through a series of more than 100 improvements and extensions. You can download a copy of this great tool from the T4 & Friends web page or from TrendsThatMatter.

CSVPNG is written in C and has been made available for a steadily growing number of platforms including DOS under Windows, Linux, HP-UX, and OpenVMS on both Alpha and Integrity servers.

Here’s a brief rundown of some of the things you can do with R-R CSV data using CSVPNG. For a full list of possible benefits, be sure to examine the CSVPNG.TXT file thoroughly after you have downloaded the kit. If you are generating readily-reusable data in CSV format, CSVPNG will surely save you time in managing this data and will help make sure that you don't have to reinvent the wheel.

1. Automatically create a selected set of graphs and format as html or PDF
2. Slice and dice files to select out just the factors you want and just the time periods you are interested in. The trimmed down result can be automatically graphed and/or used to generate a new reduced file that satisfies the R-R rules. These files could then be fed into other tools such as TLViz or Excel.
3. Combine R-R data from several files into a single synchronized file
4. Search using expert rules for “interesting” conditions and then display only those charts for cases where one or more conditions were met
5. Carryout column arithmetic to create new factors by recombining several original factors.

Here’s a sample graphic output from CSVPNG showing how it can find and highlight a peak period as well as display the original data with moving average value all in a single chart. The blue represents the raw data value, the red line shows the moving average, and the highlighted time interval identifies the peak interval.

The more R-R data you have to deal with, the more valuable you are likely to find the capabilities for CSVPNG for helping you automate and keep on top of all this data.

World Watch State of the World data in Readily Reusable form

Temporary Workaround

We are still working on finding a convenient way to share readily reusable data so that it will be as easy as possible for you to download. We're not there yet. So in the Meantime, I thought I would try an experiment of pasting the data into this blog entry. It will be clumsy but you should be able to select copy the text data (just the text in Orange) and paste it into the Notepad editor and then save the file with a CSV extension.

Once you have done that, you will be able to have fun playing with this interesting data set using TLViz.

War & Peace,Source: World Watch State of the World 2002,Composite of selected factors including ,transportation,fossil fuel use,nuclear weapons,world food supply,armed forces manpower ,population,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Sample Time,Air Billions of Freight Ton Kilometers,Air billions of passenger kilometers,Autos Passenger Cars,Autos People Per Car,Autos total,Autos World Wide Total Fleet,Bike production in millions China,Bike production in millions US,coal million tons oil equiv china,coal million tons oil equiv former soviet union,coal million tons oil equiv USA,coal million tons oil equiv world wide,Employment World Arms Industry in millions,Food - per capital food production index,Food - Prices in Constant 1999 dollars,fossil fuel consumptions millions of tons OIL world wide,fossil fuel consumptions millions of tons oil equiv ALL world wide,fossil fuel consumptions millions of tons oil equiv NATURAL GAS world wide,Grain Carryover in Days of supply,Grain Carryover in Millions of Tons,Grain Corn Millions of Tons,Grain for Feed Use in Millions of Ton,Grain Per Person in Kilograms,Grain Percentage for Feed Use,Grain Rice Millions of Tons,Grain Total Production in Millions of Tons,Grain Wheat Millions of Tons,Grain Yield in Tons Per Hectare - ALL,Grain Yield in Tons Per Hectare - Corn,Grain Yield in Tons Per Hectare - Rice,Grain Yield in Tons Per Hectare - Wheat,Nukes China,Nukes France,Nukes Soviet Union/Russia Nukes,Nukes Total World wide,Nukes United Kingdom Nukes,Nukes United States Nukes,Population - annual addition in Millions,Population Annual Growth rate in Percent,Population World Wide in Billions,Refugees per UNHCR Africa in Millions,Refugees per UNHCR Europe in Millions,Refuges per UNHCR Asia in Millions,Refuges per UNHCR World Wide in millions,Soldiers China in Millions,Soldiers India in Millions,Soldiers North Korea in Millions,Soldiers US in millions,Soldiers USSR in Millions,Soldiers World Wide in millions,Travel International Tourist Arrivals in Millions,Travel International tourist receipts in Billions of 1999 dollars
1/1/1990 0:00,58.8,1894.2,36.3,11.9,0,444.9,31.9,5.6,533.6,308,482.3,2270.3,16.3,100.8,51.7,3135.5,7150,1773.8,104,490,482,665,335,39,352,1769,588,2.54,3.75,2.4,2.54,432,504,38000,60236,300,21000,83.2,1.6,5.3,4.6,0.8,6.8,14.9,3.5,1.3,1.2,2.2,3.7,28.2,457.2,319.1
1/1/1991 0:00,58.6,1845.4,35.1,11.8,0,456,36.8,7.6,534.9,277.7,478.5,2224.5,15.4,99.4,50.1,3134.3,7165.1,1806.3,102,479,491,654,318,38,355,1708,543,2.47,3.73,2.41,2.44,434,538,35000,55772,300,19500,82.7,1.5,5.4,5.6,0.9,7.9,17.2,3.2,1.3,1.2,2.1,3.6,26.5,462.7,323.4
1/1/1992 0:00,62.6,1928.9,35.5,11.6,0,469.9,40.3,8.9,549.5,265.5,482.7,2211.3,14,100.6,49.1,3165,7185.8,1809.5,107,513,539,663,328,38,356,1789,562,2.57,4.05,1.97,2.52,434,538,33500,52972,300,18200,81.3,1.5,5.4,5.3,1.2,8.6,17,3.2,1.3,1.2,1.9,3.6,25.9,500.9,359.7
1/1/1993 0:00,68.4,1949.4,34.2,11.8,0,469.5,41,7.7,570.3,238.7,498.1,2206.1,13,100,48.6,3135.1,7190.4,1849.2,98,471,476,654,310,37,355,1713,559,2.5,3.64,2.45,2.52,434,524,32000,50008,300,16750,80,1.4,5.5,5.4,3.8,7.2,19,3,1.3,1.2,1.8,3.2,25,515.7,358.4
1/1/1994 0:00,77.2,2099.9,34.8,11.7,47.6,479.9,42,7.3,606.4,211.5,502.3,2224.4,11.6,101.6,49.3,3192.1,7274,1857.5,94,459,560,670,314,38,365,1760,524,2.57,4.15,2.47,2.44,400,512,30000,46542,250,15380,79.9,1.4,5.6,7.5,6.1,8,23,2.9,1.3,1.2,1.7,2.8,24.1,550.3,387.1
1/1/1995 0:00,83.1,2248.2,35.5,11.9,48.5,477,41,8.5,635.7,192.7,504.6,2258.3,10.6,102.3,49.4,3235.4,7406.9,1913.2,85,409,517,638,301,36,371,1713,538,2.52,3.85,2.51,2.46,400,500,28000,43200,300,14000,77.7,1.4,5.7,11.8,6.5,7.9,27.4,2.9,1.2,1.1,1.6,2.5,23.5,565.4,433.2
1/1/1996 0:00,89.2,2431.7,36.9,11.9,50.3,486,38,7.4,676.9,178.7,528.1,2341.6,9.4,105.1,58.1,3315.5,7662.2,2005.1,90,450,592,673,324,37,380,1871,582,2.66,4.2,2.54,2.53,400,500,26000,40100,300,12900,78.2,1.4,5.8,9.1,7.7,7.7,26.1,2.9,1.2,1.1,1.6,2.2,22.6,597.4,457.2
1/1/1997 0:00,102.9,2573,39.1,11.7,53.2,498,30,6,652.8,175.2,540.9,2326.8,9,106.3,54.5,3387.7,7707,1992.5,98,494,575,686,322,37,387,1880,609,2.72,4.24,2.55,2.67,400,450,24000,37535,260,12425,77.8,1.3,5.8,8.1,5.7,7.9,22.7,2.9,1.2,1.1,1.5,2,21.9,618.2,451.8
1/1/1998 0:00,101.8,2628.1,38.4,11.6,51.9,510.3,23.1,2.5,621.2,166.6,543.2,2281.1,8.5,106.8,48.5,3398.2,7694.9,2015.6,104,523,606,679,316,37,394,1872,589,2.74,4.35,2.55,2.62,400,450,22000,34535,260,11425,77.9,1.3,5.9,7.4,6.1,7.5,22.4,2.7,1.2,1.1,1.5,1.9,21.9,636,448.4
1/1/1999 0:00,108.7,2797.8,39.9,11.5,54.8,520,42.7,1.7,516.6,170.7,546.4,2159.9,8.2,108.3,42.1,3469.1,7694.2,2065.2,102,523,607,683,312,36,409,1871,586,2.78,4.26,2.56,2.65,400,450,20000,31960,185,10925,77.6,1.3,6,6.3,6.2,7.5,21.5,2.5,1.2,1.1,1.5,1.8,21.7,650,455.5
1/1/2000 0:00,118.3,3010.2,41.1,11.1,56.7,545.6,52.2,1.1,483.8,175,564.1,2186,8.2,108.2,43,3503.6,7853.6,2164,95,489,586,688,303,37,397,1840,583,2.76,4.31,2.62,2.66,400,450,20000,31535,185,10500,77.3,1.3,6.1,6.3,7.3,7.3,22.3,2.5,1.2,1.1,1.5,1.8,21.7,698,468.6