Dangerous to have too few metrics or the wrong metrics

In a previous post: Revisiting "See You In September" I noted:

There is a way forward that addresses Michael O'Hanlon's concerns

The relevant quotes from O'Hanlon were:

• "metrics are grist for a fact based debate but history shows it is dangerous to rely on too few of them"
• "metrics were used in Vietnam and we had the wrong ones"
• [the Vietnam metrics] "did net harm to the debate"
• "we can’t be exactly precise about which indicators are the conclusive ones”

Here are a few observations related to the above.

Dangerous to Rely on Too Few of Them. I agree entirely. For Iraq, we need to be able to look at at least 50-100 of the most important factors efficiently, all within a relatively small period of time. For expert use, a much higher number of metrics must be available for review in an efficient and effective manner. Michael O'Hanlon's weekly Iraq Index is one of the best examples out there that pays attention to the principle of not relying on too few metrics.

The Metrics were the Wrong Ones. This is a solvable challenge this time around. First, follow the principle above and make sure that we look at ALL of the important factors. How do we decide? We ask all the experts in all the different areas (security, economy, health, ...) to name what they consider to be the most important indicators. Then we capture and archive and report on all the factors that have been suggested, even if only once. Maybe there will be some "wrong ones" in the mix, but when we have the full set of data to work with, the chances of being led astray will be markedly reduced.

Can’t be exactly precise about which indicators are the conclusive ones. We don't have to be precise. Different people will have different opinions. If we capture all of them, record a history of each trend, and make this data available in Static Graphical format, AND as machine readable data suitable for input into a Trend Visualization application, then we can begin to have good and constructive discussions about what different trends mean and what we should do about them.

Turning the Tables

Some important data in the Brookings' Iraq Index only appears in tables (e.g. the prison population data we mentioned in the previous post). Troop strength for US forces and for other Coalition forces may be interest now because of the "surge" in in US Forces that is now reaching its stated goals. However, the troop strength trend data in the most recent report is tabular.

So we turned the tables and transformed it into a small CSV file and created some new charts (see below) using the TLViz interactive trend visualization tool.

If you would like to take a look at the CSV file yourself, you can download it from Troop Strength Table. This table includes three original trends from the Brookings' table (US Troop Strength, Other Coalition Troop Strength, and Total Coalition Force Troop Strength). To that, we have added two calculated trends. The first shows Other Coalition Troop Strength as percentage of Total Troop Strength. The second shows the ratio of US Troop Strength to a nominal value of 135,000.

If you haven't already tried out the TLViz tool that runs on Microsoft Windows as a way for looking at two dimensional tables of trend data, here's another chance. Get started by downloading the base kit. Once you have installed the base kit, you can grab the latest version of TLViz.

If you want to learn more about TLViz, check out some of our earlier posts on the subject starting in chronological order with the ones from early January.


This first chart shows a sharp downtrend in other Coalition forces dropping in half from 25,000 to 12,000 since January 2005.

This chart shows US Troop strength from the beginning of the war. While the uptick since January is sharp, the current level of US troops is not far from the middle of the range it has been operating in for the past 3 years.

This is a chart using the calculated value showing the Other Coalition forces as a percentage of total troop strength. There has been a sharp decline over the past 3 years from a maximum of 17 percent down to the current value of approximately 7.5%.

This is another calculated chart showing how US troop strength has fluctuated around the 135,000 mark for the past few years. The current peak is lower than the two previous peaks.

The final chart shows total coalition forces in Iraq over the past 4 years. The current value of 160,000 looks like it is right in the middle of the operational range for this factor since the beginning of the war.

An Old Fashioned Trend Visualization Appliance

And speaking about the excellent Calculated Risk Blog, I would like to point out the following exemplary post on the Housing market.

Using just the ordinary blog posting mechanism and the ability to embed JPG pictures as the Trend Visualization Appliance, the resulting article has a high degree of explanatory power and immediacy. You can literally see what Calculated Risk is talking about as you read as a result of the great blend of clarifying text with the easy to read trend graphics in close proximity.

Not relying on the normal conventions of providing only one or two graphics for headline numbers, Calculated Risks walks us through a series of 6 or 7 less well known but still key factors. This leaves the reader/viewer which a much more complete view of the dynamics that are at work in this one complex area.

Combine this with good references back to data sources and consistently good judgment on what would constitute a reasonable time frame, leaves with a very useful end result that provides the reader with the maximum amount of infomation in relatively short period of time.

The only difficulty with this is the same one I face with this TimelineView blog -- namely that as a limitation of BLOGGER.COM the largest allowed size of the embedded images is just too small to be fully readable and understandable in most instances. To really understand the charts, at least at first, usually requires clicking for the larger image and then using the browser navigation controls to go back to the main page. This really slows down the flow of reading and understanding.

I think we can all learn a lot by watching how Calculated Risk handles these kinds of presentations. Take a look at the post or other similar posts that appear regularly at Calculated Risk and you will see what I mean.

Iraq Trend Data Set now posted to Swivel and Many Eyes

A while back we mentioned the great new data sharing and visualization web site called Many Eyes and we pointed out an Iraq Trend data set we had uploaded so that others could more readily examine and analyze it.

We've recently come across another great new data sharing and visualization web site called Swivel and have uploaded the same data set there for comparison purposes at: http://www.swivel.com/data_sets/show/1004780.

Both Many Eyes and Swivel fall into a category I have started calling Trend Visualization Appliances. The amazing Gapminder work also fits into this category as do all the various stock market visualization tools currently available.

The old fashioned form of trend visualization appliances (e.g. a standard report including graphical output) can still be pretty useful when managed carefully. For further comparison, here is the link to a PDF report that graphically presents the same data - vizualizing-trends-ohanlon-testimony.pdf.

Similarly, for data sharing, the old fashioned URL links to csv files offer an alternative to the WEB 2.0 mechanisms such as Many Eyes and Swivel. Here's the Iraq trend data the old fashioned way: ohanlon-key-factors.csv

So many choices, so little time. How can we decide which is best for our purposes?

I'll be revisiting this topic, but for now, my own criteria for deciding which tools I will use at a given moment are:
1) ease of use,
2) shortness of learning curve, and
3) personal productivity and time saving -- the speed at which I can navigate through complex trend data sets to discover previously hidden patterns.

What do you think?