A "Manageable" Number of Indicators

In a previous post: Revisiting "See You In September" based on the Sheryl Stolberg New York Times article I noted:

There is a way forward that ... answers Rep. Kingston's call for assistance.

According to Stolberg, "some in Washington are grasping for a more complete and accurate way to quantify progress." “No one knows how to define progress in such a mixed-up situation.” Others say “we’re having trouble measuring it.

Rep Kingston (R-Georgia) wanted to "winnow down the indicators to a manageable number -- say, fewer than a dozen." He wanted to create "a standard bi-partisan metric." With such a metric in place, "then you could say who’s winning and losing.”

The key words here are: "manageable number" and the key idea is that the medium we chose to publish these numbers matters to a huge degree. The thesis of this post is that what constitutes "manageable" depends on how we chose to present the data.

Using Unquantified Indicators. If we don't get down and get numeric with the metrics we are interested in, then the manageable number doesn't matter.

Talking about Trends. If all we do is talk about numeric trend data, then maybe the manageable number is only 1 or 2 and even with that small a number of items, most listeners won't end up with a good sense of of how those two indicators are changing over time. Speech is simply not a good way to deal with understanding trend data. And because so few data points for each trend can be presented in this fashion, there is also tremendous opportunity for slanting and spinning the meaning.

Writing about Trends. If we chose to write about it, the manageable number may be 3 to 5 metrics. With written trend information, most readers will not get a full sense or understanding of how the situation is evolving. The written word and written number is not an efficient or an effective way of handling life's most important trends. As is the case with talking about trends, writing about trends normally means that very few data points are actually given for any of the important indicators. Sometimes no value is given at all, only some form of imprecise quantification. Sometimes, only a single value is provided. Sometimes only a pair of values or a single value and a comparison to a previous value. [Take a look at today's articles reporting on the DoD quarterly report to Congress for crystal clear examples of what I am talking about here.]

Documenting Trends with Printed Tables. If we decide to put the information in a printed table of data, then the manageable number may grow to 7-10 factors. This is a step forward. However, only a handful of our citizens will be comfortable with getting their trend information in this fashion and making sense of it. And for most of those included in that handful, they will rarely actually have the time to study the table in sufficient detail to actually get a sense of the trends. For those few with the skill, the interest and the substantial time required, this format will begin to allow some understanding of the trends at work. However, for most of the population, the table format will be close to unusable.

A perfect example of the difficulty of both the written word and printed tables is today's Producer Price Indexes report from the Bureau of Labor Statistics.

Static Trend Charts. If we create a set of static trend charts for each selected metric, then the manageable number can easily jump up to 10 or more and if these trend charts follow best practices, most ordinary citizens will be able to make good sense of them and gain an understanding of how the situation is progressing. Of course, if such charts violate the best practices, the utility of this approach drops off dramatically.

Interactive Trend Charts. If we have access to the trend data via a human engineered interactive trend viewing station, then it is possible for ordinary mortals to deal with dozens and dozens of factors and indicators and gain a deeper gestalt understanding of the situation, all in a reasonable amount of time. With really good interactive tools, examining and learning from hundreds of metrics is within reach.

What constitutes a Manageable Number of Metrics? It depends on the approach we take for representing the trend data.

Since the number of factors we musts look at to gain full insight and understanding into the situation on the ground in Iraq is so high, unless we begin to move towards interactive charting, our chances of success look quite slim indeed.

Right now, almost all the official and unofficial reporting of quantified results is weighted heavily towards talking, writing, and printed tables of data. Those who produce static charts are relatively few and even though we can handle maybe as many as 10 well constructed static charts at a time, the norm is to have only 1 or two as is the case with today's WaPo and NYT articles we posted about earlier. All the rest of reporting will be written, and most of the indicators will not have any numeric value associated with them all.

The Sheryl Stolberg article on the surge also included this telling quote:

"Some say measuring progress is simple: you will know it when you see it."

I agree. This is really the answer to Rep. Kingston's request. We'll know it when we see it, but we will only see it if we show it. And we will only show it if we make the effort to move beyond the current default mode of talking about it and writing about it.

For whatever trend data we gather to be useful, we need to make sure that the manageable number of indicators is actually large enough for us to be able to use that data to deal with the problems that confront us. My own estimate of how many this might be for Iraq is at a minimum in the range of 50 t0 100 of the most important factors for general use and discussion with a much higher number available for expert use on specific focused issues.