Change Over Time

Taking the long term view of real median household income

2011-10-10T11:41:00.000-05:00

See this impressive chart from the New York Times showing the drop in real (cpi adjusted) median household income since 2000. The source for this chart is a new Sentier Research report by two former officials at the U.S. Census Bureau (Gordon Green and John Coder).

Source: New York Times based on Sentier Research Report:
Household Income Trends During the Recession and Economic Recovery

This approach of showing change over time compared to a single baseline reference value often proves to be much more interesting than month-over-month or year-over-year displays.

Before even reading the full article, an underlying story with high explanatory power literally jumps off this chart at you.

You can find further details at: Recession Officially Over, U.S. Incomes Kept Falling. The article introduces some new and helpful metrics such as the average cut in pay at rehire for those who lost jobs. The NYT article also provides some disaggregation of the headline number for different groups (by age, by education, by race) but unfortunately does not provide charts for these.

Watch the trend, not just the latest data point

2011-10-09T12:55:00.000-05:00

Some excellent thinking from Barry Ritholtz in his recent post NFP Report: Trend vs Single Data Point discussing on how best to interpret the latest job numbers.

The key ideas I take away from this are to 1) watch the trend, 2) ask the right questions to establish context, 3) examine a wider range of metrics (not just the headline numbers), and especially 4) not to over focus on the latest data point.

Playing with the data, keeping context in mind, and looking for the trends and the answers to our deepest questions is the best path to more complete understanding and improved decision making.

2011-09-20T14:24:00.000-05:00

Doug Short has an excellent post up today Household Net Worth: The "Real" Story. Check it out. It provides an easy to follow example of some of the principles that we hold most strongly:

The post identifies a key metric (Household Net Worth) that has not received a lot of attention and really looks at that metric all by itself using several alternative views to aid in understanding.
It includes a long time perspective as well as zoomed in views thereby revealing patterns that would have remained invisible if only a single time span had been chosen.
The post provides a clear storyline -- sharing with us what Doug was thinking while he looked at these charts and selected them for our edification.

Providing some Context for looking at CPI charts

2011-09-18T14:40:00.000-05:00

We have posted many charts this past week looking at CPI behavior through a variety of angles and lenses. Our sources have been light on providing context and we have not filled in this area either.

So here is a first cut at the context that we were holding loosely in mind as we evaluated the incoming collection of charts and created a few of our own using the St. Louis Federal Reserves' FRED software.

To begin, CPI stands for Consumer Price Index. In our view, the emphasis is on the Consumer (real people living out their lives) and the ways in which changes in Prices (specifically price inflation) might impact the standard of living for those people.

Central Questions: The question we would like to be able to answer more fully is:

How does the cumulative change in prices affect the buying power of a range of different classes of consumers over time?

In particular, we want to be able to figure out to what degree the Cumulative Impact of Price Inflation changes each different class of consumer's standard of living over time.

Principles. The underlying principle that we believe to be at work here is that: if cumulative CPI goes up relative to the wage trend for households representing a given class of consumers, then the Standard of Living for that class would go down in proportion. We are interested in looking at the cumulative change over a variety of longer periods: 2 years, 3 years, 5 years, 10 years, 15 years, 20 years

A second key principle for us is that CPI Inflation is like Compound Interest; and so longer term cumulative views of the data should predominate.

Weaknesses of Traditional Reporting: To us, the commonly used month over month results represent little more than noise since our focus is on cumulative results.

Even the very widely used year over year reporting is not particularly useful in this regard and is usually distracting at best (1 year effects are interesting but surely not the whole story) and misleading at worst as to the important long term inflation changes that are underway and the implications of those changes on real people's standards of living and quality of life.

A long standing general purpose admonition of ours is to Look at All the Data. The traditional reporting on CPI violates this rule in spades with the focus on only headline CPI and core CPI . CPI is an aggregate of aggregates of aggregates which averages out a huge degree of variation that is present in its sub-components. To begin to understand the impact that cumulative price inflation has on Consumers (real people) we surely need to disaggregate the average CPI values and look individually at a good number of the key sub-components (e.g. Food, Energy, Health Care, Rent, Transportation, Tuition) to see their cumulative price inflation over longer periods of time. With few exceptions such as the excellent chart we posted earlier (A Unique View of CPI from Doug Short), almost all reportage we have found on CPI focuses on headline or core cpi.

Now it is true that the official report from the BLS (BLS August 2011 CPI news release) does provide an exceptionally high degree of disaggregation in its tables by sub-categories and sub-sub-categories. But there are no sub-component charts accompanying the BLS report, so looking at sub-component increases in cpi is left as a time-consuming exercise for the reader - an option that few have likely taken. Furthermore, the data presented only covers the most recent 1 year period, so if you want to explore sub-category cumulative changes over periods of 2 to 20 years, you have to look somewhere else (e.g FRED or the BLS data base).

There is a lot of fruitful potential in this disaggregation, but I have not been able to find anyone so far other than Doug Short and our recent post (Drilling down into CPI 20 year trends) tapping into this for more insights into understanding the cumulative impact of inflation on real people. We will keep on looking and would love to hear about other work in progress studying the sub-components.

Core CPI interferes with clear thinking. Finally, looking at widely discussed Core CPI and its many variants, we see Core CPI as a serious impediment keeping us from thinking more clearly about the cumulative impact of price inflation on real people who all must eat and consume energy. Rather than stripping food and energy out of the totals to create so-called CORE CPI, it would be much more useful in our estimation to include Food and include Energy as individual sub-categories - to look at the cumulative impact of Food price inflation and the cumulative impact of Energy price inflation as we have started to in our Drilling down into CPI 20 year trends post. And rather than lump all the rest of the categories together to create the "core", we would prefer to disaggregate these as well and look at them individually so we can discover patterns currently hidden..

A lot has been made about why Energy and Food are so volatile as the reason for going with Core CPI. Our investigation shows that the mostly irrelevant month to month variations smooth out incredibly well for FOOD once we look at cumulative totals. For ENERGY, there is clear volatility, but rather than hiding it, it's better to bring it out into the open. And when we look at energy using a cumulative long term 5, 10, 15, 20 year approach, there is a lot less volatility than you might expect. We'll have more on this later.

And the headline CPI number shows very low variability in trend when plotted using a long term cumulative view as we have shown in our early post ( Drilling down into CPI 20 year trends ).

There are further weaknesses we see in traditional CPI reporting regarding the limited number of different classes of consumers currently available) but we will save these for another day.

Now that we have some context on why we want to look at CPI, our plan is to revisit the CPI posts from earlier this past week. We plan to update them by showing our thinking regarding what we see as the story that might be told for each chart in the context of the ideas we have explored in this post - real people, cumulative impact over longer time spans, disaggregation. .

Update 20 September 2011: We have now updated all previous CPI posts with our further thoughts on what each chart is telling us. Please check them out.

Drilling down into CPI 20 year trends

2011-09-17T07:25:00.000-05:00

Continuing with our 20 year trend analysis, the charts below show inflation behavior for some of the most interesting major sub-categories of CPI data. For each view, the first URL is the PNG chart and the second allows you to continue the analysis and access the underlying data. All of these metrics are Not Seasonally Adjusted since when taking these long 5, 10, 15, 20 year views, seasonal adjustment is not really needed.

Each chart shows 4 trends, 1 for the past 5 years, 1 for the past 10, one for 15 and one for 20 years. It's possible to read off the approximate end value of percentage increase. It's also possible to get a feel for the overall trends by observing the slopes of each trend line.

Note that the Y axis scale max value changes with each chart based on the 20 year curve so the slopes are not directly comparable in this view. .

Update 20 September 2011

Note other updates in this post are highlighted using this same brownish orange text color

Below, we have added some comments regarding what we see as important in each chart. The key takeaways for us are:

Combining the two ideas of taking a cumulative view and drilling down on sub-components proves itself ot be most useful in helping us discover patterns previously not visible.
Food inflation turned out much smoother than expected
Energy was volatile but the cumulative approach still provided meaningful insights not otherwise available
The wide variation in cumulative effects that we witnessed in Doug Shorts's 12 year chart also showed up in the series of charts below with some factors showing inflation rates far above the average value
Many new questions were raised by the behavior we saw in the charts such as why are Medical and Tuition rising so relentlessly and smoothly and so fast while other categories show more variability?

Next steps will include comparing the longer term trend of these factors in a single picture so the relative differences will be more readily seen (our method below with the Y axis maximums varying makes that much more difficult). We will also want to see how cumulative inflation might impact different groups of citizen household's standard of living

Food

http://research.stlouisfed.org/fredgraph.png?g=2gw

http://research.stlouisfed.org/fred2/graph/?g=2gw

With this long view Food does that seem all that volatile or variable

In the Food chart, note the steady slope of the trend lines and the quick reversion to trend for the 10,15, 20 year cumulative lines following the most recent recession. If anything it looks as if cumulative food price inflation may be increasing faster than trend (greater slope).

Despite the stated volatility in food prices, when we take a cumulative view, the volatility virtually disappears so in our opinion it is perfectly fine to include Food price inflation in with a composite such as Headline CPI whenever we are using cumulative inflation thinking. .

Energy

http://research.stlouisfed.org/fredgraph.png?g=2gx

http://research.stlouisfed.org/fred2/graph/?g=2gx

Energy prices undoubtedly have been volatile, especially since 2001. But the cumulative view of this key subcomponent is still the best way to understand energy price inflation and to spot trends when they appear as in the very sharp upswing from 2001 to 2008. It also looks as if following the recent recession, that energy inflation may be trying to revert to the slope of the earlier trend line.

Key take away for me in this chart is the red line which shows doubling in energy prices over the past 10 years.

Gasoline

http://research.stlouisfed.org/fredgraph.png?g=2gy

http://research.stlouisfed.org/fred2/graph/?g=2gy

Gasoline inflation shows a pattern quite similar to overall energy but with even higher inflation rates of over 20% or the past 5 years and about by 160% over the past 10 years.

Medical

http://research.stlouisfed.org/fredgraph.png?g=2gz

http://research.stlouisfed.org/fred2/graph/?g=2gz

What's striking about this picture of Medical inflation is the absolute smoothness of the lines and the steady slope of each chart and the degree to which medical inflation far exceeds the average Headline CPI inflation.

Why is this so steady and relentless? If we can figure that out, we might be able to do something slow inflation in this category. .

Tuition

http://research.stlouisfed.org/fredgraph.png?g=2gA

http://research.stlouisfed.org/fred2/graph/?g=2gA

Tuition also shows a relentless upward pattern that looks like it is accelerating and runs far above the average headline cpi values. Why is this so? What is driving it?

Rent

http://research.stlouisfed.org/fredgraph.png?g=2gB

http://research.stlouisfed.org/fred2/graph/?g=2gB

Until 2008, rent also showed a relentless smooth upswing at just slightly above Headline CPI inflation. Following the most recent recession, it now looks like it might be reverting to its old trendline.

Transportation

http://research.stlouisfed.org/fredgraph.png?g=2gC

http://research.stlouisfed.org/fred2/graph/?g=2gC

Transportation looks very much like energy and gasoline charts, with more volatility but still some quite claer visible trends and what looks like a reversion to trend following the recent recession.No doubt this is due to the fact that the Transportation cpi includes energy components in a major way.

Public Transportation

http://research.stlouisfed.org/fredgraph.png?g=2gD

http://research.stlouisfed.org/fred2/graph/?g=2gD

It is interesting to see how different public transportation is from Transportation as a whole. Public Transportation shows a smooth upward trend line despite high month to month volatility with a version to trend line slope following the recent recession. Overall, its cumulative effects are slightly higher than headline cpi

Other Goods and Services

http://research.stlouisfed.org/fredgraph.png?g=2gE

http://research.stlouisfed.org/fred2/graph/?g=2gE

This chart for Other Goods and Services shows some interesting square wave patterns that may be a result of some change in the way this metric is calculated. Overall, the slope of the trend lines are smooth and this sub-category shows substantially higher inflation values than headline cpi

What new insights can we gain by drilling down/disaggregating?

The impact of cpi changes

2011-09-16T10:23:00.000-05:00

Calculated risk has an interesting post this morning that shows how CPI inflation values impact Social Security cost of living adjustments (COLA). This is worth checking out as it adds some important context to why we might be interested cpi inflation in the first place. The post includes lots of explanatory text regarding how all the pieces fit together to determine COLA changes.

Early Look: 2012 Social Security Cost-Of-Living Adjustment on track for 3.5% increase

A 20 year cumulative view of headline cpi from FRED

2011-09-16T10:01:00.000-05:00

Here's another view of the headline CPI number created using some advanced features of FRED (the wonderful St Louis Federal Reserve Economic Data charting program).


This chart shows the cumulative effect of cpi inflation over the past 5, 10, 15, and 20 years http://research.stlouisfed.org/fredgraph.png?g=2fo

Update: 20 September 2011

The first key thing that I see in this chart is the relatively steady slopes of the 10, 15, and 20 year cumulative changes in inflation. For example, the blue line (20 year) shows a sharp dip during the most recent recession, but then quickly ramps back to close to the same trend line. Similary for the 15 year (brown line). Even though it is claimed that headline cpi is too volatile, when we deal with cumulative change as shown above, that by itself smooths out almost all the volatility.

What this means is that if we use cumulative data, we do not need to resort to thinking we need to use Core Cpi or other less volatile measures of inflation. We can include energy and food and still get a clear, smooth picture of what's going on and how it might impact real citizen households.

The second thing that is important to note in these charts is the considerable magnitude of the cumulative change for each for these periods (e.g. 27.5% for the past 10 years) as listed below. These changes can be compared with changes in nominal income achieved over similar periods - e.g. for households that fall in different income percentile categories as shown in the recent Census Bureau report: Income, Poverty, and Health Insurance Coverage: 2010 which includes an adjustment for CPI. What we learn from that report is that cumulative changes in income over time are different for different groupings of citizen (e.g. by income level or by race). This in turn leads to differential impacts of changes in cumulative inflation on each groups. In the most recent 20 year periods, some groups have been advancing their standard of living while other groups have seen their standard of living reduced.

We plan to discuss the Census Bureau report and work up some of these figures in more detail in future posts.

The shortest red line shows 11.0 % total inflation over the past 5 years;
the green line shows a total of 27.5% total cpi inflation over the past 10 years;
the brownish line shows 43.9% increase in total inflation over the past 15 years; and
the longest blue lines shows a total increase of inflation of 65.6% over the past 20 years.

If you want to play further with variations to this chart over at FRED or if you want to download the data that was used to create this chart you can use the following link: FRED 20 year cumulative view of cpi

Key take away - except for a little hiccup in 2001 and for a dramatic drop in 2009, headline cpi inflation has increased at a rather steady and substantial rate as can be seen by the readily visible even slope of these trend lines. It also looks like the latest rise over the past year has been bring the cumulative number back previous trend line..

Calculated Risk's view of CPI

2011-09-15T14:57:00.000-05:00

Bill McBride of the always excellent Calculated Risk blog has now weighed in with his approach to the latest CPI figures with his post on Key Measures of Inflation increase in August

Bill includes both the BLS CPI number and the Cleveland Fed's median cpi and trimmed mean cpi (alternative measures of core inflation). Key take away is

"On a year-over-year basis, these measures of inflation are increasing, and are near the Fed's target."

Adding new variation to the preceding CPI charts, the Calculated Risk cpi chart covers the almost 22 year period from January 1990. All three representations of core inflation are tracking close to each other over the past year but have shown larger differences in the past.

http://cr4re.com/charts/charts.html?CPI#category=CPI&chart=InflationAugust2011.jpg

Update: 20 September 2011

From this chart, the uptick over the past year for all three measures looks quite sharp and of course we cannot tell what it will do next.

On the other hand, this 21+ year view also shows a clear downtrend in all three measures.which doesn't add up with my personal feelings about actual changes in inflation over that period of time. Remember from one of our previous charts, when we look at Headline CPI, we do not see a similar downward slope.

Of course, none of these measures directly shows the longer term cumulative impacts inflation and all three rule out aspects of inflation that actually impact ordinary citizens to end up with values that surely seem to be less than what American households really experiencing. My view is that these specialized metrics are at best a distraction from the main event: the cumulative inflation impact on each separate interesting sub-component of cpi. .

From today's series of CPI posts, we now have 5 different views of CPI

A year over year view from the BLS news release showing just the past 12 months for just headline CPI
An almost 12 year view of year over year inflation from Doug Short showing both headline and core cpi
An almost 22 year view of year over year inflation from Calculated Risk showing three different metrics that reflect core inflation
A 54+ year view of year over year inflation from Doug Short showing both headline and core cpi metrics
The cumulative effect of inflation since the year 2000 broken down by categories such as energy, health care, and college tuition.

These are all now assembled in successive posts in this blog so that we can look at and think about what story each variation might hold and switch back and forth between them to see what further insights we might glean.

What do you see?

What else do we need to look at?

How else must we transform the underlying CPI metrics to reveal even more important insights?

And recalling a recent post (Creating the context for successful analyses), what's the context for looking at this data in the first place. One place to start might be to take a look at the interesting link to the Cleveland Fed that Bill McBride provided in his post: Measuring Inflation.

A Unique View of CPI from Doug Short

2011-09-15T12:35:00.000-05:00

If you want to look at charts that help you understand what's going on in our economy, Doug Short's Updates is a great place to start. His charts are crisp, clear, concise and easy to understand and he provides a wide multidimensional view of many potentially inter-related metrics all launching from his Update landing page.

Here is one of his charts that gives a unique view to the CPI data that I have not seen expressed elsewhere and is worthy of emulation and further extension..

http://advisorperspectives.com/dshort/charts/inflation/headline-core-comps.html?inflation-since-2000.gif

In the previous two posts, we showed year over year data which is a traditional and well respected method for smoothing out the month to month variability. But as we all know, inflation is cumulative and so it makes sense to look at year over 2 years, year over 5 years, or year over 10 years changes and compare those to how our earning power has changed over that time period. E.g. for those on fixed income in retirement, the longer view translates more directly to the degree to which inflation will impact their lives. A 4 percent difference over the year will have a small effect. A 35% change over 10 years will have a huge impact for anyone whose income is not keeping up.

Doug's breakdown of the component changes is also highly revealing.

UPDATE 20 September 2011 at 3:00 PM

Of all the published charts I found online covering the latest release of CPI for August 2011, I found this chart from Doug Short to be by far the most valuable & useful by itself, while inspiring further questions encouraging the viewer to dig deeper. The big advantages of this chart compared to the others are that

it shows the cumulative impact of inflation over an almost 12 year period
it breaks down the overall cpi effects into a series of selected and interesting sub-components which show a wide range of cumulative change during this time period.
it reveals Doug's thinking about which of the subcomponents he things might be the most important to pay closer attention to (Energy, Medical Care, and Tuition)
And with a little thought it lets us get a hint about how the average headline cpi cumulative increase of 34.2% came about from a weighted average of the three (left most) componentss: a core cpi reading of 26.3% an energy reading of 124% and a food reading of 37.7%. To me this showed up the weakness of focusing on either the headline cpi number or the core cpi number whether by themselves or even when combined becasue doing so leaves the cumulative impact of energy and food invisible to the viewer.

For me, this chart inspired me to dig deeper because it was crystal clear that headline cpi value cannot be understood unless you get a feel for its key components while looking at the cumulative change. This letter eventually to the series of cumulative cpi charts we created in our subsequent posts:

A 20 year cumulative view of headline cpi from FRED - showing cumulative readings of headline CPI by itself for 5, 10, 15, and 20 years periods

Drilling down into CPI 20 year trends - showing a set of interesting sub-components and their 5-20 year cumulative history

Taking a longer view of CPI trends

2011-09-15T12:06:00.000-05:00

Doug Short has now posted some longer views of the Consumer Price Index CPI to complement those published by the Bureau of Labor Statistics (BLS). This provides additional context to help us think about what the latest numbers mean.

Here's his view of the data since 2000 with the shaded area from 1.75% to 2.00% representing the Core (CPI less food and energy) target rate for the Federal Reserve.

http://advisorperspectives.com/dshort/charts/inflation/headline-core-comps.html?CPI-headline-core-since-2000.gif

UPDATE 20 September 2011

With the benefit of this 12 year view, we can now put the 1 year view from BLS into better perspective. For example, we can see the headline value of 3.7% (in red) is now back up into it's 2004-2007 range following a 2 step process which looks like a reversion to mean process. Will it overshoot? we can't tell? We also see that the CORE CPI (in blue) looks like it is reverting to the mean of the past 12 years.

And here's Doug's really long view since 1957.

http://advisorperspectives.com/dshort/charts/inflation/headline-core-comps.html?CPI-headline-core.gif

UPDATE 20 September 2011

In this much longer view, some further nuances come into view including the obvious 3 peaks of inflation in 1970, 1975, and 1980, the much lower inflation rates from 1960 to 1965. We can also see how the Headline CPI and the Core CPI are more likely to diverge from each other after 1985. We also see the CORE rate showing a steady decline since 1990 that was not matched by the Headline CPI.

Doug also has some charts showing the PCE (Personal Consumption Expenditure) Index and the Core PCE Index as shown below since 2000.:

http://advisorperspectives.com/dshort/updates/CPI-Headline-and-Core.php

Check out Doug's original post for additional details.

We will keep our eyes open for other alternatives view of these metrics that get posted today.

CPI - year over year; not looking pretty

2011-09-15T10:03:00.000-05:00

The latest Bureau of Labor Statistics (BLS) CPI numbers are out this morning and they don't look pretty.

BLS August 2011 CPI news release

UPDATE 19 September 2011

With the benefit of the context we established in Providing some Context for looking at CPI charts we can now revisit this month's collection of CPI charts and see what stories they might have to tell. We can see there are some striking changes underway the past 9 months for both the headline CPI figure (blue line) and the Core CPI (red line).

But with such a short window, it's really hard to interpret what this chart is trying to tell us. Is this a significant inflation trend taking off and something of real concern, or is it more a matter of these metrics undergoing reversion to their mean values. We will be posting comments in chronological sequence for our other recent CPI posts that include charts over the next few days and perhaps we can better understand what this chart means once we have looked at the others.

We will be color coding our updates to distinguish them from the original post.

End Update

The all items index has increased by more than 3x over the past year
Excluding food and energy, the increase is more than 2x.

While this is a nice crisp clean easily understood chart, this view shows only the past 12 months and is way too short a window all by itself. Longer views would help put the latest numbers in better context and allow the development of a supportable storyline. But the BLS tradition is to go with their template approach using short 1 month and 1 year durations for most of their findings. For example, see the story lines shown below in the lead paragraph (emphasis added showing the small number of instances of longer views)

CONSUMER PRICE INDEX – AUGUST 2011

The Consumer Price Index for All Urban Consumers (CPI-U) increased 0.4 percent in August on a seasonally adjusted basis, the U.S. Bureau of Labor Statistics reported today.

Over the last 12 months, the all items index increased 3.8 percent before seasonal adjustment.

The seasonally adjusted increase in the all items index was broad-based, with continuing increases in the indexes for gasoline, food, shelter, and apparel.

The gasoline index rose for the 12th time in the last 14 months and led to a 1.2 percent increase in the energy index, while the food index rose 0.5 percent, its largest increase since March.

The index for all items less food and energy increased 0.2 percent in August, the same increase as the previous month. Shelter and apparel were the biggest contributors, though the indexes for most of its major components posted increases, including used cars and trucks, medical care, household furnishings and operations, recreation, tobacco, and personal care. The new vehicles index, unchanged for the second month in a row, was an exception.

The 12-month change in the all items index edged up to 3.8 percent after holding at 3.6 percent for three months, while the 12-month change for all items less food and energy reached 2.0 percent for the first time since November 2008. The energy index has risen 18.4 percent over the last year, while the food index has increased 4.6 percent.

We'll keep our eye out for other postings this morning with longer time line views which should help clarify the picture and help us evaluate just how serious the most recent 12 month behavior of CPI is likely to be.

Creating the context for successful analyses

2011-08-25T19:09:00.003-05:00

Context is essential.
If you just jump right into analysis of a complex data set and subsequent visualization and story telling without first establishing a proper context, you stand a good chance of misdirecting your focus, time, and energy.

If you are strong analyst, visualizer, and storyteller you can still end up with an interesting discovery, a good story, and exciting graphics to share. But you can have also missed hidden secrets that would have provided even greater understanding and value to your audience.

Forcing your audience to work harder. And worst of all, if you don't supply the context, you force your audience - every reader, viewer, listener, everyone who wants to interact with and learn from your visualization - to work harder with a lot of guess work and uncertainty to put what your findings into a useful perspective.

Templates shut down thinking. It is common to find examples of data analysis and visual reporting online with almost no context at all. For recurring analyses and reporting (such as various government monthly reports) it's also common to see the same basic template and boilerplate re-used verbatim month after month with no sign that any new or fresh thinking about how the context might have changed, or what had been learned in previous months, or how best to present that month's results for maximum clarity and ease of understanding.

Set the stage for discovery. What foundational context is it essential to establish in order to set the stage for the most successful exploratory analysis and discovery of new and surprising and useful domain trends, patterns, and exceptions?

Here are some ingredients that can help create a strong contextual foundation in a given data domain . These are especially important for recurring situations such as analyzing and reporting on employment/unemployment.

What's the central question? One of the best ways to supply context is to list up front the key questions that you hoped to answer as you started your analysis. Then in the storytelling and reporting that you create, make sure you establish a link back to these questions with any answers you have found, any surprises you discovered along the way, and any new questions you are keen to explore during the next round of analysis. In other words, show your thinking and link it back to the context your questions established.

Link to the mission. If the analysis you are doing is in support of an important mission, including a description of that mission and the vision and core values that the mission supports will add power and depth to the context of your work.

Think in advance of full set of key metrics. One other thing to note which we will return to in a future post is that there seems to be a connection between analyses that visualize the fewest domain metrics and the analyses that begin with the weakest contextual foundation.

What principles do you use to establish context for your data analysis/visualization/storytelling?

Showing a key metric with multiple views: a nice example

2011-08-25T18:57:00.007-05:00

Bill McBride's Calculated Risk blog has some crisp charts showing the latest new unemployment claims. The main chart shows this key metric since January 2000.

A second chart shows the same metric going all the way back to January 1971.

Both charts use a 4 week moving average to smooth out the more erratic week to week behavior. Bill's use of a dual chart approach helps present a much more complete picture of this important metric that puts recent behavior in context. Of course, even his "short" period is almost 11 years long so doesn't suffer from the common weakness of plotting too few data points.

Additional employment related charts showing other metrics and other views can be found in the Employment tab of Calculated Risk's Graph Gallery. Bill is prolific and posts some of the best looking, most unique charts related to economics and finance. Check out his gallery for yourself. You won't be disappointed

Despite these two excellent charts, one weakness I see in Calculated Risk's presentation of this important unemployment metric is that the verbal storytelling is weak. Bill's charts have potential explanatory power with important stories to tell, especially combined with the other charts in the Employment tab of the gallery, but these stories are left mostly as an exercise for the viewer.

In the blog post, the "story" told is mostly quotes from the dull boilerplate in the Department of Labor's UNEMPLOYMENT INSURANCE WEEKLY CLAIMS REPORT. This text discusses this metric with a very short term focus of only the preceding 4 weeks.

A second weakness is that the reporting (like almost all other reporting on the subject) only talks about and shows charts for this one Headline Initial Claims metric from the report while other complementary metrics are shunted aside. For example, some key missing metrics that are mentioned in the DOL report and whose short and long term time series charts could help us better understand the unemployment situation include:

insured unemployment rate - the percentage of "covered" workers collecting regular state benefits
insured unemployment - the number of people currently collecting regular state benefits
total persons claiming benefits in all programs

Some other metrics from other sources might also be added to the mix for fuller understanding such as:

total persons unemployed
percentage of total unemployed who are collecting benefits in all programs
total unemployed who are NOT collecting benefits

Note that Calculated Risk's Employment Tab does include these useful and complementary metrics shown in easy to digest graphic form but a story line to tie all these metrics together remains a challenge for another day.

headline unemployment percentage
employment population ratio
participation rate
number of workers who are part time for economic reasons
number unemployed for over 26 weeks
number unemployed for over 26 weeks as percentage of civilian labor force

What other employment related metrics would you like to see?

Do you know of others posting on the initial claims number who are crafting more complete stories than the standard laid down by the DOL report?

Reviewing the Situation

2011-08-22T17:12:00.010-05:00

It's been a while since we formulated the Trend Visualization Principles displayed in our blog's right hand panel. So we thought it was a good time to revisit them and update them to reflect both on what we have learned and on the rapid pace of change in the intersecting domains of data gathering, analysis, visualization, story telling, sharing, and collaboration.

Our revised seven principles are:

Context comes first
Create a history
Look at ALL the data
Share ALL your data
Explain your calculations
Show your thinking
Insure readability

Please check out the full statement in the right hand panel and let us know what you think.

These principles when put into practice can serve as antidotes to some of the weaknesses highlighted in the most recent post. (Common Weaknesses in Online Visualization & Storytelling)

We plan on posting some longer explanations of our thinking on these principles in coming days starting with two that are oft neglected in practice (self included) . Our experience has been that these two can reap substantial rewards when we make the time, when we take the time to remember to put them into action

Look at ALL the data
Show your thinking

What are your most useful principles that you keep in mind while gathering analyzing, visualizing, story telling, sharing, and collaborating about important trend data?

Common Weaknesses in Online Visualization & Storytelling

2011-08-19T18:30:00.001-05:00

Looking around the web we notice a growing number of examples of online visualization and storytelling.

Some of these are brilliant and incisive and easily accessible and digestible by their intended audience.

Too many however (including I am sure many of my own) exemplify one or more of a common set of weaknesses that make them harder to understand and that diminish their usefulness and value.

Here is a list of the shortcomings that appear most regularly. Once you can recognize them, all of these are correctable, often with only a modest effort that will pay big dividends. We've talked about many of these in previous posts and will no doubt return to them again to describe the particular details.

Can you think of any others? Which ones do you think are most important to correct?

Please share your thinking in the comments. Thanks.

Online Visualization and Storytelling Weaknesses and Shortcomings

Too short a time period shown
Too few metrics shown (sometimes only one or two out of thousands) and often only a single independent view of the key story telling metrics
No story presented - figuring out the story is left as an exercise for the audience/reader/viewer. This often goes hand in hand with visualizations that require a substantial time investment by the audience in order to discover messages that are not obvious at first glance. Or worse to spend time and not be able to figure out why that particular graphic was chosen from amongst all the choices available to the analyst/storyteller
Data set used to create the graphics is not readily available for further analysis by interested audience members
The larger data set used by the analyst/visualizer/storyteller is not available and not even defined or listed. Consequently the viewer has no idea of how much effort the storyteller put into the analysis before deciding to display a particular choice of graphical elements.
A standard template is re-used without any new or fresh thinking and without any sign of building on what's already been learned from previous analyses
Presenting only a single point in time for many metrics that change over time without providing the relevant time line view
Comparing just the most recent and the previous value of a particular metric without taking earlier values into account. This goes hand in hand with over use of graphs and tables showing month over month change.
When showing month over month change, failing to normalize the values to yearly percentages
Explaining time series behavior in dense text that is hard to parse and understand even for expert data analysts when a simple time series graphic would have done the job in seconds
Limited opportunities for further collaboration between the audience and those who created the visualizations and story line.
Too few data points in the time series
Use of large unsorted lists where some simple sorts and application of some variant of the 80/20 rule would have conveyed much more meaning in a much shorter time
Too may metrics all mushed together into a single indecipherable graphic. Such charts typically are ones that have no story line associated with them. What does the chart mean? You go figure it out!
Burying the lead (the potentially most interesting story element) so only audience members who invest significant time will ever have a chance to stumble across it. Everyone focuses on some headline number while the action is just a little bit below the surface and eager to see the light of day
Absence of comparisons of the result to useful baseline values
Working exclusively with the raw metrics as they arrive from their providers and missing out on opportunities to combine metrics to create calculated values that enhance the storytelling potential
Heavy emphasis on working with aggregated metrics (e.g headline numbers) and not showing whether the same patterns hold up under a variety of disaggregation approaches
Using widely varying raw metric values when a carefully selected simple moving average would have revealed greater insight
Overly tiny graphics that fail to take advantage of the full screen real estate available and make key elements more difficult to read and understand

What weaknesses would you love to see corrected?

Telling a Story with Time Series Data

2011-08-01T10:09:00.003-05:00

Once again, Barry Ritholtz over at The Big Picture shows his skills as a visual story teller in his recent post (Our Problem in Pictures) on the debt and deficit negotiations -- weaving together a series of charts produced using the powerful time series charting capabilities provided by the Federal Reserve Bank of St Louis (FRED)

Ola Rosling presentation of Gapminder at Google Headquarters

2007-07-24T19:41:00.000-05:00

Here's another excellent Gapminder presentation -- this time by Ola Rosling speaking at Google Headquarters back on March 7th, 2006. It runs approximately 1 hour and 10 minutes.

After seeing several examples of Prof. Hans Rosling taking the Gapminder tool through its paces, it's worth watching to see how the same tool in a different person's hands can produce a different and yet completely worthwhile storyline to help explain the data and bring out new details that no one has ever seen or noticed before.

The talk includes some nice examples of why the log scale is important as well as several sections that make the case for why being able to disaggregate turns out to be so important for understanding.

Along with many others who have checked out Gapminder, I personally am looking forward to the time when a new version will be available - ideally with access to a ton more of built in data as well as the possibility to allow the loading of new data sets.

Having experimented with the Gapminder World 2006 version, it is clear the possibilities of this approach are going to make a huge contribution to analysis of trend data and the creation of powerful narratives that weave those findings together. Because of its interactive nature, I am also certain that this tool will also prove to be a spring board for collaboration, for merging the best findings and story lines from many different analysts, each bringing their own unique perspective to bear on the data.

I think it is also highly likely that the next generation of Gapminder will also prove to be a tool that non-expert ordinary citizens will be able to use themselves to make sense of the world and to allow them to personally sanity check and validate the story lines of "the experts."

Some additional Gapminder tools for download are available at the Gapminder.org web site.

You won't be disappointed if you give Gapminder World 2006 a trial run. One nice feature of the tool is that as you make adjustments to what you are seeing on the screen by clicking on various options, these changes are reflected in current URL link value. If you create a situation that you personally find interesting, when you publish the URL, others will be able to begin their look at Gapminder with the way you had last set up its parameter options.

For example, check out this link to a customized Gapminder result (I recommend you do this in a new Tab or in a new browser window) where 4 countries have been selected, trailing tails have been added, the X and Y axis have been changed to represent Urban Population (linear scale) and Life Expectancy (linear scale) . We have reduced the size of each circle and zoomed in on both the X and Y axis to just contain the tracks of the 4 selected countries. We also slowed down the speed at which the trends will progress and adjusted the opacity setting for the countries not selected to a dimmer value.

The beauty of this is that when we share such a link, the recipient can actually see the last thing that we saw - what we thought important enough to capture.

They can then take that as their starting point to checkout any combination of other options - adding or removing countries, adjusting the speed, removing tails, changing to log scale on the X or Y axis, adjusting the opacity of the non-selected countries, zooming back out to a one hundred percent view of the data, or selecting new performance indicators for the x and y axis.

We recommend starting a new broswer tab or window because each time you select a new option, it is as if you had linked to a new web page, so if you make many changes, you will have to hit the back button many times to get back to your original launch page. Of course, this is also a feature as it lets you go back to previous version of option settings very easily.

S&P 500 - Comparing nominal vs inflation adjusted performance

2007-07-20T10:54:00.000-05:00

Hat tip again to Barry Ritholtz for this nice chart.

It's kind of interesting that almost all other economic metrics get adjusted (normalized) for inflation (e.g. GDP and hourly wages) but that we don't apply the same scaling adjustment to our stock market indices.

The long time scale for this chart makes it even more interesting. Note the devastation of the 1970s and how the inflation adjusted value has not yet recovered to its year 2000 peak.

Unveiling the Beauty of Statistics

2007-07-19T17:45:00.000-05:00

Jesse Robbins' excellent blog post - Unveiling the Beauty of Statistics - at O'Reilly Radar points us to another recent and Rosling video presentation - this time from the OECD World Forum in Istanbul. Unfortunately, the camera work is only a fraction as good as with the TED presentations and weather channel type presentation showing only Rosling speaking or showing a long view of the whole auditorium. In neither case is the exciting data that Rosling is talking about visible to the video viewer.

However, Rosling's message in this talk is still exactly on the money - a continued call
for more open and available data,
for better tools for visualizing it, and
for making it available to ordinary citizens around the world.

The chorus of voices echoing this message is rising. Web sites with visual tools such as Gapminder, Swivel and ManyEyes are leading the charge and changing the way we think about data.

Hans Rosling: GapCast #1 Health, Money & Sex in Sweden.

2007-07-18T20:00:00.000-05:00

Here's the latest from Hans Rosling's blog. His use of weather channel type technology for this video leads to much more vivid looking demonstrations.

This permits much more interaction between Rosling and the data as it changes over time and permits both Rosling and the data to be visible at all times. Compare it with the video capture difficulties that show up even in the very professionally cone TED talks where the camera switches back and forth between Rosling and the screen and where Rosling is not brightly lighted for fear of washing out the projection screen.

With the weather channel technology approach, both Rosling and the data are brilliant and vivid.

Definitely worth a look.

Hans Rosling: New insights on poverty and life around the world (video)

2007-07-18T19:50:00.000-05:00

Here's Hans Rosling's talk from TED 2007. Another home run from the man who brought you GapMinder.

Hans Rosling: Debunking third-world myths

2007-07-18T19:40:00.000-05:00

Don't miss this GapMinder talk by Hans Rosling at TED 2006.

20 minutes and well worth every minute you spend. GapMinder is a great interactive tool for looking and change over time. In the skilled hands of Professor Rosling, its richness and power come into full view. Don't miss it.

TED Talks Hans Rosling: Debunking third-world myths with the best stats you've ever seen (video)

Graph Showing Public Opinion During World War II

2007-07-18T19:30:00.000-05:00

Hat tip to Josh Marshall over at TPM for his blog post with the link to this excellent, easy to read, hand drawn trend graphic showing public opinion during World War II.

This chart shows the complete time range and multiple relevant factors in enough detail that the trends for each factor can be readily discerned. The use of event labeling on the Top X axis adds to the value and usefulness of this chart.

White House Economic Statistics Briefing Room

2007-07-03T17:52:00.000-05:00

Here's a link to the White House Economic Statistics Briefing Room

Their purpose is to provide easy access to current Federal economic indicators drawn from a a number of different Federal agencies. Categories of data include:

This is a nice idea but the navigation, the selection of key performance indicators, and some of the presentational choices made for each chart leaves some room for improvement.

Navigation: For example the Employment link provides access to 8 metrics and one might imagine that someone interested getting a sense of how things are going with regard to employment might actually want to look at all 8.

To accomplish you click on the employment link which brings up a web page that lists the 8 factors but due to the layout of the page, it takes almost two full screens to see the eight factors. Each factor has a chart icon to click on so you can see the actual chart.

The icon is large enough in size so that it would have been big enough to give a useful small size version of the chart. Amazingly, when you click on the icon, instead of getting the full size version of the chart, you get an intermediate screen that tells you that you are exiting the White House Server. After a 5+ second delay you are redirected to the chart.

So to look at all 8 charts that were picked to represent the most important employment related factors, you have to navigate up back and forth with the redirection delay in the middle. Not a lot of fun.

If you want to look at all the key indicators from all the categories, there is a whole lot of clicking and mousing and redirect delays ahead of you.

Contrast this to the St. Louis Fed approach of combining all the key indicators at a given point in time into a single PDF document. My take is that the PDF format is much kinder to the prospective reader/viewer and that it is much more likely to be examined and used.

It would be great if there were an option to download a single PDF or PowerPoint document that had all the charts from all the different categories so that the viewer could simply scroll through each chart and thereby obtain a composite, gestalt view of the state of the economy.

St Louis Fed July 2007 National Economic Trends

2007-07-03T08:39:00.000-05:00

The latest collection of charts from the St. Louis Fed highlighting National Economic Trends where they continue in their tradition of presenting both extended time periods and shorter intervals. This easy to download document is a good example of how it is possible to present high degrees of multi-dimensionality covering a wide range of the most important indicators, all in a single, easy to use, easy to read format.

Homicide rate since Ike

2007-07-01T16:10:00.000-05:00

Nice chart from Angry Bear (hat tip Andrew Sullivan)

"It's still too early" or is it?

2007-06-15T13:45:00.000-05:00

Here's a link to a June 13th DoD news article announcing the quarterly report to Congress on Iraq DefenseLink News Article: Quarterly Iraq Report Cites Progress, Challenges

It leads off with [emphasis added]:

It’s still too early to assess the impact of the new strategy in Iraq, but more progress is expected as additional troops come on line to boost security in Baghdad, according to the latest quarterly report to Congress, released today.

The executive summary of the quarterly report itself includes:

It is too soon to assess results.

Overall, it is too early to assess the impact of the new approach.

Although it is premature to judge whether FAQ is creating the conditions necessary for political reconciliation, ...

Just to make sure we didn't miss the point, the next day, June 14th, a new article headlined Too Early to Discern Trends in Iraq appeared and opened with paragraph 1

It is too early to discern trends out of the U.S. troop “surge” as part of the Baghdad security plan, defense officials said here today.

and continued in paragraph 3 saying

“It is too early to assess the impact of the new way forward,” a senior defense official said.

This is of a piece with the widespread and repeated propagation of the we have to wait till September meme that seems intended to deaden our ability to judge the facts so far for ourselves.

All that's needed to assess, to judge, to discern the impact of what has happened up until the present moment is to look at the freshest possible data on all the important factors. If we each available ourselves of the weekly sources of such data such as the Brookings Report, the State Dept. Report and the Baghdad Embassy report, the we can then have a fruitful discussion about what each of us thinks the data means and what we feel might be the best course forward.

On the other hand, if we listen to the siren call of the "too early to tell" and "wait till September" crowd, we will forfeit any chance we have of adjusting our behavior to increase our chances of success as we move forward.

Iraq Transition Assistance Office - Weekly Status Repot - May 29, 2007

2007-06-15T13:10:00.000-05:00

The Iraq Transition Assistance Office (part of the US Embassy in Baghdad) is now publishing a Weekly Status Report. Here's a link to the May 29th, 2007 report.

This is a continuation of the excellent work Iraq Reconstruction Management Office under a new name. This weekly report rates high on both graphical interest and on providing fresh data with an excellent score for near real time reporting.

Check out the important indicators on page 23. While the tabular format is not for everyone, there are some important indicators in this table that are worth paying attention to.

Dept of State - Iraq Weekly Status Report June 13 2007

2007-06-15T12:51:00.000-05:00

Here's the link to the latest Dept of State Iraq Weekly Status Report (June 13, 2007). This report has some much fresher (more recent) charts on oil and electricity than the quarterly DoD report. This report successfully achieves an excellent grade from delivering its data in near real time fashion.

For those interested in keeping up with what's happening on the ground in Iraq, this weekly report is one of the essential pieces to pay attention to.

Ten Trend Charts from the DoD Quarterly Report

2007-06-14T19:53:00.000-05:00

The Dept of Defense has just published its mandated quarterly report to Congress on the situation in Iraq - Measuring Stability and Security in Iraq, June 2007 (51 pages). As they state in the Executive Summary, the "report includes specific performance indicators and measures of progress toward political, economic, and security stability in Iraq"

Included as part of the report are 10 static trend graphs which I have reproduced in this post. You can click on each for a full size image. If you would like all 10 JPG images so you can look at full size with your favorite slide viewer, you can download this zip file.

Most of the important performance indicators mentioned in the report did not rate their own static trend graph chart. As we did with the UN document and our scan of recent news articles, we plan to comb through the document and come up with a list of indicators that complement the ones we have already highlighted so as to get a sense of the breadth of important indicators that are already on our radar screen.

Chart 01 - Oil Production - too short a time span. Also missing data for May 2007.

Chart 02 Electricity - Missing data for April and May 2007.

Chart 03 - Sectarian Murders and Incidents . Missing data from May 2007. Weekly data might be important to look at.

Chart 04 - Weapons Caches Found - This is looking promising. Access to weekly and daily data could be even more revealing of ongoing trends. Missing data for May 2007.

Chart 05 - Average Weekly Attacks - Missing last 5 weeks of data. This is the source for the charts that appeared in WaPo and NYT as reported earlier today.

Chart 06 - Average Daily Casualties - Missing last 5 weeks. Bar chart format with three trends makes reading trends for Iraqi Security Forces and Coalition more difficult.

Chart 07 - Hotline Tips - Interesting new performance metric. Looks positive. Missing past month of data. Weekly or daily data reporting could be even more revealing.

Chart 08 - Confidence in Iraqi Government to Improve the Situation. This looks surprisingly to me for its stability over time. I will need to read the text about this in detail. Missing most recent month's data.

Chart 09 - Assessed Capabilities of Police - Difficult to interpret trends when only two data points per item. Stacked bars make interpretation more difficult and time consuming. This is an interesting new metric and it could benefit from higher resolution, near real time reporting.

Chart 10 - Ministry of Defense Forces Assessed Capabilities. This chart also presents some important data but suffers from same failings as previous chart - only 2 data points per indicator, stacked bars are hard to read.

Summary. 10 Static charts covering a range of important indicators is more than almost anyone else except Michael O'Hanlon produces when it comes to highlighting the important trends evolving in Iraq.

So that's definite plus. Someone who spends a few minutes can examine all 10 of these (especially if they can scroll in slide show mode on full size images using this zip file).

On the minus side, the missing last month or last 5 weeks of data is a serious drawback. Many of the charts can be improved on and made more readily useful to ordinary viewers by applying some trend chart best practices principles. These include

use of full time range
use of near real time reporting for the most recent interval
selection of best resolution or choice of several different reporting intervals (day, week, month)
displaying each important indicator separately with appropriate Y axis scale
having sufficient data points for trends to be discernible - 2 is way too few

And worst of all, most of the important indicators have no charts at all and are only discussed in text or in tables showing the current values.

The indicators in this report are important, but this method of presentation falls short of ideal and short of what is needed and what indeed is possible. With the high number of factors discussed in this report, the only method that will actually make this kind of trend data usable to its intended primary audience (Congress) and to the secondary audiences including the media and ordinary interested citizens would be to deliver this data via a 21st century Trend Visualization Appliance.

What's the Best Reporting Interval?

2007-06-14T19:20:00.000-05:00

Why Wait for a Quarterly Report? Yesterday's DoD report to Congress is a mandated quarterly report. The executive summary begins with:

This report to Congress, Measuring Stability and Security in Iraq, is submitted pursuant to Section 9010 of the Department of Defense Appropriations Act 2007, Public Law 109-289. The report includes specific performance indicators and measures of progress toward political, economic, and security stability in Iraq, as directed in that legislation. [emphasis added]

My view: This report can do double duty. Quarterly reports are all well and good. They present us with specific performance indicators and measures of progress. They offer an opportunity to take a step back and look at the evolving patterns.

AND between reports, they can be even more helpful on a daily, weekly and monthly basis.

Just as we don't have to wait till July or September to find out how well the "surge" strategy is working, we don't have to hold our breath between quarterly reports. We can make double duty of the infrastructure already in place to produce these reports to create a mechanism that lets us keep track of the most important indicators and measures of progress as we go along.

Here's How.

Start where we are at. The quarterly report defines the currently understood list of the most important performance indicators and measures of progress.
High Frequency Measurement. For each of these important indicators we establish a measurement frequency that is as rapid as is practical at this time.
Near Real Time Reporting. For each of these important indicators we also establish a near time reporting interval that is as close as is practical to the time of the most recent measurement sample
21st Century Visualization. We then make the full set of high resolution trend data for all the most important performance indicators and measures of progress available. We make it available in a continuously updated form in near real time using a 21st century Trend Visualization Appliance. This makes it possible and manageable to observe the trends of a large number of factors within the time and resource constraints of the intended audiences.
Improve Frequency. For any important factor where it proves impractical today to measure at the frequency that we think best, we set in motion a process to speed up that frequency as quickly as we can bring it on line.
Improve Freshness. For any important factor where it proves impractical today to deliver the most recent measurements in suitable near real time fashion that would meet our needs for fresh data, we set in motion a process to reduce the delay as quickly as we can.
Identify and Add Missing Indicators. Finally, as soon as we notice some new and important indicator that is vital to understanding progress in Iraq, we add it to the list of indicators to be included in the Quarterly report AND to be delivered to our Trend Visualization Appliance at high resolution and in near real-time for our consideration between these quarterly reports.

Shorter Version:
If there is a performance indicator that is important enough to be reported on quarterly, then it is even more important to make the underlying trends for that indicator available interactively and efficiently, at high resolution, and in near real time for all interested parties.

The current arrangement (infrequent reports, delayed reporting, lower frequency data) does not appear conducive to the most effective oversight by congress, the media or the general public. We can and must do better if we want to make sense of what is happening in Iraq.

Near Real Time Reporting

2007-06-14T18:27:00.000-05:00

For most important situations in life where trend visualization is helpful, we are lucky that real time reporting is not required. However, collecting the data and reporting it within a reasonable time from the end of measurement is important and can deeply impact the value and usefulness of what we have collected.

An Unacceptable Delay. We noted earlier today in the post on the New York Times Iraq Progress Graphic how the underlying data from the DoD quarterly report to Congress only covered up through May 4th. So in effect we were looking at data that wasn't produced in a report till 5 and 1/2 weeks after the end of the most recently reported period.

In my opinion, this represents an unacceptable delay and that delay severely impacts the ability of any audience to gauge what the data means and to decide what to do next.

If the underlying data for attacks and for casualties were collected at the recommended rate of once per day, a near real time report on June 13th would be best if it covered everything that happened up through one, two or three days prior to June 13th.

Fresh Data. The combination of high frequency measurement and near real time reporting means that the data we are looking at is fresh - a vital quality in any rapidly changing situation. There are numerous examples in other aspects of our lives where near real time reporting and high resolution reporting are simply the norm (the stock market being a prime example).

There are sometimes situations that delay and interfere with our ability to roll up the most recent results of a particular indicator. In some cases, these are historical and can be overcome and minimized if we decide to put our minds to it and invest in the needed infrastructure to make it happen. In other cases, there are time lags that are mostly out of our control. The best we can do in these cases may be to put our energy towards devising ways to minimize the delay.

Bottom line: In a trend chart that is going to help us understand what is truly going on, timeliness is very important. When we get our measurement rate set appropriately and we make every effort to minimize the delay between collection and reporting, the trend work that we do will have the maximum value to all those who are interested.

Frequency of Reporting

2007-06-14T18:10:00.000-05:00

In the previous post, we brought up the concept of frequency of reporting on a given metric. We haven't talked much about this aspect of trend analysis so far, despite the fact that it can be quite important. And it interacts with another important topic we brought up in some previous posts: near real time reporting.

We'll take a look briefly at frequency in this post, and give a brief look at near real time reporting in the next.

Error on the Side of Too Often.
Each important factor that you wish to measure will prove most useful in the end if you make a conscious choice to consider its frequency of measurement separately on the merits. My general operational rule is wherever possible to err on the side of measuring too often rather than not often enough.

Measurement vs. Reporting Frequency. When you measure at a relatively high frequency, it's possible later to report on the data at either the high collection frequency, or at a reduced frequency in order to smooth out the trend line pattern. On the other hand, if you don't measure frequently enough, you may miss out on short-lived quantum shifts in behavior that will be lost when reporting at longer intervals.

It's unfortunately not uncommon for vital indicators that are already being collected to have their frequency set so that measurements are too far apart.

Measuring Daily is Often a Good Starting Point. Let's take a specific example: the percentage of eligible children who attend primary school. Ideally, in my opinion, a metric such as this would be best if it were measured daily. There will be times when you will want to look at the day to day trend and see if any disruptive multi-day even has shown up. You may also want to understand any patterns that might be dependent on the day of the week. There will be other times when week by week reporting will be just right and still others when month by month will give the clearest view. Before you actually have the data, you cannot really know which reporting interval will prove most revealing.

My recommendations for the important Iraq metrics of the kind we have been scraping from the recent news reporting is that we make every effort to measure them with a frequency of once per day wherever that is possible.

When is Higher Frequency Helpful? There are some indicators where it would be advantageous to measure them even more frequently than once a day. For example, electricity delivery in Baghdad is a factor that I would like to measure minute by minute during the day and to do this both for the city as a whole as well as disaggregating these results by districts in the city and/or by individual power stations. Unless I get a very high resolution frequency, I won't be able to understand the frequency of or length or breadth of districts impacted by each outage. Such diagnostic capability is likely vital to figuring out where the est opportunities are for achieving higher levels of service and moving closer to fulfilling the desired demand.

Obstacles to High Frequency. On the other hand, there will be metrics such as the percentage of Iraqi members of Parliament who are living abroad where the obstacles to measuring may mean that we only get a read out on this indicator once a month. It might be nice to see how this varies day to day, but prove to be logistically impractical to get it.

Adjusting as you go along. There is no perfect frequency and there can likely be differences of opinion as to which frequency is best. The good news is that frequency can be adjusted as needed as we go along. The really important part is identifying all the important metrics and beginning the process to measure and record their values as they change over time.

Scraping Metrics - Part III: UN Report

2007-06-14T16:50:00.000-05:00

As we mentioned in an earlier post, UN Secretary General Ban Ki-moon has recently published a quarterly report detailing the situation in Iraq from the UN's point of view.

This post continues with the idea that the news around us give us ideas that help answer the following questions:

What are the most important indicators?
What we must measure to form a fuller picture how things are going in Iraq?

When tackling a task like this, the idea is to be on the lookout for important factors that we haven't yet included in our growing composite list.

The Secretary General's report is a rich source for these kinds of ideas. Some of the measurable indicators that come up are named directly. In other cases, we use an idea expressed in the report as a springboard for creating a specific, measurable entity that would at least partially shed some light on how well things were going in that general area.

Thousands of Words and Nary a Picture. The report itself does not include a single trend graph which is a disappointment as the UN has a unique perspective and likely has access to key data that may not be available elsewhere. The report relies on the spoken word for conveying information about trends and therefore falls short of what its potential might have been.

Here then is the list of important UN indicators. Many of these can be profitably subject to disaggregation into Shia, Sunni, Kurd, or by location in Iraq.

Number of kidnappings
Oil revenues - how are these being divided today, prior to the passage of new oil revenue sharing law (by region, by ethnic category, by different government agencies)
Millions of dollars of new loans secured
Number of newly displaced Iraqis inside Iraq
Number of newly displace Iraqis who have fled to neighboring countries
Millions of dollars of financial assistance provided to internally displaced Iraqis
Millions of dollars of financial assistance provide to Iraqis who have fled to other countries
Number of Iraqis registered to vote
Number of law enforcement personnel charged with serious human rights violations
Number of attacks on journalists or media persons
Number of casualties (wounded) registered in hospitals
Number of families losing their breadwinner to death or incapacitating injury
Number of family members losing their breadwinner
Percentage of eligible children attending school (primary, secondary) (male, female)
Number of students attending college (male, female)
Number of college graduates
Number of advanced degrees awarded
Number of doctors completing their training
Number of nurses completing their training
Millions of dollars of Iraqi assistance to displaced persons
Percentage of population with safe water supplies
Percentage of displace persons having access to the public
Percentage of malnourished children
Additional contributions to the United Nations Development Group Iraq Trust Fund
Number of attacks in the International Zone (Green Zone)
Number of incoming rounds of mortars, rockets in the Green Zone
Number of deaths and injuries in the Green Zone from attacks
Number of attacks on strategic targets - e.g. bridges, major thoroughfares, checkpoints near the Green Zone, Parliament building
Number of attacks on infrastructure for electricity, water, oil, sewerage
Number of neighborhoods in Baghdad with new security outputs
Number of attacks on new security outposts
Percentage of buildings in Green Zone with hardened overhead protection capable of withstanding attack from large scale ordnance such as 107-mm and 122-mm rockets.

Imagine how much better we would understand what is actually happening in Iraq if we had a trend history and regular, moderately frequent, near real time updates for each of these factors inspired by the Secretary General's report.

Metrics inspired by the news part II - Prison Metrics

2007-06-14T13:16:00.000-05:00

Here are some additional metrics inspired by recent news from Iraq.

This batch is based on the seed idea that what is happening in the prisons in Iraq today can shed important light that would otherwise not be visible.

Current prison population under Coalition control (male, female)
Current prison population under Iraqi patrol (male, female)
Number in prison who have not yet been charged
Number in prison who have been charged
Number in prison currently under trial
Number convicted
Number found not guilty and released
Average time from arrest to being charged
Average time from arrest to trial completion
Percentage of prison population who are considered insurgents
Prison population (Shia, Sunni, Kurd, other)
Number arrested per day
Number freed per day
Number in prison who are under the age of 18
Prison population as a percentage of maximum rated capacity
Number of deaths while in prison
Number of suicides of inmates
Percentage of prisoners in solitary confinement
Prison population disaggregated by prison location

I am not an expert on prisons, but it would seem that if we could track the indicators above, we could begin to get a sense of how this aspect of life in Iraq today was evolving. Ideally, those who are expert in understanding prisons can add their favorite and most important metrics to the mix if I have not included them.

Bridging Rep. Kingston and Michael O'Hanlon's Concerns

2007-06-14T12:45:00.000-05:00

In the two previous posts, we have spoken to the concerns of Rep Kingston and Michael O'Hanlon as captured in the Sheryl Stolberg New York Times See You in September ... article.

To be successful in understanding what is happening in Iraq, we need to listen to both their voices.

A Sufficient Number of Factors. We need to pay heed to O'Hanlon and make sure we have looked at enough different factors.

Manageable Metrics. We have to hear Rep. Kingston's concern and worry about being overwhelmed with too much data or rendered ineffective by not having enough time by making whatever data we report on "Manageable" within the constraints of the time and resources that are available to us.

Eating our Cake. It is possible to meet both of these goals. We are not doing it today, but we are surrounded daily with visible proofs that equally difficult trending situations (see the St. Louis Fed Reserve FRED application as an example of what's possible) can be brought within our grasp and control by the judicious application of 21st century Trend Visualization methods.

Along the Way. Paying attention to our view of the best practices and principles of trend graphic excellence will provide the third ingredient that helps make our efforts in this arena more effective and more readily sharable with others. For a discussion of these, check out the Trend Visualization Principles displayed near the top of the right column of this blog or follow this link to trend principles for a review of some of our other thoughts on this subject.

Dangerous to have too few metrics or the wrong metrics

2007-06-14T12:20:00.000-05:00

In a previous post: Revisiting "See You In September" I noted:

There is a way forward that addresses Michael O'Hanlon's concerns

The relevant quotes from O'Hanlon were:

"metrics are grist for a fact based debate but history shows it is dangerous to rely on too few of them"
"metrics were used in Vietnam and we had the wrong ones"
[the Vietnam metrics] "did net harm to the debate"
"we can’t be exactly precise about which indicators are the conclusive ones”

Here are a few observations related to the above.

Dangerous to Rely on Too Few of Them. I agree entirely. For Iraq, we need to be able to look at at least 50-100 of the most important factors efficiently, all within a relatively small period of time. For expert use, a much higher number of metrics must be available for review in an efficient and effective manner. Michael O'Hanlon's weekly Iraq Index is one of the best examples out there that pays attention to the principle of not relying on too few metrics.

The Metrics were the Wrong Ones. This is a solvable challenge this time around. First, follow the principle above and make sure that we look at ALL of the important factors. How do we decide? We ask all the experts in all the different areas (security, economy, health, ...) to name what they consider to be the most important indicators. Then we capture and archive and report on all the factors that have been suggested, even if only once. Maybe there will be some "wrong ones" in the mix, but when we have the full set of data to work with, the chances of being led astray will be markedly reduced.

Can’t be exactly precise about which indicators are the conclusive ones. We don't have to be precise. Different people will have different opinions. If we capture all of them, record a history of each trend, and make this data available in Static Graphical format, AND as machine readable data suitable for input into a Trend Visualization application, then we can begin to have good and constructive discussions about what different trends mean and what we should do about them.

A "Manageable" Number of Indicators

2007-06-14T12:00:00.000-05:00

In a previous post: Revisiting "See You In September" based on the Sheryl Stolberg New York Times article I noted:

There is a way forward that ... answers Rep. Kingston's call for assistance.

According to Stolberg, "some in Washington are grasping for a more complete and accurate way to quantify progress." “No one knows how to define progress in such a mixed-up situation.” Others say “we’re having trouble measuring it.

Rep Kingston (R-Georgia) wanted to "winnow down the indicators to a manageable number -- say, fewer than a dozen." He wanted to create "a standard bi-partisan metric." With such a metric in place, "then you could say who’s winning and losing.”

The key words here are: "manageable number" and the key idea is that the medium we chose to publish these numbers matters to a huge degree. The thesis of this post is that what constitutes "manageable" depends on how we chose to present the data.

Using Unquantified Indicators. If we don't get down and get numeric with the metrics we are interested in, then the manageable number doesn't matter.

Talking about Trends. If all we do is talk about numeric trend data, then maybe the manageable number is only 1 or 2 and even with that small a number of items, most listeners won't end up with a good sense of of how those two indicators are changing over time. Speech is simply not a good way to deal with understanding trend data. And because so few data points for each trend can be presented in this fashion, there is also tremendous opportunity for slanting and spinning the meaning.

Writing about Trends. If we chose to write about it, the manageable number may be 3 to 5 metrics. With written trend information, most readers will not get a full sense or understanding of how the situation is evolving. The written word and written number is not an efficient or an effective way of handling life's most important trends. As is the case with talking about trends, writing about trends normally means that very few data points are actually given for any of the important indicators. Sometimes no value is given at all, only some form of imprecise quantification. Sometimes, only a single value is provided. Sometimes only a pair of values or a single value and a comparison to a previous value. [Take a look at today's articles reporting on the DoD quarterly report to Congress for crystal clear examples of what I am talking about here.]

Documenting Trends with Printed Tables. If we decide to put the information in a printed table of data, then the manageable number may grow to 7-10 factors. This is a step forward. However, only a handful of our citizens will be comfortable with getting their trend information in this fashion and making sense of it. And for most of those included in that handful, they will rarely actually have the time to study the table in sufficient detail to actually get a sense of the trends. For those few with the skill, the interest and the substantial time required, this format will begin to allow some understanding of the trends at work. However, for most of the population, the table format will be close to unusable.

A perfect example of the difficulty of both the written word and printed tables is today's Producer Price Indexes report from the Bureau of Labor Statistics.

Static Trend Charts. If we create a set of static trend charts for each selected metric, then the manageable number can easily jump up to 10 or more and if these trend charts follow best practices, most ordinary citizens will be able to make good sense of them and gain an understanding of how the situation is progressing. Of course, if such charts violate the best practices, the utility of this approach drops off dramatically.

Interactive Trend Charts. If we have access to the trend data via a human engineered interactive trend viewing station, then it is possible for ordinary mortals to deal with dozens and dozens of factors and indicators and gain a deeper gestalt understanding of the situation, all in a reasonable amount of time. With really good interactive tools, examining and learning from hundreds of metrics is within reach.

What constitutes a Manageable Number of Metrics? It depends on the approach we take for representing the trend data.

Since the number of factors we musts look at to gain full insight and understanding into the situation on the ground in Iraq is so high, unless we begin to move towards interactive charting, our chances of success look quite slim indeed.

Right now, almost all the official and unofficial reporting of quantified results is weighted heavily towards talking, writing, and printed tables of data. Those who produce static charts are relatively few and even though we can handle maybe as many as 10 well constructed static charts at a time, the norm is to have only 1 or two as is the case with today's WaPo and NYT articles we posted about earlier. All the rest of reporting will be written, and most of the indicators will not have any numeric value associated with them all.

The Sheryl Stolberg article on the surge also included this telling quote:

"Some say measuring progress is simple: you will know it when you see it."

I agree. This is really the answer to Rep. Kingston's request. We'll know it when we see it, but we will only see it if we show it. And we will only show it if we make the effort to move beyond the current default mode of talking about it and writing about it.

For whatever trend data we gather to be useful, we need to make sure that the manageable number of indicators is actually large enough for us to be able to use that data to deal with the problems that confront us. My own estimate of how many this might be for Iraq is at a minimum in the range of 50 t0 100 of the most important factors for general use and discussion with a much higher number available for expert use on specific focused issues.

NY Times charts based on DoD report

2007-06-14T11:17:00.000-05:00

Here's the New York Times graphic that accompanied their story this morning on the DoD report to Congress.

Their stacked bar chart is similar to the unstacked bar chart from the Washington post.

Geographic charts are popular these days but I find the pair of charts in this example difficult to interpret. There are some trends going on with respect to readiness to assume security responsibility, but one has to spend quite a bit of time to figure out what they are from these maps. The eye has to dart back and forth between the two maps to make a before and after comparison. If I am reading these correctly, it looks like some serious backsliding has occurred since Nov 2006. And of course, having only a pair of data points for each trend makes any interpretations or extrapolations risky.

The sectarian incident chart at the top right is useful. I would prefer to see a longer time frame and data that is more up to the minute (e.g. at least through the end of May 2007).

The first chart showing the daily casualties also falls short in only reporting data up through May 4th, 2007. While real time reporting is rarely needed in this kind of situation, important factors such as casualties must be reported in near real time for them to be truly useful.

Metric Ideas culled from the news

2007-06-14T09:25:00.000-05:00

Here is a sampling of some important metric ideas culled from news stories over the past few days. The news stories themselves may not have actually called out the specific metrics listed. Rather they reported on some aspect that might have bearing to how well or poorly things were going on the ground in Iraq.

For example, I came across an article mentioning the idea that many Iraqi parliament members live abroad and do not physically show up for sessions. That led me to craft the first 4 metrics that might give us some sense at least of Parliament activity even if we cannot easily or directly measure their progress towards oil-revenue sharing, de-Baathification and so on.

Number of members of Iraqi parliament currently living aboard
Number of hours of parliament sessions per day
Number of parliament members attending each day
Number of parliament votes each day
Number of sectarian death squad killings
Number of sectarian attacks
Number of bodies in the street in cities like Baghdad
Number of attacks in al-Anbar province
Violence in Diyala Provinces (deaths, injuries, attacks)
Violence in Ninevah Provinces (deaths, injuries, attacks)
Number of suicide bombings
Number of suicide attacks
Number of Baghdad neighborhoods protected by local militias - Shia, Sunni, mixed
Number of Baghdad neighborhoods protected by coalition, Iraqi military, Iraqi police
Number of Baghdad neighborhoods not protected
Percentage of Iraqis in favor of partition of the country by Shia, Sunni, Kurd, other
Number and % of violent deaths come from people just being shot down
Number and % of violent deaths violent deaths from US military activities
Number and % of deaths from bombing Iraqi cities (deaths from aerial strikes)
US troop level in Baghdad
Baghdad morgue data on bodies received per day
# of attacks on Sunni Arabs by the predominantly Shiite government security forces
number of Iraqis displaced from their homes per month
Number of militias currently known
Percentage of those militias that have been disarmed
Current militia troop levels
Inflation rate
Oil production rate in Millions of Barrels per day
Degree to which demand for electricity outstrips supply

For some of these, we have already suggested additional metrics based on disaggregation (e.g. by reporting on totals as well as breaking down the results by ethnic groups). Other metrics listed above are also aggregates and could similarly and beneficially be disaggregated for additional insights.

In my view, all these are important and yet even though we have listed 29 indicators, this is likely just scratching the surface.

If we wish to make sense of what's going on in Iraq, we really need to be able to literally SEE the trends in everyone of these. We're struggling right now since this is just not happening.

Measuring Stability and Security In Iraq: June 2007 DoD quarterly report to Congress

2007-06-14T07:56:00.000-05:00

Here's the link to the just published Dept of Defense quarterly report to Congress on the situation in Iraq.

It contains many interesting trend charts which we will be discussing in future posts..

Unabated Violence - washingtonpost.com

2007-06-14T07:39:00.000-05:00

Here's an important and easy to read graphic from this morning's Washington Post showing the impact of the surge so far on the number of reported violent attacks on civilians, Iraqi security forces and coalition forces.

Unabated Violence - washingtonpost.com

My preference for data like this would be to make the three separate trends individually viewable. The patterns for civilians and Iraqi forces are easy to miss because the the much reduced scale when all data is presented on the same chart.

See the accompanying WaPo article- No Drop in Iraq Violence Seen Since Troop Buildup for further details.

UN report on surge progress

2007-06-12T12:35:00.000-05:00

Juan Cole reports today in Informed Comment on the UN Secretary General Ban Ki-moon's quarterly report on Iraq.

Several things to note:

a) substantial use of important metrics with numeric quantification (see snippets below). There is also a sense that some effort was invested in sorting out which factors and indicators were the important ones relative to humanitarian concerns. Contrast this to the lack of precision or quantification with the 18 benchmarks, the See you in September article, etc.

b) much wide perspective and more holistic view on what's important to pay attention to in Iraq (beyond the 18 benchmarks).

33. ... related to a humanitarian response. For every death reported in the news, six family members on average are left without a breadwinner. The rising number of displaced persons is also a cause for concern. UNHCR estimates that displacement has continued at an undiminished pace and over 800,000 Iraqis have been internally displaced since the Samarra mosque bombing in February 2006, while 30,000 to 50,000 flee to neighbouring countries each month.

34. The violence is also having a major impact on Iraqi children and their ability to attend school. ... estimated that 17 per cent of primary school-age children were not attending school in 2005 and 2006. This translates into approximately 765,000 children, of whom 61 per cent were girls, even before the recent escalation in the numbers of refugees and internally displaced persons. Dropout rates are also increasingly outstripping school participation. Only 34 per cent of girls and 43 per cent of boys of secondary school age were attending secondary school in 2005 and 2006. . .

...

46. ... increased threat of indirect fire into the International Zone. ... The International Zone experienced 17 attacks in March, 30 in April and 39 by 22 May alone. Since 19 February, indirect fire attacks have reportedly resulted in the deaths of 26 people in the International Zone. The security situation has been further compounded by the increase in car bombs in the vicinity of entry checkpoints to the International Zone. Armed groups operating in Baghdad have demonstrated their ability to strike at well-protected, strategic targets, such as the suicide bombing inside the Parliament building on 12 April. . .

...

60. Iraq’s political and social fabric continued to come under considerable strain during the reporting period as a result of ongoing political, sectarian and criminal violence. Despite the efforts of the Iraqi and multinational security forces to stem violence, progress was slower than had been hoped when security initiatives were launched at the start of 2007. This has been demonstrated by continued attacks on the civilian population, physical infrastructure and political institutions such as the Council of Representatives. . .

This is a small step in the right direction in my opinion.

Language and the 18 benchmarks

2007-06-12T12:15:00.000-05:00

Here's a link containing the language of the 18 "benchmarks" included in the recent Iraq Supplemental Bill HR 2206

Of the 18, most are not readily amenable to quantification. The short list below are 5 of the more quantifiable targets contained in these benchmarks.

More on this in future posts. For now, the supplemental bill appears to fall afoul of the same language difficulties we have been noting in various articles from around the web - namely mushy metrics, imprecise or missing quantification. If this is the document that is going to guide our evaluation of how things are really going in Iraq, we are in deep trouble as it appears to be structured in ways that are likely to defy exactly that process.

3) equitable distribution of hydrocarbon resources

9) Providing three trained and ready Iraqi brigades to support Baghdad operations.

13) Reducing the level of sectarian violence in Iraq and eliminating militia control of local security.

14) Establishing all of the planned joint security stations in neighborhoods across Baghdad.

15) Increasing the number of Iraqi security forces units capable of operating independently.

17) Allocating and spending $10 billion in Iraqi revenues for reconstruction projects, including delivery of essential services, on an equitable basis.

However, these 5 are amenable to quantification and we could begin now (rather than waiting till September) to make these specifics, to measure them regularly, and to report on them in a way that will be easily accessible to all.

When you look at the full text of benchmarks and at the required reporting standards and timing, it's noteworthy that there are no explicit benchmarks for what the United States is required to do in this time frame. Clearly the situation in Iraq will be dependent both on what the Iraqi government does and what the United States does, but this particular bill places all the responsibility on Iraqi shoulders.

GAO May 15 Rebuilding Iraq Report

2007-06-12T11:37:00.000-05:00

Here's a link to the recent General Accounting Office Rebuilding Iraq Report - May 15, 1007.

Once chart I found enlightening and eye opening was their oil chart showing history covering the range of 1970 to 2005. [Click for larger image]

I also found the enemy initiated attack chart easily digestible although as I have commented before, I am not a huge fan of stacked bar charts unless there is also an easy way to look at the individual indicators.

Postponing the Moment of Truth

2007-06-12T11:20:00.000-05:00

Check out this excellent Postponing the Moment of Truth cartoon from cartoonstock.com.

This is a perfect match to the imprecise approach, the mushy metrics, the startling lack of numerical quantification, and the complete absence of visualization as exemplified by (but certainly not limited to) Stolberg's See You In September report in the New York Times as highlighted in the previous post.

Revisiting "See You In September"

2007-06-12T11:10:00.000-05:00

I want to return to the Sheryl Gay Stolberg article

See You in September, Whatever That Means

published in the New York Times on May 13th, 2007. There are many more important clues we can discover hiding the text. I did not have time to unravel them all during the first Charting Progress in Iraq post on this subject.

Following along the lines of the Forecast on Iraq post we'll once again de-construct and highlight some of the language and see what else that reveals. For fun, I have color coded the quoted text as follows.

Orange - vague metrics (e.g. progress, confidence, winning, losing)

Purple - vague quantification (e.g. up, down, better, more)

Green - specific metrics (e.g. sectarian murders, monthly bomb rate). Unfortunately, none of these have any specific numbers associated with them. Some have vague quantification and others have non at all.

"decisive progress"
"looking better"
“Sectarian murders are down”
"car bombings ... are up"
"deaths of American soldiers are up"
"some in Washington are grasping for a more complete and accurate way to quantify progress"
“No one knows how to define progress in such a mixed-up situation”
“We’re having trouble measuring it."
[Iraq Index includes] "the monthly car-bomb rate"
[Iraq Index includes] "how many foreign nationals are kidnapped"
[Iraq Index includes] "how many Iraqis have electricity"
[Iraq Index includes] "how many Iraqis have ... Internet access"
[the Iraq index] "is long on numbers and short on analysis"
"some signs of hope"
"more grounds for worry than for confidence"
[Rep Kingston (R-Georgia) wants to] "winnow down the indicators to a manageable number -- say, fewer than a dozen"
[Rep Kingston (R-Georgia) wants to create] "a standard bi-partisan metric"
[with such a metric in place] "then you could say who’s winning and losing”
[Micheal O'Hanlon says] "metrics are grist for a fact based debate but history shows it is dangerous to rely on too few of them"
[Micheal O'Hanlon says] "metrics were used in Vietnam and we had the wrong ones"
[the Vietnam metrics] "did net harm to the debate"
[Micheal O'Hanlon says] "we can’t be exactly precise about which indicators are the conclusive ones”
"such an index would be politically unpalatable to the White House, which does not want to back itself into a corner by agreeing to someone else’s standard for progress"
"the White House says the only progress report that counts is the one from Gen. David H. Petraeus, the new top commander in Iraq, and Ryan Crocker, the new ambassador"
"is the current strategy for waging war going well or not?"
[Stephen Biddle] did fault the White House for not being more open with the public about its own idea of what constitutes progress
“By being unbelievably vague about everything,” he said, “they’re making it very hard for congressmen and senators to go to their constituents and say, ‘Look, here’s why things are going better than you might imagine.’ ”
"Some say measuring progress is simple: you will know it when you see it."
[Metrics suggested by Senator Robert Bennett (R-Utah)] I want to see life starting to come back,” “I want to see people in markets. I want to see couples strolling down the street, folks sitting at outdoor cafes.”
[Metrics suggested by Senator Susan Collins (R-Maine) ] “a significant reduction in violence and attacks accompanied by a transfer of more and more authority to the Iraqi forces.”
[Senator Collins] “the difficult question is going to be if the analysis is mixed, and I suspect it may well be.
[Senator Lamar Alexander (R-Tennessee) says] “I don’t see any way for us to maintain a long-term presence in Iraq without more bipartisan support.”
[Rep. Kingston] "hopes to come up with some useful way of figuring out ...whether [to] remain supportive for a little while longer"
[Rep. Kingston has] " heard three years of nearly happy-talk in testimony"
“We always seem to be about to be around this elusive corner, but we never get there.”

Bottom line:
The language we use to talk about the important aspects of our lives makes a difference. When we use language about an evolving situation this way (vague metrics coupled with imprecise quantification, and lack of any clear trend quantification and visualization), we are going to have a tough time making sense of what's actually happening, what it means, and what our best next step might be.

If we continue along this path of mushy metrics, vague quantification, and numerous instances of what appears to be learned helplessness, come September, we won't be in any better state to evaluate the situation than we are now

I have some more thoughts on some of the key quotes from above for a future post. There is a way forward that addresses Michael O'Hanlon's concerns and answers Rep. Kingston's call for assistance.

Old problems plague new security plan for Iraq

2007-06-10T09:45:00.000-05:00

Here's an interesting post from Fort Wayne's News Sentinel:

KRT Wire 06/08/2007 Old problems plague new security plan for Iraq

The much more precise and quantified language describing the situation on the ground in Iraq in this Editorial by Nancy A. Youssef of McClatchy Newspapers stands in stark contrast to the fuzzy, imprecise, unquantified example we gave in the recent Washington Post Forecast on Iraq story.

Best to check these two articles out for yourself to see what I mean.

In the meantime, here are some actual numeric indicators from the article that will give you a flavor of what I am talking abut.

Number of bodies found on Baghdad's Streets
Troop levels numbers in Baghdad for US, Iraqi forces & Police
Number of neighborhoods under control
People killed in explosion
Number of car bombs

Having actual key measurable indicators mentioned by name and having actual data values for how these have changed over time is strikingly more informative and useful.

Is it good enough? Not nearly. It's a perfect example one interpretation of Von Clauswitz' "the better is the enemy of the good", but that's a subject for another post.

Hat tip to Juan Cole over at Informed Consent.

And kudos to Nancy Youssef for raising the bar.

Applying Trend Analysis Principles with Brilliant Effect

2007-06-09T16:45:00.000-05:00

Buying a camera at a fair price is not particularly earth-shaking activity, especially compared to trying to figure out whether our actions in Iraq are helping or hurting.

Nevertheless, here is a simple and striking example of the power of applying the essential principles of trend analysis to the task.

Pricescan.com did a brilliant job on this in my opinion.

The red trend line represents the high price, the blue the average price, and the green the low price. Each tick mark on the X axis represents one week's time and the chart covers the period from 9/11/2006 through 6/4/2007

What did pricescan.com do right

they identified the most important factors (in this case the minimum, average, maximum price for the Canon PowerShot AS710 IS camera.
they collected the raw data each week of the price of this item from a range of sources
rather than showing the raw data, they created calculated values of High, Average, Low price for each period
They selected a reasonable reporting rate that fit with the data (1 week in this case) and the resulting trend lines are relatively smooth while still giving us enough data points to have confidence in our understanding of the trends we see developing. A total of approximately 38 data points are plotted for each factor.
their Y axis scaling fits the data and makes it possible to see the trends in all three indicators that they displayed. The scale values are also easy to read.
they are reporting their data in near real time. The most recent data point is only 5 days ago.
They showed the entire time period ( I am guessing from product introduction right up through last Monday)

And of course, they visualized this data for us rather than telling us about it or showing us a table of 100+ numbers.

Armed with this easy to read chart, anyone interested in purchasing the Canon PowerShot AS710 camera in the next week or so will know whether the offering price is within a reasonable range and can then get on to evaluating the other factors that might impact their buying decision.

This example stands in stark contrast to the most approaches to other important areas of our lives as exemplified in yesterday's Iraq Forecast post. There are no mushy metrics or imprecise quantification or flowery metaphors. There's no postponing the moment of truth cop-outs saying we don't know exactly what the future will bring. Of course we don't know exactly. We never do.

They don't say: "we can't really tell you whether the price of this item is going to become more affordable by September".

There's just the relevant facts that help deal with the specific situation.

My two cents: the better we learn how to apply this kind of approach to the most important problems in our lives, the quicker we will begin to gain control of what's happening.

Washington Post. Forecast on Iraq

2007-06-08T17:41:00.000-05:00

Today's Washington Post featured this story with quotes from Lt. Gen. Douglas E. Lute, the new White House War coordinator (previously know as War Czar):

Nominee to Coordinate War Offers Grim Forecast on Iraq - washingtonpost.com

Here are some quotes relating to assessment of current trends and the direction we are heading. Bold emphasis has been added.

Lute said:

Iraqi factions have shown so far very little progress ... we're not likely to see much difference in the security situation" a year from now.

A CIA expert on radical Islam was quoted by Senator Bayh saying:

our presence in Iraq is creating more members of al-Qaeda than we are killing in Iraq.

The Post noted that the recently passed supplemental funding legislation

requires progress reports from the administration in July and September.

Here are some trend related quotes.

While insisting that incremental progress is being made, administration officials acknowledge that the benchmarks will be all but impossible to meet by the time progress reports are due to Congress.

"If the test is peace and reconciliation and flowers blooming by September, that will be hard to meet," said a senior administration official.

At the least, the administration hopes to be able to demonstrate movement in the right direction.

There has been little progress in four key political areas

At his hearing yesterday, Lute acknowledged that during a review of U.S. policy on Iraq last winter, he had privately expressed skepticism of Bush's "surge" proposal unless matched with more robust efforts by the Iraqis and other U.S. agencies.

"Where are we today?" he [Lute] asked. "Not where any of us would like."

Notice anything here?

Or rather do you notice anything missing here?

There's nary a single number and all the key factors and indicators referenced with regard to progress are not the kind that can normally be quantified. There's also no sense of any of these indicators changing step by step over time.

Let's list them so you can see what I mean.

Here are the metrics discussed in the article

"progress"
"the security situation"
number of members of al-Qaeda created
number of members of al-Qaeda killed in Iraq
"peace"
"reconciliation"
"flowers blooming"
"movement in the right direction"
more robust efforts by the Iraqis
more robust efforts by other US agencies
progress in four key political areas"

Revisit constitution to better balance power sharing
Oil revenue sharing
provincial elections
de-Baathifcation

And equally challenging to actually being able to use these trends to help steer our course forward is the mushy nature of the current "values" of these important indicators:

very little
not much difference
incremental
hard to meet
movement
little
more

Contrast the lack of quantification and therefore the lack of clarity with the numerical metrics we discussed in our series of posts yesterday such as:

peak megawatt hours
millions of barrels of oil per day
Iraqi prison population

The one bit of good news mentioned is that a progress report from the administration is due in July so at least we won't have to wait for September for that. How much we learn in July will depend an awful lot on whether the progress is report is based on hard metrics with clear quantification or on the kind of mushy, mostly unquantifiable indicators we find the WaPo report.

What direction are we moving?

2007-06-08T16:46:00.000-05:00

From Chart of the Day's - www.chartoftheday.com: "Quote of the Day

The greatest thing in this world is not so much where we are, but in what direction we are moving.' - Oliver Wendell Holmes"

And the best way to have a good sense of what direction we are moving is to examine (in near real time and with high resolution) the trend data for the most important factors and indicators at work in whatever situation we find ourselves.

What could be simpler than that?

Why Wait Till September in Iraq

2007-06-07T20:07:00.000-05:00

We Don't Have To Wait. There's a dangerous meme circulating that says we have to wait till September before we can learn whether the "surge" in Iraq is working. This, in my opinion is simply not true.

Many dimensions of the current situation in Iraq can and are being measured and reported on in near real time. The work of Michael O'Hanlon on the weekly Iraq Index at the Brookings Institution is one excellent proof that it is possible to know were we stand literally week by week and even day by day.

Digestible Trend Data. One principal area where we are not as good as we need to be today is in making sure that the trend data that exists is readily available in a form that is digestible by the intended audience. This intended audience includes the Congress, the main stream news media, the blogosphere, and especially ordinary citizens. Some really important data such as the example of Iraqi prison population that we showed in a post earlier today may only appear in tables. My guess is that most people are not willing or able to get what they need directly from the Iraq Index or similar reports. Those who are willing and able may not have nearly enough time. We can and must make it easier so that more citizens can be included in the process.

Assembling All Key Factors in One Place. A second principal impediment is that that the data is dispersed among many sources. We need to pull it all together so that each of us then becomes able to make a reasoned evaluation of how we are doing right now (without waiting till September) . My efforts at assembling some key trend graphs in a series of posts today and in the summary zip file is a small demonstration of what is possible.

Identifying and Acquiring Any Missing but Still Important Factors. A third thing holding us back from being able to evaluate for ourselves how things are going right now is that some key factors are not being reported in easily findable places.

Twenty First Century Visualization Tools. The fourth factor limiting our possibilities today is that most trend data is presented statically. This severely limits the number of factors that an ordinary person is going to be willing and able to look at to only a handful. On the other hand, with a modern Trend Visualization Viewing Station on the order of what's already available for stock market data from the likes of Big Charts or Prophet Net means that ordinary folk can look at and understand dozens and dozens of factors in a very short space of time.

We Don't Have to Wait. All these difficulties that are blocking us can be remedied without waiting till September. Our failure to remedy them over the past four years meant that it took that long for it to sink in that what we were doing wasn't really working. It's easy to posture that everything is ok when there is no hard data to prove you wrong.

In fact, if we don't begin to remedy them now, we will be in the same boat come September. We will find that we still do not have our arms around the entire situation. The facts we need will still not have been assembled and reported in an easily digestible way. And it will still be way too easy to claim that things are going well, or that we still don't know enough to tell.

Anyone interested in helping blast our way through these obstacles to understanding where we are in near real time, please let me know so we can join forces and see if we can make a difference.

The Data is Out There. Let's go get it and assemble it and make it easy to use and then share it with all interested parties.

Full List of Today's Iraq Trend Charts Contained in Composite Zip File

2007-06-07T19:05:00.000-05:00

Here's a screen shot showing the list of JPG and GIF files contained in the composite zip file showing all the Iraq trends that we posted about today.

The first five are from the Inspector General's report (001 to 005).

The next five are from the State Department report. (101-105)

The Deja Vu (#200) slide is from the State Dept Dec 204 report via the Matthew Yglesias post .

slides 201 -209 are from the Brookings report.

slides 301-305 were generated with TLViz using tabular data from the Brookings report.

You can find the composite zip file here.

Slide Show Viewing of Today's Trends

2007-06-07T18:53:00.000-05:00

We've covered quite a bit of ground today by reviewing several important documents that have included graphical representations of the trends at work in Iraq today.

We looked at:

The Special Inspector General for Iraq Reconstruction's April 2007 Report to Congress
The Dept. of State's Iraq Weekly Status Report for May 30th, 2007
A Matthew Yglesias blog post on Electricity goals from Dept of State's December 2004 Iraq Weekly Status and compared it to the most recent data on the same electricity factor from the most recent report.
The most recent Iraq Index from Michael O'Hanlon at Brookings Institution
Additional troop strength charts created from Tabular data in the Iraq Index report

One of the things I have noticed about blogspot blogging is that while it is easy to upload graphics and embed them in a blog post, it is difficult/impossible to retain useful control over size or placement. The maximum blog spot size usually doesn't do justice to the trend charts that I wish to present. That means that to look at a whole series of slides in a single one of my posts, the reader has to click on each one, and then click to close the window and return to the post. This is time consuming and not particularly pleasant in my mind.

down the road, I will be exploring what other blog posting capabilities are available that will let me present the data laid out the way I want it. In the meantime, I have assembled all 26 jpg and gif charts into a single zip file that I posted here in my data download area.

Once you unzip to a directory of your choice, you can enjoy a full size slide show with your favorite slide viewer - for example, Microsoft Office Picture Manager or IRFANVIEW.

I'd like to hear back whether you think this represents a useful, time saving approach for you. I am interested in learning whether you find having all the trend graphs assembled in one place makes reading them and understanding them more rapid than when looking at the individual reports. And finally, I would enjoy discovering how much value you think is added by combining charts from a wide range of resources and pulling them together in one place.

Turning the Tables

2007-06-07T17:12:00.000-05:00

Some important data in the Brookings' Iraq Index only appears in tables (e.g. the prison population data we mentioned in the previous post). Troop strength for US forces and for other Coalition forces may be interest now because of the "surge" in in US Forces that is now reaching its stated goals. However, the troop strength trend data in the most recent report is tabular.

So we turned the tables and transformed it into a small CSV file and created some new charts (see below) using the TLViz interactive trend visualization tool.

If you would like to take a look at the CSV file yourself, you can download it from Troop Strength Table. This table includes three original trends from the Brookings' table (US Troop Strength, Other Coalition Troop Strength, and Total Coalition Force Troop Strength). To that, we have added two calculated trends. The first shows Other Coalition Troop Strength as percentage of Total Troop Strength. The second shows the ratio of US Troop Strength to a nominal value of 135,000.

If you haven't already tried out the TLViz tool that runs on Microsoft Windows as a way for looking at two dimensional tables of trend data, here's another chance. Get started by downloading the base kit. Once you have installed the base kit, you can grab the latest version of TLViz.

If you want to learn more about TLViz, check out some of our earlier posts on the subject starting in chronological order with the ones from early January.

This first chart shows a sharp downtrend in other Coalition forces dropping in half from 25,000 to 12,000 since January 2005.

This chart shows US Troop strength from the beginning of the war. While the uptick since January is sharp, the current level of US troops is not far from the middle of the range it has been operating in for the past 3 years.

This is a chart using the calculated value showing the Other Coalition forces as a percentage of total troop strength. There has been a sharp decline over the past 3 years from a maximum of 17 percent down to the current value of approximately 7.5%.

This is another calculated chart showing how US troop strength has fluctuated around the 135,000 mark for the past few years. The current peak is lower than the two previous peaks.

The final chart shows total coalition forces in Iraq over the past 4 years. The current value of 160,000 looks like it is right in the middle of the operational range for this factor since the beginning of the war.

Iraq Index from Brookings Inst.

2007-06-07T17:02:00.000-05:00

Here is a sampling of what I think are the most important trend charts from the Brookings Institution's June 4th edition of Michael O'Hanlon's Iraq Index. This weekly report is still covers the widest range of Iraq trend metrics in a single place that I know of. Many of these are displayed in trend charts. Other important metrics appear in tabular form or in text description. Check out the whole report (60+ pages) for all the details.

Chart 1 - Iraqi Military and Police appear to be becoming more involved in recent months based on this chart showing their fatality rate rising.

Chart 2 - Slight downturn overall recently but still high. I would prefer to see these three factors separately so I could discern individual trends. Would also be useful to be able to normalize a given category's rate as a percentage of the total rate of attack. Rich data such as these are good when presented statically, but even better when there is a chance to interact with them (e.g. as we did in yesterday's post showing the Wall Street Journal's interactive chart on housing trends).

This chart shows the recent rapid ramp up in security stations and outposts

Stacked bars can be interesting. In addition, it regularly proves helpful to be able to examine the individual components.

This data looks like it is a very important dis-aggregation that will bring added insight to what's going on. It would be great if the data from previous years could be reconstructed and added to give more perspective.
This important chart covers the full period from the beginning of the war. My own preference would be to drop the labels with the totals for each month. I think these clutter up the chart and would be best to keep in a table for those who are interested. Since the month to month is so variable, a 3 or 4 month moving average might be a useful addition. My own preference would be to plot the non-hostile fatalities separately for greater clarity regarding any trends that might be present in that data. The disparity of scale between the two factors makes interpretation of the smaller values more difficult.

Another important chart which also covers the full time range since the beginning of the war. I believe a moving average would also help present this data more thoroughly.

While his data on the growing prison population is important and seriously under-reported elsewhere, it only appears as a table in this month's report. A chart for this factor would prove even more helpful. In addition, it would be useful to know the trends in other related criminal justice factors such as:

number in custody who have been charged and indicted
number currently undergoing trial
number found guilty, sentenced and serving time
number awaiting indictment
number awaiting trial
average number of months before indictment/trialAnother full time period chart. The pair of factors and the dual y axis work well together in this case. My view is that a moving average would be helpful in this situation as well.

Timelines in the News - Deja Vu

2007-06-07T15:10:00.000-05:00

Here's a post on Electric Goalposts from Matthew Yglesias from December 8th, 2004 commenting on the electricity trends contained in the State Departments Weekly Iraq Status Report dated December 1st, 2004.

The money quote is:

But for goalposts purposes, let's note that we are not now at the goal, have never been at the goal, and are not even moving in the direction of the goal. Instead, Iraqis have less electricity (but way more phone lines, interestingly enough) than they had before the war.

Here's the chart from December 1, 2004 followed by the one from May 30th 2007. Not much seems to have changed in the intervening 30 months.

I found the commentary added into the first chart to be most helpful and useful as an aid to understanding what was really happening. A picture is worth a thousand words but even so, many time a few extra words can add even more value.

Both these charts rate high because they show a complete time range. Of course, it would sure be interesting to see what the Iraqi electricity chart looked like starting back in 1980 so that the effects of the first Gulf War when, if memory serves, severe damage was done to the Iraqi infrastructure. It would be useful to compare the recovery or non-recovery following that war to the current recovery. If anyone has that data, please let me know.

Iraq Weekly Status Report May 30, 2007

2007-06-07T14:28:00.000-05:00

Here are some charts taken from the Department of State's May 30th, 2007 Iraq Weekly Status Report.

This first chart covers from January 2004. It appears as if demand is rising while production is falling and cannot seem to crack the 100,000 mark.

Also notice the sharp downward spikes for the light blue daily supply data. This tells me that it would be worthwhile to be able to zoom in on the supply factor and look at how supply is changing over time but using a much higher resolution than daily. An hourly trend chart for month's worth of data, or a daily chart using a 1 minute resolution would surely reveal some insights not currently available.

Similarly, when an aggregate trend factor show such sharp variation, it is frequently the case that creating additional trend charts that dis-aggregate the data can prove most helpful in understanding more fully what is happening.

This second chart makes it clear that the Iraqi currency has strengthened relative to the dollar over the past 12 months.

This third chart clearly shows the rising trend in Iraqi Commercial Bond interest rates over the past year.

This fourth chart suffers from showing far too brief a time period of only the most recent 10 weeks. Contrast this to the Oil Production chart in the previous post that showed the past 4 years and also provide useful trend data for production from the North and from the South.

This chart also has a much short sweep of time than needed for understanding the underlying trends. The related chart in the previous post showed refinery trends going back 4 years. This chart does have the positive feature that it groups all the bars for a given type of refined product together which is an improvement over the way the bars were used in the previous post from the Reconstruction report to Congress.

It's curious to note that there are no trend charts in this report from the State Department that show trends related to the first two "key areas identified as pillars of US government policy for victory in Iraq." Namely,

"Defeat the terrorists and neutralize the insurgents"
"Transition Iraq to security self reliance."

The trends in these areas are however dealt with in Michael O'Hanlon's weekly report from the Brookings Institution. We will be commenting on the May 30th version of that excellent report in our next post.

Iraq Reconstruction April 2007 Report to Congress

2007-06-07T13:01:00.000-05:00

From the Special Inspector General for Iraq Reconstruction's April 2007 report to Congress, here are some useful trend charts selected from their List of Figures page.

You can click on each chart for a larger, better view.

In this first chart, it would be interesting to break out the trend for imports separately so it could be seen more clearly.

This second chart shows substantial deterioration in average electricity load served over the past year.

In this third chart, it would again be good to separately break out the trend data for Oil Production in the North and the South to make the trends in each of these areas more clearly visible. Stacked charts often obscure the most important underlying phenomena taking place with the individual factors. A separate chart showing the percentage of total oil production accounted for by the South would also would likely be revealing.

In this fourth chart, export levels for crude have recovered to their target value in March 2007. It will be interesting to see how they progress from here on out.

In this final chart, it would be helpful in my view to show each of these factors as a separate bar chart so that the trends at work with that factor could be instantly grasped. This kind of multi-factor bar chart is much harder to understand and interpret since the trends for each factor are interspersed with bars from the other factors. When shown individually, the downward trend for each of these factors would instantly discernible. Showing data points one year apart is also not nearly as good as if each trend had been shown with month by month data points.

WSJ Housing Inventory Trends

2007-06-06T09:15:00.000-05:00

Check out this Wall Street Journal Interactive graphic showing changes in housing inventory in the past two years. It has some useful and novel features that make exploring the 18 different cities and 3 different regions easier over the past 20 months.

Here's one of the more interesting charts that I created in just a few minutes of exploration.

For the six cities shown (Miami, Orlando, Chicago, Las Vegas, Los Angeles and Phoenix) the upward trends since October 2005 are plain to see.

Try it out yourself and see what you can discover.

Here are my thoughts on the pluses and minuses of these interactive chart features

On the plus side

1. A single click selects a new factor to add to the mix of factors already selected for the chart

2. When adding a new factor into the mix, it highlights that new factor on the chart and reduces the intensity of the other factors on the chart until you move the mouse over the chart again when all factors selected come into full view. This helps you lock in on the colors for the new factor before they become blended with the other factors.

3. When you move the mouse over the check box for a factor that has already been selected, that factor is highlighted against the other factors currently selected with a little pop up text box that tells you the current inventory value exactly.

4. A single click as all that's required to de-select a factor.

5. A single click for one of the three areas (East, Central, West) selects all the factors for that area. And once a whole area has been selected, a single click de-selects all factors currently checked.

6. All of the above pluses add to ease of use and time saving when exploring this data set for the most interesting and telling patterns.

On the minus side

A. Way too short a time range. To make full sense of these trends and put the recent behavior in perspective needs at least 5 years and preferably 10 to 25 years of history.

B. Some major cities that are in the top 10 in population are missing - New York, Philadelphia, and San Antonio.

C. No way to adjust Y axis scaling. This is a serious drawback. The scale runs from zero to 110 thousand which is fine for viewing the trend for Los Angeles which has a range between 45, 000 and 103,000, but it makes it difficult to grasp the trends for such cities as Minneapolis which varies from 25,000 to 31,000 or Baltimore which ranges from 5,000 to 10,000. Both of these cities look flat line during this period even though when you examine them closely they really aren't. Y axis scaling options such as the possibility of calculating the Min-Max scaling for the factors charted would help bring the trends for all cities into clear view.

D. No available view of the data using percentage change from a baseline value for determining the Y axis value. Such a normalizing option would simplify comparisons between cities.

E. No option for showing aggregate results for a region or for stacking the factors selected.

F. If you wish to look at each factor separately, it takes two clicks. One to de-select the previous factor, and one to select the new factor.

G. No option for smoothing with features such as moving averages

H. Missing trend data for other related factors for each city such as

Inventory breakdown by New and Existing housing
Average number of months houses have been on the market (total, new, existing)
Median selling price in that market (total, new, existing)
Rate of sales per month in that market (total, new, existing)
Number of months of inventory at current sales rate (total, new, existing)

I. Missing option for downloading the underlying data set thereby enabling further analysis.

Hat tip to Barry Ritholtz at The Big Picture

The Big Picture | Some More Housing Charts

2007-06-02T12:45:00.000-05:00

Barry Ritholtz does it again with this excellent post which contains 4 excellent and easily readable charts on the trends affecting the housing market.

This time, Barry tells the story using pictures without commentary. The charts he selected are excellent and speak for themselves without commentary. However, Barry always adds something to the mix when he adds in his thoughts and perspectives and experience on what he sees in the charts and what it is most likely to mean.

Barry does give us a money quote with these few words to wrap up his post:

Funny thing: The charts tell pretty much the same story the words did: The housing story is only halfway done, going to get worse as time progresses. We are not anyway near a bottom in Residential Real Estate.

I do miss the words, so here are copies of the charts accompanied with my ordinary citizen, non-housing-expert comments on what they might be telling us and focusing on the degree to which these charts make best use of best practices for conveying trend results graphically.

This first chart is both interesting and scary. It would have been great to see what the Reset Amounts have been for the past 10 or 20 years rather than just the past 4 or 5 months. If I am reading this correctly, it sure looks like we are into a 12-18 month period with extremely high reset amounts, where a very small percent are for Prime ARM. Note that this chart is originally from Credite Suisse via Lamont Trading Advisers to At These Levels and then to The Big Picture.

Stacked bar charts are often difficult to use to interpret the behavior of the sub components such as Unsecuritized ARMs. Important trend information about these sub factors may be hiding in a chart such as this. My own preference in these cases is to use individual charts for each factor that I consider important so I can understand the individual behavior AND the composite behavior of all these factors.

This 3 year chart originally from Northern Trust is plain as day. Having the 20 or 25 year chart for this same key factor would have made this even better and clearer and more useful. Some smoothing, maybe a 3 month average could also be helpful here. It's important to note that this chart is based on combining two other factors - the actual total inventory of homes, and the most recent rate of sales of existing homes. For reference, having charts for these other two factors to look at in parallel to this calculated metric would also be useful.

This next chart gives us a nice 20 year view. having a complementary 3 year view would let us look more closely and better understand the very sharp recent spike.

For this next chart, my own preference would be to look at these two factors separately on their own chart first using a 20 to 25 year view.

As a general comment, having a shifting time window for every chart makes things much more difficult on the viewer.

Housing Freefall Continues Unabated

2007-06-02T12:02:00.000-05:00

You can find some more good charts and commentary on Housing from Barry Ritholtz over at The Big Picture.

Here's the first chart. You won't be surprised to learn that I like the long time sweep. However, this also looks like a case where having a second, shorter term chart - maybe for the past 3 years would help us better understand the more recent behavior

In contrast, the second chart shows just the 3+ year view and presents that very well. Having a 20 year year to match the first one would have been even better.

The whole article is worth a read for the extra perspective it brings.

St. Louis Fed - National Economic Trends - June 2007

2007-06-02T10:45:00.000-05:00

The newest edition of the always excellent National Economic Trends is available today from the St. Louis Fed.

Here are a sampling of the charts that caught my eye. Click for larger size version.

From their engaging cover article, this first chart shows the percentage of the population age 55 or over engaged in part time employment. The upward trend for women is particularly clear. The article discusses possible causes and notes areas for follow-up investigation.

This next chart shows trends in national income as a percentage of GDP.

Corporate profits weighs in at almost 14% of GDP and is close to the recently set 25 year highs. This is approximately double where it was in 1982.

Proprietor's Income registers at almost 9% of and it also is close to its 25 year highs. This is almost 50% higher than it was in 1982.

Compensation at a little over 64% of GDP is close to its 25 year lows. This is down approximately 3% from the 67% level of 1982.

The remaining National Income components (not shown) must by deduction be down net 7% from about 20% to 13%.

This third chart shows pretty clearly that the rise in corporate profits before tax is directly reflected in the rise in after tax profits as a percentage of GDP. After tax profits have more than doubled since 1982 from 4% to over 8%.

The fourth chart shows that the current account balance is close to its 25 year low at approximately 6% of GDP. Note the striking and steady downward trend since 1991.

And this final chart is noteworthy for the clarity with which it shows the rise of debt service payments to approximately 14.5% of personal income - close to the highest levels in 25 years. The steady upward trend since 1992 is also clearly visible.

Commentary.

1. As I mentioned in previous posts, one of the reasons that I value the St. Louis Fed's monthly National Economic Trends report is their use of long term trends. These help put the current behavior into perspective. The charts I have selected offer a glimpse at how a 25 year trajectory adds understanding to the most recent behavior.

2. That's not to say that their monthly report couldn't be improved. Here's my short list of suggestions on just how that might be done.

Show all selected factors using the same long term 25 year time frame trend chart as the default setting and template.
Show each factor on its own before combining it with other factors in a dual chart. The dual chart approach while extremely helpful for showing the possible relationship between a set of factors and simplifying comparison sometimes obscures the underlying trend in some of the factors due to the Y axis scaling used.
For the single charts created for each important factor, consider using MIN-MAX scaling for the Y axis rather than zero based scaling as the default. This is not correct in every instance but it often proves useful in helping make the trend patterns easier to read and understand.
Add at least some minimum commentary (could be 25 words or less) with 20% of each month's set of charts. Add comments for the charts that the St Louis Fed considers to be the most interesting and which have some interesting pattern to observe. E.g. follow the example we have set with the selected charts above. A picture is worth a thousand words, but not every important nuance seen by the expert creators of these charts will register immediately with every viewer, especially to non-expert viewers.
Make it easy to download a cross-tabulated table of all the data contained in all the charts for further examination by interested parties.

Charting Progress in Iraq

2007-05-14T16:35:00.000-05:00

This topic of determing how things are going on the ground in Iraq seems to be heating up in the last few days. See for example, Sheryl Gay Stolberg's New York Times article from May 13th - See You in September, Whatever That Means where she says:

There’s just one problem. Nobody in Washington seems to agree on what progress actually means — or how, precisely, it might be measured.

While I don't agree with that premise, the article opened up the door to a ton of worthwhile commentary - much of discussing possible factors that might be useful to gauge how things are going.

Another recent contribution to this topic was David Peck's Christian Science Monitor Article on why we need yardsticks for both success and failure in Iraq. David skillfully maps his experience with coaching leaders onto the pressing leadership issues involved in our presence in Iraq. The money quote of this article for me was:

there has never been agreement on the status of the war in Iraq because there are no agreed-upon measurements.

For some perspective from a couple of years back, and maybe to help understand how much or little forward momentum we are making in talking about progress, I also suggest you take a look at this Dan Froomkin article on Nieman Watchdog from a couple of years back (May 2005): Isn’t there some way to tell if we're winning or losing in Iraq?

We've delved into this topic in previous posts (see: Most Important Iraq Data) and we plan to revisit it again soon based on these new inputs and what seems to be a growing interest in the topic.

And of course in our opinion since what gets measured gets done and what doesn't get measured doesn't get done, this particular topic is of particular importance and relevance right now.

Hat tip to Dan Froomkin for pointing out both the referenced NY Times and the CS Monitor articles in his White House Watch Blog.

James Durso's Iraq Reconstruction News

2007-05-03T13:05:00.000-05:00

Here's a link to an essential resource for keeping up with the trends regarding what's happening on the ground in Iraq. It's the new Iraq Reconstruction News blog created by James Durso. The links that this blog provides are to detailed resources that go far beyond the reporting one might find in the mainstream media. As I have been exploring different and complementary sources of trend data, these updates that James provides have been particularly helpful for me in locating new sources of data.

The focus of this blog is particularly on the Reconstruction aspects in Iraq. The links however often provide excellent depth on a whole host of other related topics.

James also puts out a regular set of email updates covering the same material for those who would prefer to get their news in that manner. You can find his contact info on the blog site.

Latest Brookings Iraq Index - Key Factors for Gauging the Success of the Surge

2007-05-03T12:51:00.000-05:00

Here's the link that will always get you to the latest edition of the highly useful Brookings Institution Iraq Index that is one of the best places that I know of for learning about the trends on the ground in Iraq.

In the April 30th, 2007 edition, there is a useful essay on "Tracking the Surge" on page 4.

Unfortunately, the charts and tables that follow for the most part do not match up with the key factors noted in the essay, nor do they make it easy to for interested citizens to see the trends at work for themselves.

Here are some of the key factors that I gleaned (scraped) from the text.

number of US brigades in place
number of "joint security stations" established
extra-judicial killings
level of civil warfare
willingness of Shia militias to lie low
willingness of Sunni tribal leaders in al-Anbar to collaborate in opposing al Qaeda
level of violence in al-Anbar
daily attacks in and around Ramadi
resilience level of al Qaeda and related terrorist elements
rate of use of vehicle bombs
rate of use of vest bombs
percentage of casualties that are Shia
progress on the hydrocarbon law through the Iraqi parliament
progress towards reforming de-Bathification to allow return of lower level Baathist to public life
state of the Iraqi economy
Oil revenue received
Foreign aid received
Money available in federal coffers
Performance level of public utilities (presumable electricity, water, sewerage, sanitation, petrol supplies, public transportation)
How well schools are functioning
The overall state of the health system infrastructure
Level of unemployment

That's an awful lot of important metrics for one page of text and Michael O'Hanlon and the Brookings Institution deserve praise for consistently working to get their arms around the vast multi-dimensional range of factors at work.

And, reviewing the whole report, there are literally dozens of other important factors that show up in charts or tables.

Wishlist 1: It would be really valuable if one could turn the page of the Iraq Index report and see the chart or charts for each of the these items one by one, in the same sequence as they appeared in the essay and with the same terminology, so as to get a sense for the degree of progress that is being made, to see which things are lagging, to see what is getting worse and by how much, and so on.

This, in my mind would provide a giant step upward in the usability and the understandability of the important work that provides the basis for these weekly reports.

Comment: Many of the tables and charts in the weekly report are out of date. They represent important factors but no new data has been available to update them for quite some time.

Wishlist 2. It would be helpful if outdated tables and charts were either placed in an appendix, or if the chart could be updated to show the current end date and the missing data. Otherwise it is easy to misread the right hand side of the charts to think that the data and the trends represent the past few weeks or months instead of sometime in June 2006, or May 2005.

Comment: The chart formating for just about every chart is different and each chart seems to have its own unique starting and ending time intervals.

Wishlist 3. A move to more consistent chart formats and time intervals would greatly enhance the readability and usability of these reports.

BLS Compensation Report

2007-05-03T11:52:00.000-05:00

The latest Bureau of Labor Statistics report on wages and benefits includes these relatively easy to read charts that highlight the employment compensation trends over the past 6+ years. The moderately long time frame makes it easy to detect patterns and compare recent behavior to the situation at the turn of the century.

With the second chart, using the non seasonably adjusted numbers for the percent change over 12 months makes a lot of sense.

The red lines in the 2nd and 3rd charts plot the same metric. The blue line converts those wage gains to constant dollars and makes pretty clear how little this has changed over the past 6 years.

It would have been nice to see the total compensation chart including wages and benefits adjusted to constant dollars.

Looking behind the headline number.

2007-04-22T11:12:00.000-05:00

Here's a good example of the power that acrues the minute that we begin to look behind the headline number and see what other factors might bear on the meaning of what we are seeing.

The Big Picture Watching Trading Volume Fade . . .

In this case, insteaed of just looking at the S&P 500, Barry Ritholtz points us to a Michael Kahn Barron's article where he takes a look at the volume as well and discovers some interesting patterns.

What other factors might be worth adding into the mix and giving us even more depth of understanding of the basic workings of this important headline financial indicator?

For example, it would be interesting to see what percentage of the S & P volume was NEW short sales and what percentage of the volulme was Covering Purchases for previous short sales.

It might also be worthwhile to look at the average number of shares per trade which might give a clue into what percentage of the volulme was due to large financial instituions and what part was coming from the "little guy."

New Development at the St. Louis Fed

2007-04-20T15:24:00.000-05:00

Here's the latest on a nice new development from the St. Louis Federal Reserve.

St. Louis Fed: March 2007 Tip of the Month

It's now possible for registered users of their FRED database to create, name and save customized graphs that can then automatically be updated on later visits to the site. Looks like a great feature.

Monthly trade deficit

2007-04-14T09:51:00.000-05:00

Another great chart from Calculated Risk

You can find the full post here

Chart of the Day - www.chartoftheday.com

2007-04-13T04:41:00.000-05:00

Here's another nice chart from Chart of the Day - www.chartoftheday.com

On the plus side we have the complete time range and the use of the derived value of the inflation adjusted DOW as the variable to watch rather than the Headline raw DJIA value. In my view, the inflation adjusted number really should be the Dow value that we watch for the same reasons that we use inflation adjusted numbers everywhere else.

Historical Bear Market Contractions

2007-04-12T07:11:00.000-05:00

Barry Ritholtz continues his track record at finding and sharing interesting and useful charts that provide new perspectives at looking at important data. You can find the full post at:
The Big Picture Historical Bear Market Contractions

I liked the creative way that the width of the bars in this chart shows the second dimension of length of the contraction period.

An Old Fashioned Trend Visualization Appliance

2007-04-11T14:12:00.000-05:00

And speaking about the excellent Calculated Risk Blog, I would like to point out the following exemplary post on the Housing market.

Using just the ordinary blog posting mechanism and the ability to embed JPG pictures as the Trend Visualization Appliance, the resulting article has a high degree of explanatory power and immediacy. You can literally see what Calculated Risk is talking about as you read as a result of the great blend of clarifying text with the easy to read trend graphics in close proximity.

Not relying on the normal conventions of providing only one or two graphics for headline numbers, Calculated Risks walks us through a series of 6 or 7 less well known but still key factors. This leaves the reader/viewer which a much more complete view of the dynamics that are at work in this one complex area.

Combine this with good references back to data sources and consistently good judgment on what would constitute a reasonable time frame, leaves with a very useful end result that provides the reader with the maximum amount of infomation in relatively short period of time.

The only difficulty with this is the same one I face with this TimelineView blog -- namely that as a limitation of BLOGGER.COM the largest allowed size of the embedded images is just too small to be fully readable and understandable in most instances. To really understand the charts, at least at first, usually requires clicking for the larger image and then using the browser navigation controls to go back to the main page. This really slows down the flow of reading and understanding.

I think we can all learn a lot by watching how Calculated Risk handles these kinds of presentations. Take a look at the post or other similar posts that appear regularly at Calculated Risk and you will see what I mean.

The Importance of Mortgage Equity Withdrawal (MEW)

2007-04-11T14:10:00.001-05:00

Another hat tip to Barry Ritholtz in his post on The Big Picture The Capital Commerce Debate this time highlighting some eye opening and vital data about MEW (Mortgage Equity Withdrawal) using charts from courtesy of CalculatedRisk.

You can find a very recent Calculated Risk article here with comments on the debate between Barry Ritholtz and Don Luskin and a clarification of the meaning of the second chart here.

Business Capex - Another Nice Chart

2007-04-11T14:10:00.000-05:00

Hat tip again to Barry Ritholtz who regularly posts some charts well worth looking at as with today's post on: The Big Picture Business Capex Rolling Over

It sure does look like capital spending has turned a big corner.

The chart is courtesy of Bob Bronson. You can find some more of Bob's posts here.

For reference, here's is a similar chart from about 6 months ago.I found it interesting how well Bob's analysis back in October, before the raw data turned the corner, was able to predict the recent changes in Capex.
I also like the way Bob has integrated textual explanations and graphical pointers right in with the charts themselves.

The only negative point I see with this presentation is that the dates are hard too read clearly.

Iraq Trend Data Set now posted to Swivel and Many Eyes

2007-04-11T13:01:00.000-05:00

A while back we mentioned the great new data sharing and visualization web site called Many Eyes and we pointed out an Iraq Trend data set we had uploaded so that others could more readily examine and analyze it.

We've recently come across another great new data sharing and visualization web site called Swivel and have uploaded the same data set there for comparison purposes at: http://www.swivel.com/data_sets/show/1004780.

Both Many Eyes and Swivel fall into a category I have started calling Trend Visualization Appliances. The amazing Gapminder work also fits into this category as do all the various stock market visualization tools currently available.

The old fashioned form of trend visualization appliances (e.g. a standard report including graphical output) can still be pretty useful when managed carefully. For further comparison, here is the link to a PDF report that graphically presents the same data - vizualizing-trends-ohanlon-testimony.pdf.

Similarly, for data sharing, the old fashioned URL links to csv files offer an alternative to the WEB 2.0 mechanisms such as Many Eyes and Swivel. Here's the Iraq trend data the old fashioned way: ohanlon-key-factors.csv

So many choices, so little time. How can we decide which is best for our purposes?

I'll be revisiting this topic, but for now, my own criteria for deciding which tools I will use at a given moment are:
1) ease of use,
2) shortness of learning curve, and
3) personal productivity and time saving -- the speed at which I can navigate through complex trend data sets to discover previously hidden patterns.

What do you think?

Digging into the Job Opening Data

2007-04-10T15:57:00.000-05:00

As mentioned in the previous post, the Bureau of Labor Statistics makes detailed trend data available on all the key Job Opening and Labor Turnover Survey (JOLTS). You can take a look yourself at the BLS data page and then scrolling down to select one of the option buttons (e.g. for most requested statistics) for Job Openings and Labor Turnover Survey.

You can set the time period for the trend graphs created to be the entire time since the year 2000 that JOLTS data has been collected. Here are a few examples. It's somewhat puzzling to me why the Openings rate seems to be going steadily up from 12/2003 onward while the Hiring rate stays steady beginning around 12/2004.

There's a ton more data at the BLS web site. The biggest difficulty for me and I bet for others is just how time consuming and inflexible the BLS' trend visualization application (TVA) is to work with.

Job Openings and Labor Turnover Report

2007-04-10T14:44:00.000-05:00

Here's the two reasonably readable graphs from this morning's JOLT (Job Openings and Labor Turnover) report from the Bureau of Labor Statistics (BLS). Looks like there has been a moderately strong trend at work in the job openings picture for the past 3 years.

As has been typical in the BLS formal reports like this, the time span is too short to be able to place recent behavior in context and the number of factors shown in visual form is far too few to fully grasp what is going on in the world of employment.

In the full monthly report itself which runs to 15 pages, there are numerous tables with literally hundreds of important factors. Unfortunately, these covering an even shorter two year time period with even fewer sample trend sample points per factor. This again matches the current BLS standard not-particularly-reader-friendly approach for their "printed" reports.

In my view, this standard BLS approach is far from the best way to present important trend data. All the time consuming work of searching for and extracting meaning is left to reader (which means in most cases it will never happen) or it is left to the expert pundits who will typically comment on a few of the sub factors that they find most interesting and perhaps present one or two charts.

Of course, back at the BLS web site, just about all the trend data for all the factors for all the time periods is available for those who have the time and skill to track it down.

Another Easy to Read Housing Chart

2007-04-10T12:15:00.000-05:00

Here's another easy to read chart that I find attractive from Political Calculations' post on the Boom in Realtors. The long time sweep creates excellent perspective. It would be interesting to zoom in on the past 30 years for another view.

San Diego Mortgage Defaults & Foreclosures

2007-04-10T11:44:00.000-05:00

Here's a nice crisp chart that I think is instantly understandable. I especially like the long time sweep so the most recent data can be put into perspective. You can find the full story at:
voiceofsandiego.org: Toscano... Motivated Sellers Abound

It would be nice to be able to flip through a series of charts like this for a sampling of strategic areas or state by state to be able to compare how the San Diego trends match up against other parts of the country.

Hat tip to Kirk for this post at Reasons Unbeknownst

The Battle for Baghdad - NY Times interactive graphic

2007-04-09T11:11:00.000-05:00

Here's another New York Times graphic that provides some insight into the evolving situation.

Violence in Iraq - The Battle for Baghdad - New York Times

If you click on the Sunni Area, Shiite Area, and Green Zone labels in the upper right corner, the map will be painted with some subtle coloring that I found helpful.

The forward and back controls that that show you week by week changes did not work well for me, often not responding to my mouse clicks. The pull down menu worked better but of course is a lot more clumsy. A PLAY button with some speed control would have been really helpful.

The time span runs backward to February 11th so this chart doesn't allow comparisons of how post "surge" events relate to the pre "surge" situation.

Is the "Surge" working?

2007-04-09T10:36:00.000-05:00

Here's a before and after the 'surge' graphic from this morning's New York Time article on: Patterns of War Shift in Iraq Amid Buildup of U.S. Force that sheds some light on the degree to which the "surge" is working.

Disaggregating the figures by area is helpful at clarifying better what is happening on the ground. For example, the increase in Baghdad deaths and Diyala deaths due to IED both look substantial while the drop off in total deaths in Anbar Province is also striking.

By selecting a 7 week Before Period and a 7 week After Period, this graphic sets up the possiblity of direct and straightforward Apples to Apples comparisons .

I would also have liked to see all the same key factors represented on this chart (such as Baghdad deaths by IED, Baghdad deaths by other causes, Anbar deaths by IED) also available as a week by week trend table and charts with the time frame stretching back to the beginning of the war so that the trends showing up in the current casualty levels could be put put into a broader context.

Of course, adding in other factors such as US troops wounded, Iraqi forces deaths and wounded, and Iraqi civilian deaths and wounded would give a much fuller picture of the degree to which the "surge" has so far changed things in Iraq.

Selective Perception, Cognitive Bias, & the Recency Effect

2007-04-07T10:32:00.000-05:00

Barry Ritholtz has an excellent post this morning commenting on NonFarm Payroll (NFP) increase of 180,000 from yesterday's Bureau of Labor Statistics Employment Situation Report.

His comments on selective perception, cognitive bias, the "recency effect" and the soft prejudice of low expectations apply well beyond the NFP example.

Let's begin with a quick word on cognitive bias. Humans are guilty of this -- selectively perceiving and recalling what agrees with their world view. We are all guilty of this, and while we cannot escape it, being aware of it at least allows some measure of recognition, and perhaps, adaptation to the phenomena.
Let's use the NFP data as an example: Consider another bias, the Human tendency to overemphasize more recent data versus the totality of information and the overall trend. This is a cognitive bias known as the recency effect . Despite the overwhelming evidence showing this to be a generally weak jobs recovery (the worst since WWII), our primate brains interpret a single good data point as proof of something better. ...
Regardless, we also see some of the soft prejudice of low expectations in yesterday's data: 180k is hardly a rockin' strong number, relative to population growth,. Put that into the context of recent expansions such as the 1990s. Oh, and in that more recent period, there was no BLS Birth/Death adjustment, responsible for nearly a million fictitious jobs in 2006.

Unfortunately, the two trend chart he shows only go back three years (suffering themselves from the recency effect) so we can't compare the relatively weak current numbers to the NFP increases of the 1990s. The two charts shown in the Bureau of Labor Statistics Employment Situation Report also only go back 3 years.

So, below I have posted a 15 year view showing year over year percentage change for the seasonally adjusted value of the growth in NonFarm Payroll. It's pretty clear from this longer term chart that our growth in non-farm employment during this latest expansion never got back to the steady, multi-year 2.5% growth rate of the late 1990's and appears to be declining again.

Of course, in a subject area as complex as employment, an equally important danger in my opinion is the almost universal tendency these days to stick with just a few headline factors. If we truly want to understand employment, I believe we have to make it easy to visualize and then examine a full range of factors over a substantial time period.

For example, the recent drop off shown above does not seem to square with the reported drop in the headline unemployment rate. Maybe some other factor of the thousands of factors recorded by Bureau of Labor Statistics (BLS) can help explain it.

Too often, really important factors aren't mentioned at all, and the ones that are mentioned are not accompanied by their corresponding trend charts.

Here's a link to a short list of the most popular BLS statistics which might be a useful starting point for a deeper analysis.

Longer term view of Iraqi Oil Production

2007-03-25T07:20:00.000-05:00

Here's a longer term (10 year view) of Iraqi Oil Production in a single chart (Source is Oil Market Report, provided by the International Energy Agency). With the more complete time range view, it becomes easier to put the most recent production data as reported in the Iraq Weekly Status Report into broader perspective.

Hat tip for this link to Scott Johnson.

Michael O'Hanlon Year over Year Iraq Update

2007-03-24T16:06:00.000-05:00

Here's a recent Michael O'Hanlon New York Times article which includes year over year table showing some 17 trend metrics from Feb 2004, year by year through Feb 2007.

The State of Iraq: An Update:

We plan to graph some of the more interesting of these trends in a future post to see what patterns we can discover that may be hiding in the data once we have transformed the posted table into a more readily reusable format for further investigation and discussion.

The Brookings Iraq Index Archive

2007-03-24T15:22:00.000-05:00

Here's the link for the Brookings Iraq Index Archive: Iraq Index Archive that lets you access their entire history of this important report going back to Nov 17th, 2003.

Latest Iraq Trends from the Brookings Institution

2007-03-24T14:51:00.000-05:00

The Brookings Institution continues its weekly Iraq Trend reporting. Here are some interesting charts gleaned from their March 22nd edition. Note the link to the Brookings Latest Iraq Trends Report will always take you to their most recent version.

Security Stations Established. This is a new chart reflecting changes underway as a result of the "surge" showing the number of security stations (cumulative) that have been established in Baghdad. The new goal is to reach 70 stations (original goal of 30).

Patrols Per Week. This new chart shows patrols carried per week and shows an impressive jump up in recent weeks.

IED Fatalities & Percentage. This dual axis chart shows both the number of IED fatalities (red bars, left scale) and the percentage of all fatalities that are attributable to IEDs (blue line, right scale).

Targetted Attacks. This chart shows average weekly attack based on who the attacks target - Coalition forces, Iraqi forces, or others.

Oil Production: Iraq Weekly Status Report - March 21, 2007

2007-03-24T13:11:00.000-05:00

Here's the latest Iraq Oil production chart from the Iraq Weekly Status Report, March 21, 2007
showing production at or close to the Oil Ministry Goal for 2007 and holding quite steady .for the past 7 weeks after a slowdown period at the beginning of January.

To completely understand the meaning of current production and this year's goal, it would be helpful if the full range of production data since the beginning of the war were also available in addition to the zoomed in most recent view that is presented.

With that in mind, I assembled a mini time series history (each about 3 months apart) of this chart as it appears in weekly reports stretching back to 2005. Note that the next to the last chart overlaps the charts before and after to more clearly show the January 2007 dip.

Ideally, all this data would be on a single chart and much more readily digestible.

Iraq Reconstruction Management Office: Employment Chart

2007-03-24T12:29:00.000-05:00

Here's latest employment chart from the US State Department's Iraq Reconstruction Management Office (IRMO) Report dated March 20th, 2007.

It shows Iraqi employment in US Govt. projects and what appears to be a rapidly declining trend. It would interesting to determine whether the change in this factor was having an impact on other key success factors in Iraq.

If you would like to be on an email distribution list for the full weekly IRMO report which also contains detailed updates about Electricity, Oil, Water, and Sanitation project. you can send your request to:
jddurso AT earthlink DOT net

More BBC trend charts for Iraq

2007-03-24T12:04:00.000-05:00

BBC NEWS | In Depth | Iraq violence, in figures

The BBC link above is the source for these two easy to read charts -- the first showing the varying levels of US troop strength and the second showing the US Military Death Toll.

Good points of the BBC presentation:

1) Full range of Data. Both charts go back to the beginning of the conflict in 2003 and are up to date as of Feb 2007.

2) Combining Data from Multiple Sources. The BBC has combined data sources obtaining the troop strength information from the Brookings Institution and fatality figures from the Iraq Coalition Casualty Count. The charts are presented on the page so the single Time Axis can be used for both for more ready comparison.

3) Detailed Interactive Event Description Integrated with the Trend Data.
The BBC original interactive chart has the option to click on the event arrows to get a brief snapshot of some of the important events and see how these fit into the overall picture and relate to the underlying trends.

More Timelines in Motion

2007-03-24T10:35:00.000-05:00

BBC NEWS | In Depth | Baghdad: Mapping the violence

This interactive graphic from the BBC will repay your careful interactive examination.

Some things to try.

1) Drag the timeline slider slowly to see how the level of violence has changed month to month.

2) Also click on Ethnic Areas and then click back and forth between Current and Pre-2006 to see the dramatic changes in ethnic mix that have taken place already and to understand where the predominance of violence is taking place.

It will be interesting to compare the March 2007 view of violence when it is ready to the pictures of violence in December 2006, January 2007 and February 2007 to better gauge the degree to which the "surge" may be having a positive effect.

This would be even better if, like Gapminder, there were a PLAY button and some Speed control. Next Month and Previous Month buttons would also be handy as it is a little tricky dragging the timeline control exactly one month forward or back or clicking on exactly a given hash mark along the timeline of months.

Hat tip to: Data Mining: Text Mining, Visualization and Social Media

And speaking of gapminder

2007-03-20T10:18:00.000-05:00

Thanks to a hat tip from Pat McConnell, I've learned that the latest news is it that GOOGLE has acquired gapminder's Trendalyzer software. Here's the scoop from the the gapminder home page.

Make sense of the world by having fun with statistics

Gapminder and Google share an enthusiasm for technology that makes data easily accessible and understandable to the world. Gapminder’s Trendalyzer software unveils the beauty of statistics by converting boring numbers into enjoyable interactive animations. We believe that Google’s acquisition of Trendalyzer will speed up the achievement of this noble goal. Trendalyzer’s developers have left Gapminder to join Google in Mountain View, where Google intends to improve and scale up Trendalyzer, and make it freely available to those who seek access to statistics.

The Stockholm-based Gapminder Foundation will continue to spearhead the use of new technology for data animations. The goal is to promote a fact-based worldview by bringing statistical story-telling to new levels. In collaboration with producers of accurate statistics that are eager to give the public free access to databases, Gapminder hopes to recruit and inspire many users of public statistics.

If you haven't seen Trendalyzer in action, you are in for a treat. You can check it out for yourself with this powerful example.

Trendalyzer in simple terms is a timeline in motion machine which has all sorts of potential to help us discover previously unseen patterns. Google's stated intentions to continue to advance on this already powerful design is an exciting development for anyone interested in understanding the behavior of important factors and how they change over time.

timelines in motion

2007-03-20T09:18:00.000-05:00

A nice example of timelines in motion accompanied by an audio track from Brad Delong.

The combination of a timeline change in motion and a guided audio track is powerful.

To make this even better, I would have liked it to zip along at a faster pace and it would be great to have a higher resolution graphic quality so that the X and Y axes could be more readily examined and understood.

You won't be surprised to hear me saying that making the data available in a readily re-usable format would sweeten things even further.

Providing a mechanism similar to gapminder to be able to watch this change over time with the viewer in control of the speed would be even better.

What do you think?

Many Eyes - A New Doorway for Graphical Collaboration

2007-02-12T10:59:00.000-05:00

From our previous post we noted that:

We believe in many eyes, many hands, many voices working together on the path of mutual discovery (rather than the idea of a single expert figuring it all out and presenting it to us in final form).

To help foster exactly this kind of process, I am excited to note a new web site, Many Eyes that is now available for exactly this purpose. As they describe it on their home page: Many Eyes is a place to: view your data, ask questions, share your discoveries, and harness the collective intelligence of the net for insight and analysis.

Once you register, you can upload interesting data sets and create visualizations that you feel are meaningful. Others can view your visualization and comment. They can also create their own variant visualizations using the data you uploaded and publish these for others to view and comment. Many different visualization formats including time series graphs, bar charts, scatter charts as well as some very creative new chart types are available in an interactive fashion. Creating new charts is similar to the kinds of interactions you might experience with one of the many stock quote charting web sites such as the java charts on prophet.net

Even better, once a data set is uploaded by anyone, it is possible someone else to download that data set for further analysis on their system using their favorite tools. For example, you might want to take a set of raw time line data series and compute some additional columns of data as we did in one of our previous Iraq posts. You could then take your new data set with added columns and upload it to Many Eyes for others to work with directly.

To test out these capabilities, I uploaded the O'Hanlon indicators that we have recently posted about. You can find a visualization that shows a single indicator that I created at: Percent of Coalition Troops Appears to be Shrinking. Once you get to that page, you can select other indicators from the list box on the left side to view the other trends at work. You can also download the entire table to your computer or leave a comment or create at publish your own chart. [Unfortunately, at present the long indicator names cannot be fully seen during the selectino process. So if you want to know what they are, you can find out by clicking on the table like icon element to look at and scroll through the top row of the data set where the indicator names appear]

I see Many Eyes (what a great name) as a big step forward in thinking about, discussing, understanding, and collaborating about the most important factors in our lives. I expect I will be uploading additional data sets and creating new visualizations there in the future. I would love to hear what you think about his newly available capability.

Change Over Time: First Principles Revisted

2007-02-12T09:38:00.000-05:00

I am enjoying reading Edward Tufte's new book: Beautiful Evidence. It's the fourth in a series of wonderful Tufte volumes that exemplify the power of visual display and it certainly lives up to its title. Beautiful Evidence, like its predecessors, is simply a beautiful and inspiring work - blending art and science and philosophy and the linkages between numeric data and their visual representations into a unifying whole.

Tufte is powerfully interested in the underlying principles. For example, here's a brief passage from his introduction to Beautiful Evidence:

Evidence is evidence, whether words, numbers, images, diagrams, still or moving. The intellectual tasks remain constant regardless of the mode of evidence: to understand and to reason about the materials at hand, and to appraise their quality, relevance, and integrity.

... The point of evidence displays is to assist the thinking of the producer and consumer alike. Evidence presentations should be created in accord with the common analytical tasks at hand, which usually involve understanding causality, making multivariate comparisons, examining relevant evidence, and assessing the credibility of evidence and conclusions.

This blog is especially concerned with a particular kind of evidence: multi-dimensional time series. We are surely interested in the underlying intellectual and analytical tasks including understanding causality, making comparisons, and cross checking conclusions.

"Beauty" on the other hand, plays a much smaller role in our efforts. It is always a delight when it makes one of its rare appearances for us, but is not a primary or even a secondary goal. In fact, many of the approaches we recommend in order to meet our primary mission by their very nature tend to lead away from beauty.

Reading Tufte always encourages me to take a step back and think about the first principles that are the foundation of the work of this blog. I want to think through and examine all that his approach has in common with the work shown in this blog and all the ways we might differ.

Here are some further thoughts about the first principles that are the basis of my continuing work with the visualization of time series data (with special emphasis today on the principles that might mitigate against "beauty").

Time is of the essence - both analyst time and viewer time. While making comparisons and searching for understanding and seeking out properties of the data that have never before been seen, we will give preference to those approaches that help speed up the work of the analyst, even if some or a lot of beauty must be sacrificed.
Following Covey, we seek first to understand the data and then to present our findings in ways that make it easiest and quickest for others to understand.
Sharing the data in a readily accessible and readily reusable form along with the presentation of any findings is perhaps the core principle of our work. It's the one that allows for collaboration and for a rapid and multi-step search for meaning.
We aim to make both the data and whatever graphical representations we create accessible to experts and non-experts alike.
We care deeply about practical tools that allow ordinary citizens to participate in this work of comparison and understanding.
We believe in many eyes, many hands, many voices working together on the path of mutual discovery (rather than the idea of a single expert figuring it all out and presenting it to us in final form).
We want to apply this approach to the most important, practical, everyday problems that we face. We want this approach to help dramatically improve decision making. And we want this to happen in near real time.

For example, in hindsight, the famous Napoleon' March graphic that Tufte has written extensively about is certainly a beautiful and powerful display that promotes understanding of what happened to Napoleon's army on the way to Russia and back. But of course, it was not available in time to help Napoleon make better decisions. What we are interested in is figuring out ways in which such powerful understanding might be made available soon enough (in near real time) to actually be of use to the decisions makers and to ordinary citizens as complicated and important events are unfolding. Our recent series of posts about the key indicators at work that help us understand how things are going in Iraq is a step in that direction.

To sum up, in our approach, if we have to chose, we will chose timeliness, usefulness, ready re-usability, accessibility to the ordinary citizen, tools that are available here and now, and the ability to collaborate over beauty every time.

Edward Tufte's "Beautiful Evidence", Sparklines & CSVPNG

2007-02-12T09:21:00.000-05:00

Edward Tufte is out with a new and inspirational book: Beautiful Evidence which includes a full chapter on the idea of sparklines - small word like graphic elements that can be integrated fully with text and that allow for massive amounts of statistical data to be presented in a compact format that encourages comparison between elements.

Not many software tools are yet available to help create the sparkline graphical elements. One exception is the T4 & Friends utility: CSVPNG that we have talked about in previous posts. CSVPNG has some introduced some new basic capabilities that can let you experiment with creating your own sparklines without having to write all the code yourself. The latest version of CSVPNG can be downloaded from the Trends That Matter download page

Apprenticed Investor: Know Thyself

2007-01-24T10:16:00.000-05:00

Apprenticed Investor: Know Thyself

Hat tip to Brad Delong for pointing out and reminding me of this excellent article by Barry Ritholtz in his Apprenticed Investor. While Barry is directing his comments at our all too human behaviors when we are investing in stocks, his discussion of pitfalls to watch out for is highly relevant to the analysis of any set of complex data (e.g. the 142 Iraq factors we have so far identified).

Take a look. It's a quick read and well worth the effort

Top 12 factors (one citizen's view)

2007-01-24T10:00:00.000-05:00

Here's an even shorter list of what I consider to be the top 12 factors to watch right now.

This does not mean that we should only look at this subset. In my opinion, every item on the previous post's list of 34 is of high importance and worthy of being measured, tracked, and reported consistently in near real time.

In fact, while perhaps of not as high importance (and opinions will of course vary on this), almost every item on the list of 142 deserves regular attention - measurement, tracking, and reporting - so that the full picture can be brought into view.

From the Brookings Iraq Index Report

Iraqi Deaths from All Violent Causes

U.S Dept. of State's Iraq Weekly Status Report (30 pages, January 10, 2007).

Crude oil production

5 trend graphs drawn from Iraq Coalition Casualties

US Soldiers Killed with moving avg & event timing - March 2003 - December 30, 2006
US Soldiers Wounded with moving avg - Sept 6, 2003 - Dec 29, 2006

from Michael O'Hanlon's testimony. 30 factors

Daily Average of Inter-ethnic Attacks
Avg Hours/Day of Power Baghdad (prewar 20)
Unemployment Rate (%)

Additional factors (missing metrics) contained in the text of O'Hanlon's testimony

Rate of Jobs Creation

55 missing metrics/invisible indicators/faceless factors from Brookings Iraq Index

Iraqis Kidnapped in Baghdad (table page 15)
Number of total daily patrols (chart page 23)
People with potable water (table page 42)
Percentage of Iraqis who strongly oppose presence of Coalition troops (table page 51)

What does your short list look like?

First cut at identifying most important factors

2007-01-24T09:33:00.000-05:00

Here's one private citizen's view on which of the 142 Iraq factors identified so far are most vital to understanding what is happening on the ground in Iraq. These are the ones that I believed need to be measured and reported on regularly so we can see the trends at work. I have picked 34 out of 142 (or about 24% of the total) so this is something like an 80/20 selection.

What does your list look like?

It would be excellent to see Michael O'Hanlon's list, or Anthony Cordesman's list, or Secretary Gate's list, or Secretary Rice's list, or Lt. Gen. Petraeus' list of the top factors they will be considering as we go forward.

From the Brookings Iraq Index Report

Average Weekly Attacks by Time Period – 01 Jan 2004 – 11 Aug 2005
Iraqi Deaths from All Violent Causes – Jan 06- Oct 06
Enemy-Initiated Attacks Against the Coalition and its Partners May 2003 –Aug 2006

U.S Dept. of State's Iraq Weekly Status Report (30 pages, January 10, 2007).

Crude oil production for the past 10 weeks.
Electricity production compared to electricity demand since January 2004.

5 trend graphs drawn from Iraq Coalition Casualties

US Soldiers Killed with moving avg & event timing - March 2003 - December 30, 2006
US Soldiers Wounded with moving avg - Sept 6, 2003 - Dec 29, 2006

from Michael O'Hanlon's testimony. 30 factors

Estimated Strength of Insurgency
Estimated Strength of Shia Militias
Daily Average of Inter-ethnic Attacks
Iraqi Security Forces who are Politically Dependable
Percent of Household Fuel Needs Available
Avg Hours/Day of Power Baghdad (prewar 20)
Unemployment Rate (%)

New indicators calculated from combinations of O'Hanlon's metrics

total number of Iraqi Physicians who have murdered, kidnapped, or who have left Iraq.
percent of coalition troops that are not US troops
Percent of US troops killed by IEDs
Year over year increase in Iraqi refugees

Additional factors (missing metrics) contained in the text of O'Hanlon's testimony

Rate of Shia attacks on Sunnis
Rate of Sunni attacks on Shia
The number of children being immunized.
Rate of Jobs Creation
Degree of Overwork of our Soldiers and Marines

55 missing metrics/invisible indicators/faceless factors from Brookings Iraq Index

Civilian deaths in Baghdad (page 12 table)
Iraqis Kidnapped in Baghdad (table page 15)
Iraqi prison population in US Custody (table page 16)
Iraqi prison population in Iraqi Custody (table page 16)
Number of total daily patrols (chart page 23)
Gasoline/Benzene production (table page 29)
Total cost of US operations (table page 37)
People with potable water (table page 42)
People with sewerage system coverage (table page 42)
High School enrollment percentage (table page 43)
Percentage of Iraqis who strongly oppose presence of Coalition troops (table page 51)