Iraq Trend Data Set now posted to Swivel and Many Eyes

A while back we mentioned the great new data sharing and visualization web site called Many Eyes and we pointed out an Iraq Trend data set we had uploaded so that others could more readily examine and analyze it.

We've recently come across another great new data sharing and visualization web site called Swivel and have uploaded the same data set there for comparison purposes at: http://www.swivel.com/data_sets/show/1004780.

Both Many Eyes and Swivel fall into a category I have started calling Trend Visualization Appliances. The amazing Gapminder work also fits into this category as do all the various stock market visualization tools currently available.

The old fashioned form of trend visualization appliances (e.g. a standard report including graphical output) can still be pretty useful when managed carefully. For further comparison, here is the link to a PDF report that graphically presents the same data - vizualizing-trends-ohanlon-testimony.pdf.

Similarly, for data sharing, the old fashioned URL links to csv files offer an alternative to the WEB 2.0 mechanisms such as Many Eyes and Swivel. Here's the Iraq trend data the old fashioned way: ohanlon-key-factors.csv

So many choices, so little time. How can we decide which is best for our purposes?

I'll be revisiting this topic, but for now, my own criteria for deciding which tools I will use at a given moment are:
1) ease of use,
2) shortness of learning curve, and
3) personal productivity and time saving -- the speed at which I can navigate through complex trend data sets to discover previously hidden patterns.

What do you think?

Tapping the Power of Iraq Readily Reusable Data

In the previous post, we showed how the tabular data from Michael O'Hanlon's testimony to the Senate Foreign Relations Committee could be converted to a Readily Reusable (RR) format as a CSV file ( ohanlon-key-factors.csv ) and how the RR format then made it relatively straightforward and inexpensive to actually look at the visualization of all thirty of the reported trends, one by one as shown in vizualizing-trends-ohanlon-testimony.pdf.

We have found that when using RR data and TLViz, the time savings we achieve make it possible to look at hundreds of trends, one after the other, in a very short time so as to gain a gestalt sense of all the reported factors at work. Typically, moving from tabular form to RR form gives a productivity saving factor of at least 10 to 1. Frequently, the productivity increase is 50 to 1 or more.

After having reviewed each of the 30 reported trend factors from Michael O'Hanlon's testimony, it became apparent that there were some other interesting trends hiding amongst the original data that could be computed with simple calculations from the original data. This is readily achievable by opening the RR csv file with a tool such as Microsoft Excel.

For example, with Excel we could create a new column of data and combine US troop strength and troop strength of other non-US coalition forces and then calculate the percentage of coalitions forces that were non-US. Similarly, we could use the original data to calculate the percentage of US troops that had been killed in that month by IEDs. We could also take factors such as the cumulative number of refugees and convert them into year over year trend data.

Below, we show you the three new trend charts that bring some previously invisible trends into the light of day where we can all see them. (please click on trend graphic for full size image.)




Bottom line. Once the trend data is in RR format, all sorts of new possibilities of understanding open up that can help us better understand what is really going on.

TLViz – A Little Historical Background

Before TLViz came on the scene, I was part of a team in Hewlett Packard's OpenVMS Engineering that was already enjoying success with readily-reusable (R-R) CSV files as part of the T4 & Friends project (more on this later). These files contained detailed system performance trend data for many different factors that had been extracted from trend data that was originally stored in a proprietary, difficult to reuse format.

Originally, when we created these R-R files known as T4 files, we simply input them to Microsoft Excel. The R-R format allowed us to make full use of the trend visualization capabilities and other features already built into Excel. This immediately extended our ability to visualize the trend data beyond the limitations built into the tools that had originally collected the most important performance data that we had wanted to investigate. Conversion to R-R format by itself gave us an order of magnitude time-saving and productivity improvement compared to previous methods for looking at this trend data.

What we discovered was that Ian Megarity's creation of the TLViz utility changed everything for us, literally overnight. In particular, what TLViz accomplished was to change the productivity equation even further, allowing us to analyze and report on a complex trend data set at least 5 times more quickly than we could have done if we had continued to use only Excel. In one example, a project that had previously taken two weeks was shrunk down to less than half a day.

We also discovered that TLViz was helpful in multiple ways - it dramatically cut analyst time AND it also made it easier and faster to present the results of analysis to others (both expert and non-expert) AND it greatly increased the possibilities for collaboration.

These additional productivity improvements alone made TLViz a worthwhile tool to add to our repertoire for the OpenVMS performance work we were doing. TLViz at its conception was designed for the very specific purpose of looking at OpenVMS System Performance Data. TLVIZ not intended as a general purpose trend visualization mechanism. However, experience with TLViz in the past 5 years has shown that it can be mapped over to the general case and by doing so it can offer some of the same time-saving and productivity and collaboration advantages.

While there are certainly areas for improvement for the general case, TLViz today provides a demonstration of what might be possible to do in the general case of making trend data more accessible to a wider audience with the intention of radically improving productivity, promoting collaboration and encouraging mutual learning.

In the next post, we will outline some other key reasons why we think TLViz is such an important demonstration of what is possible for the general case of trend collaboration that we are advocating in this blog..

In the meantime, to give this a quick try yourself, we recommend you DOWNLOAD TLViz from HP's T4 & Friends page, then DOWNLOAD http://trendsthatmatter.com/nbu/t4-data/rr-example.csv (a sample readily-reusable file of World Watch data file), and then use TLViz to open rr-example.csv.

Timeline Collaboration Principles

The initial starting belief of this Change Over Time blog is that if you want to change the world, the best approach is to build better tools and then learn how to harness their power.

Peter Drucker tells us that FOCUS is the key to success and we follow his advice with a focus on timelines and trend data that tracks the areas of our lives that are most important to us. A key to unlocking the meaning of these data is a continuing search for the tools and methods and principles that best help us analyze, visualize, report and discuss our findings. We are on the lookout for tools that simplify, clarify, and especially those that save us time as we share our findings and collaborate with expert and non-expert alike. We wish to discover what the data means for us in our lives and what actional steps we might take for making the world a better place.

Why do we focus on trends and timelines?

First: Timeline data is often widely available for a substantical collection of key measures in every area of human interest. We are simply overflowing with such data. Where it is not available, it appears almost always possible to create a new data collector that will gather the missing metrics.

Second: Our observation is that most of the time, the available data is not put to its best use as key principles that would guarantee success are openly violated. Lots of opportunity appears within easy reach.

Third: In one domain after another, we have been establishing and documenting proof that substantial improvements in how we use trend data are already available or well within our reach by following a straightforward set of rules and principles and we can point to a growing number of examples on the web that show these approaches in action.

Fourth: In some cases, the existing work makes collaboration (especially between expert and non-expert) somewhat easier, but the collaboration aspect of making best use of trend data does not seem to have been actively explored. I believe that a handful of principles and standard practices can help us learn how to collaborate better by at least an order of magnitude. As we do so, we will advance towards having better and better control for shaping the future and achieving our fondest dreams.

Limited examples of how to use trend data more powerfully are popping up on the web. One of our goals on this blog is to find these examples of excellent practice. We want to use these best practices as models for what is possible if the underlying principles were applied to other domains and trend data collections. For example, we gave some examples drawn from the St Louis Federal Reserve Bank's interactive trending capabilitites named FRED. Other examples can be found at The Big Picture , the Bureau of Labor Statistics, Bureau of Justice Statistics , and Professor Pollkatz .

Here are the TimeLine Collaboration Principles that I believe are going to prove most important to the goal of this blog of making the best use of trend data. We have already discussed some of these in previous posts and will be preparing additional posts for these principles to explain the logic behind them in greater detail.

1. Share the data series with the chart. Make sure it is readily reausable
2. Multi-dimensionality is key
3. Data set includes entire time range even if chart doesn't
4. Explain how calculated quantities were obtained
5. Make sure the explanatory text is in close physical proximity to the trend chart
6. Data + Charts + Text creates a full package that encourages further conversation
7. Ask the expert in the subject matter domain: What are the most important factors?
8. Then, make sure you measure and record and create a timeline history of every one of these
9. If you have the most important factors, you'll find charts with but a single variable still tell a powerful story
10. Make sure the Axes and Titles and other text graphics are easily readable