CSVPNG – A Handy Utility for Readily-Reusable (R-R) Data

The T4 & Friends project at HP we mentioned in earlier posts was at its foundation a project that converted trend data from its original complex format into Readily-Reusable (R-R) format as CSV files. We discovered that any trend data could be so converted and that in most cases the transformation was straightforward, relatively painless and often quite rapid.

As a result of the T4 project, more complex data from a variety of sources was converted to R-R format. Those with interests in this data were then able to enjoy the possibility of looking at it with Excel, loading it into their favorite database or in many cases, analyzing and reporting on what the data meant using TLViz.

We sometimes find it convenient to use the metaphor that

1. data is collected ‘upstream’,
   
2. it is saved in R-R format in historical ‘reservoirs’ for possible future use, and
   
3. it is fed, as needed into ‘downstream’ tools for analysis, reporting, collaboration, …

Since no single such downstream tool does everything you might want, the availability of growing reservoirs of R-R data in CSV format systematically increased the possible value of building new tools or incrementally improving existing tools to take advantage and convert the potential value into real value.

CSVPNG (CSV file to Portable Network Graph Utility) is one such development that has blossomed in the world of ready-reusability. Developed in 2003 by Pat Moran of Hewlett Packard, CSVPNG’s initial purpose was to live up to its name – that is to convert CSV trend data automatically into a set of PNG graphics. These were then all embedded in a single output HTML page. Just this one feature alone added considerable value by automating the output of any sets of standard charts that you might always want to examine.

CSVPNG has then proceeded over the past 4 years to transform itself through a series of more than 100 improvements and extensions. You can download a copy of this great tool from the T4 & Friends web page or from TrendsThatMatter.

CSVPNG is written in C and has been made available for a steadily growing number of platforms including DOS under Windows, Linux, HP-UX, and OpenVMS on both Alpha and Integrity servers.

Here’s a brief rundown of some of the things you can do with R-R CSV data using CSVPNG. For a full list of possible benefits, be sure to examine the CSVPNG.TXT file thoroughly after you have downloaded the kit. If you are generating readily-reusable data in CSV format, CSVPNG will surely save you time in managing this data and will help make sure that you don't have to reinvent the wheel.

1. Automatically create a selected set of graphs and format as html or PDF
2. Slice and dice files to select out just the factors you want and just the time periods you are interested in. The trimmed down result can be automatically graphed and/or used to generate a new reduced file that satisfies the R-R rules. These files could then be fed into other tools such as TLViz or Excel.
3. Combine R-R data from several files into a single synchronized file
4. Search using expert rules for “interesting” conditions and then display only those charts for cases where one or more conditions were met
5. Carryout column arithmetic to create new factors by recombining several original factors.

Here’s a sample graphic output from CSVPNG showing how it can find and highlight a peak period as well as display the original data with moving average value all in a single chart. The blue represents the raw data value, the red line shows the moving average, and the highlighted time interval identifies the peak interval.

The more R-R data you have to deal with, the more valuable you are likely to find the capabilities for CSVPNG for helping you automate and keep on top of all this data.

TLViz - A Trend Visualization Tool for Readily Reusable (R-R) Data

TLViz - A Trend Visualization Tool for Readily Reusable (R-R) Data

The TimeLine Visualizer (TLViz) is an easy to use Windows PC tool. TLViz was originally conceived and developed for OpenVMS Engineering by Ian Megarity at Hewlett Packard in 2001. It was part of what became known as the T4 & Friends project (more about T4 in future posts).

Right from the start, TLViz was expressly designed to work directly with readily-reusable computer system trend data stored in CSV (comma separated values) files. The TLViz format for CSV files obeyed the few simple R-R rules outlined in our previous post on readily reusable data.

Since TLViz is a tool intended to dramatically enhance productivity for dealing with trend data through visualization, the best way to understand its capabilities is for you to try it out on some sample trend data yourself.

A working copy of the TLViz software can be downloaded without charge from the HP's T4 & Friends web site. It will take you just a few minutes to install it on your Windows PC.

At the HP T4 & Friends web site you will find additional background material about TLViz and about the T4 project. You might want to check out the T4 FAQ page and take a look at the Technical Journal Article on TimeLine Collaboration that's also available at that site.

In future posts, we will work through some examples showing the highlights of TLViz' capabilities. In the meantime, once you have it up and running on your PC, you can try out TLViz yourself with the following sample readily-reusable CSV trend data file. The source for this data is World Watch State of the World 2002. Many independent factors from many different areas of interest were combined together in a single file with each covering the same 1990 to 2000 time frame. The areas combined together include

• world population
• travel
• nuclear weapons
• population
• agriculture
• armed forces manpower
• fossil fuel use.

In the next post, we will show some possible impressions you might create/discover by playing with this composite readily reusable data file using TLViz by generating a few sample output graphs.

Update: January 2nd, 2006, 4:15 PM - If you would like a copy of the RR file, please send mail to steve.lieman@trendsthatmatter.com and I will forward it to you. We are working out different possible ways to make it easy for you to download such files in the future but they are not quite ready for prime time yet. My apologies for any inconvenience this may have caused you.

Timeline Collaboration Principles

The initial starting belief of this Change Over Time blog is that if you want to change the world, the best approach is to build better tools and then learn how to harness their power.

Peter Drucker tells us that FOCUS is the key to success and we follow his advice with a focus on timelines and trend data that tracks the areas of our lives that are most important to us. A key to unlocking the meaning of these data is a continuing search for the tools and methods and principles that best help us analyze, visualize, report and discuss our findings. We are on the lookout for tools that simplify, clarify, and especially those that save us time as we share our findings and collaborate with expert and non-expert alike. We wish to discover what the data means for us in our lives and what actional steps we might take for making the world a better place.

Why do we focus on trends and timelines?

First: Timeline data is often widely available for a substantical collection of key measures in every area of human interest. We are simply overflowing with such data. Where it is not available, it appears almost always possible to create a new data collector that will gather the missing metrics.

Second: Our observation is that most of the time, the available data is not put to its best use as key principles that would guarantee success are openly violated. Lots of opportunity appears within easy reach.

Third: In one domain after another, we have been establishing and documenting proof that substantial improvements in how we use trend data are already available or well within our reach by following a straightforward set of rules and principles and we can point to a growing number of examples on the web that show these approaches in action.

Fourth: In some cases, the existing work makes collaboration (especially between expert and non-expert) somewhat easier, but the collaboration aspect of making best use of trend data does not seem to have been actively explored. I believe that a handful of principles and standard practices can help us learn how to collaborate better by at least an order of magnitude. As we do so, we will advance towards having better and better control for shaping the future and achieving our fondest dreams.

Limited examples of how to use trend data more powerfully are popping up on the web. One of our goals on this blog is to find these examples of excellent practice. We want to use these best practices as models for what is possible if the underlying principles were applied to other domains and trend data collections. For example, we gave some examples drawn from the St Louis Federal Reserve Bank's interactive trending capabilitites named FRED. Other examples can be found at The Big Picture , the Bureau of Labor Statistics, Bureau of Justice Statistics , and Professor Pollkatz .

Here are the TimeLine Collaboration Principles that I believe are going to prove most important to the goal of this blog of making the best use of trend data. We have already discussed some of these in previous posts and will be preparing additional posts for these principles to explain the logic behind them in greater detail.

1. Share the data series with the chart. Make sure it is readily reausable
2. Multi-dimensionality is key
3. Data set includes entire time range even if chart doesn't
4. Explain how calculated quantities were obtained
5. Make sure the explanatory text is in close physical proximity to the trend chart
6. Data + Charts + Text creates a full package that encourages further conversation
7. Ask the expert in the subject matter domain: What are the most important factors?
8. Then, make sure you measure and record and create a timeline history of every one of these
9. If you have the most important factors, you'll find charts with but a single variable still tell a powerful story
10. Make sure the Axes and Titles and other text graphics are easily readable