Showing posts with label trends. Show all posts
Showing posts with label trends. Show all posts

Sunday, October 9, 2011

Watch the trend, not just the latest data point

Some excellent thinking from Barry Ritholtz in his recent post NFP Report: Trend vs Single Data Point discussing on how best to interpret the latest job numbers.   

The key ideas I take away from this are to 1) watch the trend, 2) ask the right questions to establish context, 3) examine a wider range of metrics (not just the headline numbers),  and especially 4) not to over focus on the latest data point. 

Related to some of our recent posts, one of the things that makes following Barry's advise more difficult these days is the way that the standard reporting modes for key metrics almost universally use Month-over-Month, and/or or Year-over-Year figures.   Other ways of reporting the data can (such as net change over 2 years or 5 years) can reveal details that would otherwise remain hidden.  

Playing with the data, keeping context in mind, and looking for the trends and the answers to our deepest questions is the best path to more complete understanding and improved decision making. 

Saturday, March 24, 2007

Latest Iraq Trends from the Brookings Institution

The Brookings Institution continues its weekly Iraq Trend reporting. Here are some interesting charts gleaned from their March 22nd edition. Note the link to the Brookings Latest Iraq Trends Report will always take you to their most recent version.

Security Stations Established. This is a new chart reflecting changes underway as a result of the "surge" showing the number of security stations (cumulative) that have been established in Baghdad. The new goal is to reach 70 stations (original goal of 30).


Patrols Per Week. This new chart shows patrols carried per week and shows an impressive jump up in recent weeks.

IED Fatalities & Percentage. This dual axis chart shows both the number of IED fatalities (red bars, left scale) and the percentage of all fatalities that are attributable to IEDs (blue line, right scale).

Targetted Attacks. This chart shows average weekly attack based on who the attacks target - Coalition forces, Iraqi forces, or others.

Iraq Reconstruction Management Office: Employment Chart

Here's latest employment chart from the US State Department's Iraq Reconstruction Management Office (IRMO) Report dated March 20th, 2007.

It shows Iraqi employment in US Govt. projects and what appears to be a rapidly declining trend. It would interesting to determine whether the change in this factor was having an impact on other key success factors in Iraq.

If you would like to be on an email distribution list for the full weekly IRMO report which also contains detailed updates about Electricity, Oil, Water, and Sanitation project. you can send your request to:
jddurso AT earthlink DOT net

Wednesday, January 3, 2007

TLViz – A Little Historical Background

Before TLViz came on the scene, I was part of a team in Hewlett Packard's OpenVMS Engineering that was already enjoying success with readily-reusable (R-R) CSV files as part of the T4 & Friends project (more on this later). These files contained detailed system performance trend data for many different factors that had been extracted from trend data that was originally stored in a proprietary, difficult to reuse format.

Originally, when we created these R-R files known as T4 files, we simply input them to Microsoft Excel. The R-R format allowed us to make full use of the trend visualization capabilities and other features already built into Excel. This immediately extended our ability to visualize the trend data beyond the limitations built into the tools that had originally collected the most important performance data that we had wanted to investigate. Conversion to R-R format by itself gave us an order of magnitude time-saving and productivity improvement compared to previous methods for looking at this trend data.

What we discovered was that Ian Megarity's creation of the TLViz utility changed everything for us, literally overnight. In particular, what TLViz accomplished was to change the productivity equation even further, allowing us to analyze and report on a complex trend data set at least 5 times more quickly than we could have done if we had continued to use only Excel. In one example, a project that had previously taken two weeks was shrunk down to less than half a day.

We also discovered that TLViz was helpful in multiple ways - it dramatically cut analyst time AND it also made it easier and faster to present the results of analysis to others (both expert and non-expert) AND it greatly increased the possibilities for collaboration.

These additional productivity improvements alone made TLViz a worthwhile tool to add to our repertoire for the OpenVMS performance work we were doing. TLViz at its conception was designed for the very specific purpose of looking at OpenVMS System Performance Data. TLVIZ not intended as a general purpose trend visualization mechanism. However, experience with TLViz in the past 5 years has shown that it can be mapped over to the general case and by doing so it can offer some of the same time-saving and productivity and collaboration advantages.

While there are certainly areas for improvement for the general case, TLViz today provides a demonstration of what might be possible to do in the general case of making trend data more accessible to a wider audience with the intention of radically improving productivity, promoting collaboration and encouraging mutual learning.

In the next post, we will outline some other key reasons why we think TLViz is such an important demonstration of what is possible for the general case of trend collaboration that we are advocating in this blog..

In the meantime, to give this a quick try yourself, we recommend you DOWNLOAD TLViz from HP's T4 & Friends page, then DOWNLOAD http://trendsthatmatter.com/nbu/t4-data/rr-example.csv (a sample readily-reusable file of World Watch data file), and then use TLViz to open rr-example.csv.

Download Readily Reusable File Example

Following up our posts from yesterday. It was our intention to make it easy for you to download a readily reusable trend data file that you could try out with the TLViz (TimeLine Visualizer) software or read into your favorite tool such as Microsoft Excel.

Yesterday's method was too complex, took too many keystrokes and sometimes ran afoul of overly smart browser software that changed the format of the file upon downloading.

We now have arranged a workable method that uses a minimum of keystrokes and that preserves the file in its original format so that it will continue to work with TLViz.

Right click on the URL below and save the results to your desktop or designated folder.
http://trendsthatmatter.com/nbu/t4-data/rr-example.csv

Alternatively, you can open the CSV text file in your browser and then select File, Save as to save it to your desktop but the right click method takes fewer keystrokes.

Have fun trying out TLViz. We'll be posting more about this tool and why we think it represents an important demonstration of the potential of readily reusable trend data combined with timeline collaboration methods.

Tuesday, January 2, 2007

World Watch State of the World data in Readily Reusable form

Temporary Workaround

We are still working on finding a convenient way to share readily reusable data so that it will be as easy as possible for you to download. We're not there yet. So in the Meantime, I thought I would try an experiment of pasting the data into this blog entry. It will be clumsy but you should be able to select copy the text data (just the text in Orange) and paste it into the Notepad editor and then save the file with a CSV extension.

Once you have done that, you will be able to have fun playing with this interesting data set using TLViz.

War & Peace,Source: World Watch State of the World 2002,Composite of selected factors including ,transportation,fossil fuel use,nuclear weapons,world food supply,armed forces manpower ,population,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Sample Time,Air Billions of Freight Ton Kilometers,Air billions of passenger kilometers,Autos Passenger Cars,Autos People Per Car,Autos total,Autos World Wide Total Fleet,Bike production in millions China,Bike production in millions US,coal million tons oil equiv china,coal million tons oil equiv former soviet union,coal million tons oil equiv USA,coal million tons oil equiv world wide,Employment World Arms Industry in millions,Food - per capital food production index,Food - Prices in Constant 1999 dollars,fossil fuel consumptions millions of tons OIL world wide,fossil fuel consumptions millions of tons oil equiv ALL world wide,fossil fuel consumptions millions of tons oil equiv NATURAL GAS world wide,Grain Carryover in Days of supply,Grain Carryover in Millions of Tons,Grain Corn Millions of Tons,Grain for Feed Use in Millions of Ton,Grain Per Person in Kilograms,Grain Percentage for Feed Use,Grain Rice Millions of Tons,Grain Total Production in Millions of Tons,Grain Wheat Millions of Tons,Grain Yield in Tons Per Hectare - ALL,Grain Yield in Tons Per Hectare - Corn,Grain Yield in Tons Per Hectare - Rice,Grain Yield in Tons Per Hectare - Wheat,Nukes China,Nukes France,Nukes Soviet Union/Russia Nukes,Nukes Total World wide,Nukes United Kingdom Nukes,Nukes United States Nukes,Population - annual addition in Millions,Population Annual Growth rate in Percent,Population World Wide in Billions,Refugees per UNHCR Africa in Millions,Refugees per UNHCR Europe in Millions,Refuges per UNHCR Asia in Millions,Refuges per UNHCR World Wide in millions,Soldiers China in Millions,Soldiers India in Millions,Soldiers North Korea in Millions,Soldiers US in millions,Soldiers USSR in Millions,Soldiers World Wide in millions,Travel International Tourist Arrivals in Millions,Travel International tourist receipts in Billions of 1999 dollars
1/1/1990 0:00,58.8,1894.2,36.3,11.9,0,444.9,31.9,5.6,533.6,308,482.3,2270.3,16.3,100.8,51.7,3135.5,7150,1773.8,104,490,482,665,335,39,352,1769,588,2.54,3.75,2.4,2.54,432,504,38000,60236,300,21000,83.2,1.6,5.3,4.6,0.8,6.8,14.9,3.5,1.3,1.2,2.2,3.7,28.2,457.2,319.1
1/1/1991 0:00,58.6,1845.4,35.1,11.8,0,456,36.8,7.6,534.9,277.7,478.5,2224.5,15.4,99.4,50.1,3134.3,7165.1,1806.3,102,479,491,654,318,38,355,1708,543,2.47,3.73,2.41,2.44,434,538,35000,55772,300,19500,82.7,1.5,5.4,5.6,0.9,7.9,17.2,3.2,1.3,1.2,2.1,3.6,26.5,462.7,323.4
1/1/1992 0:00,62.6,1928.9,35.5,11.6,0,469.9,40.3,8.9,549.5,265.5,482.7,2211.3,14,100.6,49.1,3165,7185.8,1809.5,107,513,539,663,328,38,356,1789,562,2.57,4.05,1.97,2.52,434,538,33500,52972,300,18200,81.3,1.5,5.4,5.3,1.2,8.6,17,3.2,1.3,1.2,1.9,3.6,25.9,500.9,359.7
1/1/1993 0:00,68.4,1949.4,34.2,11.8,0,469.5,41,7.7,570.3,238.7,498.1,2206.1,13,100,48.6,3135.1,7190.4,1849.2,98,471,476,654,310,37,355,1713,559,2.5,3.64,2.45,2.52,434,524,32000,50008,300,16750,80,1.4,5.5,5.4,3.8,7.2,19,3,1.3,1.2,1.8,3.2,25,515.7,358.4
1/1/1994 0:00,77.2,2099.9,34.8,11.7,47.6,479.9,42,7.3,606.4,211.5,502.3,2224.4,11.6,101.6,49.3,3192.1,7274,1857.5,94,459,560,670,314,38,365,1760,524,2.57,4.15,2.47,2.44,400,512,30000,46542,250,15380,79.9,1.4,5.6,7.5,6.1,8,23,2.9,1.3,1.2,1.7,2.8,24.1,550.3,387.1
1/1/1995 0:00,83.1,2248.2,35.5,11.9,48.5,477,41,8.5,635.7,192.7,504.6,2258.3,10.6,102.3,49.4,3235.4,7406.9,1913.2,85,409,517,638,301,36,371,1713,538,2.52,3.85,2.51,2.46,400,500,28000,43200,300,14000,77.7,1.4,5.7,11.8,6.5,7.9,27.4,2.9,1.2,1.1,1.6,2.5,23.5,565.4,433.2
1/1/1996 0:00,89.2,2431.7,36.9,11.9,50.3,486,38,7.4,676.9,178.7,528.1,2341.6,9.4,105.1,58.1,3315.5,7662.2,2005.1,90,450,592,673,324,37,380,1871,582,2.66,4.2,2.54,2.53,400,500,26000,40100,300,12900,78.2,1.4,5.8,9.1,7.7,7.7,26.1,2.9,1.2,1.1,1.6,2.2,22.6,597.4,457.2
1/1/1997 0:00,102.9,2573,39.1,11.7,53.2,498,30,6,652.8,175.2,540.9,2326.8,9,106.3,54.5,3387.7,7707,1992.5,98,494,575,686,322,37,387,1880,609,2.72,4.24,2.55,2.67,400,450,24000,37535,260,12425,77.8,1.3,5.8,8.1,5.7,7.9,22.7,2.9,1.2,1.1,1.5,2,21.9,618.2,451.8
1/1/1998 0:00,101.8,2628.1,38.4,11.6,51.9,510.3,23.1,2.5,621.2,166.6,543.2,2281.1,8.5,106.8,48.5,3398.2,7694.9,2015.6,104,523,606,679,316,37,394,1872,589,2.74,4.35,2.55,2.62,400,450,22000,34535,260,11425,77.9,1.3,5.9,7.4,6.1,7.5,22.4,2.7,1.2,1.1,1.5,1.9,21.9,636,448.4
1/1/1999 0:00,108.7,2797.8,39.9,11.5,54.8,520,42.7,1.7,516.6,170.7,546.4,2159.9,8.2,108.3,42.1,3469.1,7694.2,2065.2,102,523,607,683,312,36,409,1871,586,2.78,4.26,2.56,2.65,400,450,20000,31960,185,10925,77.6,1.3,6,6.3,6.2,7.5,21.5,2.5,1.2,1.1,1.5,1.8,21.7,650,455.5
1/1/2000 0:00,118.3,3010.2,41.1,11.1,56.7,545.6,52.2,1.1,483.8,175,564.1,2186,8.2,108.2,43,3503.6,7853.6,2164,95,489,586,688,303,37,397,1840,583,2.76,4.31,2.62,2.66,400,450,20000,31535,185,10500,77.3,1.3,6.1,6.3,7.3,7.3,22.3,2.5,1.2,1.1,1.5,1.8,21.7,698,468.6

TLViz - A Trend Visualization Tool for Readily Reusable (R-R) Data

TLViz - A Trend Visualization Tool for Readily Reusable (R-R) Data

The TimeLine Visualizer (TLViz) is an easy to use Windows PC tool. TLViz was originally conceived and developed for OpenVMS Engineering by Ian Megarity at Hewlett Packard in 2001. It was part of what became known as the T4 & Friends project (more about T4 in future posts).

Right from the start, TLViz was expressly designed to work directly with readily-reusable computer system trend data stored in CSV (comma separated values) files. The TLViz format for CSV files obeyed the few simple R-R rules outlined in our previous post on readily reusable data.

Since TLViz is a tool intended to dramatically enhance productivity for dealing with trend data through visualization, the best way to understand its capabilities is for you to try it out on some sample trend data yourself.

A working copy of the TLViz software can be downloaded without charge from the HP's T4 & Friends web site. It will take you just a few minutes to install it on your Windows PC.

At the HP T4 & Friends web site you will find additional background material about TLViz and about the T4 project. You might want to check out the T4 FAQ page and take a look at the Technical Journal Article on TimeLine Collaboration that's also available at that site.

In future posts, we will work through some examples showing the highlights of TLViz' capabilities. In the meantime, once you have it up and running on your PC, you can try out TLViz yourself with the following sample readily-reusable CSV trend data file. The source for this data is World Watch State of the World 2002. Many independent factors from many different areas of interest were combined together in a single file with each covering the same 1990 to 2000 time frame. The areas combined together include

  • world population
  • travel
  • nuclear weapons
  • population
  • agriculture
  • armed forces manpower
  • fossil fuel use.

In the next post, we will show some possible impressions you might create/discover by playing with this composite readily reusable data file using TLViz by generating a few sample output graphs.

Update: January 2nd, 2006, 4:15 PM - If you would like a copy of the RR file, please send mail to steve.lieman@trendsthatmatter.com and I will forward it to you. We are working out different possible ways to make it easy for you to download such files in the future but they are not quite ready for prime time yet. My apologies for any inconvenience this may have caused you.

Monday, August 28, 2006

Timeline Collaboration Principles

The initial starting belief of this Change Over Time blog is that if you want to change the world, the best approach is to build better tools and then learn how to harness their power.

Peter Drucker tells us that FOCUS is the key to success and we follow his advice with a focus on timelines and trend data that tracks the areas of our lives that are most important to us. A key to unlocking the meaning of these data is a continuing search for the tools and methods and principles that best help us analyze, visualize, report and discuss our findings. We are on the lookout for tools that simplify, clarify, and especially those that save us time as we share our findings and collaborate with expert and non-expert alike. We wish to discover what the data means for us in our lives and what actional steps we might take for making the world a better place.

Why do we focus on trends and timelines?

First: Timeline data is often widely available for a substantical collection of key measures in every area of human interest. We are simply overflowing with such data. Where it is not available, it appears almost always possible to create a new data collector that will gather the missing metrics.

Second: Our observation is that most of the time, the available data is not put to its best use as key principles that would guarantee success are openly violated. Lots of opportunity appears within easy reach.

Third: In one domain after another, we have been establishing and documenting proof that substantial improvements in how we use trend data are already available or well within our reach by following a straightforward set of rules and principles and we can point to a growing number of examples on the web that show these approaches in action.

Fourth: In some cases, the existing work makes collaboration (especially between expert and non-expert) somewhat easier, but the collaboration aspect of making best use of trend data does not seem to have been actively explored. I believe that a handful of principles and standard practices can help us learn how to collaborate better by at least an order of magnitude. As we do so, we will advance towards having better and better control for shaping the future and achieving our fondest dreams.

Limited examples of how to use trend data more powerfully are popping up on the web. One of our goals on this blog is to find these examples of excellent practice. We want to use these best practices as models for what is possible if the underlying principles were applied to other domains and trend data collections. For example, we gave some examples drawn from the St Louis Federal Reserve Bank's interactive trending capabilitites named FRED. Other examples can be found at The Big Picture , the Bureau of Labor Statistics, Bureau of Justice Statistics , and Professor Pollkatz .

Here are the TimeLine Collaboration Principles that I believe are going to prove most important to the goal of this blog of making the best use of trend data. We have already discussed some of these in previous posts and will be preparing additional posts for these principles to explain the logic behind them in greater detail.

1. Share the data series with the chart. Make sure it is readily reausable
2. Multi-dimensionality is key
3. Data set includes entire time range even if chart doesn't
4. Explain how calculated quantities were obtained
5. Make sure the explanatory text is in close physical proximity to the trend chart
6. Data + Charts + Text creates a full package that encourages further conversation
7. Ask the expert in the subject matter domain: What are the most important factors?
8. Then, make sure you measure and record and create a timeline history of every one of these
9. If you have the most important factors, you'll find charts with but a single variable still tell a powerful story
10. Make sure the Axes and Titles and other text graphics are easily readable