The Iraq Transition Assistance Office (part of the US Embassy in Baghdad) is now publishing a Weekly Status Report. Here's a link to the May 29th, 2007 report.
This is a continuation of the excellent work Iraq Reconstruction Management Office under a new name. This weekly report rates high on both graphical interest and on providing fresh data with an excellent score for near real time reporting.
Check out the important indicators on page 23. While the tabular format is not for everyone, there are some important indicators in this table that are worth paying attention to.
Showing posts with label near real time. Show all posts
Showing posts with label near real time. Show all posts
Friday, June 15, 2007
Dept of State - Iraq Weekly Status Report June 13 2007
Here's the link to the latest Dept of State Iraq Weekly Status Report (June 13, 2007). This report has some much fresher (more recent) charts on oil and electricity than the quarterly DoD report. This report successfully achieves an excellent grade from delivering its data in near real time fashion.
For those interested in keeping up with what's happening on the ground in Iraq, this weekly report is one of the essential pieces to pay attention to.
For those interested in keeping up with what's happening on the ground in Iraq, this weekly report is one of the essential pieces to pay attention to.
Thursday, June 14, 2007
Ten Trend Charts from the DoD Quarterly Report
The Dept of Defense has just published its mandated quarterly report to Congress on the situation in Iraq - Measuring Stability and Security in Iraq, June 2007 (51 pages). As they state in the Executive Summary, the "report includes specific performance indicators and measures of progress toward political, economic, and security stability in Iraq"
Included as part of the report are 10 static trend graphs which I have reproduced in this post. You can click on each for a full size image. If you would like all 10 JPG images so you can look at full size with your favorite slide viewer, you can download this zip file.
Most of the important performance indicators mentioned in the report did not rate their own static trend graph chart. As we did with the UN document and our scan of recent news articles, we plan to comb through the document and come up with a list of indicators that complement the ones we have already highlighted so as to get a sense of the breadth of important indicators that are already on our radar screen.
Chart 01 - Oil Production - too short a time span. Also missing data for May 2007.
Chart 02 Electricity - Missing data for April and May 2007.
Chart 03 - Sectarian Murders and Incidents . Missing data from May 2007. Weekly data might be important to look at.
Chart 04 - Weapons Caches Found - This is looking promising. Access to weekly and daily data could be even more revealing of ongoing trends. Missing data for May 2007.
Chart 05 - Average Weekly Attacks - Missing last 5 weeks of data. This is the source for the charts that appeared in WaPo and NYT as reported earlier today.
Chart 06 - Average Daily Casualties - Missing last 5 weeks. Bar chart format with three trends makes reading trends for Iraqi Security Forces and Coalition more difficult.
Chart 07 - Hotline Tips - Interesting new performance metric. Looks positive. Missing past month of data. Weekly or daily data reporting could be even more revealing.
Chart 08 - Confidence in Iraqi Government to Improve the Situation. This looks surprisingly to me for its stability over time. I will need to read the text about this in detail. Missing most recent month's data.
Chart 09 - Assessed Capabilities of Police - Difficult to interpret trends when only two data points per item. Stacked bars make interpretation more difficult and time consuming. This is an interesting new metric and it could benefit from higher resolution, near real time reporting.
Chart 10 - Ministry of Defense Forces Assessed Capabilities. This chart also presents some important data but suffers from same failings as previous chart - only 2 data points per indicator, stacked bars are hard to read.
Summary. 10 Static charts covering a range of important indicators is more than almost anyone else except Michael O'Hanlon produces when it comes to highlighting the important trends evolving in Iraq.
So that's definite plus. Someone who spends a few minutes can examine all 10 of these (especially if they can scroll in slide show mode on full size images using this zip file).
On the minus side, the missing last month or last 5 weeks of data is a serious drawback. Many of the charts can be improved on and made more readily useful to ordinary viewers by applying some trend chart best practices principles. These include
The indicators in this report are important, but this method of presentation falls short of ideal and short of what is needed and what indeed is possible. With the high number of factors discussed in this report, the only method that will actually make this kind of trend data usable to its intended primary audience (Congress) and to the secondary audiences including the media and ordinary interested citizens would be to deliver this data via a 21st century Trend Visualization Appliance.
Included as part of the report are 10 static trend graphs which I have reproduced in this post. You can click on each for a full size image. If you would like all 10 JPG images so you can look at full size with your favorite slide viewer, you can download this zip file.
Most of the important performance indicators mentioned in the report did not rate their own static trend graph chart. As we did with the UN document and our scan of recent news articles, we plan to comb through the document and come up with a list of indicators that complement the ones we have already highlighted so as to get a sense of the breadth of important indicators that are already on our radar screen.
Chart 01 - Oil Production - too short a time span. Also missing data for May 2007.
Chart 02 Electricity - Missing data for April and May 2007.
Chart 03 - Sectarian Murders and Incidents . Missing data from May 2007. Weekly data might be important to look at.
Chart 04 - Weapons Caches Found - This is looking promising. Access to weekly and daily data could be even more revealing of ongoing trends. Missing data for May 2007.
Chart 05 - Average Weekly Attacks - Missing last 5 weeks of data. This is the source for the charts that appeared in WaPo and NYT as reported earlier today.
Chart 06 - Average Daily Casualties - Missing last 5 weeks. Bar chart format with three trends makes reading trends for Iraqi Security Forces and Coalition more difficult.
Chart 07 - Hotline Tips - Interesting new performance metric. Looks positive. Missing past month of data. Weekly or daily data reporting could be even more revealing.
Chart 08 - Confidence in Iraqi Government to Improve the Situation. This looks surprisingly to me for its stability over time. I will need to read the text about this in detail. Missing most recent month's data.
Chart 09 - Assessed Capabilities of Police - Difficult to interpret trends when only two data points per item. Stacked bars make interpretation more difficult and time consuming. This is an interesting new metric and it could benefit from higher resolution, near real time reporting.
Chart 10 - Ministry of Defense Forces Assessed Capabilities. This chart also presents some important data but suffers from same failings as previous chart - only 2 data points per indicator, stacked bars are hard to read.
Summary. 10 Static charts covering a range of important indicators is more than almost anyone else except Michael O'Hanlon produces when it comes to highlighting the important trends evolving in Iraq.
So that's definite plus. Someone who spends a few minutes can examine all 10 of these (especially if they can scroll in slide show mode on full size images using this zip file).
On the minus side, the missing last month or last 5 weeks of data is a serious drawback. Many of the charts can be improved on and made more readily useful to ordinary viewers by applying some trend chart best practices principles. These include
- use of full time range
- use of near real time reporting for the most recent interval
- selection of best resolution or choice of several different reporting intervals (day, week, month)
- displaying each important indicator separately with appropriate Y axis scale
- having sufficient data points for trends to be discernible - 2 is way too few
The indicators in this report are important, but this method of presentation falls short of ideal and short of what is needed and what indeed is possible. With the high number of factors discussed in this report, the only method that will actually make this kind of trend data usable to its intended primary audience (Congress) and to the secondary audiences including the media and ordinary interested citizens would be to deliver this data via a 21st century Trend Visualization Appliance.
What's the Best Reporting Interval?
Why Wait for a Quarterly Report? Yesterday's DoD report to Congress is a mandated quarterly report. The executive summary begins with:
AND between reports, they can be even more helpful on a daily, weekly and monthly basis.
Just as we don't have to wait till July or September to find out how well the "surge" strategy is working, we don't have to hold our breath between quarterly reports. We can make double duty of the infrastructure already in place to produce these reports to create a mechanism that lets us keep track of the most important indicators and measures of progress as we go along.
Here's How.
If there is a performance indicator that is important enough to be reported on quarterly, then it is even more important to make the underlying trends for that indicator available interactively and efficiently, at high resolution, and in near real time for all interested parties.
The current arrangement (infrequent reports, delayed reporting, lower frequency data) does not appear conducive to the most effective oversight by congress, the media or the general public. We can and must do better if we want to make sense of what is happening in Iraq.
This report to Congress, Measuring Stability and Security in Iraq, is submitted pursuant to Section 9010 of the Department of Defense Appropriations Act 2007, Public Law 109-289. The report includes specific performance indicators and measures of progress toward political, economic, and security stability in Iraq, as directed in that legislation. [emphasis added]My view: This report can do double duty. Quarterly reports are all well and good. They present us with specific performance indicators and measures of progress. They offer an opportunity to take a step back and look at the evolving patterns.
AND between reports, they can be even more helpful on a daily, weekly and monthly basis.
Just as we don't have to wait till July or September to find out how well the "surge" strategy is working, we don't have to hold our breath between quarterly reports. We can make double duty of the infrastructure already in place to produce these reports to create a mechanism that lets us keep track of the most important indicators and measures of progress as we go along.
Here's How.
- Start where we are at. The quarterly report defines the currently understood list of the most important performance indicators and measures of progress.
- High Frequency Measurement. For each of these important indicators we establish a measurement frequency that is as rapid as is practical at this time.
- Near Real Time Reporting. For each of these important indicators we also establish a near time reporting interval that is as close as is practical to the time of the most recent measurement sample
- 21st Century Visualization. We then make the full set of high resolution trend data for all the most important performance indicators and measures of progress available. We make it available in a continuously updated form in near real time using a 21st century Trend Visualization Appliance. This makes it possible and manageable to observe the trends of a large number of factors within the time and resource constraints of the intended audiences.
- Improve Frequency. For any important factor where it proves impractical today to measure at the frequency that we think best, we set in motion a process to speed up that frequency as quickly as we can bring it on line.
- Improve Freshness. For any important factor where it proves impractical today to deliver the most recent measurements in suitable near real time fashion that would meet our needs for fresh data, we set in motion a process to reduce the delay as quickly as we can.
- Identify and Add Missing Indicators. Finally, as soon as we notice some new and important indicator that is vital to understanding progress in Iraq, we add it to the list of indicators to be included in the Quarterly report AND to be delivered to our Trend Visualization Appliance at high resolution and in near real-time for our consideration between these quarterly reports.
If there is a performance indicator that is important enough to be reported on quarterly, then it is even more important to make the underlying trends for that indicator available interactively and efficiently, at high resolution, and in near real time for all interested parties.
The current arrangement (infrequent reports, delayed reporting, lower frequency data) does not appear conducive to the most effective oversight by congress, the media or the general public. We can and must do better if we want to make sense of what is happening in Iraq.
Near Real Time Reporting
For most important situations in life where trend visualization is helpful, we are lucky that real time reporting is not required. However, collecting the data and reporting it within a reasonable time from the end of measurement is important and can deeply impact the value and usefulness of what we have collected.
An Unacceptable Delay. We noted earlier today in the post on the New York Times Iraq Progress Graphic how the underlying data from the DoD quarterly report to Congress only covered up through May 4th. So in effect we were looking at data that wasn't produced in a report till 5 and 1/2 weeks after the end of the most recently reported period.
In my opinion, this represents an unacceptable delay and that delay severely impacts the ability of any audience to gauge what the data means and to decide what to do next.
If the underlying data for attacks and for casualties were collected at the recommended rate of once per day, a near real time report on June 13th would be best if it covered everything that happened up through one, two or three days prior to June 13th.
Fresh Data. The combination of high frequency measurement and near real time reporting means that the data we are looking at is fresh - a vital quality in any rapidly changing situation. There are numerous examples in other aspects of our lives where near real time reporting and high resolution reporting are simply the norm (the stock market being a prime example).
There are sometimes situations that delay and interfere with our ability to roll up the most recent results of a particular indicator. In some cases, these are historical and can be overcome and minimized if we decide to put our minds to it and invest in the needed infrastructure to make it happen. In other cases, there are time lags that are mostly out of our control. The best we can do in these cases may be to put our energy towards devising ways to minimize the delay.
Bottom line: In a trend chart that is going to help us understand what is truly going on, timeliness is very important. When we get our measurement rate set appropriately and we make every effort to minimize the delay between collection and reporting, the trend work that we do will have the maximum value to all those who are interested.
An Unacceptable Delay. We noted earlier today in the post on the New York Times Iraq Progress Graphic how the underlying data from the DoD quarterly report to Congress only covered up through May 4th. So in effect we were looking at data that wasn't produced in a report till 5 and 1/2 weeks after the end of the most recently reported period.
In my opinion, this represents an unacceptable delay and that delay severely impacts the ability of any audience to gauge what the data means and to decide what to do next.
If the underlying data for attacks and for casualties were collected at the recommended rate of once per day, a near real time report on June 13th would be best if it covered everything that happened up through one, two or three days prior to June 13th.
Fresh Data. The combination of high frequency measurement and near real time reporting means that the data we are looking at is fresh - a vital quality in any rapidly changing situation. There are numerous examples in other aspects of our lives where near real time reporting and high resolution reporting are simply the norm (the stock market being a prime example).
There are sometimes situations that delay and interfere with our ability to roll up the most recent results of a particular indicator. In some cases, these are historical and can be overcome and minimized if we decide to put our minds to it and invest in the needed infrastructure to make it happen. In other cases, there are time lags that are mostly out of our control. The best we can do in these cases may be to put our energy towards devising ways to minimize the delay.
Bottom line: In a trend chart that is going to help us understand what is truly going on, timeliness is very important. When we get our measurement rate set appropriately and we make every effort to minimize the delay between collection and reporting, the trend work that we do will have the maximum value to all those who are interested.
Frequency of Reporting
In the previous post, we brought up the concept of frequency of reporting on a given metric. We haven't talked much about this aspect of trend analysis so far, despite the fact that it can be quite important. And it interacts with another important topic we brought up in some previous posts: near real time reporting.
We'll take a look briefly at frequency in this post, and give a brief look at near real time reporting in the next.
Error on the Side of Too Often.
Each important factor that you wish to measure will prove most useful in the end if you make a conscious choice to consider its frequency of measurement separately on the merits. My general operational rule is wherever possible to err on the side of measuring too often rather than not often enough.
Measurement vs. Reporting Frequency. When you measure at a relatively high frequency, it's possible later to report on the data at either the high collection frequency, or at a reduced frequency in order to smooth out the trend line pattern. On the other hand, if you don't measure frequently enough, you may miss out on short-lived quantum shifts in behavior that will be lost when reporting at longer intervals.
It's unfortunately not uncommon for vital indicators that are already being collected to have their frequency set so that measurements are too far apart.
Measuring Daily is Often a Good Starting Point. Let's take a specific example: the percentage of eligible children who attend primary school. Ideally, in my opinion, a metric such as this would be best if it were measured daily. There will be times when you will want to look at the day to day trend and see if any disruptive multi-day even has shown up. You may also want to understand any patterns that might be dependent on the day of the week. There will be other times when week by week reporting will be just right and still others when month by month will give the clearest view. Before you actually have the data, you cannot really know which reporting interval will prove most revealing.
My recommendations for the important Iraq metrics of the kind we have been scraping from the recent news reporting is that we make every effort to measure them with a frequency of once per day wherever that is possible.
When is Higher Frequency Helpful? There are some indicators where it would be advantageous to measure them even more frequently than once a day. For example, electricity delivery in Baghdad is a factor that I would like to measure minute by minute during the day and to do this both for the city as a whole as well as disaggregating these results by districts in the city and/or by individual power stations. Unless I get a very high resolution frequency, I won't be able to understand the frequency of or length or breadth of districts impacted by each outage. Such diagnostic capability is likely vital to figuring out where the est opportunities are for achieving higher levels of service and moving closer to fulfilling the desired demand.
Obstacles to High Frequency. On the other hand, there will be metrics such as the percentage of Iraqi members of Parliament who are living abroad where the obstacles to measuring may mean that we only get a read out on this indicator once a month. It might be nice to see how this varies day to day, but prove to be logistically impractical to get it.
Adjusting as you go along. There is no perfect frequency and there can likely be differences of opinion as to which frequency is best. The good news is that frequency can be adjusted as needed as we go along. The really important part is identifying all the important metrics and beginning the process to measure and record their values as they change over time.
We'll take a look briefly at frequency in this post, and give a brief look at near real time reporting in the next.
Error on the Side of Too Often.
Each important factor that you wish to measure will prove most useful in the end if you make a conscious choice to consider its frequency of measurement separately on the merits. My general operational rule is wherever possible to err on the side of measuring too often rather than not often enough.
Measurement vs. Reporting Frequency. When you measure at a relatively high frequency, it's possible later to report on the data at either the high collection frequency, or at a reduced frequency in order to smooth out the trend line pattern. On the other hand, if you don't measure frequently enough, you may miss out on short-lived quantum shifts in behavior that will be lost when reporting at longer intervals.
It's unfortunately not uncommon for vital indicators that are already being collected to have their frequency set so that measurements are too far apart.
Measuring Daily is Often a Good Starting Point. Let's take a specific example: the percentage of eligible children who attend primary school. Ideally, in my opinion, a metric such as this would be best if it were measured daily. There will be times when you will want to look at the day to day trend and see if any disruptive multi-day even has shown up. You may also want to understand any patterns that might be dependent on the day of the week. There will be other times when week by week reporting will be just right and still others when month by month will give the clearest view. Before you actually have the data, you cannot really know which reporting interval will prove most revealing.
My recommendations for the important Iraq metrics of the kind we have been scraping from the recent news reporting is that we make every effort to measure them with a frequency of once per day wherever that is possible.
When is Higher Frequency Helpful? There are some indicators where it would be advantageous to measure them even more frequently than once a day. For example, electricity delivery in Baghdad is a factor that I would like to measure minute by minute during the day and to do this both for the city as a whole as well as disaggregating these results by districts in the city and/or by individual power stations. Unless I get a very high resolution frequency, I won't be able to understand the frequency of or length or breadth of districts impacted by each outage. Such diagnostic capability is likely vital to figuring out where the est opportunities are for achieving higher levels of service and moving closer to fulfilling the desired demand.
Obstacles to High Frequency. On the other hand, there will be metrics such as the percentage of Iraqi members of Parliament who are living abroad where the obstacles to measuring may mean that we only get a read out on this indicator once a month. It might be nice to see how this varies day to day, but prove to be logistically impractical to get it.
Adjusting as you go along. There is no perfect frequency and there can likely be differences of opinion as to which frequency is best. The good news is that frequency can be adjusted as needed as we go along. The really important part is identifying all the important metrics and beginning the process to measure and record their values as they change over time.
NY Times charts based on DoD report
Here's the New York Times graphic that accompanied their story this morning on the DoD report to Congress.Their stacked bar chart is similar to the unstacked bar chart from the Washington post.
Geographic charts are popular these days but I find the pair of charts in this example difficult to interpret. There are some trends going on with respect to readiness to assume security responsibility, but one has to spend quite a bit of time to figure out what they are from these maps. The eye has to dart back and forth between the two maps to make a before and after comparison. If I am reading these correctly, it looks like some serious backsliding has occurred since Nov 2006. And of course, having only a pair of data points for each trend makes any interpretations or extrapolations risky.
The sectarian incident chart at the top right is useful. I would prefer to see a longer time frame and data that is more up to the minute (e.g. at least through the end of May 2007).
The first chart showing the daily casualties also falls short in only reporting data up through May 4th, 2007. While real time reporting is rarely needed in this kind of situation, important factors such as casualties must be reported in near real time for them to be truly useful.
Friday, June 8, 2007
What direction are we moving?
From Chart of the Day's - www.chartoftheday.com: "Quote of the Day
What could be simpler than that?
The greatest thing in this world is not so much where we are, but in what direction we are moving.' - Oliver Wendell Holmes"And the best way to have a good sense of what direction we are moving is to examine (in near real time and with high resolution) the trend data for the most important factors and indicators at work in whatever situation we find ourselves.
What could be simpler than that?
Thursday, June 7, 2007
Why Wait Till September in Iraq
We Don't Have To Wait. There's a dangerous meme circulating that says we have to wait till September before we can learn whether the "surge" in Iraq is working. This, in my opinion is simply not true.
Many dimensions of the current situation in Iraq can and are being measured and reported on in near real time. The work of Michael O'Hanlon on the weekly Iraq Index at the Brookings Institution is one excellent proof that it is possible to know were we stand literally week by week and even day by day.
Digestible Trend Data. One principal area where we are not as good as we need to be today is in making sure that the trend data that exists is readily available in a form that is digestible by the intended audience. This intended audience includes the Congress, the main stream news media, the blogosphere, and especially ordinary citizens. Some really important data such as the example of Iraqi prison population that we showed in a post earlier today may only appear in tables. My guess is that most people are not willing or able to get what they need directly from the Iraq Index or similar reports. Those who are willing and able may not have nearly enough time. We can and must make it easier so that more citizens can be included in the process.
Assembling All Key Factors in One Place. A second principal impediment is that that the data is dispersed among many sources. We need to pull it all together so that each of us then becomes able to make a reasoned evaluation of how we are doing right now (without waiting till September) . My efforts at assembling some key trend graphs in a series of posts today and in the summary zip file is a small demonstration of what is possible.
Identifying and Acquiring Any Missing but Still Important Factors. A third thing holding us back from being able to evaluate for ourselves how things are going right now is that some key factors are not being reported in easily findable places.
Twenty First Century Visualization Tools. The fourth factor limiting our possibilities today is that most trend data is presented statically. This severely limits the number of factors that an ordinary person is going to be willing and able to look at to only a handful. On the other hand, with a modern Trend Visualization Viewing Station on the order of what's already available for stock market data from the likes of Big Charts or Prophet Net means that ordinary folk can look at and understand dozens and dozens of factors in a very short space of time.
We Don't Have to Wait. All these difficulties that are blocking us can be remedied without waiting till September. Our failure to remedy them over the past four years meant that it took that long for it to sink in that what we were doing wasn't really working. It's easy to posture that everything is ok when there is no hard data to prove you wrong.
In fact, if we don't begin to remedy them now, we will be in the same boat come September. We will find that we still do not have our arms around the entire situation. The facts we need will still not have been assembled and reported in an easily digestible way. And it will still be way too easy to claim that things are going well, or that we still don't know enough to tell.
Anyone interested in helping blast our way through these obstacles to understanding where we are in near real time, please let me know so we can join forces and see if we can make a difference.
The Data is Out There. Let's go get it and assemble it and make it easy to use and then share it with all interested parties.
Many dimensions of the current situation in Iraq can and are being measured and reported on in near real time. The work of Michael O'Hanlon on the weekly Iraq Index at the Brookings Institution is one excellent proof that it is possible to know were we stand literally week by week and even day by day.
Digestible Trend Data. One principal area where we are not as good as we need to be today is in making sure that the trend data that exists is readily available in a form that is digestible by the intended audience. This intended audience includes the Congress, the main stream news media, the blogosphere, and especially ordinary citizens. Some really important data such as the example of Iraqi prison population that we showed in a post earlier today may only appear in tables. My guess is that most people are not willing or able to get what they need directly from the Iraq Index or similar reports. Those who are willing and able may not have nearly enough time. We can and must make it easier so that more citizens can be included in the process.
Assembling All Key Factors in One Place. A second principal impediment is that that the data is dispersed among many sources. We need to pull it all together so that each of us then becomes able to make a reasoned evaluation of how we are doing right now (without waiting till September) . My efforts at assembling some key trend graphs in a series of posts today and in the summary zip file is a small demonstration of what is possible.
Identifying and Acquiring Any Missing but Still Important Factors. A third thing holding us back from being able to evaluate for ourselves how things are going right now is that some key factors are not being reported in easily findable places.
Twenty First Century Visualization Tools. The fourth factor limiting our possibilities today is that most trend data is presented statically. This severely limits the number of factors that an ordinary person is going to be willing and able to look at to only a handful. On the other hand, with a modern Trend Visualization Viewing Station on the order of what's already available for stock market data from the likes of Big Charts or Prophet Net means that ordinary folk can look at and understand dozens and dozens of factors in a very short space of time.
We Don't Have to Wait. All these difficulties that are blocking us can be remedied without waiting till September. Our failure to remedy them over the past four years meant that it took that long for it to sink in that what we were doing wasn't really working. It's easy to posture that everything is ok when there is no hard data to prove you wrong.
In fact, if we don't begin to remedy them now, we will be in the same boat come September. We will find that we still do not have our arms around the entire situation. The facts we need will still not have been assembled and reported in an easily digestible way. And it will still be way too easy to claim that things are going well, or that we still don't know enough to tell.
Anyone interested in helping blast our way through these obstacles to understanding where we are in near real time, please let me know so we can join forces and see if we can make a difference.
The Data is Out There. Let's go get it and assemble it and make it easy to use and then share it with all interested parties.
Subscribe to:
Posts (Atom)