A core component of any enterprise wide improvement journey is the institution of a metrics program. A metrics program whose viability will be measured by the level of impact the program has on the decisions made, plans developed, and changes implemented – in other words, how the results are used.
The approach we'd like to start with can be described by using an analogy of a tree. Data are analogous to the leaves on the tree. Numerous and abundant, they are interesting to look at, but will not survive on their own. They are only a means for feeding the tree. The smallest branches, connecting the leaves to the tree, are measures – a little stronger, but still not the essence of the tree. The thicker, inner branches, the kind we like to hang tires from, are information. These branches are useful at times in themselves, but if taken away from the tree, they stop growing (in fact die), and are good only for specific uses. The trunk of the tree, where you can determine the age of the tree, the strength of the tree, is our metrics – a complete picture telling a complete story. The metric may be made up of many pieces of information, derived from many measures and data. But, the largest tree will whither and die without roots. The roots of our tree are the root questions the metric is designed to answer.
So, our approach is built on a simple premise. All metrics in a maturing, growing organization should tell a complete story in answer to a well formulated root question. This ensures that the metric and the supporting information, measures, and data are designed to answer a known question. Most organizations (ours included), do not have the resources (time, manpower, or funds) to partake in a metrics effort without a predetermined purpose. If we chase “interesting” data, we can collect measures non-stop, 24/7, for years, and never answer the questions the organization has. We can collect information and not have a result useful for improvement. We risk wasting vital resources.
We have documented our approach, to include: a process definition, a metrics implementation guide (worksheet), and an information grid. Our approach is a systematic, stepwise method which can be used for any type of metric. It is founded in the universal need of clearly and comprehensively identifying the customer’s requirement. The process is analogous to growing the tree. You don't start with the leaves or branches. We start with a seed that sprouts roots and a shoot. It is at the roots where we begin the process, and working up and outward until, at the last, we identify the data (leaves) needed.
We look forward to working with others who struggle with the dangers of data (especially in the hands of management), defining useful metrics, collecting accurate data, and using metrics for improvement.
In the PhD learner environment, the collection and analysis of data is even more important to success than for a business. I think the marriage of the business school with the research needs of a PhD learner make for a robust discussion opportunity.
WRONG!
I was accosted yesterday by a well-meaning manager who admonished me to “make sure the metrics are the right ones!” He went on to remind me that “we improve what we measure.” I tried (and failed) to explain to him that the adage is wrong…close, but wrong. We actually improve the measures we measure. In other words, because we focus on the “score,” “rating,” or “grade” we obtain in the metric – and not on the underlying effort, process, or activity we’re trying to improve - we are only assured of improving that score. I call this “chasing data.” Rather than improving the underlying driver for the creation of the metric - we end up trying to improve the numbers. A simpler way to look at metrics are that they are NOT used to improve anything. They’re indicators. They provide insight and visibility which in turn can be used to improve things.
A friend told me a story about an innocent example. The call center was trying to improve the way they did business. They wanted to enhance the customer experience. So among other things, they began measuring the length of help calls - with the idea that the shorter the call, the more calls the analyst could answer, and the more customers he could assist. And they only focused on those that repeatedly had “longer than average” calls. My friend was the biggest culprit. His calls lasted five times longer than the rest of his coworkers! The boss called him on it. She wanted his numbers to improve!
What she forgot was why she was collecting the data. She also forgot that the measures only provided insight, not answers. She could have asked why he was so much slower. If she investigated (instead of reacting - or “managing using the data”) she would have found that because he was the most knowledgeable and experienced, all of the difficult calls were being forwarded to him. His co-workers were passing all of the most difficult, and thereby the most time consuming, cases to him.
The measure was fine. The use of it to “manage” was not.
In this case, the measure could be used, as an aggregate - not on an individual level. The average time to resolve trouble calls could be a very good indicator for the call center as a whole. The mistake was in believing that the measure told a full story and worse - that it was an answer.
Managers manage people - managers don’t manage metrics. If we could manage by metrics, we’d not need people with people skills as managers - we could use computers.
It’s that time of year…the bird’s are back, the grass is growing, and April showers are prepping the ground for May flowers to bloom! It’s Spring! The season of resurrection! When life is renewed on our planet. What time could be more fitting for the reengaging on this effort? I’ll start with Metrics as it has been 80% of my job over the last two years and I am currently working on a book on the subject! I also recently (one year now) founded an Educause Constituent Group on IT Metrics - and we have a fairly active Listserv. Feel free to post your own ideas, questions, or conversation starters! I will wax poetic on a weekly basis (a new personal goal) so that there will be something worth reading!
Well, the intrepid team of Mike Langthorne, Don Padgett, and yours truly (Marty Klubeck) reached a serious high in 2006. After having an article published in the Educause Quarterly magazine (http://www.educause.edu/LibraryDetailPage/666?ID=EQM0639) the team was invited to present at the Annual Educause Conference, held in Dallas TX. Before the shock and joy of this completely settled in, we found out that our slot would be a full day pre-conference seminar! We considered this one of the greatest honors we could have been given.
The seminar was titled, “Do It Yourself Metrics: Developing Practical Metrics” and built off of our Educause Midwest Presentation. The good news kept coming, we found out that the full day seminar was “sold out,” even though it required separate registration and fees. What a terrific opportunity for us to spread our metrics message and to teach our process for developing metrics.
The conference itself was a rousing success, and our seminar was well received (based on the critiques). The seminar was designed as a hands-on, practical demonstration and performance session to assist the attendees in understanding the concepts behind developing useful metrics. Some of the key points taught included:
Why use Metrics?
Gain support from above
Improve processes
Provide visibility
Make better decisions
What is a Metric?
Not data,
Not measures,
Not information
A “picture” which tells a full story
Question driven
Tell a complete story
Includes data, measures, information and other metrics
How NOT to use Metrics
Not to make the results support a given case
Not to motivate the staff
Not to manager the staff or others
Not to evaluate individual performance
How to use Metrics
Explain how they will and won’t be used
Investigate (metrics in the end are only indicators…)
Share
Since Educause is an academic-based organization, serving the educational “industry” our discussion of “who uses metrics” covered:
Campus community
Management
Owners and workers
Leadership
Perhaps the most fun part of the seminar was reviewing some of the warning signs of common misunderstandings (misuses) of metrics:
“sounds interesting, so let’s collect it”
“we’ve been collecting this data for five years and no one is using it…”
“hey, do we have any data on…?”
“they don’t trust the data…”
“I’ll know it when I see it…”
After lunch, the seminar moved from a lecture format to a hands-on, demonstration/performance format. After grouping the attendees based on their most burning questions (needs) we worked with the each, stepping them through the process. One of the most well-received takeaways was our “Answer Key” which was used to help identify the level of metric being developed. Along with the Answer Key, we also provided the attendees with an implementation guide - which helps the metric designer, collector, and user to add rigor and control to the metric.
I would be very interested and excited to begin a conversation about developing practical metrics for any level of an organization…
Do-It-Yourself Metrics!
My two co-workers, Don Padgett and Mike Langthorne, and I presented a 45 minute session at the March 2006, EDUCAUSE Midwest Regional Conference. We presented a down and dirty explanation of how to implement a metrics program. We reviewed the implementation guide we use at work to build a metric:
Metric Name
Purpose (a go/no go proposition)
Metric Area/Category
Customer
Graphical Representation
Explanation
Metrics Analysis
Measures used to develop metric
Collection Schema
Schedule
Assumptions and Constraints
Related Metrics and Data Dependencies
Lessons Learned
The presentation went over quite well. You can find the slides at: http://www.educause.edu/LibraryDetailPage/666?ID=MWR0651
Let me know what you think!
A large part of understanding metrics, and using them properly is understanding why. Why we may want to report them up the chain (of command), why we ask to see metrics from below, and why we collect metrics for ourselves.
Before we examine each of these…we need to review the probable outcomes of showing metrics. When we show metrics (data) to management, the most common response is that management will try to solve our problems - even if one doesn’t exist. If we don’t have explicit reasons for showing metrics and if we do not control the event, management will ask pointed questions thereby making themselves look knowledgeable and then they will attempt to solve our problems. This is not their fault. When we lose control of the event, management assumes that the reason we are showing data is to get their inputs on how to solve our problems. The only way to combat this tendency is to know why we are showing the metrics and keep the interaction focused.
Why report metrics up the chain?
Why in the world would we want to show metrics, let alone data, to our boss? Definitely NOT to have the boss solve our problems. That’s our job and at worst, if we need their help, we want to ask for what we think is needed…not just present the problem. We have to offer solutions. So, why would we report metrics to our boss? There are two reasons I know of.
1. To inform. We show data to our bosses so that they become more aware. This is dangerous, because most managers have a hard time seeing data without reacting. If you truly want to make your boss aware of something, and that is your only intent, make sure that what you show her has no chance of being construed as a problem. One example may be to show her the number of projects or goals your group is working on. Just to keep her up to speed. Or to share information so when she meets with her boss, she’ll be well versed on the “hot” topics.
2. To gain support. This is the best reason to show metrics to your boss. This includes selling your management on a need, to gain resources, or to get a decision. These all really tie together. Once we’ve identified a problem and determined a solution, we go to management to get approval. With that approval comes the allocation of necessary resources. To get management to give us the resources and approve our solution, we have to sell them on the need (problem). It is important for you to have already solved the problem and be proposing a solution (and it’s smart to have a couple of alternatives in your “back pocket”). The only decision you should seek from management is a go or no-go for your proposal. You do NOT want management coming up with solutions to problems. Their job is to manage resources, not solve your problems.
Why a boss would ask for metrics (data)?
The reasons our boss should want to see metrics are corollaries to the reasons we should want to show them. To become informed (curiosity) or to help with making a decision. Unfortunately, the usual reason we assume we’re being asked for the metric is because our boss doesn’t trust us.
1. To become informed. From the boss’s point of view, lack of information is a driving factor. New managers especially feel like they need more information to better understand the business and better manage resources.
2. To make a decision. Again, when a new manager comes on board, many times she will ask for supporting data before making any critical decisions. While there is a lot to be said for intuition, working with data is a helpful way to build support and gain credibility.
3. To trust lower levels. This is a risk with asking for data. This risk can be mitigated through clear communication and strict adherence to proper rules for how to use (and not use) metrics. If the manager truly does not trust her direct reports…there are other problems, problems that can not be solved with metrics. The issue of trust is worthy of a whole thread of its own.
“The only valid response to data (or metrics) is to investigate.” – Martin Klubeck
I find that the misuse of data is much more prevalent than the use of it. This is probably my biggest concern when working with customers on their metrics. I’ve tried to combat this by forcing the customer to designate in advance the use they intend for the metric - before I even agree to design it.