The Tyranny of Metrics - Measure Your Way to Misery
The superior man uses his superior judgement to look superior on all the metrics.
By: Jerry Z. Muller
Published: 2019
248 Pages
Briefly, what is this book about?
The distorting effects of an over-reliance on metrics particularly when it comes to creating incentives.
What’s the author’s angle?
Muller was frustrated by the numerous metrics being imposed upon him in academia, frustrated enough to write a book about it.
Who should read this book?
If you’re in an environment where you feel like metrics are being overused and abused, this book can help you identify how that’s happening, and what you might be able to do about it.
Specific thoughts: Bad metrics are everywhere, why isn’t this problem better known?
The book opens with some stories of bad metrics taken from TV shows. I was particularly taken with this example:
The series, Bodies, written by Jed Mercurio, a former hospital physician, takes place in the obstetrics and gynecology ward of a metropolitan hospital. In the first episode, a newly arrived senior surgeon performs an operation on a patient with complex comorbidities, after which she dies. His rival then provides him with this advice: “The superior surgeon uses his superior judgment to steer clear of any situation that might test his superior ability.”
Why does the superior surgeon avoid complicated surgeries? Because it affects his success rate. A metric that will be used by his bosses, his colleagues, and his patients in future determinations of his worth. It might even be tied to his compensation. This strategy is called creaming, where rather than taking the entire universe of some category, you take only the cream off the top. In this case only easy patients. This is only one of many ways in which an over-reliance on metrics causes people to distort their behavior.
Muller points out several negative consequences of relying too much on metrics. And I know that some people are going to complain that the first example was from a TV show (though it depicts a real issue) so I’ll use some examples from books I’ve recently read.
Let’s start with the negative consequence of “Goal displacement through diversion of effort to what gets measured.” This basically means that you’re trying to accomplish X, and you come up with some metric that’s a proxy for X, but in the end you forget about X and focus directly on the metric, which is an oversimplification of whatever your original goal was. In the book Breakneck, Wang talks about China’s one-child policy and their zero-COVID effort. In both cases the metric is right in the title. And in both cases it ended up being a gross distortion of their original goal.
In the first case China wanted to be a prosperous country, and they worried that having too many mouths to feed would endanger that prosperity, so they decreed that families could only have one child. As soon as they did, that became the goal, not prosperity. In the end, the demographic collapse brought on by the policy is actually endangering their prosperity rather than ensuring it.
The zero-COVID effort also backfired. In both cases, these were top-down decrees. Chinese leaders wanted to maintain legitimacy and be seen as doing a good job, particularly since the virus originated in China. But by chasing the untenable goal of zero-COVID the government ended up with the biggest protests since Tiananmen Square.
Moving on, metrics result in the negative consequence of “Promoting short-termism.” In this case the Americans were the ones obsessed with metrics while the Chinese were the wise ones. I’m referring to the book Apple in China. This book contrasted the short term, quarterly report/metric driven approach of Apple, against the long term goals of the Chinese manufacturers. The book ended with a discussion of the precarious position this had placed Apple in. Their short-termism led to them being desperate for new customers, and China had a lot of new customers, and also desperate for large manufacturing capacity, which China also had. Now Apple is utterly reliant on them, and can’t hope to disentangle itself from the market or the manufacturers, but along the way it gave these manufacturers all the tech it needs to supplant them. Over the short term Apple looked good every quarter. In the long run they may have planted the seeds of their own destruction.
I could go on with even more examples, but I think you get the idea. Though for those that are curious here is a list of the other negative consequences of an overreliance on metrics identified by Muller:
Costs in employee time- The time required to track metrics.
Diminishing utility- Metrics become less useful over time.
Rule cascades- More rules to cover the holes in the metrics you already have
Rewarding luck- Sometimes metrics just measure the vagaries of circumstance.
Discouraging risk-taking- Metrics confine people to well-understood territory.
Discouraging cooperation and common purpose- If resources are tracked individually they’re never going to be shared.
Degradation of work- You’re going to avoid the tough surgeries.
Costs to productivity- All of the above exerts downward pressure on increasing productivity.
After reading this book I’m not sure if people don’t know about these problems; or if they know about them, but feel powerless; or if they know about them and are already working as hard as possible to fix them. Regardless of which bucket things fall into (and it’s obviously different from organization to organization) I can see where we might have created a meta-metric trap. Imagine something along these lines:
Employee: We’ve got too many metrics, they’re lowering productivity, and leading to lots of harmful distortions in what we focus on.
Boss: Okay, I hear you. But how are we going to know if we’re fixing the problem? Oh! I know we need a measure of productivity and distortions.
Employee: So we’re going to fix our problem of having too many metrics by adding more metrics?
Boss: Yeah, of course! How else are we going to know that we’re moving in the right direction?
Employee: *groan*
All of this comes from important values like accountability, transparency, and systemization. But when you combine all three the natural outcome is to establish metrics. Consequently the only way out is going to be perceived as a weakening of those values. But no one wants to do that! All of those values are important! And so we end up in the imagined conversation I just described. We can only double down, we can’t retreat. But retreat is probably the wrong framing. Maybe the answer is in the title of the book. No one is arguing that all metrics everywhere are bad. But perhaps, if we could just make them a little less tyrannical.
It is possible to go too far the other way. To eschew any metric and lurch around in the dark with no idea of where you’re going or what you’re doing. If you think this sounds like me, I won’t argue with you. But at least I’ve been lurching around for a long time, and I’ll probably be lurching around for a long time yet. Perhaps you’ll join me. The dark isn’t so bad once you get used to it.
Increasingly, I worry about the rise of measurement by omnipresent devices like the iPhone, whoop, and oura ring. When we overly quantify our life, we suck dry the nectar that is its precious quality.
Interesting a common metric used for physicians is the work revenue unit (wRVU). Basically just about every procedure from an initial office visit to a heart transplant has a number of wRVU assigned to it. The MGMA does yearly surveys and assigns benchmarks based on speciality.
So a Family Medicine without OB doctor may have 25% benchmark of 2,500, 50% of 5,000, 75% of 9,000 and maybe 90% of 12,000. Note this doesn't necessarily go up linearly. Going from 25% to 50% maybe easier than going from 50% to 75%. As often happens, a place may hire a doctor for less than a full FTE, so in that case you'd simply adjust the benchmarks accordingly.
The concept is the physician you hire at your hospital or practice can't control what type of coverage the patients have so the best way to measure them is to ask are they doing billable services? The wRVU allows you to put different services together. This also seems to drive the philosophy many pick up on with modern doctors of being quick with patients. If you think of it like NASCAR, the physician is like the driver. He has to be on the road as much as possible so if doubling the size of the pit crew to get him back on the track 5 seconds sooner will work, you do it.
This is not how every practice does it. Sometimes you get physicians just 'on staff'. Private practices may often follow "you eat what you kill", meaning the doctor is rated simply on the revenue they generate whether that's a few patients with generous insurance or lots of patients with stingy.
I think it maybe just as helpful to ask what works when metrics go right?