The Problem with Metrics

image
If you can’t measure it, you can’t manage it.” I don’t know who coined that phrase, but I have to tell you, I have some problems with the concept. My primary problem is that “it” often refers to a person, or at least the activity of a person and I think people should be managed more holistically. Another problem that I have is that since we are managing people, we are subject to the behavior of people who know they are being managed.

My first job after college was as a programmer/analyst for Burroughs Corporation, working in one of their manufacturing plants. System responsibilities had been hastily reassigned during the period between my accepting the position and my start date, so I found myself responsible for payroll, HR and work management systems. The product made at our plant was memory sub-systems and I was exposed very quickly to two problems with metrics. Problem one was the ease with which metrics can be manipulated. Our plant manager, aware that a key metric of his was the number of days between receiving components and shipping an assembly, simply purchased a used trailer from a trucking company. Material that arrived before we were ready to deal with it was unloaded onto our loading dock and reloaded into his trailer. When we were ready to start using the components, they were “received.” This little bit of subterfuge kept the metric that determined his bonus, nicely in the acceptable range.

Inside the plant, and much more problematic for me, was the fact that we measured every operation that was conducted by every person on the assembly floor. If five people were operating machines that stamped memory chips into circuit boards, one of the systems I was responsible for would calculate and report the cost per chip inserted. Later, I would use that value in other calculations. Unfortunately, the collection system didn’t differentiate between new assemblies and rework. So, while the guy building new circuits was feeding racks of memory chips into an automated press, the woman at the next station was unsoldering and re-soldering defective chips by hand. He might insert hundreds of chips per hour, while she struggled with five. Mine wasn’t an ethical dilemma though. The nature of the repair business meant that sometimes we would receive a part for repair that was no longer in active production. In the reporting side of my system, when I had to do the math that involved “average insertion times” the results were too small to measure and they would fail on a divide-by-zero error. Discarding these results made the woman look bad, including them crashed the system.

I am recalling those early struggles as we study the results of our newly implemented analysis of engineering inspection reports. One of the changes we have to make is to properly reflect inspections that people performed, prepared and participated in. These aren’t just three categories, or three bits of metadata, they are three distinctly different statements about someone’s role in a very important process. Not only is the distinction important, it’s important to know what is what, once these variables are calculated so the right metric is used in the right formula. For instance, the lead time on scheduling the inspection is properly attributed to the person who was in charge, so a guilt-by-association attribute should not be applied to every participant. There also has to be a distinction drawn between the person preparing the report and the person who performed the inspection. We aren’t talking anything on the scale of squirreling stuff away in a trailer, but if the relative performance of individuals is being compared, the right metric has to be used.

So why am I trying to commit this to memory (that is why some of us have blogs) “because it’s so easy to get this stuff wrong” and once your mistake is incorporated into a management report, nobody will ever know. I don’t know about you, but when I get a number to appear in the box on a screen prepared in SharePoint Designer, I am happy, and I feel close to being done. When I look back at that code and see “@Prepared_x0020_By” I know that it’s the right column, but when my coworker is reviewing the code where I refer to “$EngName”, she might be tempted to simply assume it’s the right variable. This is where SharePoint development has to mirror systems development. We have to document what these reports are supposed to show, and we have to document what they actually show. Then, we have to test the process, and the results.