People often develop courses of action based on data. We do this in our personal lives. It happens in organizations of every size. In fact, we think it ridiculous for people to make decisions without supporting data.
But sometimes when we develop plans based on data, we use faulty, inadequate, misinterpreted, or poorly correlated data.
For years I have camped with Scouting groups that occasionally make what we call ZIPLOC omelets for breakfast. It works like this. Each person breaks however many eggs they are going to eat into a freezer strength ZIPLOC (or other brand) bag. They add shredded cheese, bacon bits, diced ham, salt and pepper to taste. They squeeze all of the excess air out of the bag and seal it tight.
The bag is then dropped into a kettle of boiling water for about 10 minutes to cook. When there is no longer any liquid egg in the bag (you have to squeeze the concoction to make sure), it's ready to eat. You can dump it onto a plate or eat it right out of the bag. The process is quick and the mess is minimal.
On one camp a couple of years ago, one scout's mother refused to allow him to have a ZIPLOC omelet for breakfast because she had heard a news report stating that cooking food in such bags can cause cancer. Her concerns were not unfounded. As explained in this About.com article, ZIPLOC bags are not designed for boiling and can break down under such temperatures.
While we should use boil-able pouches (yes, they are available), this mother's concerns were way overblown. Her son is not going to contract cancer from having two or three ZIPLOC omelets during his time in the Scouting program. When one considers the other far more hazardous fare consumed by this family, it would seem that the anti-ZIPLOC omelet rule is like straining at the gnat while swallowing the camel.
Sometimes we see a good trend and assume that an organization or a population could achieve better results by enticing or forcing people to follow that trend. This sometimes gets the equation backwards and produces less than optimal results.
A few years ago when I was serving in a leadership position in my LDS ward (congregation), the church announced that it was raising the standard for young men to serve as missionaries for the church. When we understood the scope of the new rules, my congregation's leader confided that one missionary serving from our ward wouldn't be serving if the policy had been in place a few months earlier. Within a few months the young man left his mission early.
The rationale for the policy seemed well founded. But I wondered how the old policy had come about. After all, during my missionary days, the church would take almost anyone that met minimal qualifications. Leaders would go to great lengths to keep some of these people on their missions when common sense might have dictated otherwise.
I started to connect dots as I attended leadership meetings over the following couple of years and spoke with acquaintances that had some inside information. Church leaders had data showing that those that faithfully completed missions for the church were far more likely to remain faithful in the church and be successful in family life than those that did not. The natural conclusion was to assume that getting more young men to complete full-time missions would extend those results.
My understanding (which may be inadequate) is that church policy was based on this assumption for many years until they developed more data that demonstrated this assumption to be false. Young men that barely qualified for missionary service or that stayed on missions by the skin of their teeth fared no better in church activity and family life than those that never served as missionaries.
When I was a missionary, Elder Robert D. Hales (who was then a Seventy and now serves in the Quorum of the Twelve Apostles) visited our mission a couple of times. During one meeting he encouraged missionaries not to be "a Harry." He then related the following joke.
A man went golfing every Saturday morning with his friend Harry. One Saturday afternoon the man returned home looking awful. His wife asked him what was wrong. "You wouldn't believe it!" he moaned. "Harry had a heart attack on the 10th hole." "Oh my goodness," his wife responded, "That must have been awful." "It sure was," said the man. "For the next eight holes it was hit one, drag Harry, hit one, drag Harry."
Elder Hales said that some of our missionaries were like Harry when it came to missionary work. Their companions had to drag them around to get anything done. "You know who you are," he said.
When it became apparent that some of the "Harrys" in the mission field weren't themselves being helped by being there, raising the standards for missionary service was a natural response.
It is well known that the average bachelor degree recipient earns far more over their lifetime than the average high school graduate that has no college education (see L.A. Times article). The natural thinking is that getting more people into college and more people graduating from college would dramatically increase lifetime earnings for those that otherwise wouldn't go to college. This has driven public and private policy for many years now.
But it turns out that many degree recipients that would not have gotten a degree in earlier years do not earn more (or much more) over a lifetime than they would have earned sans a college education (see Jan 2011 post). The reason for this is that they get degrees that are not worth much in the job market. Or they get decent degrees but cannot perform jobs that produce higher income.
Not all college degrees are created equally. So, while the average college graduate out-earns the average high school graduate, there is such a broad variance in the value of a college degree that many now graduate college with mountains of debt and little or no increased earning capacity to show for it.
Perhaps it would be wise to ask what characteristics distinguish higher earning people. This includes far more than a college diploma. Giving someone a diploma that lacks the other (and probably more important) characteristics of a higher earner will not make them earn more over their lifetime.
Much of this attitude that still pervades higher education was at the root of our recent crash in the housing market. For years public and private policy assumed that society would be better off if more Americans owned their homes. Data showed that homeowners had lower rates of crime, greater family stability, and better financial performance than non-homeowners.
In an effort to try to get these kinds of benefits for those that didn't own homes, gargantuan sums of public money were poured into housing subsidies (with a fair amount of it ending up in the pockets of well-heeled bankers, heads of quasi-government agencies, and politicians).
But the nouveau homeowners did not turn out to have lower rates of social problems than their non-home-owning counterparts. A fair number of them couldn't actually pay for the homes they were enticed into buying. The market has had to absorb all of that, exacting a nasty price from almost everyone.
Maybe it would be wise to take a humility pill when thinking that we have data that proves that we can make people's lives better if only we can get them to pursue some course of action, especially when incentives and/or coercion are involved. These are too often simply demonstrations of hubris.
I have no problem with inviting people to pursue worthy actions. This is what marketing is all about. It is what missionary work is all about. But whenever we start thinking that we have ways to shortcut the system, we are creating future problems that someone will have to clean up.