Building Data Discovery into Your Organization

In the 1800s, enterprises organized themselves to use their capital assets effectively. Beginning in the mid-1900s, they organized to take better advantage of their people. Today, "data" are increasingly important to virtually all companies, government agencies, and nonprofits. They offer unprecedented opportunity and pose new risks, in turn demanding deep organizational change. The thinking here is that virtually no enterprise (whether 100 years old or yet to be established), no department therein — indeed no job — will remain untouched.

There are many ways to "put data to work," each with its own strengths and challenges. And while no one fully understands the various forms data-driven organizations will take, we can make some intelligent guesses. One option is to focus on finding and exploiting deep, fundamental discoveries in data. Such discoveries could include the best time to secure a loyal customer, a deeper understanding of how to tailor medical treatments to a patient's genome, new ways to deliver power with less loss, new ways to detect fraud, and so on.

To be clear, this strategy is not for everyone. While the rewards can be intoxicating, the strategy is fraught with danger. So those contemplating this course should find the experiences of the great Bell Laboratories instructive. Bell Labs led fundamental work in physics, information theory, computing, wireless telephony, and quality control that underpins much of today's economy. I started my career as what would now be called a data analyst, focused on network performance, essentially trying to get more bits from calling to called party. From my perspective, three aphorisms guided Bell Labs' management:

1. The secret to success at Bell Labs is working half-days. And the best thing about the Labs is you can work those 12 hours any time you want.
2. The secret to success at Bell Labs is having great ideas. You only need one every couple of years. But it has to improve phone service. And it has to be truly great.
3. The secret to being a great manager at Bell Labs is hiring the right people, giving them the tools they need, pointing them in the right direction, and staying out of their way.

During the course of my work, I wondered if the precepts of quality control, so successful on the factory floor, might apply. I had time to "follow my nose" and doing so led me to data quality. Within a couple of years we set up a lab of about 15 people — mathematicians, computer scientists, statisticians, and systems engineers — and spent half our time immersed in enormous AT&T data issues (we helped the company save hundreds of millions) and the other half extending the principles of quality management to data. This was 15 years before most others could even articulate the problem and opportunity.

You cannot replicate Bell Labs. But you can learn from its successes. Here are three steps to setting up a discovery process in your company.

First, set up a separate department, your industry's equivalent of a "laboratory." There are many reasons a certain organizational independence is important. I find the most pedestrian to be the most compelling: If you put good analysts in the XYZ department, managers will draw them into XYZ's day-in, day-out issues, and the analysts won't spend their time seeking fundamental discovery. Their own department gives them the opportunity to dedicate their time accordingly.

Second, learn to manage the data lab. This can prove a delicate balance. One the one hand, you must learn to tolerate and encourage experimentation. The search for fundamental truth is loaded with missteps, blind alleys, and encouraging results that don't pan out. It can be frustrating for the manager used to producing results on a quarterly basis.

But a lab is not an industrial sandbox. It must fully focus on the business of the enterprise, either driving down costs or creating new value for customers. It is appropriate to maintain a portfolio of opportunities, based on factors such as the time frame in which they pay out. Most critically, the interfaces between the lab, its work, and later steps in the data to discovery to dollars process must be clearly defined.

Third, manage the people. You must staff the lab with a talented and diverse group of specialists that enjoy working together. This may be the most difficult step. I believe it was Jeff Hooper, a Bell Labs' reliability specialist, who first noted, "Data don't give up their secrets easily. They must be tortured to confess." The first-rate data analysts who can extract these confessions (the vast majority of whom have advanced degrees in esoteric fields) are in short supply, and the demand is growing fast. Diversity of background and opinion is just as important. Great discoveries are most often found on the boundaries of specialties.

You don't need to do everything yourself. The sheer quantities and richness of available data, either for purchase or free, are exploding. Many organizations supplement the work of discovery through far-flung teams, including academics, suppliers, partners, customers, and even competitors. That said, you must clearly define and build a world-class "core," enabling you to attract and retain a critical mass of the best talent and build a knowledge base that provides a (relatively) steady stream of discoveries.

As a reminder, pursuing a strategy that seeks to find and exploit fundamental discoveries in data is not for everyone. Halfway measures will not do. But for those who make the commitment and do it right, the rewards will be worth it.