How to Avoid the Big Data "Gotcha's"

As exciting as Big Data is, the executives I meet with are understandably leery of jumping in with both feet. After all, some industries have been wrangling terabytes of information for years. And it's true that the costs of sourcing, cleansing, loading, querying, and storing transaction data in traditional databases often exceed the anticipated benefits. The Big Data trend is as much about new solutions for managing and storing growing information volumes as it is about the data itself.

The question on many business leaders' minds is this: Does the potential for accelerating existing business processes warrant the enormous cost associated with technology adoption, project ramp up, and staff hiring and training that accompany Big Data efforts? Compounding the skepticism, many leaders have been burned by failed IT adoption efforts in the past and have begun losing patience, suspicious that Big Data is what one of my clients calls "another intellectual exercise in IT."

Justifying the investment in Big Data isn't easy. Indeed, I've watched as companies spent many months and untold dollars researching Big Data only to struggle to define a business case. Even with a good business case, politics and ownership issues can easily torpedo Big Data projects, particularly if they necessitate changes to organizational structures and business processes. The risks inherent in adopting emerging IT solutions can quickly overwhelm the rewards.

Wondering whether Big Data is the right call for your company? See if you can answer these five questions:

1: What can't we do today that Big Data could help us do?

Business leaders must be able to provide guidance on the problem they want Big Data to solve, whether you're trying to speed up existing processes (like fraud detection) or introduce new ones that have heretofore been expensive or impractical (like streaming data from "smart meters" or tracking weather spikes that affect sales). If you can't define the goal of a Big Data effort, don't pursue it.2: What skills, technologies, and existing data development practices do we have in place that could help kick-start a Big Data effort?

In the interest of managing data as a corporate asset, many business leaders have formed discrete Data Management organizations. The mission is simple: to manage information in its own right, separate from applications and platforms. The data architecture, data quality, and metadata skills and tools these teams provide can often be leveraged to support Big Data efforts. If your company doesn't have an effective data management organization in place, adoption of Big Data technology will be a huge challenge. 3: What would a proof-of-concept look like, and what are some reasonable boundaries to ensure its quick deployment?

Big Data can take on a life of its own, evolving from informal hallway banter into a full-fledged custom development effort. Once you articulate a bona-fide business problem, be clear about the desired outcome. As with many other proofs-of-concept the "don't boil the ocean" rule applies to Big Data. Define a "bite size" effort, and expand it once it's shown value.4: What determines whether we green light Big Data investment?

Know what success looks like, and put the measures in place. At one specialty retailer analyzing social media interactions resulted in more targeted product recommendations on their website. The Big Data team projected a return of 7 percent monthly revenue uplift among the high-value customer segment. The effort would end up paying for itself in four months. The team had no trouble securing funding for the project.5: Can we manage the changes brought by Big Data?

The structural and process changes that will likely occur with a Big Data effort need to be managed very carefully. Although many will marginalize Big Data as a technology effort, as with any business initiative its anticipated benefits should be clear. Someone should then establish and communicate the progress toward the desired outcome. As change management guru John Kotter said, "Effective leaders help others to understand the necessity of change and to accept a common vision of the desired outcome." Once you've pinpointed the benefits, appoint a spokesperson — the higher in the organization the better — to sanction the effort and communicate progress. This person should have the organizational authority to ensure Big Data gets mindshare, budget, and support. With the regular communication of tangible results, the payoff of Big Data can be very big indeed.