We work with big technology companies. If there is anyone that is really doing Big Data, I would think it would be big technology companies. After all, they believe in technology, have plenty of computing horsepower, and have people that have the necessary skills to do it.
The reality is quite the opposite however. Most of the time we are working to overcome very simple problems like duplicates or obviously incorrect entries. The real data industry came up with ways to deal with these problems decades ago. Nevertheless, our clients have such low confidence in their data that they often retain us to start over.
Here are a few of the things we see preventing big companies from truly using Big Data:
- Legal Departments: The legal department does not play to win, they play to not lose. They would much rather prevent the collection of data than otherwise. After all, a company that has not collected any data does not have to worry about losing data in a breach and then getting sued.
- Poor Planning: Good data handling takes time and effort. Data initiatives invariably take longer than a quarter to implement, and longer than that to produce returns. Almost all companies are looking to hit the number this quarter.
- Internal Competition: Competition between departments can cause them to hoard data (at best) or go underground with their data (at worst), creating silos of data that is riddled with duplicates and innacuracies.
- Turnover: The people in charge of these data initiatives have their eyes on bigger and more important (more visible) jobs -- so they change often. The person taking over the job is just as uninterested in long term data health, so the problems go unaddressed.
As with many promising technologies, Big Data's biggest challenge is not in the technology but in the way people work together inside companies. There are enormous gains to be made by the companies that realize what can be done with these new tools and organize themselves in such a way to take advantage of it.