5 Things Almost Everyone Gets Wrong About Big Data and Analytics
Big data is cool. Being able to use it right and make it all actually make sense is even cooler. The problem is, there are a lot of pitfalls to working with large data sets and each minor misstep can make the next step in the process slightly off or even wreck the whole thing. People lose sight of the art in the science.
I’ve worked with data that took clusters with dozens of units and weeks to get a tiny step in the right direction. I’ve delved into analytics for GIS, oil and gas, and managed agents (among many other side projects). Turning data into something useful has been my forte and has afforded me many jobs, but the problem is that data is as fragile as it is exciting.
Let’s dive into the 5 things almost everyone I’ve met throughout my career seems to get wrong at some level about big data and analytics. We aren’t going to delve too much into algorithms and science, but more the philosophies and methodologies that can poison the whole well before the math even matters.
Trash In, Trash Out
Bad data gets you bad results. It’s obvious when put that way, but many people cling to bad data hoping that it gets “rounded out” in their data processing or if they can “finesse” the algorithm just right. I’ve seen people cling to fundamentally flawed…
