Finding the needle in the haystack is the goal of the enterprise and with that in mind ensuring the data you are analyzing has been modeled and filtered is of critical importance.
The idea that the combination of predictive algorithms and big data will change the world is a tempting one, and this may end up being true. But for right now, the industry is facing a reality check when it comes to big data analytics. Instead of focusing on what algorithms to use, big data success depends more on how well you cleaned, integrated, and transformed your data than anything else.
There is much work that goes into prepping and cleaning data before it can be analyzed. You may have the sharpest data scientists on the planet writing the most advanced algorithms the universe has ever seen, however if your data set is dirty, incomplete, or flawed than your project will not be successful. Industry reports indicate that up to 80 percent of the time and effort being spent on big data analytics projects is spent on cleaning, integrating, and transforming the data.
Instead of focusing on algorithms, enterprises should focus on validating the data. Everybody basically has the same algorithms. No Magic recommends that the Big Data teams have a good grasp on the role Ontology plays in their Big Data initiative, especially when dealing with unstructured data. In the case of financial markets the new FIBO (Financial Industry Business Ontology) standard should be well understood. Dodd-Frank brings on a whole array of new challenges.