Big Data Takeaways from 2 Events

Last week, I had the pleasure of giving a talk at Enterprise Search Europe 2012 on using search-based applications to capture business value from Big Data (specifically, machine data). This week, I had the pleasure of attending IBM’s Smarter Analytics conference in Paris, though sadly I had to drop in and out of the event to juggle commitments.

At ESE, it seemed to me about half the delegates were occupied with a longstanding debate: to make enterprise search effective, do we first need to clean out the garage (our burgeoning stores of unstructured enterprise data) and apply some old school Knowledge Management workflow discipline to keep the place tidy, or should we throw up our hands and forget about cleaning out the garage (it’s a lost cause, and storage is cheap anyway!) and just point a search engine at it, hoping for the best as we work to make engines smarter. Many, of course, took up middle-ground positions, while almost all seemed to acknowledge that the demand for global enterprise search has ceded way to tactical search tailored to the needs of individual workgroups and workflows.

The other half of the delegates seemed to have their eyes out on the horizon, beyond enterprise search as Steve Arnold would say. They were engaged in conversations about search-based applications and their role in the era of Big Data, some devoting serious attention to this role for the first time. In some ways, it seems as though the search industry itself has been slow to realizing the true value of the technologies right under its nose.

By contrast, this value was not lost on Dassault Systèmes, HP, Oracle and IBM. Their recent acquisitions of Exalead, Autonomy, Endeca and Vivisimo, respectively, are a testament to the value of search-based technologies and search-based applications in the era of Big Data.

At yesterday’s IBM event, I could certainly spot the search hole in the analytics sessions I attended. As I told Vivisimo’s amiable Mark Myers, whom I met at ESE, I hope IBM makes stellar use of Vivisimo’s talent and technology to fill the gap (and I hope, as here at Dassault, that Vivisimo survives as a core technology and standalone platform).

However, what I found most interesting yesterday was the closing session. What is IBM’s vision for the future of analytics? A search engine. Not an ordinary engine granted, rather a highly sophisticated one mounted on an MPP supercomputer, namely Watson. Ah, the beauty of the search world: ask a question in natural language, engage in a dialogue to refine your request, and get back the answer ranked as the best fit (with some good machine learning to learn from one’s mistakes).

While the road may be long, as they move with characteristic IBM deliberation toward a world “Ready for Watson,” with measured development of Watson apps and appliances while waiting for multi-threaded technology to make supercomputer power accessible to mere mortals, I think they are right-on in seeing search-based technologies as the logical future of analytics in the era of Big Data.

It’s the same argument I made at ESE in explaining Dassault’s positioning of Exalead as a key axis for the future, revealing the latent information intelligence human beings need to understand, participate, and act in a changing digital world.

I’m reminded of Sue Feldman’s comment that perhaps it’s search’s roots in probabilistic computing that have pushed it into such a pivotal role. It’s evident that probablistic computing (rooted in finding not the ‘right’ answer but the ‘best’ answer), and the recommendations and predictions it enables, are at the center of evolving ways of thinking and deciding in our ever-changing world.

What do you see as the future of search technology in the era of Big Data?

PS: ESE & IBM events participants, don’t hesitate to let me know if I missed the mark in my recap 