The Data Flow Economy

I like it when the future catches up with you.

In 2000 I read a science fiction novel by Greg Bear called Slant, and my jaw hit the floor. Why? Because it was a story that was spookily aligned to a new venture that I and others had instigated just six months before—and Slant was first published in 1997.

The wow moment came about because I could suddenly see where it could all go – Greg Bear was already painting the end game of what I took to be the start of something new.

What was more surprising was that this wasn’t a science fiction story about zap/bang technology, aliens or space wars—it was about (in part) data, and its relationship with society. Greg Bear wouldn’t have known it at the time, but he was writing about we might now call Big Data, or rather, what comes after Big Data. His term for it was The Data Flow Economy.

The Data Flow Economy is based around the notion that there is more data than you can possibly hope to manage, but from which value can be derived. There is so much data that it is a constant overwhelming stream of the stuff. A giant river. But whilst you can’t hope to consume or understand it all you can dip into that stream looking for specific things that might be passing by in the moment—passing by right now.

The stream of everything is raw Data. Observing that there are certain types of data is Information. Analysing that information generates Knowledge, and acting on that knowledge leads to Wisdom.

Each of those transformative stages adds value—creating something that can be monetised—and sold. And that is the basis of the Data Flow Economy. An economy not based on the movement of money, but the movement of data.

For a time it looked like it was already happening for real—we quickly found ourselves building data stream monitoring solutions that fished for data—streams with potentially millions of items flowing by each day. Those solutions were based around the notion of an inverted search.

Today we are used to the idea of searching for things—a few small queries searching the giant corpus that is the Internet. Inverted search turns that on its head—when there is so much stuff to search, you use the stuff to search against what you are interested in.

In fact, it’s a bit like fly fishing with a lure designed to catch a particular species of fish. Rather than seeking out the fish in the wide flowing river, you let the fish find you.

All that was back in 2004. But the aftermath of the DotCom bust, economic downturn, then crisis, put paid to a lot of that thinking. Suddenly nobody was interested anymore.

Now it’s back. It’s been back for a while—Big Data blah—but this time it is stepping on to something more akin to the beginnings of the Data Flow Economy. And as is often the case the current crop of evangelists see it as some entirely new, perhaps not realising that it has all happened once before.

Well, sort of. I am probably being a little disingenuous.

A colleague point me at this just recently——from June 2015. An interesting presentation actually—well delivered and worth a watch—but in some regards it could have been recorded over ten years ago—if not fifteen.

What goes around comes around. Perhaps we should all read more science fiction.

I can certainly recommend Slant on its own merits.