Motes in the Data Stream

First, I must say that many of my job experiences have touched on the data collection and analysis issues presented so well in a recent book called The Numerati by Stephen Baker. Whether you are technically-minded or a technophobe, Steve presents a very interesting read on an aspect of technology that has been growing exponentially for years.

Yes, big brothers or sisters are watching you, though not always in the ways you may expect or fear. In most cases, you are not being singled out for attention, but are being constantly assigned to peer groups, tribes, or market segments using statistically determined information collected about you, your family, your household, or anything else that can help a marketer target you with significant offers or options that generate revenue for the marketer's organization. More insidious uses are also implied by these techniques when used for industrial espionage. worker monitoring, political campaigning, or national security. I won't elaborate further- Steve Baker has already covered these and other uses of advanced data mining and modeling in his book. If interested, buy it and read it.

Some key technologies have made all of these manipulative approaches possible. These technologies continue to evolve. The Internet, clusters of servers, analytical databases and tools that crunch huge volumes of data, massively parallel computing, business intelligence, data warehousing, social networking, analytical customer relationship management, and on-line commerce are just a few worth mentioning. One of the points I find interesting is that all of these technologies were designed or developed to address other needs-mostly internal management requirements of large corporations or the statistical requirements of researchers. But now, very bright people, Baker's Numerati, are figuring out other applications that could use technology to manipulate micro and macro behaviors of people and groups of people.

Perhaps we should take a hard look at these outcomes before they truly result in some Orwellian or worse dystopia. As did Oppenheimer and his fellows, we may be releasing a very large and unwholesome genie from the bottle. Or possibly we already released it.