What does a really efficient market look like?

The raison d’être of hedge funds is to make the markets efficient. Or at least that’s one of the raisons d’être, the others being 1) to get rich and 2) to leave early on Fridays in the summer ( resp . winter) to get a jump on traffic to the Hamptons ( resp . ski area, possibly in Kashmir ). And although having efficient markets sounds like a great thing, it makes sense to ask what that would look like from the perspective of a non-insider. This recent Wall Street Journal article on high-tech snooping does a pretty good job setting the tone here. First, the kind of thing they’re doing: Genscape is at the vanguard of a growing industry that employs sophisticated surveillance and data-crunching technology to supply traders with nonpublic information about topics including oil supplies, electric-power production, retail traffic and crop yields. Next, who they’re doing it for: The techniques, which are perfectly legal, represent the latest advance in the longtime Wall Street practice of searching for every possible trading advantage. But the high cost of much of the new information—Genscape’s oil-supply report costs $90,000 a year—means that some forms of trading are becoming even more the province of firms with substantial resources. Let’s put these two things together from the perspective of the public. The market is getting information from hidden cameras and sensors, and all that information is being fed to “the market” via proprietary hedge funds via channels we will never tap into. The end result is that the prices of commodities are being adjusted to real-world events more and more quickly, but these are events that are not truly known to the real world. [Aside: I'm going to try to avoid talking about the "true price" of things like gas, because I think that's pretty much a fool's errand. In any case, let me just say that, in addition to the potentially realtime sensor information that goes into a commodity's price, we also have people trading on it because they are adjusting their exposure to some other historically correlated or anti-correlated instrument, or because they've decided to liquidate their books, or because they've decided the Fed has changed its macroeconomic policy, or because Spain needs to deal with its bank problems, or because someone wants to take money out of the market to rent their summer house in the Hamptons. In other words, I'm not ready to argue that we're getting close to the "true price" of gas here. It's just tradable information like any other.] I am now prepared, as you hopefully are as well, to question what good this all does for people like us, who are not privy to the kind of expensive information required to make these trades. From our perspective, nothing happens, the price fluctuates, and the market is deemed efficient. Is this actually an improvement over the alternative version where something happens, and then the price adjusts? It’s an expensive arms race, taking up vast resources, where things have only become more opaque. How vast are those resources? Having worked in finance, I know the answer is a shit-ton, if it is profitable in a short-term edgy kind of way. Just as those guys dug a hole through mountains to make the connection between New York to Chicago a few nanoseconds faster , they will go to any length to get the newest info on the market, as long as it is deemed to have a profitable edge in some time frame – i.e. the amount of time it will take a flood of competitors to do the same thing. Just as there’s a kind of false myth that most of the web is porn, I’d like to perpetuate a new somewhat false myth that most data gathering and mining happens for the benefit of trading. And if that’s false now, let’s talk about it again in 100 years, when the market for celebrities is mature , and you can make money shorting a bad marriage .