The Tech World is Overloaded with Data

I spend a lot of time writing about technology and tech policy, in books, blogs and in a weekly column I write for the Guardian newspaper’s US website. Luckily, there’s a tremendous amount of data available on this topic.

As with almost anything else, I start with the federal government. The Census Bureau has an enormous amount of useful data including (somewhat surprisingly) an informative introductory page on tech history. The Federal Communications Commission offers a great deal of data about telecommunications and broadcasting, but it is — I fear deliberately — deficient when it comes to up-to-date information about broadband access and cost at the local level. The “Broadband.gov” site is informative but also lacking in some key data.

International data is often incomplete, but the United Nations’ UNESCO  Science and Technology site is helpful.

Needless to say, Google and Wikipedia can be great places to start when looking for data in any specific area. The more detailed the search terms, as usual, the more likely you are to find something useful.

A sampling of other sources:

• For technology stock market data, CNBC has a useful page with broadly based information.
• For more detail, I often check the “CrunchBase” database at AOL’s TechCrunch tech blog, which has been compiling information about the tech scene, especially startup companies, for a number of years.
• The Wall Street Journal offers deep, granular information about companies as well, but its website has a paywall. (I subscribe online, and it’s worth it to me.)
• Many universities have research units generating significant amounts of data. Typically, these are housed in centers or institutes focusing on one topic. I haven’t found a central database of all such organizations, but there are a number of them by topic. For example, here’s Stanford University’s list of biotech research initiatives.

One of the most important emerging areas in the technology world is called Big Data — describing the massive amount of information we (and companies and governments) are creating every second of every day. It’s simply staggering to contemplate how much there is, and how it’s growing. Articles in the Economist and New York Times do a decent job of explaining the phenomenon.

Big data gives us new insights into our world (and is used for scary purposes, such as in surveillance). One way to make sense of it is through visualization — turning numbers into pictures and animations. One of my favorite ways to explore data this way is an IBM site called “Many Eyes” — take a look around to see what I mean.