Big data brings new power to open-source intelligence

"In November 2013, the New Yorker published a profile of Eliot Higgins – or Brown Moses as he is known to almost 17,000 Twitter followers. An unemployed finance and admin worker at the time, Higgins was held up as an example of what can happen when we take advantage of the enormous amount of information being spread across the internet every day. The New Yorker’s eight-page spread described Higgins as 'perhaps the foremost expert on the munitions used in the [Syrian] war', a remarkable description for someone with no formal training in munitions or intelligence. Higgins does not speak Arabic and has never been to the Middle East. He operates from his home in Leicester and, until recently, conducted his online investigations as an unpaid hobby. Yet the description was well-founded. Since starting his blog in 2012, Higgins has uncovered evidence of the Syrian army’s use of cluster bombs and exposed the transfer of weapons from Iran to Syria. And he has done it armed with nothing more than a laptop and an eye for detail. This type of work is a form of open-source intelligence. Higgins exploits publicly accessible material such as online photos, video and social media updates to piece together information about the Syrian conflict. His analyses have formed the basis of reports in The Guardian and a blog for The New York Times, while his research has been cited by Human Rights Watch. The combination of his keen interest in blogging and a burning desire to sort fact from fiction in the Syrian war has propelled this one-man intelligence unit from blogosphere obscurity to international recognition as a conflict analyst. The rise of his particular brand of social media forensics is portrayed as a triumph of enquiry – and rightly so. Higgins has successfully navigated a maze of rumours and misinformation and revealed much about the true nature of the Syrian conflict. On a larger scale, the story of Brown Moses offers a unique insight into the enormous potential of open-source intelligence. His work shows how the information posted online for all kinds of reasons can become a powerful tool in the right hands ... Two overlapping developments in particular have greatly influenced the growth of open-source intelligence.  First, the explosion of social media has given us instant access to a wealth of user-generated content ... Second, and on a larger scale, the scope of open-source intelligence has been completely changed by the rise of big data. The meaning of this term is contested but it is commonly used to describe “data that exceeds the processing capacity of conventional database systems”. According to some estimates, some 1,200 exabytes of data now exists in the world and 90% of it was created in the last two years alone ... Social media is a good example of big data in practice. Users are generating 500m tweets and 90m blog posts on Tumblr per day. Every minute, 100 hours of video is uploaded to YouTube. The social media universe is expanding at an astronomical rate ..."