Indexing Australian GLAM blogs

Last week I did a big update to the code that drives the Australian GLAM Blogs website and Twitter account. Initially I was just intending to do a bit of interface cleanup, and add a simple search tool to find blogs. In thinking through how the search index would work, however, I realised that the most effectively way to do it would be to download all the categories (i.e. subjects) each blog has used. If I was doing that, I may as well grab all the other post information (title, author, date..). Suddenly, I was creating an index of every article from every blog in the list.

The mechanics of doing this retrospectively are relatively simple, but the outcome is inconsistent. I’ve always used Feedparser to grab the blog information, but previously only used it to tweet the title of the post. Feedparser has a lot more functionality than that, but it’s limited by the RSS and Atom protocols it’s using. What that means is that how far back we can go retrospectively depends on how each feed is set up - feedparser can only work with what it’s given. Generally this is the last ten posts regardless of their age, but it varies. To get the retrospective information, I wrote a simple Meteor application that pulled in the data from each feed and feed it into a MongoDB collection. I did the same thing for the categories/tags used, either adding or incrementing the number associated with each tag. Here's the important bit:

while (item = stream.read()) {

//  add article data to the Articles and Tags collections

	// use Fibers so we can interact with the collections inside Feedparser
	// see https://stackoverflow.com/questions/21151202/async-call-generates-error-cant-wait-without-a-fiber-even-with-wrapasync
	// add any tags to the Tags collection
	Fiber(function(){
    var cats = item.categories;
    cats.forEach(function(x){
   	var tag = x.toLowerCase();
    Tags.upsert({tag: tag}, {$inc:{total: 1}});
      });
    Fiber.yield();
   }).run()

   // add the article to the Articles collection
   Fiber(function(){
    var array = item.categories;
    var cats = [];
    array.forEach(function(x){
      var normalised = x.toLowerCase();
      cats.push(normalised);
     	});
    Articles.upsert({link: item.link}, {$set: {title:item.title, author: item.author, categories: cats, blog: meta.title, blogLink: meta.link, date: item.pubdate}});
    Fiber.yield();
      }).run()
    }

Fibers

There are a few things going on here. The first thing you’ll notice is my comment about using Fibers. This is a concept created to deal with the fact that JavaScript is naturally asynchronous, but when used as a server-side language in nodejs, sometimes you really need to do things synchronously. To be honest I