Technical Development Update

As most of you know, we began the active development toward a DPLA technical prototype near the first of this year.  The interim technical development team that led this effort worked extremely hard over about 17 weeks to come up with the alpha, metadata-focused service that we demo-ed at the DPLA West plenary in April.  This team, led by David Weinberger and including Daniel Collis-Puro, Sebastian Diaz, Paul Deschner, Kim Dulin, Matt Phillips, and Nick Caramello, and supported by an external advisory group including MacKenzie Smith, Karen Coyle, Dan Bricklin, Martin Kalfatovic and Chris Freeland of the Technical Aspects Workstream, and many other project teams, did a wonderful job getting the system off to a great start and showing us the possibilities of what the back-end of the DPLA can accomplish.

This group managed the process of bringing together large and diverse sets of metadata into the platform, including the newly-released 12 million records from Harvard, 7 million records from the Library of Congress, collection records from the University of Illinois Urbana-Champaign, bibliographic records from the Biodiversity Heritage Library, a photographic archive from the San Francisco Public Library, curated Youtube and NPR book- and author-related content, and descriptive data about American public libraries. They have provided access to these records via a live, open, documented API that has been up and running since January; the data is also available through open bulk download. This team also hosted a very successful hackathon in which developers were invited to spend a day together hacking up a batch of stub applications using the API. Data visualizers, mash-ups, and API wrappers for other languages all appeared in short order at this event, demonstrating the API’s ease of use. In recent months the group has begun the initial work of bringing the platform’s bibliographic datasets towards edition and version clustering around a given title via the use of uniform title analysis within their MARC data.

This team has accomplished an enormous amount in a very short period of time, for which we are grateful.  While members of this group will largely be focusing on other projects in the months to come, we are grateful that David, Daniel and Sebastian will continue to advise the development group moving forward and that Paul Deschner will continue to work part-time on the platform’s development.

For the next 10 months, we will begin to shift our focus to create a version of the DPLA that will be “human usable” as well as an open, back-end service. We will continue to build the platform, complete with open APIs, but we will also emphasize the creation a front-end that will allow for a demo in April 2013. This front-end will not be the end-all-be-all of the DPLA, of course; it will be a gesture toward the possibilities of a fully built-out DPLA.  It will also not be the only way to experience the DPLA’s services.  The back-end work will also continue in such a fashion as to enable the beta sprinters, and others who may come along, to develop additional user interfaces and means of using the data and metadata that we will have in the DPLA over time, which continues to be a key design principle for the project overall.  There is much more to come, but we in the Secretariat and Steering Committee will be working with the workstream leaders and participants (especially Content and Scope, which is already working more closely with the Technical Aspects workstream, and Audience and Participation, which has led on the use cases that Nate Hill has drafted and circulated) to refine what we want this demo front-end to look like. The front-end design will be derived in part from the use case work.  We are also planning to put out a call to those who might like to do the front-end design work for the April 2013 demo version, working alongside those who continue to build out the platform and the beta sprinters, who we hope will continue to code against the platform.

Over the next few days, we will be updating the materials on the dp.la tech dev wikis to reflect these changes in emphasis and to describe in more detail the plan over the next 10 (!) months.  As always, we welcome your feedback, suggestions, and offers of assistance.