The Death and Rebirth of Sakai OAE

Last fall, when Indiana University, University of Michigan, UC Berkeley, and Charles Sturt University all pulled out of the Sakai OAE project, it looked like the end for the Sakai community’s next-generation platform effort. The vastly reduced project team, consisting mainly of Cambridge, Georgia Tech, and Marist College, went into quiet mode as they attempted to figure out what could be salvaged. Now, about nine months later, they have re-emerged with a late beta of a completely re-architected system, promising a 1.0 release in early July. They have also rebranded them project as Apereo OAE, named after the new organization that is the merger of the Sakai and Jasig Foundations, and signaling clearly that the project is not intended to be a successor to Sakai but instead is its own separate project with its own separate destiny.

In retrospect, it turns out that the breakup of the grand coalition and resulting near-death experience may be the best thing that ever happened to the project.

A Brief History

I wrote a long post about the lead-up to the collapse of the project a while back, which I won’t reconstruct here. But for those who aren’t familiar with the history, here’s the short version:

• OAE started as an experiment by Cambridge, whose tutorial-style education is quite different from that of a traditional American university and is particularly poorly served by a traditional LMS architecture. They wanted a platform that was well suited for fluid academic collaboration.
• Early prototypes generated excitement in the community. There was a feeling that the project had the potential of truly re-imagining the virtual learning environment. Other schools began to volunteer developers to work on the project.
• Some elements in the leadership of the Sakai community, which had been frustrated with the slowing pace of development and innovation in the Sakai platform, decided that the Cambridge project would become the future of Sakai. The project was dubbed “Sakai 3.” This created resentment among the developers who were working hard to keep the Sakai platform fresh and up-to-date while putting additional pressure on the Cambridge group to build functionality quickly.
• The Foundation board and new Executive Director, seeing the damage that was being done to the traditional Sakai CLE coalition, revised their position on the relationship between the two platforms, arguing that the old and the new would likely co-exist for many years. Sakai 3 was renamed “Sakai OAE (Open Academic Environment” to help underscore this point.
• Nevertheless, Sakai OAE continued to be seen largely as the successor platform, even if the succession was now further out into the future. Some of the big-named schools in the Sakai community began making major commitments of resources to the development of the OAE platform.
• With these new resources came new requirements. The new schools were focused on meeting their local need of having a fresh, next-generation platform that could replace their existing Sakai platform with minimum user pain and retraining required. Recall that this was precisely not what Cambridge was originally trying to achieve.
• The new resources also brought more complex governance, with multiple committees and multiple layers of approval required.
• Meanwhile, the OAE team had discovered flaws in their original architecture that caused serious scalability problems. While some attempts were made to address these problems, they were not given priority over functional requirements as they should have, probably in part because of the strong and ever-growing pressure to deliver a full LMS replacement in a short period of time based on an inflexible roadmap that was defined by an unwieldy bureaucracy.
• Predictably, the project collapsed under its own weight. OAE hit the scalability wall that they had seen coming for some time. It became clear that the project had no chance of coming close to delivering on its road map.
• The schools that had joined the coalition began leaving the coalition, in roughly the reverse order in which they had joined, leaving just a few survivor schools to figure out if they could do anything of value with the limited funding and therefore time they had left.

Back to Basics

The surviving team took several crucial steps. First, they focused on the architectural problems and, in doing so, returned to the project’s original commitment to build as little as possible on as much existing open source infrastructure as possible. In the early phases of the project, OAE was going to be built on top of Apache Jackrabbit. It turned out that Jackrabbit did not have the right performance characteristics for the kinds of usage that OAE would see. But by the time that became clear, there was a lot of code in OAE that would have to be completely refactored in order to move to another open source back end. Rather than do that in the face of mounting pressure for progress on functional requirements, one of the OAE architects wrote a custom back end that could replace Jackrabbit with relatively minimal change to the code on top of it. And this was the source of OAE’s near-fatal performance problems. This time, the team bit the bullet and re-implemented the back end on the battle-tested Node.js platform. In the process, they redesigned OAE as a true multi-tenant system.

Second, they returned to the original focus on academic collaboration. If the first generation of LMSs were inspired by nineties-era generic groupware like Lotus Notes, OAE is inspired by teensies-era generic groupware like G+ and Dropbox. But rather than just copying those systems straightforwardly and then adding features on top, the team asked the question, “What are the limitations in the basic structures of these systems that make academic collaboration more difficult?” The answer often came back to one of my favorite bugaboos in academic software—permissions. Sharing in these generic tools is awkward for academic contexts, where sometimes you need to be quite restrictive and other times you want to be quite open. The OAE team focused on the relationships among people, documents, and discussions, thinking through some common academic use cases and building a system that specifically supports those use cases.

And finally, they threw out most of the governance, letting the product team run with the vast amount of input they had already gotten regarding user needs over the past few years. The team largely set its own roadmap and made its own design decisions.

And Now…?

As I mentioned earlier, the new system is now true multi-tenant. In fact, Cambridge, Georgia Tech and Marist are sharing an instance of the platform. Furthermore, it has a feature called “permeability” which allows sharing and discoverability between tenants on the platform. (Each tenant can turn this on or off.) Functionality appears to be roughly where it was before the implosion, with some incremental improvements in both capability and usability. The system scales linearly. (You can see a recent state-of-the-project presentation here and an architectural overview presentation here.) On the one hand, the project has far fewer resources than it did a year ago. On the other hand, the current project participants seem committed to continuing the work, and the overhead of additional resources didn’t seemed to do more harm than good.

That said, OAE is at an interesting crossroads. Having caught up with its original goals, they need to decide what to do next. Even more importantly, they need to decide how they are going to decide. The biggest challenge for user-facing academic open source projects like OAE, Sakai, and uPortal is maintaining the right kind of link with the users and sponsors. These projects need continuous and serious input from their adopting stakeholders in order to feed their vision for what to do next in order to keep their projects fresh and compelling. In the Sakai and Jasig communities, I have generally seen one of two things happen in this regard:

• A top-down effort is created to drive the effort through a coalition of senior managers, which often results in inefficient development, lots of political squabbles, and clunky software. (See Conway’s Law.)
• The developers are left to drive the product progress on their own, which often results in insufficient resourcing for work that needs to be done and poor or inconsistent processes for getting user input and developing a vision for the product’s future, since it is led by developers who have not had training as product managers and often don’t see product management as what they do.

The OAE project actually has some talented product management types on the team who have actually been functioning as product managers during this last period of the project’s development. But it’s not yet clear how the product management role can function best in the context of a project that is funded by multiple institutions. I am very curious to see how the governance works out. I am also curious to see how the project pivots to its next set of goals as their foundational academic collaboration functionality matures. It is not obvious to me that moving to traditional LMS assessment and grading functionality is the next logical step. I could see, for example, how OAE could be a terrific hub for Connectionist MOOCs, particularly if it were to add RSS aggregation and sharing capabilities or even deeper WordPress integration.

Regardless, the turnaround the project has achieved to-date has been remarkable. There are some deep lessons here participants in such projects—ones that merit further study. I hope that we will hear some more retrospective analysis from project participants at some point.

The post The Death and Rebirth of Sakai OAE appeared first on e-Literate.