Opening our operations with Wikimedia Labs 2012-08-20


“For the past year and a half we’ve been working on a project named Wikimedia Labs, which enables us to invite our community to contribute to how our sites are run. Labs is a cloud computing environment using OpenStack for development, testing and deployment of Wikimedia’s infrastructure as a whole, enabling us to treat our infrastructure as an open source software project... When Wikipedia and its sister projects started, volunteers had root level access on our infrastructure. They were the only roots and most of the infrastructure they built is still in use today. Our lenient access policy made us flexible, so changes could happen quickly. Also, the sites were smaller, had far fewer users, and large, fundamental changes could be made in production. Growth has made us less willing to give out root access to volunteers. Because of the size of our sites, downtime is less acceptable. But having fewer volunteers means we have less ideas, and due to that, our ability to make changes quickly is decreased. We haven’t had a new volunteer root in years. We haven’t even had a new volunteer with shell access. Engaging volunteers and enabling them to easily contribute is a wider problem as well. Our software development community scales with volunteers. Unfortunately, operations doesn’t scale in a similar way right now. We’re limited to the staff operations engineers we currently have. The staff is great, but the fact that operations can’t scale to meet the needs of a large growth of developers means that operations is a bottleneck. Furthermore, our access policy prevents volunteer developers from learning how our infrastructure works. This leads to a situation where our staff developers and volunteer developers can’t easily collaborate. Our volunteers also have no way of appropriately testing their changes, since our infrastructure is complex and difficult to replicate. This means it’s harder to take contributions, which further slows the pace of changes on our sites. The solution we are taking for this is to open access to our infrastructure up as widely as possible. We are making it possible for volunteers to use and modify our infrastructure like they do our software. Operations work can be done via configuration management tools, orchestration, and cloud computing. Thanks to this, operations can be treated like a software development project. If it can be a software project, then we can make it an open source project, allowing us to open infrastructure development to the wider community. Opening up infrastructure development to the software development community allows developers to bypass operations for work when operations is too busy to get to a task quickly enough. It also makes it possible to take contributions from volunteer operations engineers, bringing back the flexibility we originally had. However, going staff-only to volunteer-based is hard. We still have the same concerns about site availability, so we must take a new approach. Our first step was to release our puppet configuration. We spent nearly a month going through the configuration, pulling out sensitive info, ensuring we don’t release anything that would open us to major vulnerabilities. Next we created an infrastructure, called Labs, where volunteers can test their work, document it, and eventually have it deployed live to production. Our goal is to build a self-sustaining operations community using this infrastructure in a way similar to how the operations staff team currently works... Labs has its own set of terminology, so to understand this technical overview, it’s likely good to read the terminology guide. Labs allows users to create instances (virtual machines) that multiple users can use and fully control. It does so inside of projects, where projects are maintained by a community and the projects reflect real world projects, like testing and developing MediaWiki extensions for Incubator... We launched Labs as a closed beta in October 2011, and are still in closed beta at this time. At the time of this writing, Labs has 79 projects, 129 instances, and 264 users. So far the community is very vibrant. Here’s a small select list of community maintained projects... Labs is community built and maintained. We’d love for you to help out! If you’d like to help, please request an account and come talk with us about what you’d like to help with...”



08/16/2012, 06:08

From feeds:

Open Access Tracking Project (OATP) »

Tags: oa.comment oa.crowd oa.wikimedia oa.floss oa.wikipedia oa.openstax oa.wikimedia_labs



Date tagged:

08/20/2012, 18:13

Date published:

04/19/2012, 16:13