Lessons from Yelp’s Empirical Approach to Diversity

HBR.org 2017-09-20

sept16-20-847354038

Beginning in 2013, a handful of tech companies (including Yelp, where some of us work, Google, LinkedIn, and Facebook) began to research and release data on the diversity of their workforces. The numbers were grim. For instance, in 2014 only 10% of Yelp’s engineers were female. Seven percent of Yelp employees were Hispanic, and 4% were black. These figures were similar at other tech companies, most of which had fewer than 20% of their technical positions filled by women and had low representation of black and Hispanic employees. At Yelp and elsewhere, seeing these numbers was a wake-up call.

Leaders at Yelp wanted things to change for two main reasons. First, only by bringing in candidates of all backgrounds could Yelp really make sure it was hiring the best and brightest. After all, a company that excludes women and minorities is accessing less than 40% of the labor pool. Second, having a diverse set of employees would help Yelp to customize its services for its diverse set of users. Airbnb’s experience serves as a cautionary tale in this respect. When research by one of us (Michael) and others demonstrated that there was widespread racial discrimination on the platform, Airbnb acknowledged that it might have handled the problem better if it had a more diverse workforce. According to Airbnb CEO Brian Chesky, “As a founder, I think we were late to this issue.… When we designed the platform, three white guys, there were a lot of things we didn’t think about.”

Fast forward to 2017, and things are improving after three years of work. The percentage of technical positions filled by women at Yelp in the U.S. has increased from 10% to 18%. Across the company, 10% of employees are Hispanic and 6% are black. Yelp now sets explicit targets for diversity. These goals include aiming for Yelp’s sales teams to match the composition of the diverse communities in which they operate, increasing its proportion of female engineers, and making its culture more inclusive. These goals are communicated through all-hands meetings, recruiting materials, and CEO communications, and executed by increasing both the funnel of talent (through expanded recruiting programs) and increasing Yelp’s yield (for example, doing more to convince the people who have been offered a job to take it). Finally, Yelp has instituted a number of programs to create a more supportive work environment, from employee resource groups to training programs for both managers and employees.

Yelp’s approach to improving on diversity has involved piloting a variety of strategies, and we are now analyzing the data. After scouring academic research and industry reports, leaders at Yelp brainstormed tactics to try, including varying the gender composition of hiring committees, masking voices during phone interview screening, and expanding on-campus recruiting efforts to include selected historically black universities, women’s colleges, and schools with higher proportions of Hispanic students.

As in most pilots, some of these approaches have yielded results while others have not. For example, consider a simple intervention Yelp tried: blinding résumés and having candidates use a voice disguiser in first-round phone interviews (which made the phone screen feel sort of like interviewing Kylo Ren). On the surface, it seemed like this should have alleviated bias. After all, we were eliminating race and gender from the process altogether. Similar efforts have been effective in other contexts: Economic research has found that blinding orchestra auditions greatly increased the number of female musicians hired.

But it turned out the intervention at Yelp didn’t change the gender composition of hires at all. While we are still analyzing why it didn’t work, one potential factor is that women might use different words in interviews (for example, they may be more likely to say “we” than “I,” diluting credit for their accomplishments). In this situation, blinding the process could even exacerbate bias.

This cuts to the heart of the problem in 2017, which is that increasing diversity at tech companies isn’t about finding a silver bullet. It’s about following a systematic process, involving two important steps that are too often lacking in practice.

First, companies should be as transparent as possible, not only about the problem but also about their experiences and obstacles. Both internally and publicly, companies need to share the things they have tried, the things that have worked, and just as important, the things that have failed. In 2013 Tracy Chou — then a software engineer at Pinterest, now a cofounder of Project Include — helped jumpstart the transparency movement by creating a shared spreadsheet for companies to post their employee demographics. In 2017 we’re hoping that companies will go further, listing the policies they have implemented to ensure a diverse workforce, and being clear about whether they worked. The figure at the end of this article gives an idea of what companies might report as a start.

Second, companies should be more proactive about rigorously evaluating the policies they implement, using experiments or other causal inference approaches. As a part of this evaluation process, managers must accept that not all their initiatives will bear fruit and that (as we learned) efforts that were successful elsewhere may not work at their company. For example, blind auditions may work wonders for diversity among violin players but do nothing to help the interview process at a tech company.

Currently, too many approaches are based on intuition, anecdotes, and evidence from contexts that don’t directly map to the tech industry. This means we don’t really know what the best way to increase diversity is, leaving too much guesswork in the process. Increased transparency and rigorous evidence can help to change that.

To this end, we are creating a collaboration of researchers and managers to explore what works, what doesn’t, and why. The goal will be to conduct public-facing research to surface strategies that companies have tried and provide rigorous evidence on what has and hasn’t worked in the relevant contexts. Our aim is to bring a base of evidence for tackling the diversity problem in tech. Yelp will be involved, and we hope other companies will, too. For companies, involvement will include publicly sharing practices, sharing data, and experimentally testing strategies. Together, we can develop a tool kit that tech companies can use to create a more dynamic and inclusive industry.