Few Surprises in NSF Report on San Jose State U. Test of Udacity Courses

With financing from the National Science Foundation, a San Jose State University research team has taken a close look at a high-profile experiment in which the institution offered “augmented” online courses last spring in partnership with the online-learning company Udacity. The team released its long-awaited report late Wednesday, and it contains many more tables than surprises.

In part, that’s because the university and Udacity stole much of the report’s thunder two weeks ago, when both made a big deal about how students’ grades in the online courses—dismal in the spring experiment—had improved when the same courses were offered again during the summer, this time to students who were largely better prepared.

The university’s experiment has attracted a fair amount of attention because it attempts to test whether the technology and approach that Udacity has taken for its massive open online courses—MOOCs—can be adapted to work in conventional online courses offered to a limited number of students by a traditional university.

While the spring results were indeed discouraging—students in comparable face-to-face courses did much better—the researchers say that low pass rates in the online courses “should be considered in light of the fact that the project specifically targeted at-risk populations”: San Jose State students who had failed some of the courses already and high-school students from at-risk backgrounds.

The researchers also say, perhaps unsurprisingly, that what mattered most was how hard students worked. “Measures of student effort trump all other variables tested for their relationships to student success,” they write, “including demographic descriptions of the students, course subject matter, and student use of support services.”

They go on to conclude, optimistically, that the university has learned from the spring experience: “The faculty members who taught these courses, although they had to contend with major difficulties along the way, believe that the content that has been developed has tremendous potential to advance students’ critical-thinking and problem-solving abilities.”

The report quotes one faculty member who summed it up this way: “Udacity has brought to the table ways to make the courses more inquiry-based and added real-life context.”

The principal investigator for the study was Elaine D. Collins, a chemistry professor at the university, who led a four-member team from the Research and Planning Group for California Community Colleges.