SJSU and Udacity: Poor Planning and Support, but Valuable Reviewing of Results

A couple of weeks ago I was interviewed for an article in the NY Times about recent pushback against MOOCs.

Jonathan Rees, a history professor at Colorado State University at Pueblo, who has written critically about MOOCs, said their spread is likely to lead to a three-tiered world, with a few high-status “super professors” for whom the courses provide both status and royalties; a larger pool of tenured professors who continue to teach their regular in-person classes until they retire; and “a huge army of adjuncts and teaching assistants,” whose jobs will be vulnerable to online competition.

“The problem with this MOOC-as-labor-issue argument is that it has no place for students and learning,” said Phil Hill, an education technology consultant. “Our starting point ought to be what students need and whether this is an effective form of learning.”

Perhaps to the dismay of Jonathan’s Aunt Nancy, my comments were positioned as refuting his comments. In the context of the NY Times article, that positioning was accurate; however, in Jonathan’s follow-up blog post, he misinterprets my meaning.

MOOCification is about power too. If people with access to power want to convince recalcitrant professors to accept MOOCs, arguing that they must be because this must be is a terrible idea. Yet this is precisely what the two quotes following mine in that New York Times article from last week try to do. [snip]

Whenever Phil makes that infuriating argument, I usually respond with some variation of “profs gotta eat.” Not surprisingly that argument works really well on other profs, but not so well on Phil. However, I want to point out to Phil’s clients that they can’t just wish the class divide away. Since profs gotta eat, they’re going to worry about eating whether you tell them they’re allowed to or not. Moreover, plenty of us believe that by looking out for our interests we are looking out for students and learning. Therefore, making this kind of argument simply serves as a way to shut down all discussion of the subject, which will only breed resentment. [emphasis added]

While I enjoy reading Jonathan’s blog and appreciate his real concern with student learning and the profession of teaching, he is incorrect in his assertion that I want to “convince recalcitrant professors to accept MOOCs” or that I don’t believe that “by looking out for [professors] interests we are looking out for students and learning”. I am not making “this kind of argument”.

Focus on Students and Learning First

What I meant to say is, ironically, what I said: “Our starting point ought to be what students need and whether [MOOC] is an effective form of learning”. The lens of MOOC-as-labor-issue tends to create an us-versus-them, you’re either pro-MOOC or anti-MOOC, mentality that leads to, well, nowhere except arguments. The lens of MOOC-as-teaching-method to be explored should lead us to critically look at learning outcomes and exploration of how to apply or refine the teaching method.

Michael made this same point in a more elegant fashion in his commentary on the SJSU Philosophy Department rejection of edX:

Rather than thinking of MOOCs as products to be bought or rejected, they could have approached them as experiments in teaching methods that can be validated, refuted, or refined through the collective efforts of a scholarly community.

SJSU Plus and Udacity

And this is where the story comes to San Jose State University and the recent announcement that they were “pausing” the Udacity MOOC-for-credit pilot program (SJSU Plus), as described at Inside Higher Education:

After six months of high-profile experimentation, San Jose State University plans to “pause” its work with Udacity, a company that promises to deliver low-cost, high-quality online education to the masses. [snip]

Preliminary findings from the spring semester suggest students in the online Udacity courses, which were developed jointly with San Jose State faculty, do not fare as well as students who attended normal classes — though Junn cautioned against reading too much into the comparison, given the significant differences in the student populations.

Preston Rudy, vice president of the San Jose State chapter of the California Faculty Association, said the university must be cautious with what he called its experiments on students.

“It’s wise to reevaluate and pursue something based on the evidence rather than the advertisement,” Rudy said.

The Chronicle has added more details on the actual results:

News of the break coincided with the leaking of a slide show containing preliminary data on the spring trials, which included three mathematics courses that San Jose State instructors built with Udacity. The courses were offered to a mix of students, some who were enrolled at the university and others who were not, including some high-school students.

The pass rates for the San Jose State students in those courses ranged from 29 percent to 51 percent. For nonenrolled students, the range was 12 percent to 45 percent.[emphasis added]

It turns out that the low end of the scores occurred in the developmental math course suggested by the Gates Foundation. From IHE:

The spring courses – a remedial math course, a college algebra course and an introductory statistics course – were chosen in part because of the wishes of Bill Gates, whose foundation gave the effort a grant, Junn said. Gates, the founder of Microsoft, is a fan of math and science education and wanted the offerings to include remedial math, Junn said.

And the Chronicle:

The [remedial math] instructors played to two audiences: 3,500 learners who had signed up for the course as a MOOC through Udacity, and just under 100 others who were taking the course for credit, about half of whom were enrolled at San Jose State. The two groups watched the same video demonstrations and completed the same routine assignments. The only difference was that the students taking the course for credit also were required to complete three online examinations, with monitoring by ProctorU, the online proctoring company.

Only 29 percent of the San Jose State students in the credit-bearing section passed the course, along with 12 percent of the non-matriculated students, according to Ms. Junn’s slide show.[emphasis added]

Proper Student Support

This situation should surprise no one who follows higher education and online education – remedial students need additional support, even more so if they are in an online course. The fact that the MOOCs were rushed made the situation even worse, as there was not time to design a support system for these students, per IHE:

But, because of the haste, faculty were building the courses on the fly. Not only was this a “recipe for insanity,” Junn said, but faculty did not have a lot of time to watch how students were doing in the courses because the faculty were busy trying to finish them. It took about 400 hours to build a course, though the courses are designed to be reused.

As long as I’m being self-referential in this post, I should note that the point about student support corresponds to one Michael and I made in our position paper for The 20 Million Minds Foundation regarding California online education:

It is important to remember the real goal of using online education to address bottleneck courses here. It is not to offer students seats in courses. It is to get students to complete those courses successfully so that they can complete their programs more quickly. While California cannot guarantee student success, the state can put in place provisions that guarantee students access to the kinds of support that are known to increase the likelihood of student success. This includes taking care to preserve existing campus support networks when bringing in new solutions—particularly solutions implemented by third parties—as well as taking care to provide students with extra support when it is needed. These considerations are important for locally developed solutions, but they are especially important for safety valve solutions where some of the traditional campus support and quality control mechanisms may be circumvented to achieve greater accessibility.

Online education classes typically require more self-discipline, better reading skills, and better awareness of when to seek help than traditional classes do. Offering an online class to a student who otherwise would be shut out altogether is often better than nothing. But we need to recognize that we are already starting with a solution that has its challenges for achieving a goal of high completion rates, even if everything else is equal. Not all students are equally well-prepared for online learning, and pushing students who are likely to fail into an online course may, in fact, be worse than the status quo. Online courses are not a panacea. Students will need help in evaluating whether online is appropriate for them. And if it is not, those students should be given priority access to the traditional on-campus or blended courses.

To me, this is the lens to use when evaluating new teaching methods, including online education in general and MOOCs in particular: evaluating what students need and evaluating whether the teaching method in question effective or not. Clearly for the SJSU Plus program with Udacity as currently designed, the answers are a lot more support and no.

As Currently Designed

But the story does not end in July 2013. To their credit, SJSU is launching an independent research effort to evaluate the results-to-date of the Udacity experiment, with the goal of learning from best practices and improving student outcomes. In a surprisingly good editorial from the LA Times (with the one exception of conflating MOOC and online courses):

The disappointing results from San Jose State’s experiment with online courses shouldn’t be interpreted to mean that such courses can’t help students. But the classes the university offered in collaboration with online provider Udacity were practically a model of how to do online education badly: rushed into existence and sloppily overseen. No one was even aware that some students who had signed up for the classes lacked reliable access to computers. The one thing the college did well was monitor the results of the three pilot courses and call a timeout when failure rates proved unacceptably high. [snip]

Even pilot programs must be carried out with more care. Online courses should be developed thoughtfully, from within the colleges, not as a result of top-down directives from the governor. The subjects that are offered should be based on student demand and faculty analysis of which would work best online. The preferences of even the best-intentioned billionaires should not be part of the equation. Nor should online courses be viewed as major money-savers, as Brown has pitched them. It still takes well-educated people, interacting with those who need an education, to provide high-quality courses, whether that’s via the Internet or in a classroom.

I would add that it is quite impressive that SJSU (as described in the official blog post) is “currently awaiting a more comprehensive National Science Foundation data analysis and report that will be available in August” and that they “welcome vigorous public discussion of our pilot efforts and assessments of their effectiveness”.

How often do we get the chance to review the results of traditional college courses and see institutions publicly study the learning outcomes in order to improve the course effectiveness? This open review is one innovation from the SJSU Plus program that should be extended to other courses.

Conclusion

And this gets back to my original point in the NY Times article. I don’t believe in pushing MOOCs on recalcitrant faculty, but I do believe that student needs and learning outcomes are the primary measures of whether teaching models work or not.

The post SJSU and Udacity: Poor Planning and Support, but Valuable Reviewing of Results appeared first on e-Literate.