The "Quality Control" Interview for Big Classes

Attention conservation notice: Advice on teaching, which I no longer follow myself.

I teach a lot of big classes --- the undergraduate advanced data analysis class passed 100 students many years ago, and this fall is over 230 --- which has some predictable consequences. I don't get to talk much to many of the students. They're mostly evaluated by how they do on weekly problem sets (a few of which, in some classes, I call "take-home exams"), and I don't even grade most of their homework, my teaching assistants do. While I try to craft problem sets which make sure the students practice the skills and material I want them to learn, and lead them to understand the ideas I want them to grasp, just looking at their scores doesn't give me a lot of information about how well the homework is actually working for those purposes. Even looking at a sample of what they turn in doesn't get me very far. If I talk to students, though, I can get a much better sense of what they do and do not understand fairly quickly. But there really isn't time to talk to 100 students, or 200.

About ten years ago, now, I decided to apply some of the tools of my discipline to get out of this dilemma, by means of random sampling. Every week, I would randomly select a fixed number of students for interviews. These interviews took no more than 30 minutes each, usually more like 20, and were one-on-one meetings, distinct from regular open office hours. They always opened by me asking them to explain what they did in such-and-such a problem on last week's homework, and went on from there, either through the problem set, or on to other topics as those suggested themselves.

In every class I did this in, it gave me a much better sense of what was working in the problems I was assigning and what wasn't, which topics were actually getting through to students and which were going over their heads, or where they learned to repeat examples mechanically without grasping the principle. There were some things which made the interviews themselves work better:

• Reading each students' homework, before the meeting. (Obvious in retrospect!)
• Handing the student a copy of what they turned in the week before. (Though, as the years went on, many brought their laptops and preferred to bring up their copy of the document there.)
• Putting a firm promise in the syllabus that nothing students said in the interview would hurt their grade. (Too many students were very nervous about it otherwise.)
• Putting an equally firm promise in the syllabus that not coming in to the interview, or blowing it off / being uncooperative, would get them a zero on that homework. (Obvious in retrospect.)
• Offering snacks at the beginning of the interview.

Setting aside a fixed block of time for these interviews didn't actually help me, because students' schedules are too all-over-the-place for that to be useful. (This may differ at other schools.)

Choosing the number of students each week to interview has an obvious trade-off of instructor time vs. information. I used to adjust it so that each student could expect to be picked once per semester, but I always did sampling-with-replacement. In a 15-week semester with 100 students, that comes out to about 3.5 hours of interviews every week, which, back then, I thought well worthwhile.

I gave this up during the pandemic, because trying to do a good interview like this over Zoom is beyond my abilities. I haven't resumed it since we went back to in-person teaching, because I don't have the flexibility in my schedule in any more to make it work. But I think my teaching is worse for not doing this.

Corrupting the Young