When assessing students’ learning in computing, I think we’ve a couple of approaches. One would be to look at the projects students do, whether these are open ended, design and make tasks or more constrained solutions to problems we pose, perhaps assessing these against agreed criteria or using a rubric. The other is to ask questions and use their answers to judge what they’ve learnt: these questions can be quite open, or perhaps as straightforward as multiple choice. I think a good assessment strategy ought to draw on both approaches: we want students to be able to work creatively on extended projects, and we also want to check, from time to time, whether they can remember the things they’ve been taught.

Responses to questions certainly have a place in summative assessment at the end of a course, but I think they’ve much to offer for formative assessment before, during and after lessons or units of work:

• How can we tell that students have made progress? By their doing better on questions at the end of a topic than they did at the beginning.
• How can we tell if they’ve understood the idea we’ve explained? By getting responses from a carefully designed, hinge-point question straight after our introduction.
• How can we engage students in a meaningful discussion about CS ideas? By having them work together to answer good questions?

Lots of teachers are doing this sort of thing already – writing their own questions to ask their class, or just making these up on the spur of the moment. That’s fine, but coming up with good questions is surprisingly difficult, and it’s not particularly efficient having lots of teachers all doing this independently of one another, when a divide and conquer approach to question writing would work, if only teachers could share their questions with one another.

For the last couple of years, CSTA’s UK little sister, Computing At School (CAS) has been working with assessment experts at Durham University, Cambridge Assessment and EEDI to crowd-source an ‘item bank’ of quick fire questions that teachers can use with their classes. We’ve standardised on four response multiple choice questions (a format that US-based members of CSTA are likely to be quite familiar with already), and have adopted EEDI’s Diagnostic Questions (DQ) platform for hosting the questions, making it easy for teachers to compile questions into quizzes and assign these to their classes.

Access to the questions, and use of the DQ platform is free for anyone. The questions are released under a Creative Commons licence, so teachers are able to embed these in their own virtual learning platform or presentation software if they wish, but our hope is that students attempt these on the DQ site, so we can use the data from hundreds of thousands of students attempting thousands of questions to work out how hard each questions is, whether a question is good at discriminating between stronger and weaker students, and where common misconceptions are in school level computing.

As I write, we’ve some 8,049 questions online: mostly covering middle / high school CS, but there’s some coverage of elementary school CS and of information technology and digital literacy – I’d really encourage you to register on the DQ site and have a browse of what we’ve got: you can filter down through different aspects of CS, and sort questions by most likes, most answered, most misconceptions etc. It’s easy enough to add questions to a quiz of your own, and we’ve got 384 shared quizzes which are free to use too. Once you’ve registered, you can access the questions at bit.ly/quantumquestions.

We’re already getting some insights from students’ answers to the questions, highlighting the areas of the CS that students seem to struggle with, such as understanding variable assignment, code tracing and data types. We’re also running Rasch analysis on students’ responses, and plan to use this to identify lower quality questions, as well as making it easier for teachers to find questions suited to their students’ current level of achievement.

It’s a crowd sourced project, and so we’d be very glad to have more questions: I’d be glad to support anyone interested in getting their questions onto the site, or who’d be interested in learning more about writing good questions. If you’d like to learn more about the project, check out bit.ly/projectquantum, or watch the seminar Simon Peyton Jones and I gave at Cambridge Assessment last month.

Originally published on the CSTA Advocate blog