Jupyter Notebooks

Course Multiple: Netwerkanalyse (BSc Informatiekunde Year 1, period 2), Data Science (BSc Informatiekunde Year 3, period 4), Data Mining (BSc Informatiekunde Year 2, period 4), Collectieve Intelligentie (BSc Informatiekunde Year 1, period 5). Afterwards, also courses in Bachelor AI, minor programming and probably even in other schools or even faculties.
Lecturer Maarten Marx
Context Bachelor Information Science
Challenge Provide students a secure and stable environment to make their digital (coding) exams.
Intervention Since February 2018 it is possible to run Jupyter Notebooks on UvA/HvA computers in the digital exam rooms.
Evaluation Students love it (and seem to get higher grades).
Related Topics Large groups, Assessment and feedback, Digital formative assignments and feedback

Interview with Maarten Marx

Can you tell something about your course before the innovation? What was the issue you were facing in your course?
I give courses in which algorithmic thinking and problem solving play a crucial role. I test this in group homework assignments, but wanted to test it also using real life puzzles and data during exams. With Jupyter notebooks students can solve complex problems in a natural and familiar environment. Manually grading the solutions scales well.

What intervention was chosen? Why?
Jupyter Notebook (winner of the 2018 ACM Software Award) is installed on all computers in the digital exam rooms at the UvA/HvA since February 2018. Jupyter Notebooks provide an interactive problem solving environment, with support for 100s of programming languages such as Python, R, and much more (although at this moment only the Python kernel is installed). The notebook has lots of low level support helping the student to focus on deeper skills than rote-learning (e.g., autocompletion on variables and methods of objects, complete manual inside the notebook). Students can experiment with an answer and receive immediate feedback when they try it out on data. This is the natural way of solving problems.

Furthermore, notebooks provide a better measurement of learning objectives, teachers can pose more realistic (e.g. data driven), and exciting problems, and it uses auto grading (although manual grading can be done too, and is remarkably fast). As you can see, this tool provides many advantages!

Did it solve the issue? How?
Definitely! Students can now make their exams on the Jupyter Notebooks, allowing them to both save time and formulate better answers. That’s the good news for them. Good news for teachers is that this is a way of examining which scales well with student numbers without sacrificing quality of the questions asked.

What was the students’ experience? Did they like it and/or did they perform better?
Students seem to love working with Jupyter Notebook, not only because they seem to get higher grades. In my (limited) experience, all students remained until the last moment of the exam: because of the feedback of the notebook, they can keep on trying and improving answers.

Are you going to use it again? If yes, what would you change in the next iteration?
Yes. I will spend more time on the unit tests which allow intelligent autograding.

How much (extra) work did it cost you? Does it outweigh the benefits?
My homework assignments and model answers are also in Jupyter notebooks, so making an exam as a notebook is natural. Getting to know the setup for the autograding system has a learning curve and needs some time investment.

Do you recommend this approach to other lecturers? Why?
Yes, because you provide a natural and realistic problem solving environment to the student which allows you to ask challenging questions.

Is there anything else you’d like to add?
I predict that within five years all courses in which students are taught active knowledge will use Jupyter notebooks, and many of them for exams too. Then also most course material (including textbooks) will be available as notebooks, as we see now already at Kaggle with the kernels.

Further information
Software for easily setting up autograding is available at https://github.com/UvA-FNWI/notebook-exams.
For further information, see https://github.com/UvA-FNWI/notebook-exams/blob/master/Docs/NotebookExam_presentation.ipynb.