Announcing Exercism Research

In August, we announced our partnership with James Evans and Gary Lupyan to carry out research into how people's programming experience affects how they learn languages.

We spent some time earlier this month meeting with James and Gary and planning out how we achieve some of their wider research goals alongside the redesign of Exercism tracks I discussed in the previous post. It was clear that there are many areas where Exercism will be able to help their research but that those might not fit directly into core Exercism product. We have therefore decided to create a new area of Exercism specifically centred around research, called (unimaginatively) Exercism Research. This subsite will host experiments that aim to gather data on a specific research goal. The experiments will also have the added benefit of allowing us to test out new technology on a less critical area of the site.

The first experiment, which we aim to launch in the first week of January, will be centred around investigating how someone's previous programming experience impacts the reusability of the code they produce in different languages. The experiment will be structured as followed:

• Users will be asked to complete a short survey outlining their programming experience (e.g. the languages they know, the order they learnt them, and their fluency in each).
• Users will then be presented with a choice of X languages to participate in. It does not matter how well the user knows the language(s) they choose - a variety of previous knowledge will help the breadth of data gathered.
• The user will be given an exercise to complete, which should take 10-15 minutes.
• The user will then be given someone else's solution to a different exercise, and asked to extend it in a certain way. This should also take 10-15mins. After each step we will ask the user questions about how difficult they found the exercise/extension.

Gary and James' teams will then crunch the data, investigating how the code submitted differs based on different backgrounds, and on how extensions changed based both on the initial submitter's background, and the extender's background.

This experiment will be the first testbed for our new in-browser coding before that is integrated into the main site in 2020. We hope to have 5-10 languages that participate in the experiment. To take part a track will need to have a functional test-runner, and two exercises for the two parts, which we will help maintainers create. If you are a maintainer and interested in your language being part of this, please join the #experiment-1 channel on Slack and let us know.