2015 has arrived. While we are some way off from hoverboards and flying cars, the advent of data-driven technologies is cause for excitement.
Whether through our personalized social media feeds, talking smartphones or targeted email ads, the growing influence of machine intelligence is felt by us all.
But with this promise comes stark warnings on the limitations and even the dangers of over-reliance on computers…
“Summoning the demon”
The battle of ‘Man vs Machine’ has been a regular fixture on the big screen, and particularly in Hollywood blockbusters, over the past thirty years. More recently, however, a number of thought leaders have echoed these stark warnings on the existential threat of self-learning machines.
Although somewhat less apocalyptic, the stakes are still rather high in the case of adaptive tutoring. As long as students’ engagement in mathematics is at stake, so too is the next generation of potential scientists, engineers and astronauts. It is up to us as educators to push for the right balance between technology and human judgment.
Here at Whizz, both technology and human judgment are key to our educational mission of delivering the right lesson at the right time for every child.
Trialing sequencers at Rosemead School
As a partner of the EU-funded iTalk2Learn research project, we have privileged insight into the research and development of machine learning applications in areas such as adaptive sequencing, speech recognition and exploratory learning environments. Our experiences as part of this project also allow us to tell a cautionary tale in how to approach data-driven research…
As part of iTalk2Learn, we conducted a study with students at Rosemead Preparatory School last year. The study was based upon a comparison between the Whizz sequencer (the algorithm that determines the order in which students receive topics and lessons with the Math-Whizz Tutor) and the iTalk2Learn machine-learning sequencer. The goal? To find out which sequencer recommends lessons that are most in line with each student’s individual learning needs, and is therefore more effective as an online virtual tutor.
The results revealed no significant differences between the two sequencers in students’ learning growth during the course of the trial.
So what does that mean?
On the one hand, it gives us encouragement in the effectiveness of the Whizz sequencer, since it holds its own against even the most cutting edge of tutoring algorithms. But more importantly, the trial revealed a key insight into the best way to approach adaptive tutoring.
Balance data-driven tutoring with human judgment
One revealing finding from the trial was that many students on the new sequencer complained that the tutoring experience was too repetitive. The new sequencer is a prediction model based entirely on students’ past performance data. The model leverages more data than ever before and uses cutting edge techniques from machine learning to determine the next lesson within each topic. From that standpoint, it may be reasonable for the Tutor to deliver the same lesson multiple times in succession.
However, we have long recognized the importance of variety in order to keep students engaged. The Whizz sequencer has hard-coded policies to ensure that content is not repeated in quick succession. Very rarely, for example, do students visit the same topic twice in a row.
As demonstrated by the trial, such hard-coded policies are needed as a sanity check against the hard science of performance prediction. To rely upon data-driven models alone would be to ignore the humanity of the human learning process.
Putting these findings into action
The stated goal of the Math-Whizz Tutor is to replicate the behavior of a human tutor.
Data can be extremely powerful in revealing hidden insights into student learning. We continue to explore ways in which aspects of the new sequencer may be incorporated into the Math-Whizz Tutor. The trial reminds us, however, that a data-driven approach must be balanced with our own experience and judgment in order to stay connected with the human side of learning.
As a recent example of the importance of finding this balance, let’s think about a recent improvement to the Math-Whizz Tutoring algorithm. Math-Whizz students can now choose to skip parts of some lessons when the Tutor detects that they are ready to move on. Data cannot tell us about the importance of allowing students to pass some exercises early – that’s a decision that we made based upon human feedback alone. Data can, however, inform us of the optimal way to leverage this insight, bound by the knowledge that we have gained over the past decade and the crucial balance between structured and autonomous learning.
With that balance, Math-Whizz students can look forward to an evolving tutoring experience that is richer than ever before – led by both ground-breaking research and human expertise.