August 2005
Keri Facer and Mary Ulicsak, Futurelab
Paul Howard-Jones, University of Bristol

The full version of this report is available to download in pdf format - see box below. On this page you'll find the report's executive summary.

Executive summary

Debating the Evidence is designed to raise awareness of risk and uncertainty in scientific reasoning and support students' collaboration in engaging with these issues.

The learning environment comprises three stages: 1) in pairs, students aged 11 to 14 first complete training exercises which involve analysing data with covariance between cause and effect. To assist the students the software provides feedback about the strategy they employed based on their inputs; 2) then, once mastered, students analyse findings from a data set in which there is limited covariance between factors, that is, there is an imperfect relationship between cause and effect; 3) finally, the students have to draw conclusions and present their findings and recommendations. Awareness of risk and uncertainty is raised in the second activity, which is closer to 'real' scenarios such as those dealing with evidence related to genetically modified food or vaccination. In this second activity, the students have to reason why discrepancies occur and how much evidence is sufficient, given the cost and consequences of delaying recommendations. To support students' collaboration, the software requires that both students participate, as individuals and by providing an agreed hypotheses for causes and outcome. The software provides feedback on the apparent amount of cooperation between students' when providing 'agreed' responses.

The software is accessed via a webpage, however, the inputs and feedback are logged on a server and can be accessed by the students and teachers at a later time. Ideally, the personal computer is set up with two mice so that students have an independent means of entering a decision.

After consultation with teachers and a usability study with gifted and talented students, Debating the Evidence was trialled with a Year 8 class. The study focused on the impact of the software rather than the creation of a learning environment in which the software would be embedded with additional teaching and material resources. This report summarises the process and findings.

Key innovations of Debating the Evidence

Debating the Evidence was designed to be innovative in three key areas, as summarised below:

Engagement with uncertainty

It was intended that this would be achieved via collaborative prediction-making based on evidence that was partially inconsistent. It was intended that such an experience would improve students' awareness of the importance and limitations of scientific evidence.

Dual responsibility

The software was designed for pairs, but theories and predictions were entered first individually and then as a response agreed by the pair. Ideally, each student would have their own mouse for inputting their individual response. This was intended to support collaboration through committing individuals to form personal as well as negotiated theories. It also facilitated analysis of inter- and intra-individual strategies by the system and the production of formative feedback.

Automated formative feedback

The software provided students with prediction feedback and also formative feedback regarding the problem-solving strategies demonstrated and the extent of their peer co-operation, that is, how they appeared to choose the joint response. Formative feedback was carried out by the system via a dynamic analysis of the students' responses. This was intended to improve students' consideration of scientific evidence and the extent to which they collaborated on the problem-solving task. Analysis of the students' behaviour used the relationship between their theories and their predictions, and the sequence in which these occurred, to characterise their thinking strategies and provide appropriate advice to the students about how these could be improved.

Key learning findings and recommendations

The process of developing and trialling the software led to a better understanding of how students comprehend uncertainty and the impact of the feedback. It also highlighted how such software should be designed and areas for future investigation. The key findings are listed below.

For the educational research community

• Interaction with a computer simulation of a scenario where there is an imperfect cause-effect relationship appears to improve students' ability to sceptically examine how evidence is used.
• Rapid feedback from their prediction outcomes engages and supports students in revising their theories.
• Requiring individuals to enter their own responses, as well as their agreed ones, allowed the system to monitor inter- and intra-individual performance as well as allowing individual explicit expression. The pupils appeared motivated to do this, and may even have been motivated by this in their collaboration, but were not motivated to do this as soon as the decisions became routine.
• In the absence of teacher-intervention, instances were recorded where students improved their thinking strategies following relevant automatic formative feedback from a dynamic computer-based analysis of their behaviour. However, further research is needed to investigate the role that the automated feedback played in this improvement.
• Although completing the training facilitated the dynamic analysis of the students' strategies and thus the provision of formative feedback, the students' motivation dropped during training as soon as they were confident that they had identified the two causes. This contrasted with the mission, where the element of unpredictability, even when causes had been correctly identified, maintained the students' interest in sharing opportunities to make further predictions and increase the evidence base.

For teachers, advisors and head teachers

• Students appear to find interactive encounters with simulated scientific problems involving uncertain cause-effect relationships interesting and challenging, and this project has produced some evidence that this interaction improves their critical consideration of how evidence is used.
• The heterogeneity of outcomes within the class and the diverse experiences that different pupils had with the same software also suggest that the experience could be a useful precursor to classroom "science in society" debates about the importance and limitations of scientific evidence.
• Debating the Evidence supports the 21st Century Science curriculum being introduced in September 2006, as it focuses on the understanding that can be applied rather than the acquisition of facts.
• The training was of most benefit to students who had most difficulty hypothesis testing and prediction making. While the mission was of most benefit to those students who already had some understanding, albeit implicit, that inconsistent evidence weakens confidence in findings.

Policy makers and industry

• Students often demonstrated an unreasoned mistrust of authority, echoing some of the popular rejections of government and medical advice on recent issues such as the measles, mumps and rubella (MMR) vaccination. This may reflect a need for greater attention to be given in the curriculum to the critical analysis and appreciation of scientific evidence in areas of social concern.
• Students are not generally prepared in schools to make reasoned assessments of confidence based on incomplete evidence and interactive software may play a key role in helping them make reasoned judgements about situations involving uncertainty.
• Even without teacher intervention children can make gains by working collaboratively using a system that provides feedback on their performance and teamwork.

top of page