Designing Good Assessment Questions Plymouth University 10th October
Designing Good Assessment Questions Plymouth University 10th October 2013 Phil Race Follow Phil on Twitter: @RacePhil (from Newcastle-upon-Tyne) BSc PhD PGCE FCIPD PFHEA NTF Visiting Professor: University of Plymouth and University Campus Suffolk (Adjunct Professor; James Cook University, and Central Queensland University) Emeritus Professor, Leeds Metropolitan University www.phil-race.co.uk The curriculum enrichment project 2013-2015:
Ambition To provide excellent learning and stimulating student experience (Plymouth University Strategy 2013-20) that is genuinely at the national cutting edge of student learning and delivers the University Teaching, Learning and Student Experience Strategy initiatives. To support all our students consistently with curricula and co-curricular opportunities across a 30 week learning year, through alignment of the academic year in semesters.
To prioritise inclusive assessments, and minimise our MAP (modified assessment provision) needs, so that all our students are treated as equally as possible in all aspects of their programme. The curriculum enrichment project 2013-2015: Overview 1 The Plymouth University Curriculum Enrichment Project (PUCEP) will review and revise the level 4, year 1 curriculum to offer a full 30 week learning experience including opportunities for a broad menu of co-curricular activities and the introduction of interdisciplinary Plymouth Plus modules. A priority is to enable all level 4, first year students to complete and pass two modules
before the Christmas break. Benefits will include greater student engagement with their learning, active learning and research experience opportunities, and improved student retention. The curriculum enrichment project 2013-2015: Overview 2 At levels 5 and 6 we will introduce a two semester structure, where programme can choose to teach in sequence or parallel to enable placements, performance, fieldwork elements to be undertaken in term time. The opportunity for a four or five week term-time research placement with a member of staff or off-campus is enabled through the revised timetable
patterns. The curriculum enrichment project 2013-2015: Overview 3 This proposal recognises that knowledge in the 21st-century is easily available, what matters is developing the personal and IT skills to work with it. Modules will probably address less stuff, covering fewer topics but in greater depth. Students will develop their ability to be critically engaged with the material and to tackle larger and more difficult problems through group research. Working full-time on one module at one time will allow double loop learning to be more explicit, engagement with alumni and community groups,
and greater opportunities for students to reinforce practical experience. The curriculum enrichment project 2013-2015: Overview 4 This proposal recognises that modern academic research in most disciplines is a team activity benefiting from shared discussion, argument and debate. It also recognises that businesses need people who can be flexible, innovative and confident in team settings (CBI 2011).
These changes will affect all undergraduate learning from September 2014-5. Parallel changes to Taught Masters programmes should follow in 2015-16. The curriculum enrichment project 2013-2015: Some Issues driving this proposal: 1 Our students do not have a 30 week value for money learning experience. Student and parent dissatisfaction. Level 4 student engagement, and class
attendance is not good in many programmes there's no real reason to attend unless it really engages you, or its a compulsory assessment session. First year isn't consistently developing good practice study habits in preparation for second and third year. First-year academic experience is perceived to be of limited value. The curriculum enrichment project 2013-2015: Some Issues driving this proposal: 2 Many students are not receiving feedback until
after Christmas of year 1, and have few marks by which they can assess their progress. In some cases first feedback is after Easter. This affects students confidence in their academic ability and retention. Plymouth has a very high proportion of disabled students; we should aim to provide everyone with the same learning, support and inclusive assessment opportunities. The curriculum enrichment project 2013-2015: Some Issues driving this proposal: 3
Arguably there is over assessment in some modules, and over-reliance on terminal examinations in some Faculties. Returning to Plymouth for August resit examinations is very expensive for our students. Co-curricular activities need timetabled weeks to encourage student engagement. (Co-curricular activities are not for credit, recognised through the Plymouth Award and transcript). The curriculum enrichment project 2013-2015: Some Issues driving this proposal: 4
We dont have opportunities for students to learn outside their programmes. Introducing interdisciplinary Plymouth Plus assessed modules in level 4 will also enable some staff to teach with colleagues from other Faculties. The time table structure could allow Plymouth Plus modules to be available for level 5 where desired. Current six modules in parallel teaching pattern makes in-term work and researchplacements, theatre, performance, and fieldwork activities impossible to timetable. Many staff have horrendous marking loads in MayJune, which currently means teaching stops for most students at Easter. Intended learning outcomes By the end of this workshop,
you should be better-able to 1. Link assessment and feedback to processes underpinning student learning. 2. Design assessment questions and tasks with due regard for achieving validity, reliability, transparency, authenticity, manageability and inclusiveness; 3. Align assessment constructively with the evidence of achievement of intended learning outcomes, and with teaching and learning processes. www.phil-race.co.uk This afternoons workshop How many of you have been to sessions Ive run here previously? How many of you will also be coming to
this afternoons workshop in Babbage 002 on Involving students in their own assessment? www.phil-race.co.uk About About Phil Phil Born a Geordie First a musician Then a writer Then a scientist Then a researcher Then a lecturer and warden
Got interested in how students learn And the effects assessment and feedback have on them And how we teach them Gradually became an educational developer And an And now retired! (1995, 2009) Working with students on learning techniques And lecturers on teaching and assessment strategies
And trainers on training design Currently Visiting Prof: Plymouth Emeritus Prof: Leeds Met Travelling around as usual! Based at Newcastle, UK And on trains expert And still a Geordie. on train routes and timetables! Thanks to students I could not do what I do nowadays if I did not
continue to spend a fair bit of my time working with students theyve taught me most of what I know. www.phil-race.co.uk Whats an Emeritus Professor? Someone really old? Someone who has not been sacked for gross moral turpitude? Someone whos still on the books, but not on the payroll? Or perhaps as in edentate, the e signifies without. (C.f. e-learning?).
www.phil-race.co.uk You will be able to download my main slides You dont need to take notes. Ill put the main slides up on my website before the day is out, so sometimes I may go through them very fast! www.phil-race.co.uk 12 key sources about assessment, feedback and learning in higher education 1. Knight, P. and Yorke, M. (2003) Assessment, learning and employability Maidenhead, UK SRHE/Open University Press. 2. Bowl, M. (2003) Non-traditional entrants to higher education
they talk about people like me Stoke on Trent, UK, Trentham Books. 3. Flint, N. R. and Johnson, B. (2011) Towards fairer university assessment recognising the concerns of students London: Routledge. 4. Gibbs, G. (2010) Using assessment to support student learning Leeds: Leeds Met Press. 5. Boud, D. and Associates (2010) Assessment 2020: seven propositions for assessment reform in higher education Sydney: Australian Learning and Teaching Council. 6. Joughin, G. (2010) A short guide to oral assessment Leeds: Leeds Met Press. www.phil-race.co.uk 12 key sources about assessment, feedback and learning in higher education
7. Petty, G. (2009) Evidence-Based Teaching: A Practical Approach (Second Edition) Cheltenham UK: Nelson Thornes 8. Race, P. (2010) Making learning happen: 2nd edition London: Sage Publications. 9. Blue Skies (2011) new thinking about the future of higher education: download free from http://pearsonblueskies.com/ 10. Hunt, D. and Chalmers, L. (eds) (2012) University Teaching in Focus: a learning-centred approach see Chapter 5: Using effective assessment to promote learning, Sally Brown and Phil Race Australia: ACER Press, and London: Routledge. 11. Price, M., Rust, C., ODonovan, B. and Handley, K. (2012) Assessment Literacy, Oxford: the Oxford Centre for Staff and Learning Development (based on the ASKe Project). 12. Sambell, K., McDowell, L. and Montgomery, C. (2013) Assessment for Learning in Higher Education, London: Routledge.
www.phil-race.co.uk Much of this module is based on Making Learning Happen 2nd edition Phil Race 2010, Sage, London Several more aspects are
covered in The Lecturers Toolkit 3rd Edition Phil Race Routledge, London, 2006. Name labels Please write your first name, big and bold, on a coded sticky label, and stick it to your clothing (not onto a fabric which would be damaged). A 3
Phil www.phil-race.co.uk Getting to know each other In turn, please.. 1. Say one or two sentences about... who you are, where you come from, and what you do. 2. (Optional) Share with us one thing that not many people know about you! www.phil-race.co.uk
Post-it exercise On a post-it, in your best handwriting, please write your own completion of the starter: Designing assessment questions would be much better for me if only I Please swap post-its so that youve no idea who has yours. If chosen, please read out with passion and drama the post-it you now have. Please place them all on the chart as directed. www.phil-race.co.uk Massive, Open Online Courses
www.phil-race.co.uk Learner destinations: Planet MOOC? About you and MOOCs Hands up if you dont know what a MOOC is. Hands up if youre already studying on a MOOC. Hands up if youre already contributing stuff to a MOOC. Hands up if youre already assessing learning which has taken place with a MOOC? Thoughts about MOOCs are already going viral on the Twittersphere. www.phil-race.co.uk
MOOCs and higher education in 2013 We are now in an era of mass education with significantly increasing sizes of Higher Education Institutions but some already with fewer students!! Distance education is seen as a panacea in some areas with the increasing promotion of Massive Open On-line Courses (MOOCs) currently gaining much publicity; Higher Education Institutions including Harvard, MIT and the UK OU offer free on-line content, with some offering peer review opportunities for assignments; Currently most MOOCs dont offer assessment or credit opportunities and retention can be a significant problem: but this will change. www.phil-race.co.uk
MOOCs There is not yet a business model which works. 2m of OU capital has been invested in FutureLearn, seeking 8m return. Worries about types of assessment: how good are digital MCQs? Peer-reviewed essays cant really count towards accreditation, but are excellent for feedback to learners. But Pearson are involved, who suggest possible that several middle-ranking universities in UK will close. www.phil-race.co.uk Diana Laurillard Speaking on Radio 4 on July 25th, on news
that a particular MOOC pilot for disadvantaged learners in the US had a retention rate of around 20%... Higher education systems operate on a ratio of around one academic to 15 students overall this is elite. Technology can help, but needs to be used well to address the problem. www.phil-race.co.uk This is what MOOC users commit to: Validation the only validation on this MOOC was online acceptance
of an honor code. If MOOCs are to provide credit with any currency, then this will have to change. (Martin Hall) www.phil-race.co.uk Helen Barefoot (Uni of Herts) on MOOC completion rates Reported completion may be very low (1-10%) Does that matter? With very large starting numbers, there are still many learners completing. Maybe learners achieve personal goals even if they dont complete. But: Can MOOCs encourage access to HE if >90% have an experience which is a failure?
www.phil-race.co.uk Prof Martin Hall on MOOCs They could finish off the lecture as a form of conveying information, and the teacher becomes an expert guide in a world of abundance rather than a lecturer passing down wisdom from the podium. www.phil-race.co.uk
Now is the decade of Post-compulsory Educations dis-content! Never mind the content feel the learning www.phil-race.co.uk I predict a move of post-compulsory education providers away from being the guardians of content, (where everything was about delivery), towards two major functions: 1. Recognising and accrediting
achievement, wherever learning has taken place (i.e. getting the assessment right); 2. Supporting student learning and engagement (not least by making feedback work well for students). www.phil-race.co.uk David Willetts, 2011 we are relying on student choice to drive up quality. Students will control a much larger proportion of the investment in higher education. They will decide where the funding should go; and institutions will compete to get it. As
students will be paying more than in the current system, they will demand more in return. Blue Skies: New thinking about the future of higher education A collection of short articles by leading commentators: Pearson www.phil-race.co.uk What do you get for 9k or 27k? 9000 buys you: Ford Ka, Fiat Panda, Citroen C1, Peugeot 107 27,000 buys you: BMW 3 series, Audi TT, Mercedes C Class, Land Rover Freelander, Ford Galaxy
What kinds of service standards, warranties and guarantees are you going to want? So now, its time to re-think: 1. How students really learn and how we can help make learning happen for them; 2. How to make feedback really work for students; 3. How best to measure and accredit students achievement (not just more of the same old tired methods). In this module, well touch on all three of these areas. www.phil-race.co.uk
One of my main worries... We still tend to try to measure... ...whats in students heads, and what they can do with whats there in terms of two unsatisfactory proxies ... 1. what comes out of students pens in exams; (Henceforth called Govian Assessment?) 2. what comes out of their keyboards in essays and reports. www.phil-race.co.uk [The] pupils got it all by heart; and, when Examination-time came,
they wrote Activeit down; and the Experimentati Examiner said Beautiful! What on depth! They became teachers in their turn, Abstract Concrete and they said all these things over Conceptualisati
Experience on again; and their pupils wrote it down, and the examiner accepted it; and nobody had the ghost of an ideaReflective what it meant Observation Lewis Carroll, 1893 Teaching Other peoples knowledge is just information. Teaching is helping people to turn information into knowledge by getting them to do things with the
information and giving them feedback about their attempts. www.phil-race.co.uk Albert Einstein It is simply madness to keep doing the same thing, and expect different results www.phil-race.co.uk Understanding Experience
Competence Evidence Slow learning Achievement Knowledge The language of learning? Repetition
Information Skills Feelings Judgements Dialogue People Desire Whats missing in
learning cycles? Feedback Purpose Communication Surface learners? Deep learners? Strategic learners?
Rote learners? What kinds of learners have we got? Cueoblivious learners? Cueseeking learners? Cueconscious learners?
Validity Whodunit? Fairness Six aspects of quality of assessment Real-world links Transparency
Manageability Using what comes out of their pens? Using what they say? Using things they make? Using how they work with each other? Using how
they assess each others learning? How can we assess what students have learned? Using how they do practical things? Using how they assess their own
learning? Using how they solve problems? Using how they answer questions? Using what they put online? Drives most learning Is assessment broken? Assessment
as learning Some assessment issues Whodunit? What students think Validity Fairness Group task
Design and re-design..(e.g. 20th version). the wording of an assessment question, to make it valid, reliable, address whodunit, real-worldly, and manageable. It needs to link very well indeed to one or more intended learning outcomes (which you will explain). Two or three of you are interrogators, helping the writer of the question to improve and adapt the question. Please write the final version of the question itself on a flipchart, for later plenary discussion. Deadline: 1120 please. www.phil-race.co.uk
A question [30 marks] Please develop an annotated bibliography using exactly 300 words, on wobulatory kinetics: Top five sources Exactly referenced (Harvard)  In order of priority (best first)  Explaining why the order is as given. www.phil-race.co.uk Back to our intended learning To what extent do you now feel better able to... outcomes
(two hands = much better, one hand = somewhat better, no hands = no better) 1. Link assessment and feedback to processes underpinning student learning? 2. Design assessment processes and instruments with due regard for achieving validity, reliability, transparency, authenticity, manageability and inclusiveness? 3. Align assessment constructively with the evidence of
achievement of intended learning outcomes, and with teaching and learning processes? www.phil-race.co.uk Action planning statements One thing Im going to do is One idea Im taking away is Im going to think more about I have found out that Id like to know In future, Im not going to 2/26/20 www.phil-race.co.uk
Thank you www.phil-race.co.uk Twitter: @RacePhil e-mail: [email protected] www.phil-race.co.uk
Zportal / Xportal (FDI) WorldCat Link Resolver. All Legacy ILS. VDX. WorldCat. WorldShare. Speculative. Changes afoot at Ex Libris. Eventual product consolidation. Alma for resource management. Eventual transition of Voyager and Aleph . Immediate transition of Verde.
Le trafic du bois de rose à Madagascar. 1. Introduction. Les bois de rose, dont le bois a une valeur commerciale, sont très menacés à Madagascar, à cause d'une surexploitation, à leur coupe illégale et à leur trafic.
When Sin Entered the World Satan's Lie Satan denied the reality of the curse of death and God's veracity - 3:4 He added only one word to God's statement - 2:17 Man was seduced into believing he would not be...
To a minor or within 1000 ft of a school, or other specified areas carries a double penalty. ... (BSA), requires financial institutions to file Suspicious Activity Reports (SARs) on businesses they suspect to be engaged in potentially illegal activity....
Naming Aromatic Compounds Aromatic compounds are named with benzene as the parent chain. One side group is named in front of the name benzene. methylbenzene chlorobenzene (toluene) Naming Aromatic Compounds When two groups are attached to benzene, the ring is...
Notes. It is easy to see the quality impact of initiatives that promote health and prevent direct harm to patients; more difficult to get a clean cost measure.. It is easier to have clear cost measure for interventions aimed at...
Yet in the Justice trial the US Military Tribunal, with reference to the UN General Assembly's Nuremberg Principles, noted that: "The General assembly is not an international legislature, but it is the most authoritative organ in existence for the interpretation...