In this week's subscription episode, I unpack the complexities of assessing design and technology capability, emphasising the significance of real-time assessments. Using published literature, I question England's current non-examined assessment (NEAs - coursework) approach
Listen in and be inspired to rethink how we evaluate our students' grasp of design and technological knowledge.
(Text generated by AI, edited by Alison Hardy)
Links/ Mentioned in the episode
R Kimbell, & K Stables, Researching design learning: issues and findings from two decades of research and development. Springer, 2008.
R. Kimbell, Examining the reliability of Adaptive Comparative Judgement (ACJ) as an assessment tool in educational settings. International Journal of Technology and Design Education, Volume 32, pages 1515 to1529.
Ciaran Ellis posted a thought-provoking question on LinkedIn recently: Do design decisions involve value judgements?
What do you think? Join the conversation over on LinkedIn and let us know what you think.
If you like the podcast, you can always buy me a coffee to say 'thanks!'
Please offer your feedback about the show or ideas for future episodes and topics by connecting with me on Threads @hardy_alison or by emailing me.
If you listen to the podcast on Apple Podcasts, please take a moment to rate and/or review the show.
If you want to support me by becoming a Patron click here.
If you are not able to support me financially, please consider leaving a review on Apple Podcasts or sharing a link to my work on social media. Thank you!
Speaker 1:
On Tuesday I had Neil Wright, a teacher from William Farron, lincolnshire in England, on the podcast talking about a paper that had really got him thinking from Pat Forty. It was a paper about assessment and very specifically about an assessment method for teachers to use, which was adaptive comparative judgement. I've got to say that really slowly. Acj Now people in England might have heard this in use across other subjects, but actually it's been around for design and technology for quite a long time. Richard Kimball at, who was at Goldsmiths Professor at Goldsmiths, has led on this in England for design and technology, using D&T as a case study in particular. But I don't want to go into here the pros and cons of adaptive comparative judgement. It's not an area that I'm particularly skilled at I've never practised it or been involved in a research group about it but I do think it has a lot of merits and I know that Richard and his colleagues have done some really rigorous research about this and I'd encourage you to go back to Tuesday's episode and have a look at that in more detail. But instead I want to focus on a couple of things that I picked up from talking to Neil and just explore those in a little bit more detail. The first is about being clear about when we're doing any formative assessment formal assessment or whether that formative is for an external or an internal audience. We need to be really clear about what is being assessed and I think in Neil's conversation we did touch on this briefly, thinking about what is being assessed.
Speaker 1:
But I think there's two things that we assess in design and technology, and this does come from the literature. The first is about design and technology capability. Whether that is children's capability in design and technology is developing over time, so there's different points where we would assess that. Secondly, it's about the individual components of the curriculum, that's design knowledge or technological knowledge. So those are the two aspects that I think we need to assess in design and technology and we would use different methods for assessing that. So the first is around the design knowledge or the technological knowledge and you'll have to listen to me talking about knowledge in other episodes to be really clear about the fact. I'm not talking about factual knowledge, like regurgitation of knowledge, but I'm also talking about procedural knowledge, where children learn the processes of designing and the processes of technology. So that's one component is these design knowledge or technological knowledge? And the second component is their design and technology capability, and it's in this area of design and technology capability that Richard Kimball and Co have done a huge amount of research over the years around. How do you assess this? And the main way that this is assessed is through a real time design context how pupils respond to this design context.
Speaker 1:
And in England that manifests itself as a non-examined assessment, a piece of coursework. Now there are issues with the way that's been taken in England, but we have to be really clear that this is the component that we're assessing how good are children at autonomously responding to an unfamiliar-ish design context? And what I mean by that is a good teacher will pick those design contexts really carefully and sometimes the children will be more familiar or less familiar, I think, depending on where they are, what stage they're at in their development. And what I mean, therefore, by their in real time is I think we've gone a little bit too far in England with the NEA this coursework assessment is. It takes so long, it's over so much time that it becomes unrealistic. If what we're actually trying to do here is demonstrate how pupils can respond to a design context, the research says that there needs to be done more in real time, so that is more of a concentrated time.
Speaker 1:
And I remember back in the day when I taught A-level design and technology for edXcel that testing of their design and technology capability was done over six hours, two, three hour blocks in real time. And I know that in--- England OCR, the exempled OCR also set up these challenges that were done in real time and that was very much based on the research from Kimball and Co. So I am calling to the examples a little bit that we need to rethink the way design and technology papers are assessed formatively, that's, at the end, and in that case, of Key Stage 4, at the end of their formal education, compultrary education at the age of 16, where it's a piece of coursework at the moment, where it takes weeks and weeks and weeks. That is that really assessing their de-intercapability or is it almost more assessing what teachers are managing over those weeks? I also have an issue and I've raised this in a couple of meetings over the past year that actually, where we have such restricted time with children in the classroom in design and technology, is it really good use of our time to be using 25 hours of that taught time to be an assessment? So I think this real time thinking has to be real rethought and I also think that's something that needs to happen lower down in education I might say lower down with younger children, whether that's in Key Stage 1, 2 or 3, that's middle of primary, end of primary, middle of secondary for those who are listening or international thinking about this formative assessment, this stock take in real time, whether that's a two hour, whether that's a three hour, whether that's a whole day, and I think that needs to be something that is done in negotiation with school senior leaders to have that time to do a proper, thorough, holistic judgment of pupils design and technology capability. So that's not an exam as in pupils showing that they've learnt design knowledge or technological knowledge, but this is about assessing their design and technology capability and it has to be done, I believe, in real time. And that's what the literature says, and I'll put some links to the key literature in the show notes. So we need to be clear.
Speaker 1:
So, going back to my first point about this response to Neil's podcast it's not response to Neil, but some of the things we talked about is that we need to be clear about what's been assessed and then how we are assessing it, and I think that is absolutely crucial and I think it's something that's been lost a little bit is that this coursework is being extended, and extended where I don't think anymore that's being assessed. The other things and I've just touched on this, the other things that the literature says is about finding an optimal context for the assessment. So I think a good teacher and this is what the literature says again is that a good teacher will choose design context for the assessment that allow pupils to use what they've learnt in an appropriate way, that they're drawing on that, that they're not having to particularly learn new knowledge to respond to that. The only new knowledge they might be learning is an understanding of the context, but they're still drawing on design and technological knowledge that they've previously learnt. That's really difficult, I think, for a teacher to plan.
Speaker 1:
And let's use an example here that again comes from a literature. It comes from Richard Kimbell and Kay Stable's book Researching Design Learning, and this is one that they used have used with younger primary age pupils, given the context of what a surprise. Okay, that's the context, and it's used to assess pupils' design technological capability. They have some resources that they gave to the pupils to draw on in this task some handling collection of greeting cards, some pop-ups and a story about a user, and the idea was that the pupils were generating ideas and solutions to something that would be a surprise card for somebody that they knew within their family. So it was all kind of a relatively familiar context. So that's the surprise. So this card would have a surprise in it. Hence the B models of paper pop-ups is like something to handle and inspire.
Speaker 1:
So what the teachers were also assessing within this is they previously learned knowledge, what children had previously learned about some simple mechanisms, some basic properties of paper and card and how that paper card could be shaped, joined and cut. And therefore but the context was new this water surprise was the new bit, this kind of optimal context. So very clearly here what's been assessed is the way the children can draw on that previously learned knowledge about mechanisms, properties of paper and card, shaping and joining paper and card. But in this new context of a surprise card for somebody special to them and to help them with that, they had these different resources that they could draw on. So that's really the way that teachers I think can think about is okay, they've learned this previous knowledge and we might have done a design and make activity to structure learning of that knowledge, but we can't use that same context for the assessment. What's a new context? Whether children have to then choose what they draw on and what they've proven to learn to demonstrate in this formative assessment?
Speaker 1:
I'm hoping I'm making sense, but this is a complex thing. I think assessing design and technology capability and I really would recommend having a look at Richard Kimberlin case table's work on this that example I've given has come from them, it's not one that I've generated. So I'm just going to end there really with this thing about design and technology capability, assessing it. I'm going to talk in another podcast another time about how do we assess design knowledge and technological knowledge. But this assessment that we need to think about that can be used with adaptive, comparative judgment, as Neil spoke about is assessing their design and technology capability.
Speaker 1:
It's not assessing what they know. It's assessing how they use what they know to address the context, and teachers need to pick that context very carefully that something that is relatable but is a little bit out of their comfort zone, and then the time that they have the research says needs to be real time, which I think is a challenge to what is happening in England, particularly around post 14, that age 14 to 16, informal education of a qualification where they're doing formative assessment. So there's lots to think about. It is complex and that's where teachers professionalism comes in and, as I said, I'll put some links in the show notes, but I'm hoping my prompt teachers to think about this in more detail, based on what the research says, as usual. Thanks for listening. Any comments, questions or feedback, please do get back to me.