Learning Experience Designer Kristin Torrence joins me to discuss stealth assessment, particularly within immersive environments. Kristin has designed learning solutions ranging from educational games, mobile applications, e-learning, Instructor-Led Training (ILT), and virtual reality (VR).
In this episode we talk about what stealth assessment is, the history of its development by Valerie Shute, and why it is the optimal form of assessment for learning within immersive environments. We also talk about the importance of the state of flow for immersion, the future of adaptive learning, and some of the drawbacks of stealth assessment...mainly, it's a lot of work!
The following transcript was auto-generated and may contain typos or spelling errors.
Leslie Early
Okay, today I am super excited to be joined by the awesome Kristen Torrence. Kristen is a learning experience designer who has designed learning solutions, ranging from educational games to mobile applications to elearning, and instructor led training, and also virtual reality. So you've kind of done it all, haven't you, Kristen?
Kristin Torrence
Yeah, I'd say so.
Leslie Early
Well, thank you so much for joining me today.
Kristin Torrence
Yeah, it's my pleasure. Thanks for having me.
Leslie Early
Yes. And I'm also super excited for this episode, because we're talking about something that is somewhat mysterious sounding, at least to me. And that is stealth assessment for immersive environments. So I think people may be familiar with immersive environments, maybe not so much self assessment. But before we jump into that, why don't you take a couple of minutes and introduce yourself?
Kristin Torrence
Yeah, so I'm a learning experience designer. I've been in the instructional design field now for about seven years. And I started in the K 12. Space designing edtech for middle school and elementary school. And I recently switched over to corporate enterprise space just about a year ago, or less than a year ago.
Leslie Early
Okay. Yeah. So and so I think I saw you were doing game based kind of learning experiences in K 12. And that sort of got carried over into what you're doing now.
Kristin Torrence
Mm hmm. Yeah, so I was designing a science game for middle school and elementary school students, and the company I was at before, and was always, you know, super interested in game based learning. And from that experience, you know, had had the opportunity to really try out some virtual reality games, which just blew my mind and had that interest actually had brought me into really the role that I'm in now, which is a learning experience designer for a company called tailspin, and they create xR solutions for enterprise clients.
Leslie Early
Very cool. Okay, so that's kind of the space you're in now is this xR space? So let's dive into today's topic. Because it's, it's sort of around this concept of stealth assessment, which I had never heard of before, before you told me about it. So I'm assuming people who are listening, maybe don't know what this is. So do you want to go give a little definition of what is self assessment? And how did you become interested in it?
Kristin Torrence
Yeah, so um, just to kind of tell you a little bit more of how I kind of stumbled on to self assessment. So when I was designing for those educational games, I really dove into the literature and self assessment was just really something I stumbled and stumbled upon, while doing my research, and it's actually this evidence centered design framework applied to game based learning, which I thought was just fantastic. Because it's essentially this method of assessing that's woven into the game mechanics and the game design. And it was coined by Dr. Valerie shoot out of Florida State University. She's, I think, really the pioneering researcher in what she coins self assessment. And she calls it that because you can actually assess learning without the learner ever knowing they're being assessed, which is pretty cool. So it's like stealth mode evaluation.
Leslie Early
Yeah, so that makes a lot of sense why in a game based learning context, you would want the assessment to be sort of incognito, like you don't want people who are playing a game to be interrupted, and reminded that actually, this is sort of a formal learning experience. And they're being assessed, right?
Kristin Torrence
exactly. And, you know, that was actually the key thing that I think sold me on it with that you're able to assess learning without essentially disrupting flow or, you know, getting the players out of the flow state. And if you know, you're familiar with chicks at Mihai, and the whole, like flow theory, flow, it's really, in games, the really the most optimal state that we want players to be able to reach, which means you know, they're fully engaged, they're completely motivated and immersed in the gameplay. And so, you know, I just just thought, you know, maintaining flow is super essential. I You know, just to keep them playing, keep them motivated. And if you can evaluate learning in the process, then what better way to assess
Leslie Early
Right? And also, maybe not just evaluate, but you don't want to interrupt the learning process itself. Like, first, if it's a game based learning, they should be learning as they're going. Right. And so you don't want to interrupt that process either to stop and assess them. So that makes a lot of sense. Um, so I, again, I don't know that much about this. But I also thought it was very interesting. You said that there's sort of a controversial history to what this stealth assessment sort of comes out of, do you want to talk about that a little bit? Oh,
Kristin Torrence
I wouldn't say controversial, it's just a little surprising, which I thought was really cool. And actually pretty enlightening, in a sense. So self assessment research, II think started in the early 2000s. And really, the foundation of it is based off of the evidence centered design for assessment framework. And it actually turns out that ECD had, like was originated, and I think even like, created by the Educational Testing Service. So you know, that the company that uses or creates the JRE. And they had used it really to, to drive their Adaptive Testing within the like, online assessment space.
Leslie Early
So like, if you're applying for college, no one does like the fill in the blank, I mean, the fill in the bubbles anymore, it's mostly an online assessment for the GRE or whatnot. Okay, that makes a lot of sense. The reason I thought it was controversial is because it's coming out of the standardized test space, but I guess the GRP is a little different. I was thinking more like AC T and sa t, which, you know, people have opinions about. Yeah, so I guess, yeah, that makes a lot of sense why it would come out of that, because it is another form of incest, a set adaptive assessment, my snap, does that make sense? I'm not sure.
Kristin Torrence
And, you know, it's like, it's kind of cool to know that background just because it shows that, you know, as technology advances, and modalities and applications advanced, like the underlying mechanisms of measurement, and research still kind of hold true and remain valid, which I think is really cool.
Leslie Early
So I guess that leads me to the second part of our topic here, which is self assessment for immersive environments, so I can sort of see how this plays out in just a regular game. But how does this play out? Even more? So in an immersive environment? Does it look the same? Is it different? or different things to consider?
Kristin Torrence
Oh, that's a good question. Um, I would say, probably the underlying architecture remains the same, but maybe the triggers are a little different. Right? So, you know, similarly to game based learning, learning in virtual reality or other sort of immersive environments. Flow is still a very important factor, I would say. And then even more, so a factor for immersive environments is immersion. Right? So immersion is like really closely related to flow, and that it's really this feeling of being totally absorbed by an activity. And I know that there's been research in research in VR that suggests that interactivity, and like interactivity within the virtual environment, influences like a person's level of perceived presence. And both interactivity and presence both have a really strong influence on your sense of immersion. And so I would, I kind of feel that if self assessment could similarly be woven into the fabric of virtual reality tasks, similarly to games then learning can be you know, measured also without breaking flow, or immersion, which I think is really powerful specifically in the immersive space.
Leslie Early
Especially because, as you're saying, like that the point of virtual reality and xR, and there's a lot of research coming out that this immersion state that you get in does a lot of interesting things for learning and for for our brains, right? Like especially for things like empathy, or like different things. We're having that really like, close the second closest thing we can get to actually being there in person and going through it yourself as if it's something you're really experiencing yourself. Yeah, that is is so important to the selling point of VR, like, like, people who are using it like doctors or med students who are like, you know, you can't put them in a high stakes situation, but you can put them in this and then you know, they can get some of their learning that way. So you don't want to interrupt that feeling of like, this is really happening. And these are really my high stakes. And I really need to make life or death decisions here by like stopping and being like, let me at, give you a pop quiz, like, right in the middle of cutting up on this.
Anyways, sorry, that was me thinking out loud about why this would be important.
Kristin Torrence
Now, and just to sort of build on that another thing that's cool about self assessment, and I know, we talked about it in the sort of online adaptive testing, but in games, and then also in virtual reality, it has that potential to fuel that adaptive learning, based off of really the evidence that you that the system collects based off of learners actions that they've demonstrated in the virtual reality space, which is, I think, is really powerful.
Leslie Early
Yeah, now that you mention it, like, people have been talking about adaptive learning, for a long time, it feels like for a while, but actually, you can't really have successful adaptive learning if you're not constantly assessing the person who's going through it. Right? Unless there's like self selecting, I guess that's another version of adaptive, like, which module Do you want to do now? But um, yeah, if it's like real, the way we're thinking of how adaptive learning is going to evolve with like, AI and things in the future, you would have to be constantly assessing the learner.
Kristin Torrence
Absolutely. And and, you know, assessing accurately and having a very accurate depiction of a player or a learner's level of skill at any given point in time.
Leslie Early
Yeah. Hmm. Very interesting. Well, so I think we've sort of, we sort of already been talking a little bit about this, but what do you think are some of the benefits? Or I think we've been talking about benefits, mostly, but I think are some of the benefits or drawbacks of using this type of assessment?
Kristin Torrence
Sure. So we can start with some drugs. I think it's really time consuming, I would say and costly. And then one of the other drawbacks that I had read and Dr. Shoots book was that, that it's actually really difficult to do retro actively. And so it's really optimal, if you could design self assessment concurrently with the game design, which I know isn't like always super possible, you know. So I think that's a pretty big drawback. And a lot of time goes into creating self assessment in and of itself, because it is this like evidence centered design framework, that whole framework consists of multiple models within that. And I'll just name like, the really the three main ones, which are the competency model, the evidence model, and the task model. And each of these takes a lot of time to create, and then also a lot of time to validate. Yeah, and so just to kind of do a brief overview of what those models are the competency model, it really represents, you know, what you want to evaluate within? within the game within the virtual reality space, it's, it's sort of like, what do you want to claim about the learner what what skills or competencies Do you want to be able to assess and claim that they have, and that model in itself, it kind of moves from the more abstract concepts like problem solving, right to like, really more tangible. And so that model is pretty intricate. And then the evidence model, it really defines what qualifies as evidence in your training. So how those competencies can then be measured with these observable things that you could see a learner do and then be able to assessit.
Because the evidence that you collect, you know, also feeds into the competency model through the statistical probability relationships, to be able to then say, if we saw learners do these things, then we could print we predict that okay.
Leslie Early
That's so interesting, because it's just, it's so interesting to me. I'm going to go on a little tangent right now, but yeah, what you're describing, and the amount of effort like the teams of people probably that have to be involved in this and the amount of planning of like, through this experience, these are the places where we're going to be assessing, and these are the things we're assessing for. And this is the behavior we're looking for. And all of that sounds like a lot for like, designing a training or learning experience. But that's the exact thing that's happening in social media. That's the exact that's happening in consumer games. Like, I know, we've talked about this briefly before, but like, that call of duty is now assessing people and like, you know, when they realize somehow the algorithm realizes that you're getting tired, and you're not playing as well, they're gonna bump you down and put you in a different tier of players so that you can be successful again, so that you keep playing longer. And it's like, if they can do that and Call of Duty, then it means it is possible. I think that's essentially what you're describing. Yeah, there are some target behaviors that they're assessing. And it's all it's all secretive. Yeah, so I just see, it is possible. It's just the question always is how much time? How much money? How many resources? Are people willing to throw at that type of problem? I guess?
Kristin Torrence
Exactly. And, and, you know, I definitely do think it takes a good cross functional team, right? There's learning designers, there's developers, you know, this whole element of probability, statistical probability will have data sciences and machine learning algorithms, you know, and then even with the whole adaptive learning piece, you'll need to build a system that has that sort of feedback loop, right. And so that is where you can adaptively like present tasks at a specific, you know, within the CPD of for the learner based off of a, their, their level of competency at that time. And so it, I definitely would say it's a cross functional team.
Leslie Early
For sure. Well, anything in VR, and xR feels like it's in a cross functional way. But yeah. Okay, so I mean, I guess we're talking like, very like, this is what a big team can achieve maybe, if they have the right time, money, resources, but if someone was thinking of experimenting, or how might someone apply, maybe let's just focus on self assessment, because maybe I didn't have enough environments. We know, that's like a big undertaking. But if you are trying to somehow implement self assessment in your own, you know, training or learning experience, what what are some tips that you would give people?
Kristin Torrence
Yeah, you know, I would actually say, if you don't have the time or resources to integrate self assessment fully into your learning solutions, I would just say, like, stick to the fundamentals, you know, it's grounded in measurement research. And, you know, one of the things I think, from learning more about self assessment and evidence centered design, I think, you know, I'm just a little bit more mindful now about, like, what and how learning is evaluated. And really the importance of how you make claims based off of empirical evidence. And I think, like that, as sort of a design principle, or an assessment principle is like, really, I would say, it's a good, it's pretty good practice. And then, I guess even without the adaptive learning piece, I'd probably recommend going through an exercise of evidence centered design alone in just creating a competency model, creating an evidence model, creating a task model. It doesn't need to be fully fledged, but I think just going through that exercise can really bring focus on you know, what needs to be learned how it should be evaluated, and what tasks you can give a learner that would allow them to demonstrate evidence of that learning.
Leslie Early
Yeah, definitely. That's a good place to start for sure. Yeah, and it's funny because that's as t as a teacher, that all makes perfect sense. To me. It's sort of like the Understanding by Design also sort of goes along those steps of like, make sure you know what you're evaluating before you start designing your curriculum. Yes, good advice. My last question, we're going to get a little bit like prophetic trying to envision the future. Let's do it. How do you think or if you let your imagination wander, how do you think that self assessment It might affect or even in immersive environments to either one, either one or both. Self Assessment immersive environments. What do you think that looks like in the future for learning?
Kristin Torrence
Oh, for learning, I think where it's really powerful is that in the meeting we kind of touched on this earlier is that it's really its potential to drive continuous adaptive learning. And then I think just the fact that it could provide pretty accurate depictions of what a learner's competency level is, like, I think that's pretty powerful. It might not seem like it, but like, accurate data is like, really powerful. Yeah. I think also, from a design perspective, it could really drive artificial intelligence within virtual reality environments.
Really just in the form of, you know, interacting with the world, or, you know, the virtual non player characters and the things that they're saying the dialogue or the tasks that, you know, you're now given, or even the supports that are made available to you.
Leslie Early
Yeah. Yeah, I, again, I'm going back to like, this medical student as being my that's my, for right now. That's who I imagine using VR, like, realistically, right now, when they're training, like medical people, or maybe people in like, technical jobs, like, you know, like, electrician, h fac, people. Because they, they're very task oriented. So I imagine that being just continuing that development, and like you said, getting data and saying for sure that this person who passed their certification exam, actually successfully hooked up, you know, or, or successfully removed an artificial appendix and like, or whatever the case. Yeah, I see that being super beneficial that we can through this self self assessment can say for sure, like, yes, this person passed this, and it makes sense. But yeah, I'm just, I'm not getting more creative from that. I feel like I should be more creative, but I can't think of anything right now. Yeah, so anyway, I think it's gonna be really cool. I mean, everything's evolving. Right now, xR is evolving so fast. I think self assessment, adaptive learning, all of these algorithms that are in social media, I think it's all coming together. And I'm just really excited to see what happens like the next five to seven years.
Kristin Torrence
Me too! I'm really excited. I feel like adaptive learning and you really utilizing AI and machine learning for the l&d space, you know, I think I think that's where we're headed. And I'm excited to see where we're where we go.
Leslie Early
Me too. All right, well, anyway, I I'm gonna stay in touch. I love hearing your thoughts on all of these things. And there's a couple of us, I think, right now on LinkedIn are sort of into the XR stuff right now. And it's just, it's again, I'm super excited to see how it's all developing. But if listeners would like to connect with you to continue, if there's other enthusiasts out there, where can people connect with you and reach you?
Kristin Torrence
Sure. So I'm on LinkedIn, Christian torrents, and it's a kr is t i n and t o r e n c. So Kristen Torrence on LinkedIn, and then I'm also Kris, K-R-I-S Torrence, LXD on Twitter.
Leslie Early
Okay, great. Well, thank you again, so much, Kristen, for joining me.
Kristin Torrence
Yeah. Thanks so much for having me. This has been so fun.
Transcribed by https://otter.ai
Comments