Assessing Job Readiness with VR

Learner success and job readiness is easier to assess with the rich data that can be collected in VR. Careful planning for assessment can help you generate meaningful results and feedback for your learners.

xAPI Results

The xAPI profile includes an optional “result” property that you can leverage to extract results from the learning experience. xAPI results can be in the following forms:

• Success: a binary variable, also known as a boolean value, that tracks the “success” or “pass/failure” of an activity
  • Note that success can be tracked on a more complex activity (made up of smaller, sub-activities) and cumulative “success” of a larger activity could depend on the success/failure of a combination of sub-activities.
• Completion: a boolean value criteria for completion can be defined by the learning designer and can depend on the results of a set of activities in the scenario
• Score: a numerical result which can be represented as a raw score (one number) or a scaled score (percentage)

Rubrics & Criteria Based Assessment

Defining what success looks like in a training scenario will vary from project to project, but there are some guiding principles and techniques that you can use to ensure that your assessment reflects the learning goals you set in the planning phase. In order to demonstrate how you might build a thoughtful and informative assessment, we will use a food safety example from a fictional food preparation company. This company wants to reduce contamination from 1% to below 0.2%. They currently use a Foodbot 3000 machine for food preparation. 

The company’s overall learning goal is: “Learners will be able to disassemble, sanitize, and reassemble a Foodbot 3000.” This learning goal was set to address a business goal of decreasing the problematic contamination of food equipment. To read more about setting goals for your VR training program, view our blog post here.

In order to properly achieve the learning goal, employees need to demonstrate the following key performance indicators:

• Learner will disassemble the Foodbot for cleaning
  • Learner must take safety precautions, specifically donning safety gloves and unplugging the machine before they begin cleaning
  • Learner will loosen all 6 screws on the cover plate to reveal the inner chamber of the Foodbot
  • After unscrewing the cover plate, it must be removed and placed in the sink
  • When reassembling, screws on the cover plate must be in the locked position
• Learner will scrub all widgets in the inner chamber a minimum of 10 times using the Foodbot cleaning solvent
  • There are 8 widgets, each needs at least 10 passes with the cloth
  • Learner must check pH level of the cleaning solvent before using it
• Learner will test the surface of the Foodbot upon completion to check for contaminants
  • Swab surface
  • Place swab in reagent
  • Read results

Example Rubric and Assessment Strategy

You can use a combination of “result” types to define what overall completion and overall success of this scenario means (note that this an example only – there are many ways to plan the assessment).

Performance Indicator	Result Type	Score
Disassembly/Reassembly
Unplug machine	Success	2
Put on safety gloves	Success	2
Loosen screws and remove plate	Success	2 for screws 2 for plates
Lock screws on reassembly	Success	1 point per screw locked for up to 6 points
Cleaning
Check PH of solvent	Success	2
Wipe each widget 10 times	Success	0-3 times: 0 points 4-7 times: 1 point 8-10 times: 2 points 8 widgets, maximum of 2 points each => 16 possible points
Check for Contamination
Swab surface	Success	2
Dip swab in reagent	Success	2
Read results	Success	2
Total Possible Points		38

Overall Success: Achieve a scaled score of over 90% or 35/38

Overall Completion : Success on ALL steps.

Breaking down assessment in this way checks for learner action as opposed to memorized knowledge. Traditionally, you might check for job readiness by having a quiz at the end of a video. Questions like “How many times should you wipe each widget?” can easily be answered. But checking if the learner actually wipes the required number of times to achieve sanitation is a true measure of success.

You will want to ensure that you have a solution in place that can handle this data and allow you to visualize it for each learner. Analyzing results will allow you to refine the training as you go so that the skills that need the most focus are highlighted.

Teach Mode vs Test Mode

The above noted assessment strategy will be a lot more meaningful if you create a version of the training that is unguided. At Motive, we refer to this as “test mode”. Generally, trainees experience a guided, “teach mode”, first where they can learn the procedure or task and then attempt test mode to see if they are able to complete the task with no help. Multiple repetitions of teach and test mode may be required before trainees can achieve completion and success in test mode.

Assessing the ways your learners perform will be key in determining the job readiness of your employees. Using your own key performance indicators alongside the data extracted from your employees VR training scenarios will help you better understand your employees training success.