Status: a plan, not a result. No learner has taken this training and no data exists. Everything below says what would be measured, from where, and what decision each measure informs. Numbers that appear are sample sizes for a pilot, not findings.
1. The question
The business goal is more hazards reported and fixed before anyone is hurt (see the action map). The evaluation asks three things, in order:
- Did people practice the behaviors? (Did they get through the training, and what did they choose?)
- Did the behaviors show up on the job? (Are cords tagged, edges stopped, reports sent, and are the reports any good?)
- Did it change what the business cares about? (More reports early, fewer injuries later.)
The first is cheap and fast. The second is the one that matters most. The third is slow, noisy in a small workforce, and should be watched, not promised.
2. Framework: LTEM, with Kirkpatrick alongside
The plan uses Will Thalheimer's Learning-Transfer Evaluation Model (LTEM), which separates attendance from learning from doing. A Kirkpatrick crosswalk is included because many clients ask in those terms.
| LTEM tier | What it measures | Kirkpatrick level | In this plan |
|---|---|---|---|
| 1. Attendance | Showed up | (none) | Recorded, not treated as success |
| 2. Activity | Took part | (none) | Recorded, used to find where people drop out |
| 3. Learner perceptions | Felt it was useful | Level 1 | Three questions, in the pilot only |
| 4. Knowledge | Can recall facts | Level 2 | Measured, but kept to what a worker needs in order to act: the practice checks, the welcome-back quiz and the test |
| 5. Decision-making competence | Chooses well in realistic situations | Level 2 | Main in-training measure |
| 6. Task competence | Performs the task | Level 2 to 3 | 30-day ride-along and a report-writing task |
| 7. Transfer | Does it on the job | Level 3 | Main real-world measure |
| 8. Effects of transfer | The results follow | Level 4 | Watched over 6 to 12 months |
Tier 4 is measured, but only for facts a worker needs to act (what an anchor must hold, which signs mean call 911). A recall quiz alone predicts little about whether a worker will tag a cord with the crew waiting, and v1's one question was that kind of item. So the knowledge checks are practice, not the success measure; the scenes and the test ask the learner to decide.
3. What is measured, where the data comes from
| Tier | Measure | Data and source | When | Decision it informs |
|---|---|---|---|---|
| 1 | Completion rate | completed statements ÷ people assigned |
Weekly during rollout | Whether people can reach the training at all |
| 2 | Where people stop | Last answered statement before a session ends; timestamps between statements |
Pilot, then monthly | Whether a decision is confusing or too long |
| 3 | Perceived usefulness and realism | Three end-of-training questions in the pilot: "I could use this on my next shift," "The situations felt like my job," "What was unclear?" | Pilot only | Wording and realism; not a success measure |
| 4 | Check results and welcome-back results, per question | answered with stage check or return, score, grid cell |
Pilot, then quarterly | Which fact is not landing; whether a lesson's knowledge holds a few lessons later or a day later (warm-ups and the welcome-back quiz ask about earlier material in new wording) |
| 5 | First-try best rate, per scene decision | answered with stage apply, first-try = true, grouped by grid cell |
Pilot, then quarterly | Which behavior is weakest; which decisions to revise |
| 5 | Most-picked wrong option, per scene decision | answered responses grouped by option |
Pilot, then quarterly | Which risky belief is most common |
| 5 | Test score, score by grid cell, and dangerous answers | scored, passed or failed, the per-question scores and the count of dangerous answers |
Pilot, then quarterly | Which of the twelve skills is weakest across people; the pass rate; how many otherwise-passing attempts fail on a dangerous answer; whether the pass mark is set well |
| 5 | Test item analysis | Share correct per item and per version, which wrong option is chosen, whether high scorers get it right, whether the versions of one cell behave alike, and whether any "dangerous" option is chosen by strong learners (a sign the item or the flag is flawed) | Pilot, then quarterly | Items that are too easy, too hard, confusing, or that a strong learner misses; versions that are not equivalent; flags a safety professional would not defend |
| 5 | Retakes and review use | Number of attempts, and which lesson pages were opened from the results screen | Pilot, then quarterly | Whether the diagnostic grid sends people back to the right place |
| 6 | Cord, chemical, edge and heat practice observed | A 10-item supervisor ride-along checklist within 30 days: cord tagged and removed; stored fuels reported; stop-work called; heat plan known | Day 30 | Whether the behavior shows up when observed |
| 6 | Report quality | A report-writing task on a photo of a hazard, scored 0 to 4 for the four parts by two raters; rater agreement checked. The training's own write-your-own boxes record only the length and the word-check score, never the text, so they cannot stand in for this | Day 30 | Whether people can write a four-part report |
| 6 | Delayed retest | Twelve test items from versions the learner has not seen, taken at day 14 | Day 14 | Whether the skills hold after two weeks, and whether the grid still points at the right pages |
| 7 | Hazard and near-miss reports per month per 10 workers, by property | The report channel, or the work-order system, or a text log | Monthly, against a 60-day pre-launch baseline | The main outcome measure |
| 7 | Reports with all four parts | Same source, sample of reports read against the rubric | Monthly | Quality of what is reported |
| 7 | Time from noticing to report | Timestamps on the report versus the work order or the note | Monthly | Whether reports arrive the same shift |
| 7 | Tags issued and spare cords used | Stock counts of "Do not use" tags and replacement cords | Monthly | Whether the environment support exists and is used |
| 7 | Stop-work calls and early heat stops | A simple crew-lead log | Monthly | Whether speaking up is happening |
| 7 | Day-7 follow-up: "Did you do your first move?" | One text message | Day 7 | Whether the training's last step took |
| 8 | Injuries and first-aid cases | The company's injury log | Quarterly, watched over 6 to 12 months | Whether outcomes move; not used to judge the training in the first quarter |
| 8 | Workers' compensation claims | The insurer's report | Annually | Long-term effect |
| 8 | HIOSH inspections and citations | Company records | Annually | Regulatory exposure |
4. Baseline and comparison
- Baseline. Before launch, collect 60 days of the tier 7 measures as they are today. Where no report channel exists, the baseline is the count of hazards a supervisor finds on a walk-through that no one had reported.
- Staggered rollout. If the company has several properties, train them in waves. The properties not yet trained act as a comparison for the ones that are. This is simple and honest, and it does not need a statistician.
- Reading the results. With a small workforce, injury counts are too few to test. Report rates and direction, by property, with the raw counts beside them. Say what the numbers cannot show.
5. Pilot
Before rollout, a small pilot with real workers on their own phones. The training is about 60 minutes by the word-count model, so the pilot also answers the question the model cannot: where do people stop, and what would they cut?
- Who. 5 to 8 workers across roles, including at least one who has never done training on a phone. (This is a planned sample size.)
- How. Ask them to think aloud while they go through it. Time each person. Note every place they hesitate, re-read, or say "that is not how it goes."
- Check. Do the options feel like real choices? Is the consequence believable to someone who has seen the real thing? Is each lesson close to the 9 to 11 minutes the model says, and where does attention drop? Do the picture questions work with a thumb on a small screen, and with a screen reader?
- Revise. Rewrite any decision where nearly everyone chose the same first option, or where a wrong option was chosen because it read as the more natural sentence.
6. Item analysis rules of thumb
The test bank has 32 items (12 per attempt, one per cell). Each is a design guess until real learners answer it.
Design heuristics, not statistics:
- A decision that nearly everyone answers correctly on the first try is probably too easy. Make the wrong options more tempting, or cut it.
- A decision that few answer correctly may be confusing or may be hard for a real reason. Read the think-aloud before changing it.
- If the same wrong option leads the count in every property, the belief behind it is worth naming in the job aid.
- A "dangerous" option that more than a few strong learners choose may be reasonable in practice. Take it to a safety professional before keeping the flag.
- If the best answer is picked mostly by longer-answer habit, the item is not measuring safety.
tools/metrics.jschecks option lengths before any learner does.
7. Data handling
- The demo stores anonymous, random identifiers in the learner's own browser. Nothing leaves it.
- In a real deployment, tell learners plainly what is recorded, who sees it, and how long it is kept.
- Use the data to improve the training and to see where the workplace makes the safe choice hard. Do not use it to discipline anyone. Individual first-try results belong to the learner. Supervisors see property-level patterns.
8. What success would look like
Set with the client before launch, from the baseline. In outline:
- Reports per 10 workers rise within 90 days, and the share with all four parts rises with them.
- Time from noticing to reporting falls to the same shift.
- Tags and spare cords are being used.
- Nobody reports being punished for stopping a job or reporting.
- If the first-try rate on a behavior stays low after training, fix the environment before rewriting the training.
Failure looks like a high completion rate and flat reports. That would mean the training got done and the workplace did not change, which the needs analysis predicts when the environment actions are skipped.
9. Where the data goes
The tracking in the demo is shaped so a dashboard can read it: completion, the path each learner took, the most-missed decision, and first-try rates by behavior. That dashboard is a separate project (P6). Any dashboard built from the demo's data would be simulated, and would say so.