Frameworks for benefits, risks and KPIs
Framework 1: Task decomposition (automate, augment, or keep human) Jobs and processes are bundles of tasks, and AI affects tasks, not whole jobs. Write the process out as steps, then label each step:
- Automate: rule-based, repetitive, low judgement, low harm if wrong. Software does it; a person monitors exceptions.
- Augment: AI produces a draft or analysis; a person reviews, edits and decides.
- Human only: needs judgement, empathy, negotiation or legal accountability.
Example: hiring a customer support specialist at Brightwater.
| Step | Label | Why |
|---|---|---|
| 1. Define the role with the hiring manager | Human only | Requires judgement about business need. |
| 2. Draft the job description and posting | Augment | AI drafts quickly; a person confirms requirements. |
| 3. Post to job boards | Automate | Rule-based. |
| 4. Acknowledge applications | Automate | Standard message, no judgement. |
| 5. Screen applications against requirements | Human, with caution about any AI support | A decision about individuals. High risk. |
| Step | Label | Why |
|---|---|---|
| 6. Schedule interviews | Automate | Calendar logic. |
| 7. Prepare the interview guide | Augment | AI proposes; a person selects job- related questions. |
| 8. Interview and assess | Human only | Judgement and relationship. |
| 9. Summarise interview notes | Augment | AI structures; interviewers verify. |
| 10. Decide and make the offer | Human only | Accountability. |
| 11. Send the offer letter from a template | Augment or automate with approval | Template-driven, but errors are costly, so a person approves. |
Framework 2: Value, feasibility and risk scoring When you have several candidate use cases, score each from 1 (low) to 5 (high) on three dimensions:
- Value. How much time is saved, how often the task occurs, and whether quality or the experience of candidates and employees improves.
- Feasibility. Whether you have the tool, the data, the skills, and a clear, stable process.
- Risk. The impact on individuals if it goes wrong, the sensitivity of the data, legal exposure, and how visible errors would be.
A simple priority score is Value + Feasibility − Risk. Add one gate: any use case with Risk of 4 or 5 goes to legal and privacy review before any pilot, whatever its score. The numbers are judgements, so the benefit comes from the discussion they force and from comparing options on the same scale.
Framework 3: The use-case canvas For each shortlisted use case, write one page that answers these questions. This is the core of your first playbook assignment.
| Section | Question to answer |
|---|---|
| Process and pain point | What happens today, how long does it take, and what goes wrong? |
| AI approach | What exactly will the AI do, with which tool? Is each step automate, augment or human only? |
| Benefits | Time, quality, experience, consistency. Be specific. |
| Risks | Accuracy, bias, privacy, legal, trust, over-reliance. |
| Guardrails | Who reviews what, which data is excluded, what people are told. |
| KPIs | What you will measure, the baseline today, and the target. |
| Owner and next step | Who is accountable and what happens in the next 30 days. |
Choosing KPIs A key performance indicator is a measure that tells you whether something is working. Choose a small set that covers four angles, so that an efficiency gain cannot hide a quality or fairness problem. Always record the baseline (the value before the change); without it you cannot show improvement.
| Angle | Example KPI | How it is calculated |
|---|---|---|
| Efficiency | Time to fill | Days from approval of the job opening to the candidate accepting the offer. |
| Efficiency | Time per task | Minutes to produce a job description, answer a query, etc. Time a sample before and after. |
| Efficiency | Cost per hire | (Internal recruiting costs + external recruiting costs) ÷ number of hires in the period. |
| Quality | Revision rounds | Number of review cycles before a document is approved. |
| Quality | Answer accuracy | Share of an assistant’s answers judged correct against the source policy. |
| Quality | Quality of hire | No single standard. Often a composite of new-hire performance, retention at 12 months and manager satisfaction. |
| Experience | Candidate or new-hire satisfaction | Average survey rating, or share giving a favourable rating. |
| Experience | Employee Net Promoter Score (eNPS) | Percentage of promoters (9 to 10 on a 0 to 10 scale) minus percentage of detractors (0 to 6). |
| Risk and fairness | Adverse impact ratio | Selection rate of one group ÷ selection rate of the group with the highest rate (Lesson 1.3). |
| Risk and fairness | Error or escalation rate | Share of outputs corrected by a reviewer, or share of queries handed to a person. |
A worked example of return on investment Brightwater writes about 40 job descriptions a year. Each takes 90 minutes. With an AI-assisted process, a pilot shows each takes 30 minutes including review.
- Time saved: 60 minutes × 40 = 40 hours a year.
- Value of time at a loaded cost of $45 per hour: 40 × $45 = $1,800.
- Tool cost attributed to this use: $300 a year.
- Net benefit: $1,800 − $300 = $1,500. Return on investment: $1,500 ÷ $300 = 500%.
Two cautions. First, time saved is only worth money if it is put to better use, so say what the 40 hours will be spent on. Second, pair every efficiency number with a quality measure (for example, revision rounds or hiring-manager satisfaction), or you may simply be producing worse documents faster.
Framework 4: Risk tiers and matching controls
| Tier | Examples | Minimum controls |
|---|---|---|
| 1. Low | Drafting postings, emails, checklists. | Human review before use. No personal data in unapproved tools. |
| 2. Medium | Employee question assistants; survey analysis. | Named owner, tested against a question set, clear route to a human, periodic sampling, people told they are using AI. |
| 3. High | Anything that screens, scores, ranks or predicts about individuals. | Legal and privacy review, bias testing, notice to those affected, a human who genuinely decides, records kept, vendor due diligence, regular re-testing. |
If your organisation wants a recognised structure for this, the US National Institute of Standards and Technology publishes a voluntary AI Risk Management Framework built around four functions: Govern (set policy and accountability), Map (understand the context and who is affected), Measure (test and monitor) and Manage (act on what you find). The international standard ISO/IEC 42001 sets out requirements for an AI management system.
Jot down key ideas in your own words. Saves automatically.