The typical awareness programme starts at the end. A platform is bought, courses are assigned to the whole organisation, completion is chased for three months, and a compliance percentage is reported to the Board.
The next year it repeats. Nobody knows whether it worked, because nothing other than attendance was ever measured.
The problem is not the training. It is the order.
What a controlled phishing exercise measures
A simulated exercise does not measure knowledge: it measures behaviour under real working conditions. Which is the only thing that matters, because the incident does not happen during the course.
Four metrics count, and only the first is the one everybody looks at:
Open rate. How many people opened the message. It is the least informative: opening an email is, to a large extent, doing your job.
Click rate. How many followed the link. This is where real exposure begins.
Credential submission rate. How many went as far as completing the form. It is the metric that correlates best with the risk of an actual incident, and it is usually considerably lower than the click rate.
Report rate. How many people alerted the internal channel that something looked wrong. This is, for us, the most important metric of the exercise, and the one almost nobody measures.
Why reporting matters more than clicking
An organisation where nobody clicks is a statistical fantasy. With enough volume and a well-built lure, someone will always click. The operational question is not how to get to zero, but how long it takes the security team to find out.
If the first report arrives seven minutes after the send, the organisation has an early detection capability no tool replaces: it can block the domain, find out who else received it, and force a credential change before those credentials are used.
If nobody ever reports, the organisation finds out when there are already consequences.
That is why the goal of the programme should not be “reduce clicks” but “increase reporting and reduce the time to first alert”. It is a different objective and it produces a different design: you have to provide an easy-to-use channel, communicate it, and —above all— thank every report, false positives included. If someone who reports a legitimate email is made to feel foolish, they stop reporting.
The assessment changes the content
When you measure before you train, the training stops being generic and becomes specific.
Results are almost never evenly distributed. Departments show markedly different exposure, and the reasons are structural, not a matter of “carelessness”:
- Procurement and finance are sensitive to lures about invoices, bank account changes and urgent payments. Opening those emails is literally their job.
- HR opens CVs and documents from unknown senders all day long.
- Sales opens messages from new contacts by definition.
- The board is the target of lures personalised with public information, and usually has less time to stop and verify.
A single course for the whole organisation wastes some people’s attention and falls short for others. With the assessment in hand, the programme assigns paths by role and concentrates effort where the risk actually is.
How to do it without breaking trust
A badly run simulated phishing exercise leaves lasting damage: people feel deceived by their own employer and the security function is thereafter seen as an adversary. Recovering from that takes years.
Five rules we always apply:
1. Agree the framework beforehand. The board, HR and —where applicable— staff representatives must know about and approve the exercise, its purpose and what will be done with the data, before it starts.
2. Announce that exercises will happen, without saying when. Communicating the programme does not invalidate the measurement: nobody stays on high alert for months. And it removes the sense of entrapment.
3. Never use lures that exploit sensitive ties. Bonuses, redundancies, medical results, family matters. They raise the click rate and destroy trust. The metric you gain does not compensate for what you lose.
4. Aggregate metrics, not name lists. Results are reported by department and by trend. Individual identification only makes sense for assigning training, and must stay within the agreed boundaries. A league table of “who fell for it” circulating around the organisation guarantees nobody ever reports anything again.
5. Close with useful information, not reproach. Whoever clicks should see, at that very moment, what signals the message carried that should have raised suspicion. The exercise is educational or it is nothing.
The curve to expect
In sustained programmes, the evolution follows a fairly stable pattern: the click rate drops sharply between the first and second exercise, then flattens, and from there sustained improvement comes from the reporting side, which rises more slowly but more durably.
That flattening is not a failure: it is the realistic floor. From then on, the organisation stops chasing an impossible zero and starts building what does scale —early detection— while technology takes care of reducing how many messages reach the inbox at all.
The auditable close
For anyone who has or is implementing an ISMS, there is an additional benefit worth taking. ISO/IEC 27001 requires competence and awareness, and has a specific control on information security awareness, education and training.
A programme with assessments, named certification, metrics by user and department, and simulated exercises with their progress report produces exactly the evidence the auditor asks for, with no extra work. The alternative —an attendance sheet from a talk— complies formally and demonstrates nothing about effectiveness.
The Complete level includes SORT SecurityAcademy, controlled phishing and social engineering awareness campaigns.