Introduction
The main objective of security awareness training has always been to reduce organizational risks related to employee actions. However, legacy training metrics may not always correlate with real employee behavior when faced with a real-life threat.
To determine whether the training positively impacts reducing actual human risks, monitoring the right security awareness metrics for employee progress is key.
Yet, only some training software get it right. This guide will explain the shortcomings of the metrics legacy security awareness training vendors utilize, and provides an alternative method for measuring risk reduction through behavior change.
The shortcomings of legacy phishing metrics
As part of security awareness programs, organizations may test people’s knowledge and skills by sending out phishing simulations. When sending these simulated phishing emails, aspects such as difficulty, variety of content, or the individual point of view means that the results won’t provide a realistic outlook on each employees’ security skills. With infrequent, one-size-fits-all training, you can’t reliably track human risk or progress to reduce it.
In the following sections, we will explain the pitfalls of the metrics that organizations frequently use to report on infrequent, one-size-fits-all employee phishing training.
The failure rate per campaign
The failure rate, also known as the click rate, measures how often an employee performs an unsafe action when they conduct a phishing testing campaign. The failure rate is a very common metric, and many organizations rely on it heavily.
Many believe that the failure rate per campaign is the best metric to describe their organization’s risk regarding phishing and social engineering attacks. However, a common misconception is that the lower the failure rate, the better the training performance must be.
This is not true. A low failure rate does not necessarily tell you anything about the success of the training.
When only a small percentage of employees report the phishing simulation, the failure rate doesn’t necessarily represent the majority’s skills and success. You don’t know how the other employees would react and, therefore, don’t have metrics for how the organization would perform as a whole.
The failure rate also depends on the simulation’s difficulty level and the employee’s skill, knowledge, and experience level. It’s a volatile metric because it’s easy to fabricate low or high failure rates artificially. Do you want to lower the failure rate? Send out easier phishing simulations that even the least experienced employee can easily recognize. The moment you start sending out more difficult ones, the failure rate could jump up significantly.
With traditional security awareness training, another issue is that some companies stop training those that reported the test successfully. They only train that part of the employee population who failed the test by clicking an email link, downloading a file, or giving away their credentials.
The moment you stop training those that did not fail the test, the reporting rate only tells the story of those that failed. As they may attach negative feelings toward security training, they may stop reporting suspicious activity, lowering their engagement with their security responsibilities.
The failure rate can be a useful metric when the training is right. Each employee should receive frequent and personalized training in terms of content, difficulty, and individual points of view. We’ll shortly explain when failure rates can provide useful and descriptive metrics.
The pass rate
The pass rate represents the number of employees who did not perform an unsafe action. It's a common metric, but this approach means that you can’t measure if they know how to identify the threat correctly.
For example, if an employee, for some reason, misses or ignores the training email, this metric would mark them as a success, even if they didn't recognize the email to be suspicious.
When people ignore the test, the pass rate could be outstandingly high. It may look like your organization is in safe hands. In reality, employees may easily fall victim to a cyber attack the next time a well-crafted spear-phishing email catches their attention.
At Hoxhunt, we don’t use the pass rate. The reporting rate replaces it entirely. Those people that did not report the test do not automatically pass the training. This means that if 70% of the population correctly reports, there are still 30% that are not actively learning. We include people that are not actively reporting or clicking the simulation in the miss rate.
The reporting rate
Infrequent phishing simulations can also use the reporting rate. Ideally, the reporting rate is the primary metric to follow. It’s an important metric as it tells you how many people engaged with the training. The goal should always be to engage as big a part of the population as possible so that you know that those people are actively shielding your organization and its assets from attacks.
For the reporting rate, a good quality reporting process is mandatory. With a simple process, people will encounter a lower barrier when it comes to reporting phishing emails. When people report, you know that they are engaged, learning, and acquiring the knowledge and skills to face actual security threats.
When the reporting process is complicated or intrusive on their normal routine, such as calling the security team or finding the reporting email address and following the instructions on what to do, people may not report anything. You won’t know whether they identified the threat, they didn’t notice it, or they just by chance didn’t interact with it.
Important metrics for modern SAT programs
A modern security awareness training program means employees know how to spot and report threats because they learn the habit through frequent and personalized simulations. When people start correctly reporting both simulations and real threats, you will have guaranteed human risk reduction, compared to checking a box with traditional security awareness training for employees.
With frequent training, you will have more data points available to measure both simulated and real threat reporting, which makes it possible to track the progress of individual employees and the company as a whole.
Measuring frequency
Frequent and practical training that reflects the sophisticated and ever-improving real-world threat landscape provides you with data on how your people regularly perform. Based on the results from a few simulations, it’s hard to tell whether the training has been positively impacting human behavior and reducing the cyber risks for your organization. With frequent training and measurement, you can follow how people react to the training and what it means in terms of the overall assessment of security risks.
Reporting rate
When the security program/training focuses on behavior change, the reporting rate is the most important metric. The goal must be to engage as many of the employees as possible to obtain data on how people develop their threat recognizing and reporting skills, both in the simulations and real life.
A good reporting process is a must for improving the reporting rate of both simulations and real threats. When the process is simple, it is a lot easier to report phishing emails. It's common to overlook having a reporting button, but it's a simple solution for encouraging people to report threats.
A good reporting process should look like this:
- Open the email - The email could be an everyday email, a phishing simulation, or an actual phishing email. Always be mindful when you open an email.
- Recognize the danger - Whenever you open an email, think critically before you click. Could it be a possible threat? Take the normal precautions before you click on links or attachments
- Report the email - If the email is suspicious, report it. If it's a simulation, you will get immediate feedback. If it's a real threat, you just saved the day!
Average simulation reporting rate per employee
The goal of any security awareness & phishing training program should be to aim for high engagement. When people correctly report, you will have data on their progress. It’s vital to engage all employees, not only those that previously failed the test.
The average simulation reporting rate tells about how many people have engaged with the training. At Hoxhunt, we advise our clients to aim for at least an average 70% reporting rate. This is because when over 70% of employees keep reporting threats, you know that the chances of them making an error and falling for an attack are lower as they are adopting safe email security practices.
The reporting rate is almost the only consistent metric. With people-first training, you keep educating all people regardless of if they’ve failed the simulation or not. This is why the reporting rate gives you a good indicator of the whole population’s level over time.
The reporting rate tells you that people are actively learning, and you can be more confident that they will do the right thing when they face real threats.
Real threat reporting rate
The end goal of training employees to report phishing simulations is to make sure they recognize and report real cyber threats, too.
Motivating your employees to report all the threats they encounter can be invaluable for preventing a breach and can help you gather data on attacks that get through your email filters.
Average dwell time
Dwell time is the time elapsed between a phishing email landing in an inbox and a user reporting it.
In cybersecurity, time is essential for mitigating risk. Improvements in the amount of time it take employees to report threats can help keep the whole organization safe, as it can accelerate response to real attacks. If reported quickly, the SOC team can mitigate multi-pronged attacks before others in the organization have had a chance to click on it.
Failure rate
A low failure rate is not always an indicator of successful training. The failure rate depends on factors such as difficulty level, variety of the content, individual points of view (also referred to as personalization), timing and frequency, and positive reinforcement and feedback. However, the failure rate can be a crucial metric if used in a meaningful way.
Difficulty level
People can have widely varying skills based on the time spent in training or any previous security education. When employees receive simulations that are too easy, it will likely lower the failure rate. They may also start developing negative feelings towards the training because it’s not challenging enough for them, and they may not feel like it’s worth their while engaging with it.
When you adjust the simulations’ difficulty level for each individual, the average failure rate will become a far more useful metric.
Variety of content
In another of our guides, we’ve written in-depth about why the test emails’ content is so important. It’s not useful to send the same content to everybody for two main reasons.
First, people with different backgrounds and roles need different content tailored to their experience. For example, content should vary based on who they typically interact with or what tools and software they use.
Second, attackers are quickly coming up with new attack vectors. When people don’t frequently see relevant and up-to-date simulations, they are more likely to fail the test or not interact with the test at all.
An individual point of view
Personalization matters. Don’t send the same simulation to the finance department and the software engineering team. People must learn what types of messages, specific for their role in the organization, they should expect to receive from attackers.
Even if people fail a reasonably challenging but personalized phishing exercise, they fail in a safe environment and learn from the feedback.
Timing and frequency
When you send out the phishing test for the whole company at the same time, the word may get around fast to watch out for a phishing email. These warnings will likely lead to an artificially lower failure rate.
When the simulations’ frequency is as often as once every ten days, you can follow the failure rate’s progress. It will provide a better indication of how the organization is progressing. Attacks also arrive randomly: the test gives the most realistic result in terms of failure rate metrics when it appears realistic and catches people off-guard, for example, during a stressful day.
Positive reinforcement and feedback
Using positive reinforcement and giving people feedback can have a significant beneficial impact, whether they fail or not.
People will understand that it’s okay to fail a simulation because they will know that they can do it without consequences; instead, they will be encouraged to try harder to succeed the next time.
Don’t aim for a zero failure rate
It may sound like a good idea to aim for a zero failure rate, but that shouldn’t be your goal. When people fail a simulation, it’s not the end of the world. When the process is reasonable and provides feedback or micro training, they can better learn from their mistakes.
You will also have more visibility into how your organization performs or which attack vectors are leaving your organization more vulnerable. When you know that you can provide more training on those specific topics.
Failing a test shouldn’t be a negative experience. It should be a means of teaching everyone how to defend the organization better together. After all, the individual’s improvement and dedication to participating in the training are more important than lowering the failure rate.
Miss rate
The miss rate is the percentage of users who did not click or report the simulated attack within a set period from receiving it. Typically we use a period of four days.
People may be out of the office for many reasons such as annual leave, sick leave, or traveling, so having a miss rate is natural. The miss rate is problematic because you don’t get data, don’t know whether people are learning, and report a real threat.
You still want to monitor this metric because you want to make sure that people did not stop engaging with the training. Comparing the miss rate with absenteeism rates will indicate this. If they do stop engaging, it’s good to plan how you will reconnect with them.
Measuring cybersecurity sentiment
Even when you focus on behavior change and risk reduction, measuring cybersecurity sentiment with qualitative and quantitative post-awareness surveys can be useful. You want to know how people feel about the training, their attitude toward cybersecurity, or their motivation to participate. Do they understand that it’s important? Do they find it interesting? Is the service delivery good?
Don’t create a survey just for the sake of gathering insights. Make sure when you ask for employee feedback on something, it’s actionable, so you can react and improve the experience for the employees if necessary.
Change behaviors to drastically improve risk metrics
Methodologies broadly vary on how you assess organizational cyber risk. If you want a quantifiable approach to the human element of risk, measuring human behavior change helps with that. Using metrics is essential to identify whether your employees may jeopardize security.
Engaging people is essential. People need to participate in training to track how they are performing. Employees need to continually learn to develop the right skills and knowledge to defend the organization from attacks.
Through constant improvement of the reporting rate and creating an environment where it’s acceptable to fail the simulations, you can develop a security culture where people feel motivated to be active participants of your layered security defenses.
When people get into the habit of reporting real threats, you will lower the risk of them clicking on malicious links and attachments or giving away credentials. You will also collect invaluable data from the threat reports for use in improving your incident response.
