Are your phishing tests worth the money you are spending on them?
Please don't misinterpret that as suggesting you shouldn't be testing your users. To the contrary, I think you should be testing all your users (executives of all ranks included) on a regular basis. What I mean by that question is; are you really "testing" your users, or are you merely spot quizzing them?
Both can be effective; but one without the other is a waste of both time and money.
If you have never tested your entire user base (without exceptions) then you should run a fairly simple test; let's call it a "Level 1 sophistication" test, just to see how effectively your users critically think about email. Be sure to whitelist the email campaign so it actually reaches your users, as you are not testing your anti-spam filter (this time). Once you start to see improvement, start increasing the sophistication on subsequent tests to get outside their comfort zone. The goal of this training and testing is all about critical thinking. Your users need to critically think about each email they receive, and it should become second nature to them. I describe the ‘Levels’ of these tests, and what they might include below.
After each test, track how many of them click the link or open the attachment, how many of them actually provide credentials, or run the macro. All of these data points are valuable when you go to train them about the decisions they are making. You should train users both before and after each test, since as the techniques and sophistication change in real life, they should change for your tests as well.
If you are not testing your users appropriately, then what is the output of money and time really getting you?
Every user you fail to test becomes your biggest risk. Everyone is susceptible to phishing and social engineering in general. I mean everyone. If it is just the right time of day, or the email appears on that one day where everything is going wrong, or right, your emotions will greatly affect your ability to critically think about what you are looking at.
If you regularly test them and provide feedback about their performance (not firing the employees who fail, but teaching them) they will be your sounding board in the break room. This will decrease your overall risk to these attacks and may even stop that breach that would have happened otherwise. Not only will this increase the security of your business, but it will also positively benefit these users on a personal level, further securing their and potentially their families' personal email habits.
Sophistication Levels: Training Users Incrementally
I mentioned above about using a Level 1 phish to start your testing. The concept of sophistication levels is not new, or mine*, but may not be widely utilized as much as they should.
If your first test uses a Level 3 phish, everyone will fail. That doesn’t really help, or provide a meaningful teaching moment. Instead start simple and increase difficulty once they are able to process what they are seeing and critically think on each one consistently. With enough training and testing, you will see compromise numbers drop, significantly.
Think of the Level 1 phish as the classic 419 scam or the FedEx invoice, with bad spelling and grammar. It’s a completely impersonal, almost random email that typically ends up in your spam folder. Some get through though, and the reason we whitelist this testing is to simulate that failure of your technology to stop the threat. You are trying to train people, not software filters in this case.
A Level 2 phish should be a bit cleaner, fewer or no spelling/grammatical errors, still not very personal, but not random. It could come from an unnamed person in your HR or IT department (i.e., from "IT Staff" or "Human Resources"), or from a known provider, but from their "Shipping Department." It asks you to follow this link, enter your credentials, open this document—and don't forget to run the macros.
Level 3 is much cleaner, with perfect spelling and grammar. It may come from "John Smith, IT supervisor" (John actually works for the company in that position, found through Open source intelligence (OSINT)). He may be asking you to do something not that out of the ordinary: follow this link to one of your service providers (a similar, but different, domain of an actual service provider for the company, also found via OSINT). Open a document, that may actually be hosted on the company website, but this copy is trojaned and not hosted on the company domain.
Level 4 is what is called spear phishing. All previous levels can be sent to the masses, in bulk. This one is user specific; perhaps only 1 to 4 people receive the same email. It is very personal, based on data found during extensive OSINT, referencing activities they are involved with or charities they contribute to. It may appear to come from friends or coworkers, asking them to perform some action that is part of their day-to-day activities (like the "wire transfer request from the CEO to this other account" scams). These can be the most devastating for a company and should be part of your periodic phishing tests (after the users are passing the previous levels consistently).
Winning Is About Education (Not Shells)
From a penetration tester’s perspective, winning is not about the number of credentials/shells you get, or by making the test so easy that they look great on paper. Winning is testing the users at the appropriate level for their current inclination to critically think about each and every email they encounter and providing that teachable moment. There is a balance, and as a tester you need to determine how they behave before you test, so that your test actually tests them. None of that can be accomplished without repeated campaigns and analyzed data. This is one area of computer security where you can prove value and improvement without having to suffer a breach first.
*These concepts were first introduced to me in the book “Phishing Dark Waters” by Chris Hadnagy and Michele Fincher, 2015