Warfighters have used offensive engagements to improve their planning and execution for centuries. Military planners “red team” battle plans with dedicated teams designed to emulate the adversary’s capabilities and behaviors. After World War II, Admiral Chester Nimitz famously stated, “The war with Japan had been enacted in the game rooms at the War College by so many people and in so many different ways that nothing that happened during the war was a surprise—absolutely nothing except the kamikaze tactics toward the end of the war. We had not visualized these.”
In our infosec universe, penetration testing or “pen testing” has been around in various forms for decades. When I was at Microsoft, I started the first internal offensive testing program that targeted Windows NT 5 (which became Windows 2000). Microsoft, like most product companies, then focused on “functional testing”—verifying that features work as specified. It did not formally consider overtly hostile attacks until after the Melissa and Love Bug mass mailers of the early 2000s. I called my program “malicious testing” to differentiate it from benign “functional testing” and to highlight the critical difference in mindset.
What I saw at Microsoft still largely holds true today: defenders don’t understand attackers. The “defenders” I interacted with in 1996 were young developers trying to get their cool new features to work and checked into the nightly builds. They weren’t thinking about security, and they weren’t specifically trained in attacker techniques. Similarly, today, the average enterprise IT department is consumed with daily firefighting and doesn’t understand how attackers abuse its systems.
Today, when a classic “pen test” is performed, it almost always proves that similar, difficult-to-detect attacker techniques continue to work time after time. Without timely detection, attackers have unlimited time to breach defenses, and they will always be successful. Tightly scoped, short-term penetration testing engagements don’t present a realistic test of defensive people, processes, and technologies, and often don’t result in improvement, as evidenced by the fact that the same attacker techniques tend to work over and over.
Today’s classic “pen tests” do not result in an improved defense. They are activities without progress. Admiral Nimitz learned from the valuable red team exercises at the War College that activities made them better and directly contributed to the outcome of the conflict. However, classic “pen tests” aren’t making us better as infosec defenders.
Offensive Exercise Maturity Model
There are several stages enterprises go through with respect to offensive testing; most enterprises are stuck in Type I and Type II engagements that provide limited value. The maximum security value comes from Type III and Type IV engagements.
I want to encourage all defenders to increase the utility of offensive engagements in 2020. Think about how to create progress and improvement from these activities. Don’t settle for classic “pen tests” that generate the same result over and over. Don’t accept a “pen testers always win” defeatist attitude—gain confidence in and tighten your detection timelines. Like Admiral Nimitz, use offensive engagements to understand the playing field before you find yourself in a firefight.