Recently, during a consulting meeting, I was asked whether you are given a guarantee after a penetration test that the tested application is absolutely secure. From a customer's point of view that question is perfectly understandable. Anyone who spends anything up to several thousand euros on a security analysis, depending on the effort involved, expects at least subconsciously that every possible vulnerability has been identified and that the tested application is 100% secure. The uncomfortable truth, unfortunately, is that this cannot be the case.
As a rule, almost every penetration test that is conducted is a compromise between testing effort and how much the results actually say. At a professional pentest provider, a test includes at least,
- a test performed according to a standardised methodology that follows best practices
- a tool set that allows automated checks for typical vulnerabilities
- extensive manual testing on top of that
- certified penetration testers (OSCP & OSCE rather than CISSP / CEH & Co)
- a quality assurance review
- a detailed report as the deliverable
If a test satisfies the criteria above, then with a high degree of certainty:
- all easily discoverable vulnerabilities (low hanging fruits) have been found
- attackers at script kiddy level, at the very least, no longer stand a chance
- at the very least, every unspecific automated attack has been deprived of its foundation
- the effort for an attacker has been raised to at least the effort the penetration tester invested
- a genuinely meaningful indication has been produced as to whether the target system or target application was set up or developed at a professional level of security
The honest admission that an attacker with considerably more effort might still find a vulnerability does not exactly help sell the service.
A higher level of assurance - developing zero day exploits or even buying them, for example - is unfortunately not realistic in the vast majority of cases. Carrying out an APT (advanced persistent threat) at NSA & Co level is very, very laborious and would drive the cost of a penetration test to extreme heights. For a customer that would be an enormously bad price-performance ratio, and the money could certainly be invested more sensibly in their own security.
But let us assume that a pentest provider really did give a 100% security guarantee: what would happen if a system was compromised anyway? How would you prove that the penetration tester had overlooked something? How would you prove that the customer had not modified the system since then? How would you prove that the customer possibly missed a security update? How do you deal with vulnerabilities in the frameworks or libraries in use? A single penetration test with a 100% security guarantee and, as a provider, you carry incalculable legal risks for all eternity. Even if you estimate the risk roughly and spread it over all customers on average, it would drive the cost of a penetration test up disproportionately. No customer would want to pay for that. A 100% security guarantee is unfortunately unrealistic.
Nevertheless, most penetration tests do identify vulnerabilities or weaknesses that make the security posture more transparent and reveal the potential for improvement. In practice, pentest reports with exactly 0 findings are very rare and point to an extremely high level of security. What ultimately matters for the customer is knowing both the limitations mentioned and the benefits outlined, so that they can take the right decision for their situation: pentest or no pentest?