Can Penetration Testing Be Automated?

May 23, 2022 Han Wallace

Can penetration testing be automated image

What is automated penetration testing?

Vulnerability scanners

Automated penetration testers

The ideal formula

How does it work

Continuous security testing in response to real security challenges

How to implement cybersecurity testing effectively


“Automated penetration testing” has become something of a buzz phrase in the cybersecurity services space. For IT and security leaders researching ways to cut costs and resource management, and add momentum to their testing programme, it surely sounds like a no-brainer. However, as this article will explain, not everything can or should be fully automated – and that’s not a bad thing for your business. Automation can be used to optimise parts of the penetration testing process, making it more efficient and improving its effectiveness as a risk identification exercise. What it can’t do, is replace manual activity whilst delivering the same outcomes. 

What is automated penetration testing? 

The market is congested with cybersecurity solutions that overdress their functionality and purport to solve challenges they weren't built for. And anyone who’s done even a little research will know that automated penetration testing is no exception. Thankfully, most solutions with that name sit within one of two categories: 

Vulnerability scanners 

Vulnerability scanners monitor networks, devices, and applications for known vulnerabilities and then score them (usually as per CVSS). In this sense, they can identify but they can’t investigate.  

Identification is, of course, crucial. And for this, vulnerability scanners are a powerful tool. However, they deliver high volumes of data that must be followed with the heavy lifting of verification, deduplication, and then intelligent prioritisation, before you’re left with actionable data. As our UK Cybersecurity Services Director, Dave Ashton, says: “Most customers who use automated scanners are not managing the remediation effectively. This is down to volume and/or a lack of understanding the contextual impact of a given vulnerability or how its relationship with other vulnerabilities creates a wholly more worrying exploitable risk”. The TLDR: the more scanners you use, the more findings you’ll uncover, creating a greater data overhead to investigate, prioritise, and action. 

There are three approaches to tackle this specifically: 

  1. Deploy vulnerability scanning with pragmatic (risk-based) objectives in mind (ergo, don’t scan without a plan for processing the data it will generate). 
  2. Develop a process for removing false positives (most likely using manual analysis). 
  3. Consolidate scanner reports with detection activities to ensure high-risk assets and areas of your estate are being properly monitored. 

It's crucial to note that even when it comes to vulnerability identification, scanners struggle if there are dynamic variables. Such variables can be found in situations where systems and their data change according to different inputs or sequences of inputs. In the example context of an e-commerce website, a human penetration tester could input different data into the customer checkout fields on each page form to see what effect this has. A scanner, however, doesn't see the potential to programme different outcomes by interacting with the fields. Without context, it can only observe and perform limited random actions, rather than objectively testing the effect of variables. 

One final point. Using scanners on authenticated systems carries some business risks (our engineers avoid doing it at all if possible). This is because they can’t intelligently differentiate between “safe” and “unsafe” options, instead of performing a pre-scripted set of (“if”) instructions. Deployed on a live system with authenticated access, where the user is authorised to perform critical actions (such as deleting data), this can result in operational disruption and irreversible harm. 

Automated penetration testers 

A more accurate representation of the name they go by, automated penetration testers are essentially a tooling stack operated by an agent or virtual machine (VM) to emulate a manual penetration test. The typical process – reconnaissance, testing, and reporting – is loosely followed, whereby the agent scans the customer’s attack surface, identifies targets, performs exploits, and then generates a summary report of the vulnerabilities identified. 

Automated penetration testers offer speed and scale, deploying hundreds of attacks across different assets and from multiple endpoints into the network. Unlike scanners, they can perform some level of investigative testing. So, as a security housekeeping activity, they perform well – to an extent. 

Like scanners, these automated solutions frequently generate false positives that require more post-test analysis. They also falter when it comes to deep testing; they are ill-equipped to understand the context of a system or asset and the connections between multiples of those. This limits their ability to tactically chain together vulnerabilities and pivot, making them more likely to miss complex attack paths.  

In fact, any techniques that realistically emulate social engineering and hands-on keyboard techniques will be unachievable, which is a problem. While bot-operated attacks may dominate the threat landscape, it’s the threats at the more sophisticated and severe end of the scale that use human intervention to achieve maximum damage and/or gain. This explains why we’re seeing many bot-operated threats evolving to incorporate that human element. In Microsoft’s own words, “Human-operated ransomware campaigns pose a significant and growing threat to businesses and represent one of the most impactful trends in cyberattacks today.” Most targeted real-world attacks use a mix of manual and automated techniques. If organisations’ testing activities can’t emulate this kind of intentional blended attack, there’s no way to account for the risk they pose. 

The final limitation of automated penetration testers is perhaps the most remarkable: they can’t be used on web applications. Right now, they are only effective within the internal network and can’t test the external perimeter. The deficiency this would create within your cybersecurity testing programme doesn’t need an explanation. 

The ideal formula 

The industry has come a long way from the early days of penetration testing, but penetration testing still has a place. Despite claims that “traditional penetration testing is dead”, the long-established process of blending manual and automated attacker emulation remains an effective means for reducing risk and evaluating compliance. Deployed strategically, it can even enable digital transformation and enrich B2B partnerships. What it is not is a catch-all solution to solve every security need. 

The integral truth about penetration testing in the age we work in is that it can’t provide continuous and dynamic assessments that keep pace with how digital environments now change and grow. It isn’t fast enough, nor can it be affordably scaled to this magnitude. This is where new, continuous security testing activities come in. These take the best bits of penetration testing – the scoping; the scanning; the deep, human-led investigation and experimentation; the detailed reports provided with business context – and make them more dynamic. This always-on activity can then be used to underpin your security programme, continuously monitoring and prioritising both simple and more complex vulnerabilities and alerting security teams when action is required. It does not replace penetration testing, but it can help you refine your penetration testing strategy and optimise your costs. 

How does it work? 

Continuous security testing varies from provider to provider but can be loosely outlined as follows: 

  1. A scope is developed to model testing activity around requirements and desired outcomes. 
  2. Testing begins:
    2.1 Vulnerability scanning is deployed across the designated attack surface
    2.2 Security engineers verify scanner alerts
    2.3 Security engineers run vulnerability hunting across the attack surface to supplement scanning and vary coverage

  3. Confirmed vulnerabilities are analysed and prioritised ready for remediation 
  4. Performance and recommended remediation reports are generated 

Unlike penetration testing, this takes place in a non-stop cycle; alerts and other findings are continuously fed back to your team so that patching and other remediation activity can be carried out in a more agile way. You begin to work dynamically, reflecting both the continuous integration/continuous development (CI/CD) process used by application developers and the changing nature of your estate. 

All continuous security testing “runs in the background”, always on. However, whilst some solutions provide very little person-to-person interaction, others build this in as an essential component of the service. For security and IT leaders who want a consultative layer – advice, support, education – to help them use the service to develop their organisation’s security posture, naturally, the latter is best. 

Continuous security testing in response to real security challenges 

The most effective security testing services are the direct result of recognising where improvement is needed and where cybersecurity teams are struggling. We’ve broadly categorised these as follows: 

Business challenge

Provision of continuous security testing

New vulnerabilities emerge daily, constantly adding risk to the business. 

Continuously seek, identify, and recommend remediations for new (known and unknown) vulnerabilities

Potential for unknown risk because of limited security visibility.

Increase visibility of vulnerabilities and other weaknesses across the entire estate.

Limited budget and unfulfilled remediations leading to security debt.

Helps prioritise remediations according to likelihood of exploitation and impact.

New requirements to quantify operational resilience.

Reveals a dynamic view of the organisation’s security and how remediations are having an impact over time.

Create a business-wide security culture.

Analyse new parts of the estate, uncover shadow IT, and align security with development speed.

How to implement cybersecurity testing effectively 

In the book How Great Leaders Deliver High Quality Software & Accelerate Growth, a former Head of Testing at eBay argues that there are two types of application testing:   

  1. Investigating: building hypotheses, testing these, and experimenting to uncover information about a product. 
  2. Verifying: authenticating (confirm/deny) pre-conceived expectations of how a product will behave.   

He goes on to say “The problem with verifying activities is that you can only verify what you know needs to be checked. But once a problem is uncovered, you then have to investigate it. And that is why you will always need some form of manual testing.”   

The same is very much true of cybersecurity testing. Skilled cyber attackers are strategic, creative, and often innovative, which must be reflected in the way organisations test their systems and assets. Only with human testers can you truly emulate the same deliberate, sometimes intuitive, decision-making. Remove that, and you remove whole swathes of information on if, how, and why your organisation could fall victim to a more intelligent, targeted, persistent, and potentially more damaging attack. 

Automation has its place. In cybersecurity testing, it enables penetration testers to deploy single and repeat attacks whilst also taking the heavy lifting out of identification, analysis, and enumeration. These are the tasks that humans don't do so well, and which become unaffordable when deployed manually and scaled. 

Implementing the most effective blend of manual penetration testing and continuous security testing requires a clear security strategy and an understanding of the requisites of cyber resilience. This takes time and the input of different stakeholders. For any security or IT leader, the consultancy of an unbiased partner who can help identify the right mix of tactics is often indispensable.  

To discuss your cybersecurity strategy needs with a team of technical and business specialists, get in touch. To find out more, explore our penetration testing and continuous security testing services. 

Get your attack surface appraisal. Book now button

Previous Article
What is penetration testing?
What is penetration testing?

Penetration testing (aka pentesting) is an offensive security exercise using manual and automated (tooling-...

Next Video
Microsoft Sentinel: what, why, and how it works with your business
Microsoft Sentinel: what, why, and how it works with your business

With an estimated 85%* of organisations expected to embrace a cloud-first principle by 2025, and with new c...