In this long technical read Sumit (Sid) Siddarth, Director at Claranet's NotSoSecure, looks at how DevSecOps does not need to cost a fortune. As he explains, freely available open-source tools can be used for organisations to improve their software security posture.
Section One: Introduction
Today, DevOps is enabling organisations to deploy changes to production environments at blazing speeds. There are many different ways this can be achieved, but a typical DevOps process flow could look like this:
- A developer writes code using any development environment of their choice and pushes it to a central source code repository.
- The code is merged into a central repository management tool for the purpose of versioning.
- The CI/CD server then pulls the code from the source code repository and packages the build artifacts/binaries.
- These artifacts/binaries are then pushed onto a binary repository manager against the commit ID.
- These artifacts/binaries are then pulled out to be deployed back into staging and production servers which are spun up using docker.
- The service uptime is monitored regularly using a monitoring service.
For illustration purposes, let’s use the environment and tools below to represent a typical DevOps process. The tools will vary for each organisation but the process depicted here and in the rest of this blog will be more or less the same.
The same process as seen on a “Jenkins Build Pipeline” would look like this:
"When we integrate more security controls, tools, and processes in an automated fashion, our DevOps pipeline metamorphoses into a DevSecOps pipeline."
Adding Security to the mix
Once the application is running in a production or staging environment, it's common for a penetration test to be scheduled. Imagine a scenario where a production application is tested and a high-risk vulnerability, like an SQL injection is discovered. In such a case, the team will be required to re-run the entire DevOps pipeline in order to fix this one vulnerability. A time-consuming process and clearly not an ideal way to deal with the issue.
So, what if we bring security a little closer to the development cycle by embedding security testing within the DevOps pipeline? For instance, the SQL injection could have been easily discovered at an early stage, with the help of a source code scanner and fixed even before packaging the artifacts:
The benefits and implications of shifting left
As a consequence, we are shifting security more towards the left in the DevOps pipeline. And, when we integrate more security controls, tools, and processes in an automated fashion, our DevOps pipeline metamorphoses into a DevSecOps pipeline.
In other words, DevSecOps is the methodology of integrating security tools within the DevOps process in an automated fashion. Importantly, it is also about having the required knowledge around how to use those tools. This necessarily leads to a cultural change in the normal functioning of DevOps, and teams need to be trained so they understand what tools are available, what they can achieve, and how they work. This allows colleagues across the teams to collaborate more efficiently, thereby creating a stronger "security culture". As a result this multicultural, multi-disciplinary, automated security environment makes security everyone's concern and not just that of a single team. This is one of the major drivers of DevSecOps.
Combining expertise, knowledge, and skills
image source: Claranet | NotSoSecure
Touchpoints for Security testing
In a DevOps model, the places where security can be injected are shown below. Once again, the tools chosen at each stage will vary for each organisation. In the examples here, we have used open-source tools and other free alternatives to show how DevSecOps does not need to cost a fortune. Even with the free alternatives and adequate training, amazing results can be achieved to mature the software security standard.
Section Two: Tools of the Trade
Now let’s look at some of the common ways for teams to keep their DevSecOps pipeline rolling. For each area, we’ve suggested a range of open-source tools to help everything move fast and stay secure without breaking the bank.
Sensitive information such as the AWS keys, access tokens, and SSH keys are often erroneously leaked via the public source code repositories due to accidental git commits. This can be avoided by using pre-commit hooks like “Talisman” which checks for sensitive information in the files before commits or push activity.
Open source tools for Pre-commit hooks:
Talisman - https://github.com/thoughtworks/talisman
Crass - https://github.com/floyd-fuh/crass
Git Hooks - https://githooks.com/
Git Secrets - https://git-secret.io/
Pre Commit - https://pre-commit.com/
Detect Secrets - https://github.com/Yelp/detect-secrets
Git Hound - https://github.com/ezekg/git-hound
Truffle Hog - https://github.com/dxa4481/truffleHog
With automation, it is a common practice followed by developers and administrators to store credentials in configuration files and environment variables for accessing various services. However, storing credentials in the files or configuration can lead to exposure of credentials to an unintended audience. This can be segregated by leveraging secret management services like “Hashicorp vault”. This allows segregation of credentials on separate levels and every environment can pull credentials from a specific environment and use it programmatically.
Open source tools for Secrets Management:
Hashicorp Vault - https://www.vaultproject.io/
Torus - https://www.torus.sh/
Keywhiz - https://square.github.io/keywhiz/
EnvKey - https://www.envkey.com/
Confidant - https://github.com/lyft/confidant
AWS Secrets Manager - https://aws.amazon.com/secrets-manager/
Software Composition Analysis
Let’s start with an example here. In September 2017 Equifax, a popular credit rating agency, was breached leading to leakage of 143 million credit card numbers, personally identifiable information (PII), and much more. The vulnerability that was exploited was found in the “Struts2 Web Framework” (due to a missing security patch) on which the primary web application was built. Just like Equifax, a lot of organisations also make use of open source libraries, frameworks, and solutions like Wordpress, Magento, Drupal or even jQuery in which new vulnerabilities are being discovered every day.
image source: snyk
For these reasons, it is necessary to perform an analysis of all the dependencies being utilised in the application and check them for vulnerabilities arising from missing security patches. For Java and .NET applications, we can make use of a tool called “Dependency-Check” which can be run before creating the builds to identify if any vulnerable software is being used in the application. A decision can be made depending upon the number and severity of vulnerabilities to either continue running the pipeline or fail it to fix the vulnerabilities. Below are a few tools to perform a software composition analysis for security vulnerabilities:
Open source tools for Software Composition Analysis:
OWASP Dependency Check - https://www.owasp.org/index.php/OWASP_Dependency_Check
SonaType (Free for Open Source) - https://ossindex.sonatype.org/
Snyk (Free for Open Source) - https://snyk.io/
Bunder Audit - https://github.com/rubysec/bundler-audit
Rubysec - https://rubysec.com/
Retire JS - https://github.com/RetireJS/retire.js
Static Analysis Security Testing (SAST)
Using automated tools to perform a security code review flushes out many low-hanging fruit like SQL injection, Cross-site scripting, Deserialisation vulnerabilities, and many more. For Java based applications we can make use of a tool called “FindSecBugs” which performs an in-depth analysis of the code (without giving out too many false-positives) and gives a comprehensive report for all the vulnerabilities that have been identified in the code. A decision can be made, depending upon the number and severity of vulnerabilities to either continue running the pipeline or fail it in order to fix the vulnerabilities before the pipeline can continue. Below is a list of some open source tools that can be used for SAST.
image source: Source Code Analysis Tool
Dynamic Analysis Security Testing
Web application scanners are an important part in doing a vulnerability assessment of a web application. A good majority of these scanners have API end-points or CLI access which can be leveraged to initiate scanning on target applications. OWASP ZAP is one such tool which can be used for initiating a web application security scan on an application in the QA/Staging environment to iron out a vast number of security misconfigurations such as disclosure of sensitive information in a backup file, insecure HTTP header etc.
Open source tools for Dynamic Analysis Security Testing:
OWASP ZAP - https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project
Arachni Scanner - http://www.arachni-scanner.com/
Nikto - https://cirt.net/Nikto2
Security in Infrastructure as Code
Containerisation solutions like Docker are very popular while building the infrastructure by simply using a few lines of code. As an example, “Docker Hub” is a popular public docker repository from where docker images are pulled to run containers. However, there have been various instances in which these docker images are laced with malware or are riddled with various vulnerabilities as shown in the image below. Therefore, it is extremely important to scan these docker images that are being pulled from the public repository.
image source: snyk
One solution which provides a good insight into the security stature of the Docker containers/images is “Clair”. Clair scans the raw docker images and gives an exhaustive report highlighting the vulnerabilities that exist in the image. It is thus important to run a tool like Clair on the production docker images before deploying them in the infrastructure.
Open source tools for Security in Infrastructure as Code:
Clair - https://github.com/coreos/clair
Anchore Engine - https://github.com/anchore/anchore-engine
Dagda - https://github.com/eliasgranderubio/dagda
Open-Scap - https://www.open-scap.org/getting-started/
Docksan - https://github.com/kost/dockscan
Vulnerability Assessment (VA)
It is a common practice to perform vulnerability assessment on the production systems to identify various services running in the environment and the associated vulnerabilities.
While pointing a VA tool on the servers that have been created using Docker, it would execute the scan only on the service that is being exposed on that host. However, if we attach the tool to the docker network and then execute the scan, it would then give us a good picture of services which are actually running.
This can be done using various solutions like OpenVAS which can easily integrate into the pipeline.
Compliance As Code
Organisations need to apply compliance controls to their IT infrastructure to abide by industry best practices and various regulations like PCI DSS, HIPAA, and SOX. With “Infrastructure as Code” in DevOps, the production environment is never retained, it is always torn down and re-created again and hence it is a strong requirement to test the updated or newly created environment after it has been set up. “Inspec” is one such tool which can help us in performing these tests as we only need to supply a ruby file containing the tests to be conducted in a very simple and lucid manner, which is easy for every audit professional to write and code.
The tools to create a DevSecOps pipeline will generate plenty of vulnerabilities and each will have its own separate format. Because of this, it becomes difficult to manage this data, let alone track and remediate the vulnerabilities. Hence, vulnerability management solutions are at the core of a DevSecOps process where all tools are required to spool their data into those solutions so that it can be centrally managed, triaged, tracked, and remediated.
“ArcherySec” is one such tool which not only has good integration with most of the tools mentioned above but we can also initiate scans such as Zap and OpenVAS through ArcherySec.
Below is a screenshot of the ArcherySec dashboard showing the vulnerabilities aggregated across all the tools.
And here's a list of the tools that ArcherySec supports for pulling data:
Open source tools available for Vulnerability Management:
ArcherySec - https://github.com/archerysec/archerysec
DefectDojo - https://www.defectdojo.org/
JackHammer - https://github.com/olacabs/jackhammer
Alerting and Monitoring
Production applications are always faced with new threats from unknown and unforeseen vectors. This can be mitigated by having an active intrusion monitoring and prevention solution. One such open source solution is the “ModSecurity WAF(Web Application Firewall)” which detects when OWASP Top 10 vulnerabilities - such as SQL injection and Cross-site scripting - are being attempted against the application. Below is a screenshot of ModSecurity in action.
One open source tool for Altering and Monitoring:
ModSecurity WAF - https://modsecurity.org/
In this piece we have focused on the technical aspects of how DevSecOps can operate in an environment. But just having the tools and techniques is not nearly enough. That’s because DevSecOps requires a cultural change to promote a “secure by default” way of working. This can be achieved by creating security champions within each domain, increasing collaboration with the security team, and leadership from the top.
In a world where cybercrime is on the increase, while rapid iteration and development is needed to enable businesses to adapt to the constantly changing demands of their digital consumers, a new way to build in continuous testing is essential for organisations that have adopted DevOps, but don’t yet have agile security measures in place.
That’s why DevSecOps could be the new plug to secure the gaping gap, and the only way to handle “security at scale”.
- DevSecOps can be achieved through integration of simple open source tools and collaboration in existing DevOps process.
- DevSecOps can be fine tuned to eliminate plenty of monotonous security checks by automating them in the DevOps pipeline, giving time to focus on other serious issues.
- With limited resources, DevSecOps can ensure that DevOps "scales securely".
- With DevSecOps, security becomes everybody's responsibility.