DevSecOps Pipeline: Delivery
Over the coming weeks, Cybrary will be posting a series of blogs that correspond with the newly released course, DevSecOps Fundamentals. This post is the fifth in a series covering core principles to assist in the automation of a secure pipeline.
- Securing the Development Cycle
- What Are We Defending?
- Pipeline: Planning and Awareness
- Pipeline: Development
- Pipeline: Delivery
- Pipeline: Deployment
- Pipeline: Operation & Monitor
- Summarize the learned concepts
Please follow the posts and provide feedback or questions to the author via LinkedIn.
In the previous post, we discussed the requirements for the Development phase needed for a modern DevSecOps pipeline. In this article, we start integrating security into the evaluation of the Delivery phase.
The article will cover the following topics:
- Test and release
- Dynamic App Security Testing (DAST)
- Testing business logic flaws
- Automate testing and defect tracking
- Jenkins pipeline orchestration
- Interactive App Security Testing (IAST)
- OWASP DevSecOps maturity model for Delivery
Test and Release
For this blog post, we define the Delivery stage as they transition from Build to Deployment. This stage tests the readiness for release and produces a go/no-go decision. Remember that test procedures should have already been created and the testing tools selected in the planning phase. The tools and testing procedures are used during the delivery phase.
The app will be packaged and will go through various testing, including our security review before the release is ready. If all testing exit gates are satisfied, the app will proceed to the final process before moving to the release. In the release, the artifacts will be packaged and digitally signed. The artifacts will also be tagged with a release and moved to the repository.
Dynamic Application Security Testing (DAST)
In the previous phase (Development), the code was scanned before compiling into the application. In the delivery phase, the app is tested in the running state using DAST tools. This phase involves testing the app, the web server, app server, third-party libraries, images, and any other components that are part of the final release.
Vulnerability scanning can be performed with commercial tools such as Nessus or open-source scanners such as Nexpose or OpenVAS. WebApp scanning can also be performed with commercial tools, but several open-source tools are available, such as Arachni, Nikto, OWASP Zed Attack Proxy (ZAP), or OWASP Xenotix XSS. Testing the strength and quality of encryption can also be performed with Nmap scripts.
These tools will test for known vulnerabilities or patterns of insecure app behavior, which produces vulnerabilities such as Cross-Site Scripting (XSS). Additional tools are available for Content Management Systems such as WordPress and Joomla to look for vulnerable themes or modules. While not recommended for production systems, SQLMap can be used in this phase for an extensive check for SQL injection vulnerabilities. SQLMap is very powerful and can drop web shells and inject data into vulnerable apps/databases that could introduce new vulnerabilities or corrupt existing data.
Business Logic Flaws
DAST tools perform use patterns to check for known types of vulnerabilities based on responses or app behavior. Business logic flaws are more difficult to find since the goal is to identify authorization flaws. Identifying business logic problems will require a strong knowledge of the app to design the security tests. They may involve running the tests with multiple user accounts to identify access to unauthorized portions of the application or data. This phase may also require performing tests against an API to identify endpoints that are not allocated to a specific role. Rapid7 identified several types of logic flaws:
- Cookie tampering
- Business flow bypass
- Profile extraction
- Unauthorized URL access
- Privilege escalations
- And many more.
Automate Defect Tracking
Several open-source tools are available to automate the testing and tracking of defects. SecureCode Box developed a complete solution for testing in the delivery phase with a product that includes multiple security tools and integration with existing pipelines. SecureCode Box contains microservice instances of ZAP, Nmap, Nikto, Arachni, and more. It can integrate with pipeline automation tools such as Jenkins, Travis CI, and Bamboo.
Defect Dojo can complement an existing solution and provide tracking of defects across multiple phases. This means vulnerabilities can be associated with the phase it was discovered to track fixes. It can also be used to identify weaknesses that manifest in different phases or were marked as resolved. Defect Dojo offers bi-directional integration with Jira.
Jenkins Pipeline Orchestration
In the previous blog post, we used the pre-built plugin to execute the test tools and parse the output. To demonstrate the flexibility of Jenkins, we execute the Arachni scanner via the plugin. We then use a custom Python file to parse the results as shown in the example Jenkins pipeline script:
The Arachni scan is performed against our internal Tomcat server hosting the JavaVulnerable app. The Arachni scan can be configured with multiple options, such as the depth of the URLs to scan and the max number of pages to test. Jenkins determines whether the test passed or failed based on the exit code of the executing program. In this case, the Python script parses the JSON to look for vulnerabilities. If any high or critical vulnerabilities are found, it produces a system exit code, sys.exit(-1), which Jenkins identifies as a failure and fails the stage. If no vulnerabilities are identified in the report, then an exit code of 0 is returned to notify Jenkins that the stage should be marked as successful.
The Jenkins pipeline was continued from the previous blog post, and two new stages were added. The ‘DeployDev’ stage copies the .WAR file to the webapps folder of Tomcat to deploy the webapp before Arachni scans the endpoint. In this case, Arachni found some vulnerabilities, which caused Jenkins to fail the stage and stop the build. A full demonstration of the activity is shown in the DevSecOps Fundamentals course.
Interactive Application Security Testing (IAST)
An IAST is an interesting addition to security testing because it replicates some of the testings from the development phase and creates a method for validating DAST scans. The IAST is an application that is deployed inside the app server, in this case, Tomcat. Since the tool runs inside the app server, it has elevated access to the other running webapps. The IAST monitors activities such as input, output, tokens, changes in state, database access, third-party libraries, and more. All of the activity is sent as telemetry to an analysis engine.
The analysis engine contains rules that are run against the telemetry to identify malicious patterns of behavior. The IAST does not perform any scanning, so a DAST tool must be used to scan the webapp. For this example, we used the open-source tool Contrast Community edition:
In a later blog post, where we discuss the running application, we will revisit the IAST tool, which can also perform real-time blocking of an attack when it runs in a Runtime Application Security Protection (RASP) mode. In the RASP mode, the tool monitors for attacks just like an IAST, but instead of passively reporting vulnerabilities, it can stop malicious requests.
OWASP DevSecOps Maturity Model
The OWASP DevSecOps maturity model will be used again, just as it was in the Development phase. The model defines four activities for application tests and nine activities for dynamic depth for applications. At level 2, security unit tests should be performed for important components. A maturity level 3 or 4 requires integration and coverage of all security-related modules. The maturity model defines the need for client-side testing and not just DAST tools running against the server-side components. To reach levels 3 and 4 of the maturity model, security tests must be performed against hidden components and use multiple scanners to provide an overlap of testing. At the highest level of maturity, security tests will be performed against microservices to monitor service-to-service communication.
Cybrary offers courses that cut across the entire DevSecOps topic. You can take classes on secure programming, hacker fundamentals, system administration, cloud certifications, and much more. The knowledge will be fundamental to implementing a secure application across the lifecycle of DevSecOps. Signup for Cybrary to learn more about the topic and stay informed of the continuous release of cybersecurity training at Cybrary.
About the author
Dr. Philip Kulp is a Cybrary Fellow and instructor of several courses on the platform. In his current role as a cybersecurity architect and incident responder, he combines his passion for IT and cybersecurity to develop realistic approaches to secure the enterprise. He performs the roles of an independent assessor, incident responder, and secure code reviewer. Philip seeks opportunities to balance his cybersecurity skills between academic, technical, and compliance roles. He holds the CISSP certification and two Offensive Security certifications of OSCP and OSCE. In his educational capacity, Philip serves as a chair, committee member, and mentor for doctoral students in the Ph.D. and D.Sc. programs at Capitol Technology University. Visit his author page on Cybrary or contact him via LinkedIn.
References  OWASP DevSecOps Maturity Model. Retrieved from: https://owasp.org/www-project-devsecops-maturity-model/