By: Philip Kulp
September 23, 2020
DevSecOps Pipeline: Deployment
By: Philip Kulp
September 23, 2020
Over the coming weeks, Cybrary will be posting a series of blogs that correspond with the newly released course, DevSecOps Fundamentals. This post is the sixth in a series covering core principles to assist in the automation of a secure pipeline.
- Securing the Development Cycle
- What Are We Defending?
- Pipeline: Planning and Awareness
- Pipeline: Development
- Pipeline: Delivery
- Pipeline: Deployment (current post)
- Pipeline: Operation & Monitor
- Summarize the learned concepts
Please follow the posts and provide feedback or questions to the author via LinkedIn.
In the previous post, we discussed the requirements for the Development phase needed for a modern DevSecOps pipeline. In this article, we integrate security into the evaluation of the Delivery phase.
The article covers the following topics:
- Deployment terminology
- Cloud-based delivery in AWS and Azure
- Infrastructure as Code (IaC)
- Jenkins pipeline orchestration
- OWASP DevSecOps maturity model for Deployment
- Verification scanning
For this blog post, the deployment stage is defined as the handoff of the release to the operations staff for running in production. The release is the composition of all binaries and other artifacts needed to run the application. Deploying artifacts in the release include Virtual Machine (VM) images, or if the application is more mature, it could be composed of containers for each of the microservices. Containers are recommended over VM images since they require fewer resources to run, start, and have improved scaling capabilities when leveraged with Kubernetes and Docker. The containers consist of a small operating system with only the minimum libraries and binaries needed to run a service.
Deployment is triggered after a successful gate exit from delivery. Delivery was discussed in the previous blog post and consisted of functional, security, and other tests to verify an application is secure and meets the requirements. Once the review is complete, a release package is created with the required artifacts. The artifacts are delivered to the repository for deployment to the production environment. Depending on the maturity level of the DevSecOps process of the organization, IaC (discussed later in the post) may be used to create the infrastructure, test policies, and perform other automation to prepare the production environment for running the application. Finally, after the application is deployed to the production environment, a post-deployment verification scan is performed. The post-deployment scan verifies the same version of the tested application is deployed into production, and no new vulnerabilities are introduced into the environment.
While some organizations prefer to maintain the development locally with Jenkins, Gitlab, or GitHub, cloud-based solutions are available to integrate with or replace code management. Some cloud-based capabilities are AWS CodePipeline and Azure Pipelines, which can automate the entire process or work with existing tools. For example, AWS Code Commit can function as the source repository to work with BeanStalk to build, test, and deploy. Additionally, organizations can use AWS CodePipeline to maintain the GitHub code, S3 buckets, or other supported environments. Azure Pipelines has similar support with their version of a repository called Azure Repos.
Both AWS and Azure offer software release pipelines with similar functionality to solutions such as Jenkins or GitLab. Organizations may be able to improve their automation and scaling by incorporating all components into a single solution. Other organizations may have an existing source repository and pipeline phases, and moving them would be too disruptive. Furthermore, cloud solutions have build services, such as Azure Kudu, which can build, deploy, and run applications in ASP.NET, PHP, Node, Python, and more. AWS offers BeanStalk with similar support for running webapps developed in most modern programming languages.
Infrastructure as Code (IaC)
IaC can be used to deploy applications, build the entire production infrastructure, and verify compliance. The most mature instance will use IaC to build the development environment's infrastructure and use the same scripts in production. The production system will be immutable, and system administrators will not be allowed to make changes. Any required changes will be made in the development environment and propagate with the release, enabling the code, artifacts, and infrastructure to maintain the same version tags.
Why use IaC to automate all aspects of the release? If development and production are both built from the same code, environment drift can be reduced. Identical environments lessen the chance of unknown errors in production, make deployments easier, and improve resilience. If a disaster occurs, the production environment does not need to be built from backups, delta notes for recent changes, and system administrator's knowledge of changes. Ultimately, using IaC is the only reliable method to deploy services at scale.
IaC can also be used as a verification and compliance tool. The DevSec Hardening Framework offers recipes for compliance configuration. The recipes are available for compliance with PCI DSS, HIPAA, SOX, and others. DevOps teams can use the security recipes to configure databases, web servers, operating systems, and individual components, such as SSH. Other tools, such as Chef InSpec, can be used for Compliance as Code (CaC) to validate the existing configurations to support audit reviews.
Jenkins Pipeline Orchestration
The DevSecOps Fundamentals course continues the Jenkins pipeline script for delivery to include two additional stages: Docker image deployment and post-verification scan. An excerpt from the Jenkins pipeline stage is included in the following image.
A Dockerfile already exists to copy the built application (JavaVulnerableLab.war) into the Tomcat version 9 image. The image is then built with the Dockerfile. Next, the Docker container is created, and a port forwarder is defined to allow the traffic into the container from the network. Finally, the container is started. In a robust environment, the deployment phase would be much more involved, but the simplified version was used here to provide an example of the pipeline orchestration. Once the container is deployed and a curl script is run to verify the port is available, Arachni is executed on the running image to verify no vulnerabilities exist in the production version.
OWASP DevSecOps Maturity Model
The OWASP DevSecOps maturity model will be used again for the Deployment phase. The model defines ten activities for deployment and 17 activities for infrastructure hardening. For deployment activities at level one, the maturity model identifies the need for a defined deployment process and an inventory of running artifacts. At level two, the system should be backed up, and deployment should use trusted images. As the DevSecOps matures to level 3, organizations should rely on rolling updates to avoid downtime and use the same artifacts for dev/prod. Finally, in level four, the system should be mature enough to have blue (testing) and green (live) environments in production to eliminate downtime during deployment. Both environments run the same webapp and infrastructure. The latest version of the webapp is deployed to the blue environment for testing before acceptance and activation as the green environment.
The maturity model defines level one mature systems as containing a segmented network for virtual environment and dev/prod segregation. Level two requires running webapps in a virtual environment, performing a security review of third-party library sources, and hardened clusters. A system reaching level three maturity will require multi-factor authentication, immutable infrastructure, and deployment versioning. Level four maturity reduces system calls in clusters, deploys microservices, and employs Chaos Monkey testing. Chaos Monkey is designed to introduce delays randomly, stop services, and cause other problems to simulate real-world failures. How the system reacts to unknown errors will help identify areas that require additional maturity.
The same scanning tools and procedures used during the delivery phases should be executed again to verify the deployed application has not changed since it was tested. If you are using IaC for deployments, the operations team cannot make changes to production. The continuous monitoring activities will regularly scan assets, but a verification scan should also be used instead of regular scans. The DAST tools will be run against the web application with the same configuration as the delivery phase.
Kubernetes (K8S) can improve an application's security by reducing assets needed to test, limiting ports with microservice segregation, and re-using secured images. Images can be run with Docker, tested, made immutable, and then deployed in production without continuously testing for deviations. K8S manages the provisioning, networking, load-balancing, scaling, and all aspects of the services.
The use of Kubernetes and microservices will drive the division of logical work units, which may simplify the security review of an application since components may be easier to understand and test. Deployment can be accomplished with a trusted image repository, which K8S uses to avoid introducing insecure components into production. K8S supports rolling updates for faster patching by updating the repository image, deploying new versions as the old services are destroyed. The rolling updates avoid downtimes, which allows the operations team to deploy updates without the need to coordinate disruptions.
Cybrary offers courses that cut across the entire DevSecOps topic. You can take classes on secure programming, hacker fundamentals, system administration, cloud certifications, and much more. The knowledge will be fundamental to implementing a secure application across the lifecycle of DevSecOps. Signup for Cybrary to learn more about the topic and stay informed of the continuous release of cybersecurity training at Cybrary.
About the author
Dr. Philip Kulp is a Cybrary Fellow and instructor of several courses on the platform. In his current role as a cybersecurity architect and incident responder, he combines his passion for IT and cybersecurity to develop realistic approaches to secure the enterprise. He performs the roles of an independent assessor, incident responder, and secure code reviewer. Philip seeks opportunities to balance his cybersecurity skills between academic, technical, and compliance roles. He holds the CISSP certification and two Offensive Security certifications of OSCP and OSCE. In his educational capacity, Philip serves as a chair, committee member, and mentor for doctoral students in the Ph.D. and D.Sc. programs at Capitol Technology University. Visit his author page on Cybrary or contact him via LinkedIn.
References  OWASP DevSecOps Maturity Model. Retrieved from: https://owasp.org/www-project-devsecops-maturity-model/