What is continuous testing in DevOps?

Historically, QA has started testing the application only when features were pushed to the QA environments. The problem in this approach was that defects were found late in the development process, at which time they proved too time-consuming and expensive to organizations.

With continuous testing (CT), testing is performed early, right from the planning phase. It continues throughout the development process until the production and monitoring phase. This aligns with the “shift-left” paradigm.

With CT, teams get immediate feedback on the application at each stage of the development process. Automated tests run each time a code gets checked in, so teams get quick insights to make informed decisions.

CT also reduces the risk of finding critical defects in production that could cause unnecessary cost, time, and effort for the organization. The key is to test as early as possible and find defects as soon as possible.

1. Identify gaps in the existing test process

Before planning steps to implement CT, you need to get a handle on the existing problems that CT could help solve. Look through your current testing artifacts; your test process, which may include manual, automated, and exploratory testing; the types of production defects found; and the bottlenecks that are preventing the timely release of features with high quality.

This information will help determine how your team can implement CT and what tests need to be included and run in the different stages of the development process.

2. Set up the infrastructure

Implementing a seamless CT pipeline involves having the right infrastructure to support the effort. This may include:

• Realigning the teams
• Unifying development, QA and staging environments with production-like data
• Changing the deployment process
• Setting up automated tests to run each time code gets checked into different branches
• Using service virtualization to test APIs
• Having CI/CD tools such as Travis CI to automate build, deploy and release cycles
• Implementing containerization using Docker and Kubernetes
• Having a cloud infrastructure to deploy and run tests quickly
• Setting up monitoring tools to get more insights into the development and release process

3. Identify the right tools and resources

Teams need to invest in using the right tools for various activities. This could include:

• Testing tools and frameworks for unit testing, UI testing, API testing, performance testing, security testing and accessibility testing
• Version control tools
• CI/CD tools
• Monitoring tools
• Security scanners

Teams also should have skilled employees who can use these tools effectively throughout the development process. They should be capable of analyzing reports, troubleshooting problems, and fixing issues as soon as they occur.

4. Give sufficient training

Having an effective CT pipeline involves a mindset shift. Instead of thinking of testing as something that happens only during a certain time, you need to reimagine testing being an integral part of every single stage of the development process.

This means sufficient training should be given to people to help them adapt to the new way of thinking. Teams have to embrace that shift-left testing will be the norm moving forward.

5. Automate as much as possible

Automation is the glue that binds CT and the DevOps pipeline. When set up the right way, automated tests help reduce the time between release cycles and make them more dependable.

Everything that can be automated should be automated. This may include unit, deployment, integration, smoke, regression, security, and performance tests. Testers play an integral part in the DevOps and CT process by starting automation early in the process so defects can be found sooner rather than later.

6. Continuously monitor results

It is wasteful to set up everything for a seamless CT pipeline and not measure the progress to know whether the implementation is successful.

Here are some steps you can take to continuously monitor the effectiveness of your CT implementation:

• Use daily standups, team meetings, retrospective meetings, and other channels to gather feedback on what is going well or what can be improved
• Have weekly check-ins with your team to analyze and discuss the results of the CT process
• Come up with key performance indicators (KPIs) to measure progress
• Make changes to the CT pipeline based on the gathered feedback

Setting up a good CT process is an iterative approach, and the more you look at the data, the better insights you will get into optimizing the process.

KPIs to track for product quality

Since CT is part of the DevOps process, the metrics that apply to the DevOps process apply to CT as well:

• Test case coverage
• Unit test coverage
• Pass/fail rate
• Number of tests executed
• API pass/fail rate
• Requirements covered by tests
• Blocked test cases
• Percentage of automated test cases
• Successful code builds
• New defects
• Critical defects
• Automated tests prioritized by risks
• Code coverage
• Release deadlines
• Total number of defects discovered
• New API defects found

Using these metrics, teams can measure gaps in the implementation process and take corrective actions immediately.