Software testing in Continuous Delivery: beyond unit and integration testing
Many developers are still stuck in the past, using obsolete testing practices from an era when projects focused on long-term releases. In those days, we relied on Quality Assurance (QA) professionals for manual testing, or Test Automation Engineers (TAE) for building automated tests.
Today, every technology company aspires to implement Continuous Delivery, but not all of them are truly prepared to achieve this goal. The pursuit of Continuous Delivery has led to a decrease in the presence of QA professionals within organizations. TAE professionals have also faced challenges, becoming a bottleneck for this Software Engineering practice, since the pace of development often exceeds these professionals’ test automation capacity.
To successfully implement Continuous Delivery, it is essential to automate as many tests as possible, reducing the dependency on manual testing. To achieve this, it is crucial to involve all Software Engineers in building all types of tests, not just unit and integration tests. This aims to balance the relationship between delivery speed and software quality.
When we prioritize Continuous Delivery without implementing all the necessary tests, we compromise software quality and operational continuity, which can introduce new problems in the system, similar to what happened when teams lacked QA professionals.
If organizations wish to do without Quality Assurance (QA) professionals, they must ensure their software has adequate automated test coverage. Otherwise, we would be moving backward instead of forward.
Benefits of software testing
Software testing allows organizations to save time and money, guaranteeing software stability and maintainability, ensuring that features work as intended, and reducing production errors. This, in turn, lowers development and support costs.
The development time for new features is reduced when there is a test suite that ensures the new functionality is complete and ready for delivery. This enables developers to make more accurate time estimates and reduces the probability of bugs. Additionally, overall maintenance costs are reduced and it ensures that previously delivered features continue to work correctly.
Software testing levels
There are various levels of software testing, and each one complements the others. I could go on here about testing patterns, the testing pyramid, or the inverted pyramid (ice cream cone testing), but in the end, it all comes down to the strategy a team or organization adopts to prioritize tests. In my opinion, regardless of the strategy, it is essential to perform them. Therefore, I leave it to each organization to decide what is best for their context and agreements.
Let’s look at the types of tests:
Unit testing
Unit tests are fundamental in software development, as they determine code coverage and should be the only ones to do so. It is common for developers to include other types of tests in coverage, but this distorts the real measurement.
In unit tests, the inputs and outputs of individual code units are verified. Classes or modules are tested completely independently, checking each line of code and its possible variations. It is essential to include exceptions and error scenarios in the tests.
User interface (frontend) components should also be considered, not just the backend. Unit tests are executed using mocks, simulating all necessary integrations to isolate the code unit. This ensures that only the unit in question is being tested, making the tests faster and more efficient.
A good practice to improve the efficiency of this type of test is to follow the SOLID principles.
Integration testing
Integration tests verify the interaction between two or more modules within the system, using mocks to simulate data and external communications.
Personally, I consider these tests the least attractive and, in some cases, they can be redundant depending on the implemented test suite. Well-designed functional tests can easily cover integration test cases. Integration tests become necessary when a specific scenario not covered by functional tests needs to be tested, or when modules are shared with external applications.
Functional testing
Functional tests simulate the complete user experience and can easily be confused with end-to-end tests due to their similarity. The main difference is that functional tests run locally using mocks, while end-to-end tests are performed in a real environment with real data.
In functional tests, the correct operation of all features, use cases, validations, alerts, exceptions, and any other aspect relevant to the application is verified.
Examples:
- Frontend: Sequence of steps (clicks and field inputs) to execute the required action and verify a feature in the user interface (UI).
- REST API: Testing of different endpoints with their input data and responses, validating both data and exceptions.
- Worker: Execution of a job or process, verifying its input data and the responses or changes generated.
- Backend MVC: Similar to REST API tests, but also verifying the rendering of views based on the executed action.
It is essential to have use cases and tests defined by the product team to ensure the correct operation of the application and to establish a minimum set of necessary tests.
End-to-end testing
End-to-end tests are similar to functional tests, but they run in a real environment with real data, usually in a staging environment. It is also possible to run them in production with a controlled test suite.
These tests are fundamental to verify that all components necessary to execute a feature are correctly configured. For example, they can detect the omission of configuring a secret or an environment variable in production, which could cause problems for users.
Exploratory testing
These tests consist of assigning tasks to users to test the application, looking for bugs, security flaws, unexpected behaviors, or unconsidered use cases. Some organizations even offer rewards to users who identify problems.
They are common tests in the User Experience (UX) area. It is a test that encourages continuous learning by the tester, is highly adaptable, and can be used to test any type of software. Exploratory tests often focus on the user experience, which allows discovering usability problems that might not be evident in other tests.
Other tests
Coverage
Generally, an analysis is run with Sonarqube (or another similar tool) to obtain the code coverage result. If this result is equal to or greater than the threshold established by the team or organization, the test is considered passed. Some organizations configure this test as a blocking requirement for deployment.
Contract testing
These tests aim to guarantee the consistency of the inputs and responses of an API throughout the lifecycle of an endpoint, thus preventing problems with future or existing integrations.
Data types and their formats are verified to avoid inconsistencies.
API Lint
These tests ensure that our API complies with REST standards or other necessary rules. This allows developers to work under a common standard, avoiding future problems in endpoint integration and facilitating the understanding and use of the API.
Performance
These are tests that help us evaluate how a system or application behaves under a specific workload. Their main use is to identify and resolve performance issues before the software is released to the public or when we have significant changes.
They help us ensure user satisfaction, plan system scalability, prevent production failures, and optimize resources correctly.
Profiling POD testing
These tests are similar to performance tests, but their main objective is to measure the resource capacity that a POD will require to meet a specific demand. They help us precisely define the amount of CPU, memory, and the scalability strategy for our PODs.
Conclusion
A system should have, at a minimum, Unit testing and Functional testing for production deployment. A unit test coverage of 80% is recommended, and critical features should be covered by functional tests. However, to achieve successful software development with Continuous Delivery, it is necessary to maximize the number of tests possible, thus guaranteeing the quality and operational continuity of the system.
All types of tests should be executed across all services of a system, not just the frontend. This means that functional and end-to-end tests should also be applied to APIs, Workers, and any backend component. Each part of the system must be tested to ensure software quality and operational continuity, leveraging the speed at which deployments are generated through Continuous Delivery.