End-to-end tests provide the best confidence that a software functioning for the end user as expected. But they are also the slowest and the most unreliable tests because the results rely on everything: databases, backend, networks, 3rd party integrations, front-end, content, clients.
The slowness can be resolved by parallelization of tests. Parallelization ensure fast feedback loop when end-to-end testing is part of the continuous integration process. However parallelization brings new challenges. Because tests are often assuming a specific state of the application under test, they can easily step on each others toes. Generally this problem is called poor test isolation.
One of our clients was having test flakiness with Paypal’s payment process. Paypal allows only one open session per payment credential. When running tests in parallel, there are several browser sessions open at the same time, doing the same thing. When PayPal tests are run individually everything works perfectly. When running in parallel, we get randomly failing tests because PayPal sessions get logged out.
A solution we created for this, was to use predefined data sets as a collection for the program to choose from. Instead of using hard-coded PayPal credentials per test, we made it possible to create sets of user values, identified as virtual users. The program knows how to select a unique virtual user per browser session. There is no need to modify the test itself in any way. This is how we solved the challenge of simultaneous use of user credentials in several open browser sessions.
This was the first post of a new blog series where we introduce real customer cases that sparked us to create a solution. Real customer cases also help to illustrate the benefits of end-to-end test automation: helping with the development process, as well as play an instrumental part in monitoring your software’s health.
