Sanoma Pro: Leading platform for virtual learning
Sanoma Pro, the front-runner in digital education in Finland, provides an eLearning platform with study materials, a learning environment, and a web store. The platform is built upon D2L’s Brightspace and extended with multiple vendors’ components. Testing is outsourced to Symbio, a software consulting firm.
Sanoma Pro’s platform:
Problem: Faster releases demanded faster testing
Sanoma Pro faced a sudden release dilemma. Previously, Sanoma Pro had upgraded its platform with vendors’ software once per year. Last year, however, its supplier D2L announced it would switch from yearly to monthly releases. To bring D2L’s latest improvements to users, Sanoma Pro needed to start updating its own platform once a month. Sanoma Pro had six months to speed up its release cycle 12-fold before D2L’s upgrades would start.
Releasing much more often meant that Sanoma Pro had to accelerate testing. Before Usetrace, testing had been done by three to five manual testers. However, manual work was too time-consuming to test the platform thoroughly every month. On top of new features, manual testers had to check that all previous features worked. To test faster, Sanoma Pro chose to automate testing for existing features. Meanwhile, manual work could better focus on new improvements.
Solution: Test automation in the cloud
Sanoma Pro reached out to Symbio to get expert advice on automated testing tools. Automation specialist Leo Hynninen evaluated several options to speed up testing for Sanoma Pro. The top three candidates were Robot Framework, an open-source solution; Squish, a commercial GUI testing tool; and Usetrace, a commercial cloud service.
As Sanoma Pro only had six months to start monthly releases, fast setup was a must. It was estimated that Robot Framework would take three months to install and code before tests could be made. The second option, Squish, did not demand coding to get started, but it required installations. While the two had useful features, Usetrace was deemed best for rapid automation. In Usetrace, testing is done on the web with a visual editor.
“Usetrace was a practical solution. Testing can be started right away in a browser, and most cases don’t require any programming.” Leo Hynninen, Senior Test Engineer, Symbio
Usetrace also made sure that Sanoma Pro’s testing would actually stay on schedule. With complex tools like Robot Framework, automation can fall behind if an expert is suddenly unavailable. In contrast with coding, even a non-technical person can test with Usetrace.
“New people get the hang of Usetrace on the first day. It makes us confident that automation stays up-to-date and everything gets tested on time.”
Anne Kylmälä, Release Manager, Sanoma Pro
Implementation: Finishing quickly with little experience
Symbio’s testing specialist Olli Hautamäki automated Sanoma Pro’s 58 use cases. Although he had little experience in web-based automation, he finished the task in six weeks with Usetrace. With Robot Framework, it would have taken three months only to get started.
The secret to fast automation was test reuse. By splitting tests into reusable modules in Usetrace, he made new tests without having to click through the same user paths again. His very first automated test was a user login. The login check was turned into a module which he added into 80% of Sanoma Pro’s tests.
“I recommend Usetrace for projects that demand fast automation. Even without much experience, it’s possible to automate a large set in a short time.”
Olli Hautamäki, Senior Test Engineer, Symbio
Results: More upgrades, lower testing costs
With Usetrace, Sanoma Pro was able to set up monthly releases on time. In the new release cycle, Sanoma Pro integrates D2L’s Brightspace and vendor components in the integration server each month. Once feature improvements are done, the build moves into QA for final checking before release. Usetrace speeds up testing in three environments: integration, QA and final release.
Sanoma Pro’s monthly release cycle:
Olli Hautamäki now handles Sanoma Pro’s testing on behalf of Symbio. While exploring new features manually, he runs automation in the background to know that all existing features work. With Usetrace, he can test Sanoma Pro’s use cases in 57 minutes across browsers. It would take him three to four days to test the 58 cases manually.
Besides speeding up testing, Usetrace cuts Sanoma Pro’s costs. According to Sanoma Pro, Usetrace saves 220,000€ (61%) per year as compared to the previous, fully manual testing*. Usetrace has also been estimated 186,000€ (83%) less expensive than Robot Framework in the first year of usage**.
“Automation gets done much faster with Usetrace than with coding. It’s really efficient. We’re already looking to harness Usetrace’s automation in our next large project.”
Anne Kylmälä, Release Manager, Sanoma Pro
*) Three testers’ salaries compared with one tester’s salary, Usetrace’s annual license fee, and a training cost.
**) One automation specialist’s salary compared with Usetrace’s annual license fee, a training cost, and the portion of one tester’s time and salary spent on automation.
