It’s no secret that the payments landscape has changed massively over the past decade or so. Consumers expect to be able to pay anywhere, anytime, using any device. First came the surge in online commerce and the myriad of needs for security and authentication it opened. Recently, innovation has shifted to the physical point of sale, with a seemingly endless procession of phone-based “Pay” apps.
Consumers have become conditioned to expect more from these solutions – not only should each one work wherever they decide to use it, but the experience should be seamless and hassle-free. Unfortunately, each new channel also represents a new avenue for fraud, requiring merchants, banks and their service providers to think through and lock down all conceivable opportunities for malfeasance. The evolution of open platforms and API connectivity has reduced the effort required to bring new solutions to market. This is a benefit in that it reduces the demand on already overtaxed IT staffs, but by speeding the pace of change, it also places new burdens on risk and compliance groups.
It seems natural to assume this new landscape would bring along with it new testing regimes. Unfortunately, that hasn’t been the case. Most FIs continue to rely on payments testing procedures that have existed since before the digital revolution we are undergoing today. Such old-school testing involves the running of isolated routines, often performed manually by QA staff in test labs. This manual method leaves many of the systems that touch the transaction flow untested.
Precisely at the time when the number of test scenarios is expanding dramatically, testing capacity faces new challenges. Cost and logistics limit testing range – even in a fantasy world of unconstrained headcount, doubling the size of a test team would NOT halve the execution time of a testing cycle. Each manual test requires its own equipment, including physical devices. As a result, we’ve found that many institutions relying on old-school testing methods achieve only 20% coverage of potential scenarios. Those are pretty frightening odds, when account balances and FI reputations are at stake.
Old school methods largely limit routines to acceptance testing what has already been built. In the new payments universe it’s equally important to test for what wasn’t built, i.e., the unintended collateral consequences of code changes. This is where the value of automated, continuous end-to-end testing becomes apparent. In an iterative, test-driven development process, developers run tests while they’re coding. The ecosystem can be set up to trigger testing whenever a change is made to the system.
An automated approach also enables more effective, comprehensive testing, an important factor in today’s marketplace and one that’s nearly impossible to replicate manually. One new paradigm involves setting up a test harness – an automated framework comprised of software and test data configured to test a program under varying conditions, monitoring its behavior and outputs in a single view.
In evaluating automated testing software alternatives, consider the following:
- In simulating and evaluating various network protocols, some companies offer broader testing to encompass all touchpoints. This generates greater scenario coverage in the same amount of testing time – a big win for IT shops.
- In implementing an automated testing regimen, it’s important that institutions not confuse tools and strategy. Strategy involves people and processes. Understand your automated test strategy first and then look for tooling that facilitates the strategy.
- Understand your existing tools in your testing organization. When possible, leverage existing tooling. Most modern enterprise systems expose services through an API. You don’t have to shop for a full replacement of your existing test management infrastructure. Through those APIs you can implement automation by complementing existing test management systems.
- Examine all manual steps in the testing process. Don’t just focus on test execution. Make sure the testing solution allows other systems to automatically setup test sessions, execute them and access the results. If other systems cannot perform those three operations automatically then the ability to automate the testing workflow will be limited.