5 Testing Challenges For Your Test Automation Framework

Jan 14, 2020 9:00:00 AM / by Dora Zambrano

At telco operators around the world, test engineers and operations managers are experiencing a bit of a conundrum. With networks growing in complexity by the day and manual testing getting more time consuming than ever, the need for automation is becoming obvious. At the same time, not all automation is created equal, and testers need to make sure they’re equipped with the right tools for the challenges that await them as modern networks continue to evolve.

Failure crisis concept and lost business career education opportunity. Lonely young man on a rock cliff island surrounded by an ocean storm wavesObviously, everyone wants a solution that’s future-proof—but what does that really mean? On some level, future-proofing is about technology integration, i.e. the ability to adapt to changing technological realities and incorporate new devices, network elements, and protocols as they arise. On another level, future-proofing is a matter of setting testers up for continued success by laying the groundwork for agile, integrated testing workflows.

The right telco test automation solution will do both of these things. How do you know if a particular solution meets those requirements? For starters, it should help you to address these 5 testing challenges.

1. Testcase Reusability

One of the biggest factors correlated with overall testing velocity in an automated environment is how much manual effort has to go into creating and maintaining a library of testcases. Plenty of companies have adopted a new automation solution only to find that each new round of tests required a laborious round of testcase scripting, either because the proprietary coding language is too complicated for easy use or the test cases aren’t designed to be reused in the first place. When that happens, the ROI for test automation can go down significantly as a result of decreased time savings. For this reason, creating reusable testcases is one of the most impactful hurdles that testers can work to overcome. How can an automation solution help with this? For starters, it can offer up an environment in which testcases are easy to script and can be adapted to new testing scenarios as needed. This is the goal of keyword-based testing, for instance, which provides users with a modular scripting language based on pre-defined, readable keywords that are swapped in and out based on the particular test scenario. On top of that, automation frameworks can help by offering ready-made test libraries for common telco use cases.

It’s key that the test results themselves be readable and accessible—otherwise, the most comprehensive tests in the world still won’t lead to speedy bug fixes. 

2. Readable Reporting

This second challenge is directly related to the first one in pretty obvious ways. Just as it’s imperative that you leverage test scripts with a low barrier to entry and relatively easy reuse, it’s key that the test results themselves be readable and accessible—otherwise, the most comprehensive tests in the world still won’t lead to speedy bug fixes. This is another area where keyword-based tests can have real value. How? By leveraging those same keywords that make testcase scripting so easy (so much so that even non-technical users can script up simple testcases) in its reporting. In this way, testers can gain visibility into the results of the automated tests and provide visibility into those tests for coworkers outside their department. Rather than trying to pick relevant details out of dense, opaque reports, testers using keyword-based frameworks can quickly see which tests were run, which ones failed, and what part of the test the failure occurred at. From there, they can find the root causes more quickly and easily.

3. Testing on Flagship Devices

The sections above mostly speak to our second definition of future-proof: laying the operational groundwork for continued testing success. But what about the other side of that coin—the ability to keep pace with emerging technology? For this side of the equation, it’s useful to talk about a given solution’s ability to automate new devices as they come onto the market. Of course, not all of your tests need to be performed with flagship devices—but the ability to use the latest technology is crucial for test labs, and any delay in incorporating flagship devices into a particular automation workflow has the potential to slow down test lab operations. As 5G and the IoT become more and more of a reality for network operators to grapple with, the rate of change for devices is going to speed up considerably, making the ability to automate them quickly more crucial than ever. If your automation provider can’t make that happen, it suggests the possibility of significant operational issues down the road.

4. Automating Mobility Management

This one might seem like somewhat of a niche topic, but in its way it’s suggestive of the kind of synthesis between forward-thinking technology and real telco expertise that you should look for in an automation framework. Mobility management is a particularly tricky use case for a handful of reasons—chief among them the fact that, by definition, you’re testing functionality (e.g. SRVCC handovers from LTE to 2G/3G networks in areas of poor coverage) that only arises in certain poorly-defined geographic areas. Where, once upon a time, testing in this area required lengthy and uncertain drive tests, it’s now possible to achieve the same results within an automated environment using out-of-the-box end-user devices and attenuators. This is just one potential example of the sort of telco-specific use case you should look for when assessing different automation providers’ respective solutions.

5. Achieving High Test Coverage

Last but certainly not least, perhaps the most important challenge that faces modern telco testers is overcoming the growing number and complexity of use cases requiring verification to achieve a high level of test coverage. This is easier said than done in a network environment that’s increasingly inundated with new devices and protocols—but it’s crucial to avoiding disruptions, maintaining high network quality, and retaining your subscribers. Whatever automation framework you’re considering, it should showcase its ability to improve test coverage. Ask any potential vendors how many use cases per day their solution can cover, how quickly they can set up a test environment, and how quickly testcases can be scripted up. The answers to these questions will tell you a lotnot just about the platform's potential future value, but about how useful it will be to your test flows at this very moment.  

Learn more about circuit switched core test automation

Topics: Network Quality, End-to-End Testing

Dora Zambrano

Written by Dora Zambrano

Dora has over 10 years of experience in leading marketing initiatives, project management, content strategy in the IT & Telecom industry. As a Marketing Manager at Segron, she currently leads the marketing strategy across all channels including offline and online actions.