Let’s say you’re a telco service provider: after careful deliberation, you decide to migrate your network in order to improve bandwidth for your growing customer base. After some time, the hard part appears to be over—you developed a plan that involved key stakeholders, you sketched out the scope of the migration, and you updated all of your switches and other equipment as needed. Now, it’s just a matter of verifying that you’re still providing all of the services you think you’re providing.
Like virtually everything in the telecom industry, it turns out that this is not as easy as it looks. A human tester can only work through so many test cases in a given day (anywhere from 6 to 10 on average), which means that verifying network quality after a full migration could take inconvenient amounts of time. So, feeling savvy, you decide to automate the tests. Congratulations, you’ve just taken a crucial step towards digital transformation! But even this is not necessarily as simple as it looks. How exactly should you approach the automation of your network migration tests?
1. Establish a Clear Testing Framework
By switching to automated testing, you can begin to cover hundreds of test cases per day easily—a measurable improvement over the fewer-than-a-dozen per test engineer per day that you would be stuck with otherwise. But that doesn’t meant that you’re off the hook when it comes to developing a systematic framework to guide testers through the process and set clear expectations for the rest of the organization. Start by identifying which particular areas are the most critical for service verification, and develop a plan to address those areas first. You could consider abiding by the 80-20 rule, in which the riskiest or most impactful 20% of cases get 80% of your attention and resources. From there, establish a clear timeline for the tests (which should be comparatively easy with the tests automated) and a straightforward hierarchy for who reports to whom and how bugs are going to be addressed as they’re found. This will put you on solid footing as you wade into what may be uncharted territory for your organization.
2. Identify Use Cases
Because network quality is increasingly measured in terms of customer perception, it’s crucial that your migration tests reflect the actual expectations and needs of those customers. On the one hand, this is one of the ways in which automation can really add value: because you’re able to perform so many tests, you can cast a wide net and cover a lot of functionality that matters to your end users. On the other hand, it means that effective testing now relies largely on adopting a use-case based test structure, i.e. a test structure that verifies service for a particular set of real-world uses across different configurations and protocols. Does the new network really add bandwidth for streaming users on iOS-powered devices? What about Android? Is your quality of service maintained across all of the devices that your users utilize? Can users transfer money securely? All of these questions will require answers, which means that before you configure your automated tests you’ll need to identify all of the use cases that are relevant to your network migration and document them for testing.
3. Utilize Out-of-the-Box Devices
Often, when people talk about testing automation they’re talking about tests that are either simulated digitally or run on rooted or “jailbreaked” devices. This might make life easier in some sense for testers, helping them to tick off all of the boxes more quickly, but it comes at the cost of effectively recreating the situations that your users actually care about. Especially with 5G looming, the small differences between modified or simulated devices and the real thing can quickly become amplified. That’s why it’s critical for testers—especially in automated test environment—to make use of out-of-the-box devices of the sort that end users will have access to. Not only does this improve the accuracy with your test device-specific use cases, it also provides a better window into things like latency times that have an outsize impact on customer perception of network quality.
4. Mind Your Documentation
Remember how we said above that you need to start with a clear framework for who does what, when? This is where the scope and precision of your plan becomes particularly important. Especially for something as complicated and multi-faceted as a full network migration, you’ll need to ensure that whatever automation solution you’re using is able to provide clear and understandable documentation as it verifies service and uncovers potential issues. After all, what good is finding all of the bugs in your new network if none of the engineers can replicate those bugs later in order to fix them? Not only does high-quality documentation help you perform maintenance on your switches and other elements of your infrastructure as issues crop up, it also helps lay the foundations for increased efficiency in future tests. For your first go around with automation, things might seem a little confusing or disorganized, but if you document the first set of tests effectively, in future tests you’ll be able to use your past efforts to guide your new workflows.
5. Follow Up With Regression Tests
Okay, you’ve created a test framework and you’ve run through hundreds or thousands of different use cases. Anywhere your network performance didn’t meet expectations was flagged for the relevant department to be looked into. Time to call it a day, right? Not exactly. Because automated migration testing can be performed quickly and repeatably, test engineers have no reason not to follow up their initial migration tests with ongoing regression tests aimed at maintaining a high quality of service. As new updates are rolled out and protocols for network functionality from VoIP to mobile broadband change and evolve, you’ll want to keep testing different use cases on an ongoing basis. In this way, you greatly decrease the likelihood of a large scale service outage or a sudden, unexpected dip in network quality. Even once your network migration is ostensibly complete, automated testing can be an extremely useful tool for keeping tabs on network performance and thereby helping to ensure customer happiness.
LEARN MORE ABOUT TEST AUTOMATION
Are you looking for a solution to automate your service verification that includes both true end-to-end testing and full access to the systems under test.