Testing the Internet of Things? It’s time to plan your test strategy

What is your reaction to a technology that is likely to impact economy by $6.2 trillion by the year 2025 as valued by McKinsey Global Institute? Of course I am referring to the “Internet of Things” or “IOT” as it is commonly called. Though 6.2 trillion may be a long shot, the fact is that IOT has huge market potential that is expected to connect anywhere between 20 to 30 billion devices within the next couple of years.

Some of these devices are consumer-focused like smart phones or industrial devices, for example; medical equipment that can communicate in real time with providers  and insurance companies while monitoring patient health, or cars that will stream driving patterns to auto insurance companies, engine and operational details to service centers or prescribe alternative travel routes by reading traffic patterns. While these changes are real and set to completely alter the way we look at business models and use products, they also pose some real technological challenges; especially testing and validating the functional accuracy of such complex, highly interdependent, data intensive and real-time communicating devices.

Venkata Ramana Lanka
Venkata Ramana Lanka

The world of IOT is all about interconnected devices communicating, analysing terabytes of data and providing information and taking appropriate actions in real time. In short, it is all about smart devices connecting smart cities, communicating with smart buildings and smart industries to provide smart transport, smart energy  and smart health, enabling us a smart life. Smart Buildings could consist of communicating devices that monitor and communicate the status of HVAC (heating, ventilating and AC), lighting, security, and building access. Smart energy could include devices that monitor and communicate the status of UPS, batteries and generators, while smart homes could include devices that monitor and communicate the status of digital cameras, computers, TV’s, lighting and alarm system. Smart health could consist of devices that monitor and communicate the status of MRIs, implants, surgical equipment, lab equipment and other health monitoring devices.

As different types of devices communicate and simultaneously exchange data, relying on a strong analytics engine and act independent of human intervention, testing and certifying such a complex environment will be a huge challenge for organisations. It is critical to implement a fool-proof test strategy. In order to do this, one must consider certain technology factors , including:

  • Pricing model: The standard T&M or fixed bid pricing model, may not be mutually beneficial to the testing team or the client, unless there is already some implementation in place to understand the different dynamics. Pricing models based on output or preferably based on outcome, may be good to start with. Output-based pricing can be measured by the number of integration points tested or the number of use cases tested. Outcome based pricing can be based on the meeting product quality SLA’s. Alternatively, variable pricing models, with a fixed price to charge for infrastructure and software licenses costs along with an output of outcome for work, should also be considered.
  • Skill set: This is a very critical factor, especially considering the complexity of onboarding team members. They must have adequate experience in different domain specific business process, an understanding of different devices and how they communicate, and a good understanding of systems engineering (architecture, algorithms, analytics etc.) The traditional model of onboarding people with only black box testing skills will not work.
  • Test environment management: Factors to consider when setting up an effective testing environment are:
    • Procuring necessary hardware, buying is not an option because there will be too many devices with different versions and newer models will keep getting launched.
    • Integrating the devices to form a network of interconnected systems. For example, hospital management system connected with your car, home and wearable health monitoring systems.
    • Mimic operating devices in real world conditions, (example: heat / cold, power out situations, electronic disturbances etc)
    • Simulating different network capabilities (broadband, 3G, 4G, 5G, low bandwidth etc.)
  • Test data management: This is probably one of the most complex needs in the case of IOT based applications. It involves:
    • Creating terabytes of data that mimics the behaviour of multi-domain specific business processes, like banking specific data, insurance specific data, hospital management system care specific data, patient information, demographic data, device specific data, for example, different devices in a hospital generate different types of data.
  • IOT Standards: There are no industry accepted standards at this point of time. Until these standards are identified and universally adopted, clear agreement on the standards to be adopted must be established. This can include protocols supported, data exchange formats, data periodicity and device performance standards.
  • Data Privacy: This is one of the most critical factors in determining the success of IOT implementations, and requires significant effort from the testing and certification perspective. This means, testing must validate compliance with medical, insurance, banking, financial services and host of other government & industry regulatory frameworks. Think about distributed data getting stored on cloud infrastructure as well as other servers worldwide.
  • M2M & M2H interaction: Machine to machine, and machine to human interactions, relies on a machine’s artificial intelligence capabilities and its ability to interpret situations, based on data and trigger appropriate response. Minimum human interference means, most of the actions are performed automatically, giving less time for human intelligence and emotions to respond.
  • Self-healing systems: In the interconnected world, systems will be designed to auto-heal (recover) and start performing their activities. From a testing perspective, this means designing scenarios to bring down devices and validate that continuity of business processes without failure. This is going to be truly complex as you are dealing with hundreds of thousands of devices and you will need a very meticulous mechanism to sequence series of device failures.

Having carefully considered the above factors, the next important step is to determine the test types. Though no new type of testing is needed, the testing team needs to focus on running different types of tests with varying degrees of detail, in order to reach a decent level of test coverage. Here are some of the common tests that are relevant in the context of IOT.

  • Compatibility and Interoperability Testing (C&I testing): C&I testing focuses on validating data flow between different devices, OS, and following different messaging protocols over different types of networks. Validating compatibility standards and interfaces is a key factor for this testing type.
  • Performance testing: No longer restricted to simple load generation for a small set of business scenarios, IOT takes performance testing to a different dimension. Testing of volumes of varying types of data, running business scenarios over different devices and networks, you will have to set monitors to not only measure response times between different layers in the network, but also to carefully watch system statistics of the devices including, power usage, changes in device temperatures, memory usage, disk space usage etc. Reliability testing, endurance testing, scalability testing, failover and disaster recovery testing are all essential.
  • Analytics and business rules testing: As the success of IOT rests on real time-analysis and provisioning of real time responses, testing analytics algorithms and the ability to validate thousands of critical business rules will pose a huge challenge. Imagine the outcome if a wrongly implemented business rule administered more than the needed dosage of medicine to a patient or an air conditioning unit in a smart building getting switched on during a fire accident.
  • Application security and Data privacy: With pretty much everything we do being connected and information passing between different systems spread geographically, application security and data privacy testing are non-avoidable tests.
  • Accessibility Testing: Smart cities connected to smart buildings and communicating with wearable devices means, you will definitely have to test for accessibility. This will include testing for much more complex regulations beyond what the current WCAG 2.0 standards define.
  • I18N and L10N: This is short form for internationalisation and localisation testing, which is another very important test. Validation of content delivery that adheres to different geographies, cultures and government regulations is essential.
  • Regulatory compliance testing: Testing is never complete without validating for compliance of the application with regulatory requirements, and so this is a crucial testing type.
  • Upgrade testing: You must think about your regression testing strategy when different devices get upgraded or replaced with better solutions.

In summary, while IOT is exciting, it puts a huge responsibility on the testing team. Complexity created by multiple devices, regulations and networks communicating in real-time need a foolproof test strategy. This needs to be supported by a relevant test environment. Testing teams must look at solutions to solve these challenges and have tools to support these tests and frameworks that will be able to run volumes of test automatically and the ability to validate the results.

The writing on the wall is clear, model driven, highly robust automated scripts that can determine what tests to run, based on dynamically changing situations, are needed. Also needed, are solutions to create and manage test environments (something like a TaaS model) that mimic hardware and other devices, as well as a robust test data management solution. Manual testers, with no specialisation or domain knowledge will become redundant unless they reskill themselves with technology and specialisation.

The author is Venkata Ramana Lanka of Virtusa on Testiong the IoT


9 IoT applications that will change everything

Posted on: September 1, 2021

Whether you are a future-minded CEO, tech-driven CEO or IT leader, you’ve come across the term IoT before. It’s often used alongside superlatives regarding how it will revolutionize the way you work, play, and live. But is it just another buzzword, or is it the as-promised technological holy grail? The truth is that Internet of

Read more

Which IoT Platform 2021? IoT Now Enterprise Buyers’ Guide

Posted on: August 30, 2021

There are several different parts in a complete IoT solution, all of which must work together to get the result needed, write IoT Now Enterprise Buyers’ Guide – Which IoT Platform 2021? authors Robin Duke-Woolley, the CEO and Bill Ingle, a senior analyst, at Beecham Research. Figure 1 shows these parts and, although not all

Read more

CAT-M1 vs NB-IoT – examining the real differences

Posted on: June 21, 2021

As industry players look to provide the next generation of IoT connectivity, two different standards have emerged under release 13 of 3GPP – CAT-M1 and NB-IoT.

Read more

IoT and home automation: What does the future hold?

Posted on: June 10, 2020

Once a dream, iot home automation is slowly but steadily becoming a part of daily lives around the world. In fact, it is believed that the global market for smart home automation will reach $40 billion by 2020.

Read more

Partnership with Nokia helps Clavister expand in Australia

Posted on: January 19, 2022

Ornskoldsvik, Sweden. 19 January, 2022 – Clavister, a provider of European carrier-grade cybersecurity solutions for mission-critical applications, announced an order intake from a Railway Infrastructure company in Australia

Read more

Wi-Fi 6E trial in Turkey hailed a success by Turk Telekom

Posted on: January 19, 2022

London, UK. 19 January, 2022 – Wireless Broadband Alliance (WBA) member, Turk Telekom, announced the successful completion of Wi-Fi 6E trials designed to demonstrate how the technology can be used to enhance speed and capacity in a variety of different end-user scenarios. Wi-Fi is vital to the success of Turk Telekom, who has the Wi-Fi footprint in

Read more