Despite the attention that IoT has garnered over the past years, the mass adoption remains challenging. IoT is an interdisciplinary domain and requires more thorough design process than a traditional IT system. Diversity in applications, lack of standardized framework and security concerns add a layer of complexity over IoT which makes testing and validation process super challenging.
In this post, we are addressing some of these challenges from an IoT Testing point of view. Testing plays a vital role in certifying the system. However, unlike a traditional IT system, IoT poses some specific challenges. If you are a tester tasked with certifying or validating the functionality of an end to end IoT system, then this post is definitely for you.
Introduction
Unlike a traditional IT system, IoT systems are cyber-physical systems involving both humans and machines as end users. Their interaction forms a complex web of M2M (Machine To Machine) and H2M (Human To Machine) transactions.
Right from device firmware, to network interfaces, extending all the way to business logic defined in cloud application and user app, software remains the most critical driver in IoT. Needless to say, testing the various software components is essential to ensure a safe and reliable IoT system.
How different is IoT testing compared to other software testing approaches? How many of the existing tools could we use to test the IoT software? What are the blind spots when it comes to testing IoT systems? These are the kind of questions that we are going to address in this post.
A Typical IoT System Architecture
An IoT system architecture can be split in the following way as shown in this picture below.
Here is a short explanation of each of the stack components.
For a more detailed overview, you can check out the IoT Wikipedia page or this IoT Explained article.
To test an IoT system, it is imperative to split it into a set of logical components. There can be further split, depending on the objective. However, let’s keep the scope at an end-to-end system level.
Setting The Stage For IoT Testing
The first thing to consider, before the commencement of testing an IoT system is: What is the test objective? That is, what are we going to test the system for.
The very first stage of the testing process is to find out the top level test objectives. As an experienced tester, you are probably aware of it. However, if you are a newbie, you can get excited initially. Depending upon your experience with IoT testing so far, you could be in any of these mental states.
If you are completely confused or new to the topic, don’t panic. The key here is to focus on functionality and performance of your IoT system. Let’s look at some analogies to understand this concept.
The Meaning of Functionality
The functionality encompasses the ACTIONS that are performed by a system in response to a TRIGGER or EVENT.
If you want to test a car, then you know what it does in response to your actions on it.
So now your test objective is to verify the functionality of these basic operations of a car that makes it a useful means of personal conveyance.
It's always a good idea to start with a use case and break it down further to define the test objective at a granular level. Broadly, there are two way to achieve this.
Horizontal Split
Horizontally split the system in the form of an end-to-end flow that cuts across all the system components and defines separate objectives for each such flow.
This approach takes a user story involving the entire system and maps it into a flow across the system. Each story becomes an end-to-end flow comprising of steps ( shown by the arrows above) that indicate the flow of messages from one component to other.
Thus the test objective is to test the correctness of the flow and all associated functionality of each component of the system concerning the flow.
As an example, consider the popular AWS IoT button. A user presses the button, and an order for a product gets generated. In this case, the flow constitutes all the steps right from the pressing of the button to the final validation of the user's mobile app updating the product order.
Pros
Cons
Vertical Split
The test objectives are centered around each component of the system with a separate objective for testing the individual component's functionality.
This approach splits the system into its major components. Test objectives are defined for each component separately, such that its functionality is verified against all external interactions.
Pros
Cons
Note: The pros and cons mentioned above are situational. They depend on the testing methodology. We will cover more on this at a later post.
Performance Requirements in IoT
To build a robust system, testing the functionality alone is not enough. There are scenarios beyond the usual functional scope of a system that must be tested.
Take the case of Google. Google can accept a search query from a user and present the results within a few seconds. That's all fine. However, what if, there are millions of users submitting search queries at the same time. Will Google be able to fetch the results within a few seconds for all the users? How would the performance deteriorate if the number of simultaneous queries exceeds the total server capacity? Would the system be able to recover from the outage?
These are the performance aspects that we need to address. So in addition to the base functionality, we have to look for quantifiable aspects that can manifest themselves with a multiplication factor.
In the case of IoT, the performance aspects can be any one of the following
-
1 Scalability - Getting data from one sensor viz-a-viz multiple sensors
-
2 Response Time - TIme to receive data from one sensor at a time viz-a-viz multiple sensors at a time
-
3 Heterogeneous Interfaces - IoT systems are multi-vendor and multi-protocol. We need to make sure that one system could easily interface with another, without any loss of performance.
For an in depth coverage of the real challenges in IoT testing, check out this page
The Best IoT Testing Tools
As already stated, we will focus on the software testing of an IoT system. This means that we will concentrate on the IoT system components that are software driven and have an impact at the system level.
For this reason, we are not covering the tools for testing hardware device that interfaces with sensors and actuators or the network. This is a different domain altogether. We will also not cover the software/firmware testing of hardware devices and gateways since these devices are unit entities. They constitute the IoT endpoint and connectivity components and on their own, they do not have a system-wide impact on the overall IoT system.
Considering the architecture, it is safe to assume that the cloud and user interaction related components have the most notable impact on the performance of an IoT system. These components are what we call as the IoT platform components and they constitute the core of any IoT system.
Let’s take a closer look and review the IoT testing tools for IoT platform components.
The Contenders
The tools we are examining in this review are:
In deciding the evaluation criteria for the tools, we have considered the most essential parameters a company will look for. This is applicable for small and medium-sized companies wanting to build their own continuous integration workflow for their IoT product or even large enterprises wanting to have their own IoT testing practice.
The tools have been reviewed based on the following criterion.
|
Criterion |
Description |
|---|---|
|
Ease of Use |
Ability to onboard the tool quickly and effortlessly |
|
Open Source |
Whether the tool is open sourced |
|
Scalability |
Capacity to simulate virtual IoT endpoints |
|
Functionality Testing |
Ability to cover functional functional aspects apart from load and performance testing |
|
Popularity |
Popularity of the tool |
|
Cloud Support |
Support for on-the-fly cloud deployment |
|
Pricing |
Relative indication of product pricing |
Interpreting the Criteria
Throughout the review, you'll find that for every IoT Testing tool, we have listed the above criterions in a tabular form which represents features with symbols. Here's what the symbols in the tables mean.
|
Symbol |
|||||
|---|---|---|---|---|---|
|
Meaning |
Feature not available |
Feature is available |
Feature rating is low |
Feature rating is medium |
Feature rating is high |
1. BlazeMeter
Blazemeter is one of the popular API testing platforms built on top of open source JMeter and has been around for a long time. It defines its own domain specific language for writing tests that can be converted to JMeter format.
Blazemeter also supports IoT testing through MQTT JMeter plugin. You can learn more about how to perform MQTT based testing with Blazemeter and JMeter here.
|
Ease of Use |
Open Source |
Scalability |
Functionality Testing |
Popularity |
Cloud Support |
Pricing |
|---|---|---|---|---|---|---|
|
5K per instance |
2. IoTIFY
IoTIFY is the only testing tool that was designed from the grounds up to be cloud native focused on realistic IoT simulation instead of only load testing. As a result IoTIFY provides several unique features such as real time GPS coordinates, battery modeling, device to device interaction and a rich set of APIs to create complex functional test cases.
With IoTIFY, the entire test setup can be hosted and operated on the cloud or hybrid environment. Moreover, it supports Jenkins integration and easily integrates with any CI/CD workflow.
|
Ease of Use |
Open Source |
Scalability |
Functionality Testing |
Popularity |
Cloud Support |
Pricing |
|---|---|---|---|---|---|---|
|
50K per instance (Upto 1M) |
3. Gatling
Gatling is yet another tool for load testing of websites and web applications. It also has an enterprise version, called Gatling Frontline which supports end to end test process management with enhanced dashboards for reporting and metrics.
Gatling has an MQTT extension for IoT, however, please be aware that this is a third party extension and not directly supported by the company.
|
Ease of Use |
Open Source |
Scalability |
Functionality Testing |
Popularity |
Cloud Support |
Pricing |
|---|---|---|---|---|---|---|
|
20K per instance |
4. LoadRunner
LoadRunner is one of the most popular and the oldest among all the testing tools mentioned here. It is loved by the DevOps communities all around the world. It supports the most extensive set of protocols for testing any kind of software application built on any platform that you can think of.
LoadRunner support both MQTT as well as CoAP, which make it a more versatile testing tool for IoT, compared to the others. A video of using LoadRunner for IoT testing can be found here
|
Ease of Use |
Open Source |
Scalability |
Functionality Testing |
Popularity |
Cloud Support |
Pricing |
|---|---|---|---|---|---|---|
|
1K per instance |
5. Locust
Locust is a python based tool which is open sourced and designed for scalability. It supports load testing over distributed machines and you could write your own protocol plugin to enable IoT testing with Locust.
Locust provides a nice web-based UI to monitor the test execution in real time. When it comes to performance, Locust is way ahead of the other open source tools in the market.
|
Ease of Use |
Open Source |
Scalability |
Functionality Testing |
Popularity |
Cloud Support |
Pricing |
|---|---|---|---|---|---|---|
|
1K per instance |
All In One Comparison of IoT Testing Tools
If you want to cut the chase then here is the all-in-one comparison of the IoT testing tools discussed above.
|
Ease of Use |
Open Source |
Scalability |
Functionality Testing |
Popularity |
Cloud Support |
Pricing |
|
|---|---|---|---|---|---|---|---|
|
5K per instance |
|||||||
|
50K per instance |
|||||||
|
20K per instance |
|||||||
|
1K per instance |
|||||||
|
1K per instance |
Considering price & popularity, LoadRunner offers the best choice to deploy IoT test bed. However, it does not have a native support for cloud based deployment.
IoTIFY offers the best value in terms of price & features. Moreover, if your IoT platform is running on one of the popular PaaS cloud like AWS & Azure IoT Hub, then IoTIFY has a native support for them. It is build for the cloud era and truly leverage the massive scale and elasticity offered by cloud based application deployment.
