Don’t let the scale of IoT data sink your business

The Internet of Things is conjuring up new challenges. Many IoT services are still in their infancy, and we’re only just hitting the teething problems – one of the most significant of which will be architecting suitable data platforms.

Over the past few years, like many companies under the IoT umbrella, Temetra has undergone an intense period of growth. The volume of data we collect has increased by a factor of five. Temetra now ingests, stores and analyses pressure and flow data from over 12 million water meters, which our customers use to do everything from automatic meter reading to identifying potential leaks, says Paul Barry, director at Temetra.

The reliability of readings and the availability of their data is absolutely critical to our customers, who must be able to access meter data across all their sites at all times. However, delivering optimum service reliability was becoming increasingly difficult as our customer base, and our customer’s businesses grew. For some data types, our underlying PostgreSQL database was starting to show the strain, so we knew we needed to find a solution or risk sacrificing our growth.

The IoT has phenomenal potential to transform many aspects of all our lives – a decade from now almost everything could be connected – but we have found as a business that there are lessons the service providers need to learn. If you can’t scale an application with the infrastructure, the huge datasets introduce new problems associated with managing larger backups, and delayed master-slave replication.geo

In our case, we knew the situation was likely to get more difficult the faster we grew our service, so we began investigating new ways to distribute our data. In 2010, we began looking at the emerging wave of NoSQL technologies, as the scalability they promised seemed geared towards the specific business challenges of the IoT.

NoSQL favours availability, scalability and resiliency having been designed to solve the data challenges of Facebook or Google, which required a database that could write and read data anywhere, while scaling and delivering across large data sets and millions of users.

We knew we needed to handle more and more so-called time series data such as meter readings or alarms produced by transponders that can arrive at high speeds, and so required a database that could support very rapid reads and writes.

During the assessment process our IT team looked at all the main NoSQL players including Riak, MongoDB, Cassandra and Redis. We got to grips with Riak the quickest, and had an evaluation up and running within an hour.

The other serious contender was Cassandra, but in our experience it proved to be time consuming to administer, so we ultimately opted for Riak. The next step was to move our core data from Postgres to NoSQL, although I should note we still run Postgres for certain types of data as it has some very desirable features when dealing with data integrity and transactions.portal

We then went through an extended period of testing which lasted approximately two years, during which time we found that NoSQL was able to scale whenever we added readings and had the potential to become a very resilient foundation to our application.

Like many others in our position we kicked off by using the open source version, but as we got more comfortable with the software, and more of our customers began to rely on it, we moved to the enterprise grade version with per-node support from Basho to provide expertise in the event of any problems.

Like a lot of companies, we’re now interested in pulling in more and more sensor data. In our case it will likely be specific data types that have value to utilities that we can aggregate and analyse alongside the meter data. Now that we have a distributed application platform that can manage a high volume of time-series data reliably without much administration, we’re also interested in seeing how we can add value in the future and take advantage of the development hours that are freed up to work on new features.

Ultimately, our customers are after a boringly reliable service. In addition to our NoSQL platform, we’re looking closely at open source projects such as Apache Spark and Apache Storm, which will allow us to carry out more distributed compute time analysis.mapsearchsqrfadbasvg

Our aim is to continue using these emerging technologies to get the scalability right so we can add millions of meters. Our growth is predictable, so the key is making sure our search and data analysis still works when we get to the next milestone.

Temetra operates a software-as-a-service, delivering edge data management tools to individual, discrete metering networks without the need for costly infrastructure at every site. We built our business back in 2002 around the Internet’s potential to collect and analyse data.

Our web-based Meter Data Management service is designed for water, gas and heat meter utilities, storing a mixture of eye-ball, low-power radio Automatic Meter Reading (AMR), and fixed network high-frequency meter index data, along with meter asset and network management data. Beyond billing, this mass of data allows us to do a number of things, including identifying potential leaks and alerting the relevant parties.

The author of this blog is Paul Barry, director at Temetra.

Comment on this article below or via Twitter: @IoTNow_ OR @jcIoTnow


Smart metering: Load balancing leads to annual savings of almost £5,000 for a large UK bakery

Posted on: August 10, 2022

Energy management is becoming an increasingly important component of business strategy for today’s FMCG companies, aimed largely at reducing energy costs, improving sustainability, and achieving ESG goals. According to the 2020 Deloitte Resources Study, half of industrial companies report incorporating energy management at the corporate strategy level, says Matthew Margetts, director of sales and marketing

Read more

Digi-Key exclusively stocks new XPLR-IoT-1 Kit from u-blox for purchase globally

Posted on: August 10, 2022

9 August 2022 – Digi-Key Electronics, which offers the selection of electronic components in stock for immediate shipment, announced that the XPLR-IoT-1 explorer kit from u-blox, a global technology specialist in positioning and wireless communication, is now available for purchase globally, exclusively from Digi-Key.

Read more

9 IoT applications that will change everything

Posted on: September 1, 2021

Whether you are a future-minded CEO, tech-driven CEO or IT leader, you’ve come across the term IoT before. It’s often used alongside superlatives regarding how it will revolutionize the way you work, play, and live. But is it just another buzzword, or is it the as-promised technological holy grail? The truth is that Internet of

Read more

Which IoT Platform 2021? IoT Now Enterprise Buyers’ Guide

Posted on: August 30, 2021

There are several different parts in a complete IoT solution, all of which must work together to get the result needed, write IoT Now Enterprise Buyers’ Guide – Which IoT Platform 2021? authors Robin Duke-Woolley, the CEO and Bill Ingle, a senior analyst, at Beecham Research. Figure 1 shows these parts and, although not all

Read more

CAT-M1 vs NB-IoT – examining the real differences

Posted on: June 21, 2021

As industry players look to provide the next generation of IoT connectivity, two different standards have emerged under release 13 of 3GPP – CAT-M1 and NB-IoT.

Read more

IoT and home automation: What does the future hold?

Posted on: June 10, 2020

Once a dream, home automation using iot is slowly but steadily becoming a part of daily lives around the world. In fact, it is believed that the global market for smart home automation will reach $40 billion by 2020.

Read more

5 challenges still facing the Internet of Things

Posted on: June 3, 2020

The Internet of Things (IoT) has quickly become a huge part of how people live, communicate and do business. All around the world, web-enabled devices are turning our world into a more switched-on place to live.

Read more

What is IoT?

Posted on: July 7, 2019

What is IoT Data as a new oil IoT connectivity What is IoT video So what’s IoT? The phrase ‘Internet of Things’ (IoT) is officially everywhere. It constantly shows up in my Google news feed, the weekend tech supplements are waxing lyrical about it and the volume of marketing emails I receive advertising ‘smart, connected

Read more
IoT Newsletter

Join the IoT Now online community for FREE, to receive: Exclusive offers for entry to all the IoT events that matter, round the world

Free access to a huge selection of the latest IoT analyst reports and industry whitepapers

The latest IoT news, as it breaks, to your inbox