The top 3 challenges of preparing IoT data

Sean Kandel of Trifacta

The Internet of Things (IoT) is already entrenched in our everyday lives – from wearables and smart watches through to connected TVs and smart home appliances.

Businesses, too, are utilising the technology; in a B2B context, says Sean Kandel, CTO and co-founder of Trifactaconnected devices refer to machines and sensors that are used to track everything from machine performance to maintenance requirements.

For instance, sensor devices might be found on a production line to track the readiness of the machines and automate predictive maintenance. Or, a hospital might use IoT devices for remote patient monitoring, robotic surgery or dispensing medication.

All of these growing sensors, devices, and other connected “things” ultimately mean more data. And lots of it. But with more data come more complex challenges in preparing it. To harness the value of IoT and big data—and deliver innovation-driving insights— industrial organisations must quickly prepare all of this disparate, unstructured data. Below, we’ve named some of the top three challenges in preparing IoT data to leverage it for analysis.

1. Huge volumes of data

International Data Corporation (IDC) market research estimates that IoT devices will create 40,000 exabytes of data by 2020. To keep this in perspective, in the year 2000, three exabytes of information were created globally. That is a lot of data to prepare—and under many current processes, organisations won’t be able to keep up. This is particularly challenging in the industrial world, where manufacturers and other large industrial organisations typically collect billions of data sets from machines, sensors and internal business applications.

Data preparation still accounts for up to 80% of the time and resources involved in any data project, and the more data you add, the more time-intensive that process will become. As organisations take on new IoT data initiatives, it’s important for them to consider new technologies and processes that will allow them to keep up with this huge influx of data.

2. Complexity

Another challenge in preparing IoT data is its complex nature. Often, organisations must not only prepare timestamp or geotag data, but combine it with more structured sources, such as csv files. This complexity is only multiplied when factoring in the rate at which this data is being generated.

Finding a solution to this problem is tricky. The technical resources within an organisation that could handle this complexity are typically limited, and scaling out those resources are costly. Using common data preparation tools like Excel can’t handle this complexity, which leaves skilled analysts locked out of working with this data. Today’s organisations must figure out a way to leverage the resources they have in order to prepare the increasingly complex IoT data.

3. Interoperability

Business computer systems—both hardware and software—aren’t made to exchange or process the vast amounts of complex information pulled from sensors and connected devices. It’s difficult to quickly integrate and enrich machine generated data with data from business applications such as Salesforce and Marketo for example, and other data repositories. Therefore today’s organisations must look for solutions that better allow data to talk to each other, so that the entirety of an organisation’s data can be leveraged.

Data preparation platforms for IoT initiatives

Many organisations that are spearheading IoT initiatives have turned to modern data preparation platforms to ease these challenges. With an intelligent data preparation platform, some of Trifacta’s customers have seen time spent preparing data reduced as much as 90%, while also allowing nontechnical resources to prepare large quantities of complex data themselves. In addition, we have partnered with Sumo Logic to offer clients a solution to prepare complex log data with business application data.

For instance, a large European rail company is using Trifacta to prepare sensor data generated from monitoring 8,000 locomotives across 32,000 miles of rail tracks in order to predict when they required maintenance. Prior to adopting Trifacta, the company was preparing this data ad-hoc across multiple people and with many different tools, which ultimately delayed analysis and their responsiveness to necessary repairs. Now, this company can prepare 100% of complex sensor data and has rapidly decreased their time spent preparing data.

Another customer, Kuecker Logistics Group (KLG), is using the Trifacta platform to prepare a multitude of sensor data generated from warehouses owned by the world’s largest retailers. These customers have extensive, complex supply chain operations and one faulty or inefficient link in the chain can cause a ripple effect downstream.

By using an intelligent data preparation platform, Kuecker has been able to scale data preparation processes without hiring costly developers, which has dramatically improved their efficiency. Now, they’re preparing customer warehouse data and more quickly identifying the necessary changes that need to be made within the warehouses.


IoT data is an exciting opportunity, but its benefits can only be realised with a proper data preparation strategy in place. Organisations must equip their team with data preparation platforms that can handle the volume and complexity of IoT data, as well as understand how this data can and will be joined with other sources across the organisation. By adopting intelligent data preparation solutions, the universe of IoT and big data no longer overwhelms. Sensor data becomes the key to innovation, not an impediment to it.

The author is Sean Kandel, CTO and co-founder, Trifacta.

Comment on this article below or via Twitter: @IoTNow_OR @jcIoTnow


Army’s new next generation squad weapon programme to launch ARC’s weapons intelligence platform

Posted on: August 12, 2022

Washington – Armaments Research Company, Inc, a technology and data company serving national security and public safety customers, announced their Internet-of-Things (IoT) full-stack technology will be introduced in the Next Generation Squad Weapon (NGSW) programme of record, in partnership with Sig Sauer. For the first time in 65 years, the U.S. Army’s initiative will replace

Read more

Connected logistics market to hit $47.6bn valuation by 2029 backed by MaaS for fleet management

Posted on: August 12, 2022

The global connected logistics market stands at a valuation of US$22.2 billion (€21.61 billion) in 2022 and is projected to reach $47.6 billion (€46.34 billion) by the end of 2029. Demand for connected logistics is estimated to increase at a compound annual growth rate (CAGR) of 11.5% over the forecast period (2022-2029).

Read more

9 IoT applications that will change everything

Posted on: September 1, 2021

Whether you are a future-minded CEO, tech-driven CEO or IT leader, you’ve come across the term IoT before. It’s often used alongside superlatives regarding how it will revolutionize the way you work, play, and live. But is it just another buzzword, or is it the as-promised technological holy grail? The truth is that Internet of

Read more

Which IoT Platform 2021? IoT Now Enterprise Buyers’ Guide

Posted on: August 30, 2021

There are several different parts in a complete IoT solution, all of which must work together to get the result needed, write IoT Now Enterprise Buyers’ Guide – Which IoT Platform 2021? authors Robin Duke-Woolley, the CEO and Bill Ingle, a senior analyst, at Beecham Research. Figure 1 shows these parts and, although not all

Read more

CAT-M1 vs NB-IoT – examining the real differences

Posted on: June 21, 2021

As industry players look to provide the next generation of IoT connectivity, two different standards have emerged under release 13 of 3GPP – CAT-M1 and NB-IoT.

Read more

IoT and home automation: What does the future hold?

Posted on: June 10, 2020

Once a dream, home automation using iot is slowly but steadily becoming a part of daily lives around the world. In fact, it is believed that the global market for smart home automation will reach $40 billion by 2020.

Read more

5 challenges still facing the Internet of Things

Posted on: June 3, 2020

The Internet of Things (IoT) has quickly become a huge part of how people live, communicate and do business. All around the world, web-enabled devices are turning our world into a more switched-on place to live.

Read more

What is IoT?

Posted on: July 7, 2019

What is IoT Data as a new oil IoT connectivity What is IoT video So what’s IoT? The phrase ‘Internet of Things’ (IoT) is officially everywhere. It constantly shows up in my Google news feed, the weekend tech supplements are waxing lyrical about it and the volume of marketing emails I receive advertising ‘smart, connected

Read more
IoT Newsletter

Join the IoT Now online community for FREE, to receive: Exclusive offers for entry to all the IoT events that matter, round the world

Free access to a huge selection of the latest IoT analyst reports and industry whitepapers

The latest IoT news, as it breaks, to your inbox