O&G Company Fuels Safety Monitoring with IoT

The Customer’s Challenge

To minimize risks to workers and equipment, a large U.S. Oil & Gas (O&G) company needed more from their sensor-based safety monitoring system. But adding capabilities like automated alerts and forecasting required building an all-new data streaming architecture, able to handle high performance and uptime requirements and niche O&G data form factors.

phData’s Solution

The phData team delivered an IoT solution into production, providing a unified view of both real-time and historical sensor data — with custom software handling ingestion from the niche WITSML form factor. With forecasting and automated anomaly detection, the company can now act decisively to protect their people, their capital, and their brand.

The Full Story

In the unpredictable world of oil and gas (O&G), the safety of onsite workers is paramount; and with all the variables at play in the extraction process, successful operators know to expect the unexpected. That’s what drives one Fortune 500 O&G company based in the U.S. to do whatever they can to improve their safety monitoring systems.

Their drilling rigs are equipped with a variety of sensors that transmit status readings such as well pressure, flow rate, and temperature to a third-party data vendor. This allows the O&G company’s operations team to monitor status indicators — looking for aberrations that might suggest any potential danger to personnel or equipment — via dashboards on their vendor’s web application.

However, because the company was unable to build additional automation via the vendor tooling, they decided to build a more robust solution that went beyond basic dashboards to actually flag anomalies and send out urgent alerts automatically, in order to minimize risk in the event of an incident. In addition, their data scientists sought a long-term solution to store the sensor data coming in, in order to power historical analyses and predictive analytics to help them preempt future accidents even before they happened.

Complexities of real-time data streaming for O&G

To support monitoring and forecasting across all their drilling sites, the O&G company would need to build out a new data streaming framework that could process the high volume of data coming from their IoT sensors in real time — all while ensuring a consistent, unified view across their internal teams, their third-party data vendor, and their ecosystem of smaller contractors and subcontractors.

Challenges included:

  • A gusher of data volume — The solution needed to process a massive volume and frequency of IoT data from dozens (often hundreds) of wells very day, each of which generates sensor values every single second. To stream that kind of data in real-time, architecture design, technology selection, and performance tuning would all be paramount.
  • O&G industry complexities — The O&G world necessarily involves integrating industry-specific technologies, parsing jargon, and wrangling different subcontractors with differing standards for metric naming and taxonomy. For example, the solution would need to ingest sensor data from the idiosyncratic WitsML form factor used in O&G, which isn’t natively supported by most streaming platforms like Streamsets.
  • High fault-tolerance requirements — Because drill sites are often in remote locations — typically lacking in server and network infrastructure to transmit this sensor data (more often, it might be coming from a laptop) — it would be critical for the solution to seamlessly account for any potential disruption, without losing data or disrupting visibility in such a way that risks a preventable disaster.

Although the O&G company did have a Cloudera data platform in place, they lacked specific data streaming expertise they needed around technologies like Streamsets and Kudu. Realizing they were out of their depths, they turned to phData.

Digging deeper with IoT

The phData team designed a solution architecture to deliver both real-time and historical data into production, using Streamsets and Kudu to handle the sensor data coming in from the field.

  • Hot-to-cold data — phData chose Apache Kudu for storing hot data and HDFS for long-term storage, with Apache Spark, Impala and Kudu all running on Cloudera CDH. They implemented a lambda architecture between Kudu and HDFS for cold data, and a unifying Impala view to query both hot and cold datasets.
  • Data ingestion — phData built a custom StreamSets origin to read the sensor data from the O&G industry’s standard WitsML format, in order to support both real-time alerting and future analytics processing.
  • Designed for fault tolerance — To account for the high likelihood of downtime, the solution includes two Streamsets and two Cloudera pipelines, as well as buffering messages in Kafka, to minimize data loss if one system goes down.
  • Tuned for performance — To process the enormous volume of IoT data coming from all those sites, the team used their deep knowledge of Kudu, HDFS and Streamsets to tune the solution for optimal performance at every layer.

As a result, the company’s monitoring systems and dashboards can continuously query the Kudu table where this data was stored to compare it against the historical data now being compiled by their data science team.

Safer extraction through smarter technology

With the solution designed by phData, the O&G’s company’s safety monitoring system now goes far beyond simple dashboards to include automated anomaly detection and alerting, as well as the foundation for future forecasting and historical analysis.

  • Automated alerts for reliable detection — The ability to automatically trigger safety alerts when an anomaly is detected at a drill site is a boon for worker safety, safeguarding against human error and protecting workers when monitoring teams may be short-handed.
  • Historical data analysis for future worker safety — By rolling cold sensor data into long-term HDFS storage, the O&G’s data science team can now create all-new historical forecasting and analysis capabilities, such as identifying sensor patterns that may indicate a given piece of equipment is likely to begin failing.
  • High fault tolerance for high confidence — With high-redundancy built into their solution architecture, the O&G company can have faith in their safety monitoring systems.
  • Better performance for better accuracy — Supporting a higher frequency of data gives the O&G operations team a more fine-grained view into the status of drilling sites, getting the most out of their sensors’ capabilities to help ensure that no anomaly goes undetected.


With their new automated alerting and historical forecasting capabilities — made possible by the data streaming and IoT solution delivered by phData — the company has vastly improved ability to know what’s going on at their drilling sites. And in the unpredictable world of O&G exploration and drilling, knowledge is power. Detecting issues faster empowers them to act more decisively to protect their people, their capital investments, and their brand.


9 IoT applications that will change everything

Posted on: September 1, 2021

Whether you are a future-minded CEO, tech-driven CEO or IT leader, you’ve come across the term IoT before. It’s often used alongside superlatives regarding how it will revolutionize the way you work, play, and live. But is it just another buzzword, or is it the as-promised technological holy grail? The truth is that Internet of

Read more

Which IoT Platform 2021? IoT Now Enterprise Buyers’ Guide

Posted on: August 30, 2021

There are several different parts in a complete IoT solution, all of which must work together to get the result needed, write IoT Now Enterprise Buyers’ Guide – Which IoT Platform 2021? authors Robin Duke-Woolley, the CEO and Bill Ingle, a senior analyst, at Beecham Research. Figure 1 shows these parts and, although not all

Read more

CAT-M1 vs NB-IoT – examining the real differences

Posted on: June 21, 2021

As industry players look to provide the next generation of IoT connectivity, two different standards have emerged under release 13 of 3GPP – CAT-M1 and NB-IoT.

Read more

IoT and home automation: What does the future hold?

Posted on: June 10, 2020

Once a dream, iot home automation is slowly but steadily becoming a part of daily lives around the world. In fact, it is believed that the global market for smart home automation will reach $40 billion by 2020.

Read more

Workz launches eSIM cloud

Posted on: November 29, 2021

Dubai. 29 November 2021 – IoT solutions provider Workz has expanded its eSIM subscription management solution to offer a GSMA certified cloud-based platform in the US. The platform, which is certified by the GSMA’s Security Accreditation Scheme (SAS) and hosted at Microsoft Azure’s Virginia, USA site enables mobile network operators (MNOs) to remotely manage consumer

Read more

Laiye partners with HUAWEI CLOUD to drive Brazil’s digital transformation

Posted on: November 29, 2021

Laiye, a provider of intelligent automation, and HUAWEI CLOUD, a global provider of cloud services, announced a strategic alliance to drive digital transformation in Brazil through of cloud computing, artificial intelligence (AI) and big data. The alliance will be implemented in the rest of Latin America. 

Read more