The manufacturing data platform

Manufacturers understand the importance of leveraging technologies to stay ahead in the market. Whether referred to as Industry 4.0 (I4.0), smart manufacturing, or the 4th industrial revolution, data is the biggest asset, says Francisco Almada Lobo, chief executive officer and co-founder at Critical Manufacturing.

With only a small percentage of companies succeeding in digital transformation, however, how can manufacturers take advantage of the massive amounts of data they are creating?

Like oil, data has a great deal of value, but only if it is refined to release the information it holds. To build a transformational data platform requires a combination of the Industrial Internet of Things (IIoT), a future ready manufacturing execution system (MES) and advanced analytical tools.

How smart is your factory?

The ultimate smart factory has action-oriented data and artificial intelligence (AI) controlled production lines. But how do we get there? What solutions are needed? Some predict that the answer lies solely in the IIoT and that MES is no longer required, but this is simply not true. In fact, the MES needs to evolve to include data platforms.

Building a manufacturing data platform

The main elements for a successful Manufacturing Data Platform are a natural extension of the modern MES.

On the edge

A Manufacturing Data Platform combines solutions for processing, storing, and analysing data from huge numbers of resources. Edge solutions run close to the place where data are generated, with some local processing and analysis before sending to a central system. As such, they reduce latency, enable faster responses to changing process conditions and can reduce the costs of central processing and analysis.

Data ingestion

One of the most critical functions of data ingestion is the meta data registry, which enables the platform to understand what data is being sent. The meta data registry refers to a schema. Based on the schema ID, schemas are added to messages before sending them on. Later, an application can read a schema and know which data it contains.

Data brokering

The data platform needs to deliver data from numerous data sources, including equipment, process data, MES and ERP, to applications such as historians, dashboards, alarms, analytics, and data reporting. Data warehouses and subsequently data lakes were used, but, for data platforms, Apache Kafta was a gamechanger.

It decouples data streams from systems and is distributed, fault tolerant, has incredibly high performance, extremely light weight consumers, and easily scales horizontally with the addition of hardware. Kafka acts as a nervous system, managing streams of information from various applications, processing each piece of data, and sending it to where it needs to go and has the dual capability to process data in real-time and ‘replay’ data from any given point in time.

Data processing

Data processing includes batch and stream processing. Batch processing processes large groups of transactions in a single run, involving multiple operations and handling heavy data loads. This may be used to run a report or aggregate data on a data warehouse. Stream processing deals with transformations that require extremely fast handling, usually involving less data.

Higher stream processing speeds and configurable, automatic rule-based actions (e.g. ‘if this then that’) reduce latency between an event and subsequent action, thereby adding value. For exceptional processing speed with in-memory processing, the Critical Manufacturing platform uses the powerful and feature-rich Apache Spark to handle batch and stream processing.

Data enrichment

Data enrichment is invaluable for manufacturing. It merges third-party data from an external authoritative source with the existing database of first-party customer data.

Take the example of the temperature profile of a machine. Alone, there is little analysis that can be done. However, If the system understands the processes being carried out, historical temperature profiles, maintenance activities, etc., more can be understood about the readings.

The MES provides data for enrichment and contains all the necessary contextual information. An event received into the data platform has a name, value, timestamp and MES object. It is written into a raw topic, stored in a data lake and sent into stream processing. An MES data enricher then appends contextual data to the message. This new, event-enriched topic is written back into Kafka, where it can be consumed again by stream or batch processing.

Advanced analytics

Descriptive, diagnostic, predictive and prescriptive analysis help us understand what has happened, why it happened, what will happen and what actions should be taken.

One of the most common uses of predictive analytics is machine maintenance. Data is collected over time from sensors and machine actions and merged with previous maintenance activities. Correlations between variables and results can then be made to determine causes of machine failures.

Predictive analysis then creates a data-driven model to calculate the probability of machine failure or remaining useful life, thereby anticipating maintenance needs or postponing routine maintenance if not required.

Machine learning (ML)

ML is used to analyse large data sets and learn patterns to help make predictions about new data sets. Using Big Data, it requires a data platform that scales accordingly and is the most promising techniques to gain hidden insights and value from data.

ML comprises several levels of analysis. ‘Detection’ of anomalies to identify faulty products, predict machine maintenance needs, and detect possible safety issues. ‘Classification’ then organises information and looks between categories to identify correlations. ‘Probability’ functions test how changes to specific variables will impact outcomes and ‘Optimisation’ can then be achieved by calculating the probability of various outcomes and adjusting parameters accordingly.

Given enough relevant data, learning algorithms can approximate almost any function. Correlation, however, does not imply causation. Initial hypotheses need to be tested for significance, and the more statistically relevant investigated further.

Serving and output applications layers

The final block of the Manufacturing Data Platform makes outputs available to applications such as third-party solutions, alarms, or visualisation tools, through a serving layer into the applications layer.


Francisco Almada Lobo

It is a combination of MES, IIoT, equipment integration and data platform elements that distinguish a Manufacturing Data Platform from a generic one. The use of a data platform designed specifically for the manufacturing environment is a massive accelerator in providing insights to manufacturing processes, continuous improvement, and competitive advantage.

It is a combination of MES and Manufacturing Data Platform that will enable manufacturers to seize the huge advantages I4.0 has to offer.

The author is Francisco Almada Lobo, chief executive officer and co-founder, Critical Manufacturing.

About the author

Francisco Almada Lobo is recognised as a top strategic thought leader and evangelist on digital transformation, specifically Industry 4.0, manufacturing operations and the factories of the future. He holds an MBA and an Electrical Engineering Degree from the University of Porto. He started his career in a CIM R&D Institute and joined Siemens Semiconductor in 1997.

Throughout his tenures at Siemens, Infineon and Qimonda, he specialised in optimising highly complex, discrete manufacturing operations. In 2004, he led the migration of an MES system in a running high-volume facility. Francisco Almada Lobo holds various positions within the smart manufacturing and venture capital industries, including being a Member of the 200M Fund’s Investment Committee, executive committee member of SEMI Smart Manufacturing Technology, member of the Forbes Technology Council and advisor to many Industry 4.0 startups. Francisco Almada Lobo co-founded Critical Manufacturing in 2009 and has been CEO since 2010.

Comment on this article below or via Twitter: @IoTNow_OR @jcIoTnow


9 IoT applications that will change everything

Posted on: September 1, 2021

Whether you are a future-minded CEO, tech-driven CEO or IT leader, you’ve come across the term IoT before. It’s often used alongside superlatives regarding how it will revolutionize the way you work, play, and live. But is it just another buzzword, or is it the as-promised technological holy grail? The truth is that Internet of

Read more

Which IoT Platform 2021? IoT Now Enterprise Buyers’ Guide

Posted on: August 30, 2021

There are several different parts in a complete IoT solution, all of which must work together to get the result needed, write IoT Now Enterprise Buyers’ Guide – Which IoT Platform 2021? authors Robin Duke-Woolley, the CEO and Bill Ingle, a senior analyst, at Beecham Research. Figure 1 shows these parts and, although not all

Read more

CAT-M1 vs NB-IoT – examining the real differences

Posted on: June 21, 2021

As industry players look to provide the next generation of IoT connectivity, two different standards have emerged under release 13 of 3GPP – CAT-M1 and NB-IoT.

Read more

IoT and home automation: What does the future hold?

Posted on: June 10, 2020

Once a dream, iot home automation is slowly but steadily becoming a part of daily lives around the world. In fact, it is believed that the global market for smart home automation will reach $40 billion by 2020.

Read more

Password managers go mainstream and reveal need for C-suite driven security consistency

Posted on: December 7, 2021

London, UK. – Bitwarden, an open source password manager for businesses and individuals, announced the results of its 2022 Password Decisions Survey. The survey, which polled more than 400 U.S. IT decision makers across a wide range of industries, shows that password managers are a near-defacto standard for organisations, with 86% reporting they are being put to use. This reflects

Read more

OCF celebrates pulse systems’ installation of newly certified IoT platform for smart lighting

Posted on: December 6, 2021

The Open Connectivity Foundation (OCF) has announced the first installation of the recently-certified low power IoT platform, from OCF member Cascoda, that combines the end-to-end security benefits of OCF and the low power, wide-area coverage advantages of Thread.

Read more