Multimodal sensor-to-machined surface image diffusion for defect
detection in industrial processes
Jae Gyeong Choi
1
, Yun Seok Kang
2
, Hyung Wook Park
3
, Sunghoon Lim
4
1,2,3,4
50, UNIST-gil, Eonyang-eup, Ulju-gun, Ulsan 44919, Republic of Korea
choil6043@unist.ac.kr
yskang@unist.ac.kr
hwpark@unist.ac.kr
sunghoonlim@unist.ac.kr
ABSTRACT
Generative models, particularly diffusion-based approaches,
have gained significant attention recently due to their ability
to create realistic outputs. Despite their potential, the appli-
cation of these models in manufacturing remains largely un-
explored. This work presents a framework that addresses this
gap by generating machined surface images guided by mul-
tiple sensor inputs in manufacturing. The proposed model
integrates information from multiple sensors with varying
sampling rates using multimodal embedding and employs a
latent diffusion model to translate the fused sensor embed-
ding into an image embedding, which is then converted into
a machined surface image. The effectiveness of the frame-
work is validated using real-world time-series data, including
force, torque, acceleration, sound, collected from various in-
dustrial processes, such as a carbon-fiber-reinforced plastic
drilling process. The results demonstrate the model’s ability
to predict defects from the generated machined surface im-
ages. The proposed approach can potentially revolutionize
prognostics and health management (PHM) in smart manu-
facturing by enabling sensor-guided visual inspection, defect
detection, process monitoring, and predictive maintenance.
1. INTRODUCTION
Prognostics and health management (PHM) have emerged as
a critical aspect of modern manufacturing to improve sys-
tem reliability, reduce maintenance costs, and minimize un-
planned downtime (Lei et al., 2018). Accurately detecting
defects is crucial for effective PHM, enabling proactive main-
tenance and preventing potential failures. However, rely-
ing solely on a single sensor or limited modalities for defect
detection can be challenging due to the complex nature of
manufacturing systems (Choi et al., 2024). Moreover, tradi-
Jae Gyeong Choi et al. This is an open-access article distributed under the
terms of the Creative Commons Attribution 3.0 United States License, which
permits unrestricted use, distribution, and reproduction in any medium, pro-
vided the original author and source are credited.
tional approaches that provide binary predictions (i.e., defect
present or absent) or numerical estimates of defect severity
may not offer sufficient insight into the specific nature and
location of the defects. These limitations hinder the effec-
tiveness of PHM, as they do not provide a comprehensive un-
derstanding of the system’s health status and may not enable
targeted maintenance actions.
Recent advancements in generative models, particularly
diffusion-based approaches, have led to significant break-
throughs in various domains, including text-to-image gener-
ation (Ramesh et al., 2022; Rombach et al., 2022). These
models have demonstrated remarkable performance in cap-
turing complex data distributions and generating high-fidelity
samples. While they have shown outstanding performance
in creating realistic images from textual descriptions, their
potential in manufacturing applications has not been fully
explored. In manufacturing sites, the ability to generate ac-
curate visual representations of machined surfaces based on
sensor data can significantly benefit process monitoring and
predictive maintenance, highlighting potential defects and
facilitating effective PHM.
This work proposes Sensor2Image++, a framework for gen-
erating machined surface images guided by multimodal sen-
sor inputs. Building upon the success of the previous work,
Sensor2Image (Choi et al., 2023), which translates single sen-
sor data into images, Sensor2Image++ excels in synthesizing
high-fidelity machined surface images while effectively ad-
dressing the challenge of integrating information from mul-
tiple sensors with varying sampling rates. This enhancement
ensures that our model comprehensively captures the intri-
cacies of the manufacturing process, thereby advancing the
state-of-the-art in sensor-guided image synthesis. By provid-
ing a powerful tool for PHM, Sensor2Image++ has the poten-
tial to significantly enable defect detection, process monitor-
ing, and predictive maintenance.
1
