Citation: Kiya, H.; Nagamori, T.;
Imaizumi, S.; Shiota, S.
Privacy-Preserving Semantic
Segmentation Using Vision
Transformer. J. Imaging 2022, 8, 233.
https://doi.org/10.3390/
jimaging8090233
Academic Editor: Silvia Liberata Ullo
Received: 14 July 2022
Accepted: 28 August 2022
Published: 30 August 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Privacy-Preserving Semantic Segmentation Using
Vision Transformer
Hitoshi Kiya
1,
* , Teru Nagamori
1
, Shoko Imaizumi
2
and Sayaka Shiota
1
1
Department of Computer Science, Tokyo Metropolitan University, 6-6 Asahigaoka, Hino-shi,
Tokyo 191-0065, Japan
2
Graduate School of Engineering, Chiba University, 1-33 Yayoicho, Chiba 263-8522, Japan
* Correspondence: kiya@tmu.ac.jp; Tel.: +81-42-585-8454
Abstract:
In this paper, we propose a privacy-preserving semantic segmentation method that uses
encrypted images and models with the vision transformer (ViT), called the segmentation transformer
(SETR). The combined use of encrypted images and SETR allows us not only to apply images without
sensitive visual information to SETR as query images but to also maintain the same accuracy as that of
using plain images. Previously, privacy-preserving methods with encrypted images for deep neural
networks have focused on image classification tasks. In addition, the conventional methods result in
a lower accuracy than models trained with plain images due to the influence of image encryption. To
overcome these issues, a novel method for privacy-preserving semantic segmentation is proposed
by using an embedding that the ViT structure has for the first time. In experiments, the proposed
privacy-preserving semantic segmentation was demonstrated to have the same accuracy as that of
using plain images under the use of encrypted images.
Keywords:
semantic segmentation; vision transformer; segmentation transformer; privacy-preserving
1. Introduction
Deep learning has been deployed in many applications including security-critical ones
such as biometric authentication and medical image analysis. Generally, data contains
sensitive information, so privacy-preserving methods for deep learning have become an
urgent problem. In particular, data including sensitive information cannot be transferred
to untrusted third-party cloud environments even if they provide a powerful computing
environment. Therefore, it has been challenging to test a deep learning model in cloud
environments while preserving privacy. To address the privacy issue, researchers have
proposed various solutions. However, cryptographic methods such as fully homomorphic
encryption are still computationally expensive [
1
–
3
], and moreover, the encrypted images
cannot be directly applied to models trained with plain images. To protect privacy, federal
learning (FL) has also been studied as a type of distributed machine learning [
4
,
5
]. FL is
capable of significantly preserving clients’ private data from being exposed to adversaries.
However, FL aims to construct models over multiple participants without directly sharing
their raw data, so the privacy of input (query) images is not considered. For these reasons,
numerous learnable perceptual encryption methods have been studied so far for various
applications [
6
–
11
] that have been inspired by encryption methods for privacy-preserving
photo cloud sharing services [
12
]. Most perceptual image encryption methods aim to
realize the secure transmission/storage of images as in [
13
]. In contrast, learnable image
encryption is encryption that allows us not only to generate visually protected images to
protect personally identifiable information included in an image such as an individual or
the time and location of the taken photograph but to also apply encrypted images to a
machine learning algorithm in the encrypted domain.
Perceptual encryption-based methods have been verified to be effective in image
classification tasks [
7
–
9
,
14
], but other tasks such as semantic segmentation have never been
J. Imaging 2022, 8, 233. https://doi.org/10.3390/jimaging8090233 https://www.mdpi.com/journal/jimaging
