SimMIM: a Simple Framework for Masked Image ModelingSimMIM：一种用于掩码图像建模的简单框架Zhenda Xie 1* Zheng Zhang 2* Yue Cao 2*Yutong Lin 3 Jianmin Bao 2 Zhuliang Yao 1 Qi Dai 2 Han Hu2*1 Tsinghua University 2 Microsoft Research Asia 3 Xi’an Jiaotong University¹ 清华大学²微软亚洲研究院⁸ 西安交通大学{t-zhxie,zhez,yuecao,t-yutonglin,jianmin.bao,t-zhuyao,qid,hanhu}@microsoft.comAbstract摘要This paper presents SimMIM, a simple framework for masked image modeling. We have simplified recently proposed relevant approaches, without the need for special designs, such as block-wise masking and tokenization via discrete VAE or clustering. To investigate what makes a masked image modeling task learn good representations, we systematically study the major components in our framework, and find that the simple designs of each component have revealed very strong representation learning performance: 1) random masking of the input image with a moderately large masked patch size (e.g., 32) makes a powerful pre-text task; 2) predicting RGB values of raw pixels by direct regression performs no worse than the patch classification approaches with complex designs; 3) the prediction head can be as light as a linear layer, with no worse performance than heavier ones. Using ViT-B, our approach achieves 83.8% top-1 fine-tuning accuracy on ImageNet-iK by pre-training also on this dataset, surpassing previous best approach by +o.6%. When applied to a larger model with about 650 million parameters, SwinV2-H, it achieves 87.1% top-1 accuracy on ImageNet-1K using only ImageNet-1K data. We also leverage this approach to address the datahungry issue faced by large-scale model training, that a 3B model (SwinV2-G) is successfully trained to achieve state-of-the-art accuracy on four representative vision benchmarks using 4o× less labelled data than that in previous practice (JFT-3B). The code is available at https://github.com/microsoft/SimMIM.本文提出了SimMIM，一个用于掩码图像建模的简单框架。我们简化了最近提出的相关方法，无需采用特殊设计，例如按块掩码以及通过离散VAE或聚类进行 tokenization。为了研究是什么使掩码图像建模任务能够学习到良好的表征，我们系统地研究了框架中的主要组件，并发现每个组件的简单设计都展现出非常强的表征学习性能：1)以适度较大的掩码 patch大小(例如32)对输入图像进行随机掩码，可构成一个强大的 pre-text 任务；2)通过直接回归预测原始像素的RGB 值，其表现不逊于采用复杂设计的patch 分类方法；3）预测头可以轻量至仅使用一个线性层，性能不逊于更重的预测头。使用 ViT-B，我们的方法仅在ImageNet-1K数据集上进行预训练，即可在ImageNet-1K上实现 83.8%的 top-1 Fine-tuning 准确率，比此前最佳方法高出+0.6%。当应用于一个拥有约 6.5 亿个 Parameter的更大模型SwinV2- H 时，仅使用ImageNet-1K 数据，即可在ImageNet-1K上实现 87.1%的 top-1 准确率。我们还利用该方法解决了大规模模型训练面临的数据饥渴问题：一个3B 模型(SwinV2-G)仅使用以往实践(JFT-3B)中1/40的标注数据，便成功训练并在四个具有代表性的视觉基准上取得了最先进的准确率。代码可在 https://github 获取。com/microsoft/SimMIM。1. Introduction1.引言“Masked signal modeling” is one such task that learns to create: masking a portion of input signals and trying to predict these masked signals. In NLP, following this phi-“掩码信号建模”便是这样一种任务，其学习方式为：遮蔽一部分输入信号，并尝试预测这些被遮蔽的信号。在NLP中，遵循这一理-Figure 1: Figure 1. An illustration of our simple framework for masked language modeling, named SimMIM. It predicts raw pixel values of the randomly masked patches by a lightweight one-layer head, and performs learning using a simple l1 loss.图1：图1. 我们提出的用于掩码语言建模的简单框架 SimMIM 示意图。它通过轻量级的单层头部预测随机遮蔽图块的原始像素值，并使用简单的l₁损失进行学习。losophy, self-supervised learning approaches built on the masked language modeling tasks have largely repainted the field [2,12,32], i.e., learning very large-scale language models by using huge amounts of unlabeled data has been shown to generalize well to a broad range of NLP applications.念，基于掩码语言建模任务的自监督学习方法在很大程度上重塑了该领域[2,12,32]，即，利用海量无标注数据学习超大规模语言模型已被证明能够很好地泛化到广泛的NLP 应用中。In computer vision, although there are pioneers leveraging this philosophy for self-supervised representation learning [13, 60, 61], in previous years, this line of work was almost buried by the contrastive learning approaches [8, 21, 51]. The different difficulties of applying this task to the language and visual domains can be explained by the differences between two modalities. One of the differences is that images exhibit stronger locality: pixels that are close to each other tend to be highly correlated [27], so the task can be done by duplicating close pixels rather than by semantic reasoning. Another difference is that visual signals are raw and low-level, while text tokens are human-generated high-level concepts. This raises a question of whether the prediction of low-level signals is useful for high-level visual recognition tasks. A third difference is that the visual signal is continuous, and the text token is discrete. It is unknown how classification-based masked language modeling approaches can be adapted to handle continuous visual signals well.在计算机视觉中，尽管已有先驱利用这一理念进行自监督表征学习[13，60，61]，但在过去几年中，这一研究方向几乎被对比学习方法[8,21，51]所掩盖。将该任务应用于语言和视觉领域时所面临的不同困难，可以通过两种模态之间的差异来解释。其中一个差异是，图像表现出更强的局部性：彼此接近的像素往往具有高度相关性[27]，因此该任务可以通过复制邻近像素来完成，而非依赖语义推理。另一个差异是，视觉信号是原始且低层级的，而文本Tokens是由人类生成的高层级概念。这引出了一个问题：对低层级信号的预测是否对高层级视觉识别任务有用。第三个差异是，视觉信号是连续的，而文本Token是离散的。尚不清楚基于分类的掩码语言建模方法如何能够适配，以良好地处理连续的视觉信号。Until recently, there have been trials that attempt to bridge modality gaps and resolve the obstacles, by introduc-直到最近，才出现了一些尝试通过引入来弥合模态差距并解决这些障碍，ing several special designs, for example, by converting continuous signals into color clusters [7], by patch tokenization using an additional network [1], or by a block-wise masking strategy to break short-range connections [1], etc. Through these special designs, the learned representations proved to be well transferable to several visual recognition tasks例如，通过将连续信号转换为颜色簇[7]、使用额外网络进行图块标记化[1]，或采用块级掩码策略以破坏短程连接[1]等多种特殊设计。通过这些特殊设计，所学习的表示被证明能够很好地迁移到多个视觉识别任务中。In contrast to requiring special complex designs, in this paper, we present a simple framework which aligns well with the nature of visual signals, as shown in Figure 1, and is able to learn similar or even better representations than previously more complex approaches: random masking of input image patches, using a linear layer to regress the raw pixel values of the masked area with an l1 loss. The key designs and insights behind this simple framework include:与需要特殊复杂设计的方法相比，本文提出了一个简单框架，该框架与视觉信号的本质高度契合，如图1所示，并且能够学习到与先前更复杂方法相当甚至更好的表示：对输入图像图块进行随机掩码，使用线性层以l1损失回归被掩码区域的原始像素值。该简单框架背后的关键设计与洞见包括：- Random masking is applied on image patches, which is simple and convenient for vision Transformers. For masked pixels, either larger patch size or higher masking ratio can result in a smaller chance of finding visible pixels that are close. For a large masking patch size of 32, the approach can achieve competitive performance in a wide range of masking ratios (10%-70%). For a small mask patch size of 8, the masking ratio needs to be as high as 8o% to perform well. Note that the preferred masking ratios are very different from that in the language domain, where a small masking ratio of o.15 is adopted as default. We hypothesize that different degrees of information redundancy in two modalities may lead to the different behaviors.在图像块上应用随机掩码，这对于视觉Transformer而言简单且方便。对于被掩码的像素，更大的图像块大小或更高的掩码比例都会降低找到邻近可见像素的概率。当掩码图像块大小较大(32)时，该方法可在较宽的掩码比例范围内(10%-70%)取得具有竞争力的性能。当掩码图像块大小较小(8)时，掩码比例需要高达80%才能获得良好性能。请注意，优选的掩码比例与语言领域中的情况截然不同，后者默认采用较小的0.15掩码比例。我们假设，两种模态中不同程度的信息冗余可能导致了这种不同的行为。- A raw pixel regression task is used. The regression task aligns well with the continuous nature of visual signals, which possesses ordering property. This simple task performs no worse than the classification approaches with classes specially defined by tokenization, clustering, or discretization采用原始像素回归任务。该回归任务与视觉信号的连续性及其所具有的有序性质高度契合。与通过分词、聚类或离散化专门定义类别的分类方法相比，这一简单任务的表现并不逊色。- An extremely lightweight prediction head (e.g., a linear layer) is adopted, which achieves similarly or slightly better transferring performance than that of heavier prediction heads (e.g., an inverse Swin-B). The use of an extremely lightweight prediction head brings a remarkable speedup in pre-training. In addition, we note that a broad range of target resolutions (e.g., 122 -962) perform competitive with the highest 1922. While heavier heads or higher resolutions generally result in greater generation capability, this greater capability does not necessarily benefit down-stream fine-tuning tasks.采用极轻量级的预测头（例如线性层），其迁移性能与更重型的预测头（例如逆向Swin-B）相当或略优。使用极轻量级预测头可显著加速预训练。此外，我们注意到，较宽范围的目标分辨率（例如12²- 96²）与最高的192²具有竞争力。尽管更重型的头或更高的分辨率通常会带来更强的生成能力，但这种更强的能力并不一定有利于下游 Fine-tuning 任务。Though simple, the proposed SimMIM approach is very effective for representation learning. Using ViT-B, it achieves 83.8% top-1 fine-tuning accuracy on ImageNet-1 K , surpassing previous best approach ([1]) by +0.6%. SimMIM has also shown to be scalable to larger models: with a SwinV2-H model (658M parameters) [33], it achieves 87.1% top-1 accuracy on ImageNet-1K classification, which is the highest number among methods that use ImageNet-1K data only. This result encourages the use of self-supervised learning to address the increasing datahungry problem caused by quickly rising model capacity. In fact, with the help of SimMIM, we successfully trained a SwinV2-G model with 3 billion parameters [33] using ~40× smaller data than that of Google’s JFT-3B dataset, and set new records on several representative benchmarks:84.0% top-1 accuracy on ImageNet-V2 classification [42], 63.1/54.4 box/mask mAP on COCO object detection [6,31], 59.9 mIoU on ADE2oK semantic segmentation [52, 63], and 86.8% top-1 accuracy on Kinetics-400 action recognition [28,35].尽管简单，所提出的 SimMIM 方法在表征学习方面非常有效。使用 ViT-B，它在ImageNet-1 K上实现了 83.8%的top-1 Fine-tuning 准确率，超过了此前最佳方法（[1]) +0.6%。SimMIM 还展现出可扩展至更大模型的能力：使用SwinV2-H 模型(658M Parameter)[33]，它在 ImageNet-1K 分类上实现了 87.1%的 top-1 准确率，这是仅使用ImageNet-1K数据的方法中最高的结果。这一结果鼓励使用自监督学习来应对因模型容量快速增长而导致的数据饥渴问题。事实上，借助SimMIM，我们成功地使用比 Google的JFT-3B 数据集小~40×的数据训练了一个拥有30亿Parameter 的 SwinV2-G 模型[33]，并在多个具有代表性的基准上创下新纪录：在ImageNet-V2 分类[42] 上达到84.0% top-1 准确率，在 COCO 目标检测 [6,31] 上达到 63.1/54.4 box/mask mAP，在 ADE20K 语义分割 [52, 63]上达到 59.9 mloU，以及在 Kinetics-400 动作识别 [28,35] 上达到 86.8%的 top-1 准确率。While in recent years we have witnessed an increasing overlap between NLP and computer vision in both basic modeling and learning algorithms, as well as in multi-modal applications, which aligns well with how human brains achieve general intelligence capabilities, we hope that our demonstration of “masked signal modeling" in computer vision can drive this trend a bit further and encourage deeper interaction of different AI fields.近年来，我们见证了NLP与计算机视觉之间在基础建模和学习算法以及多模态应用方面日益增多的交集，这与人类大脑实现通用智能能力的方式非常契合。我们希望，在计算机视觉中对“掩码信号建模”的展示能够进一步推动这一趋势，并促进不同 AI 领域之间更深入的交互。2. Related Work2. 相关工作Masked language modeling (MLM) Masked language modeling [12,32] and its auto-regressive variants [2] are the dominant self-supervised learning approaches in the field of natural language processing (NLP). Given visible tokens in a sentence or a sentence pair / triplet, the approaches learn representations by predicting invisible tokens of the input. This line of approaches has repainted the field since about 3 years ago [12], that it enables the learning of very large language models and generalizes well on broad language understanding and generation tasks by leveraging huge data.掩码语言建模(MLM) 掩码语言建模[12,32]及其自回归变体[2]是自然语言处理(NLP)领域中主流的自监督学习方法。给定一个句子或句子对/三元组中的可见 Tokens，这些方法通过预测输入中不可见的 Tokens 来学习表示。自约 3年前以来，这一类方法重塑了该领域[12]：它能够利用海量数据学习非常大的 Large Language Model，并能在广泛的语言理解和生成任务上实现良好的泛化。Masked image modeling (MIM) Masked image modeling [7, 13, 23, 38, 47] progressed in parallel with the MLM task in NLP but located in a non-mainstream position for a long time. The context encoder approach [38] is a pioneer work in this direction, which masks a rectangle area of the original images, and predicts the missing pixels. CPC [23,47] predicts patches via a verification task in each batch with a contrastive predictive coding loss. Recently, iGPT [7], ViT [15] and BEiT [1] recall this learning approach on the modern vision Transformers, and show strong potential in representation learning by introducing special designs on some components, such as clustering on pixels [7], prediction of mean color [15], and tokenization via an additional dVAE network with a block-wise masking strategy [1]. In contrary to these complex designs we present an extremely simple framework, SimMIM, which shows similar or even slightly better effectiveness.掩码图像建模(MIM) 掩码图像建模[7，13， 23， 38， 47] 与 NLP 中的 MLM 任务并行发展，但长期处于非主流位置。上下文编码器方法[38]是该方向的开创性工作，它对原始图像中的矩形区域进行掩码，并预测缺失的像素。CPC[23,47] 通过每个批次中的验证任务，利用对比预测编码损失来预测图像块。近年来，iGPT[7]、ViT[15] 和 BEiT[1] 在现代视觉Transformer上重新采用了这种学习方法，并通过在某些组件中引入特殊设计展现出强大的表征学习潜力，例如对像素进行聚类[7]、预测平均颜色[15]，以及通过额外的 dVAE 网络结合块级掩码策略进行标记化[1]。与这些复杂设计相反，我们提出了一个极其简单的框架SimMIM，其效果相当甚至略优。Reconstruction based methods are also related to our approach, particularly the auto-encoder approaches [24, 30, 37,41,49,50]. Similar as in our approach, they adopt a reconstruction task to recover the original signals. However, they are based on a different philosophy of visible signal reconstruction, other than the creation or prediction of invisible signals as in our approach. They thus progress in a very different path, by studying how to effectively regularize the task learning byproper regularization or architecture bottlenecks.基于重建的方法也与我们的方法相关，特别是自编码器方法[24，30，37,41,49,50]。与我们的方法类似，它们采用重建任务来恢复原始信号。然而，它们基于的是一种不同的可见信号重建理念，而非像我们的方法那样生成或预测不可见信号。因此，它们沿着一条截然不同的路径发展，通过研究如何借助恰当的正则化或架构瓶颈来有效地正则化任务学习。Image inpainting methods Beyond representation learning, masked image modeling is a classical computer vision problem, named image inpainting. This problem has been extensively studied in computer vision for a long time [39, 55, 56], aiming for improving the inpainting quality and without connecting to self-supervised representation learning. While we advocate image inpainting as a strong self-supervised pre-text task, we also find stronger inpainting capability does not necessarily leads to stronger finetuning performance on down-stream tasks图像修复方法除了表征学习之外，掩码图像建模还是一个经典的计算机视觉问题，称为图像修复。该问题长期以来已在计算机视觉领域得到广泛研究[39，55，56]，旨在提高修复质量，而未与自监督表征学习建立联系。尽管我们提倡将图像修复作为一种强大的自监督前置任务，但我们也发现，更强的修复能力并不一定会带来下游任务上更强的微调性能。Compressed sensing The approach in this paper is also related to compressed sensing [14], which affirms most of the data we acquire including image signals can be thrown away with almost no perceptual loss. Such claim is also partly supported by recent works of sparse inference [2o] that the recognition accuracy has very little drop after throwing a large portion of image features [25,43,54]. The observation in this paper goes further for the input signals, that with an extremely small portion of randomly selected input image patches as input, i.e., 10%, the inpainting task can still be learnt to produce good visual representations.压缩感知本文的方法也与压缩感知「14]相关，该理论指出，我们获取的大多数数据（包括图像信号）都可以在几乎不造成感知损失的情况下被丢弃。近期关于稀疏Inference的研究工作[20]也在一定程度上支持了这一主张：在丢弃大部分图像特征后，识别准确率仅有极小幅度的下降[25,43,54]。本文的观察进一步延伸至输入信号：仅使用极小一部分随机选择的输入图像块作为输入，即10%，仍然可以学习图像修复任务并产生良好的视觉表征。Other self-supervised learning approaches During the last two decades, there have been numerous pretext tasks to learn visual representation in a self-supervised way: grayscale image colorization [6o], jigsaw puzzle solving [36], split-brain auto-encoding [61], rotation prediction [18], learning to cluster [4]. Though very different from masked image modeling, some of them interestingly also follow a philosophy of predicting the invisible parts of signals, e.g., [6o,61] use one or two color channels as input to predict values of other channels. Another large portion of works lie in the contrastive learning approaches [3,8,16,19,21,51,53], which are the previous mainstream. We hope our work can encourage the study of masked language modeling as a pretext task for self-supervised visual representation learning.其他自监督学习方法在过去二十年中，已经出现了许多用于以自监督方式学习视觉表征的预训练任务：灰度图像着色[60]、拼图求解[36]、分脑自编码[61]、旋转预测[18]、学习聚类[4]。尽管与掩码图像建模非常不同，其中一些方法有趣地也遵循了预测信号不可见部分的理念，例如，[60,61]使用一个或两个颜色通道作为输入来预测其他通道的值。另一大类工作属于对比学习方法[3,8,16,19,21,51,53]，它们是此前的主流。我们希望我们的工作能够促进将掩码语言建模作为自监督视觉表征学习预训练任务的研究。3. Approach3. 方法3.1. A Masked Image Modeling Framework3.1. 掩码图像建模框架Our approach SimMIM learns representation through masked image modeling, which masks a portion of input image signals and predicts the original signals at masked area. The framework consists of 4 major components:我们的方法 SimMIM 通过掩码图像建模学习表征：它掩盖部分输入图像信号，并预测被掩盖区域中的原始信号。该框架由4 个主要组件组成：Masking strategy. Given an input image, this component designs how to select the area to mask, and how to implement masking of selected area. The transformed image after masking will be used as the input.掩码策略。给定一张输入图像，该组件设计如何选择要掩盖的区域，以及如何对所选区域实施掩码。掩码后的变换图像将作为输入。Encoder architecture. It extracts a latent feature representation for the masked image, which is then used to predict the original signals at the masked area. The learnt encoder is expected to be transferable to various vision tasks. In this paper, we mainly consider two typical vision Transformer architectures: a vanilla ViT [15] and Swin Transformer [34].编码器架构。它为掩码图像提取潜在特征表示，随后用于预测掩码区域的原始信号。所学习的编码器应能够迁移到各种视觉任务中。本文主要考虑两种典型的视觉 Transformer 架构：标准 ViT [15] 和 Swin Transformer[34]。Prediction head. The prediction head will be applied on the latent feature representation to produce one form of the original signals at the masked area.预测头。预测头将应用于潜在特征表示，以生成掩码区域中原始信号的一种形式。Prediction target. This component defines the form of original signals to predict. It can be either the raw pixel values or a transformation of the raw pixels. This component also defines the loss type, with typical options including the cross-entropy classification loss and the l1 or l2 regression losses.预测目标。该组件定义要预测的原始信号形式。它可以是原始像素值，也可以是原始像素的变换形式。该组件还定义损失类型，典型选项包括交叉熵分类损失以及l1 或l2 回归损失。In the following subsections, we will present typical options of each component. These options are then systematically studied. By combining simple designs of each component, we have been able to achieve strong representation learning performance.在以下小节中，我们将介绍每个组件的典型选项。随后将对这些选项进行系统研究。通过组合每个组件的简单设计，我们能够实现强大的表征学习性能。3.2. Masking Strategy3.2. 掩码策略For input transformation of masked area, we follow the NLP community [12,32] and BEiT [1] to use a learnable mask token vector to replace each masked patch. The token vector dimension is set the same as that of the other visible patch representation after patch embedding. For masking area selection, we study the following masking strategies (illustrated in Figure 2):对于掩码区域的输入变换，我们遵循 NLP 社区 [12,32] 和 BEiT [1] 的做法，使用一个可学习的 mask Token 向量来替换每个被掩码的 patch。该 Token 向量的维度设置为与经过 patch Embedding 后其他可见 patch 表示的维度相同。对于掩码区域选择，我们研究了以下掩码策略（如图2所示）：Patch-aligned random masking We first present a patch-aligned random masking strategy. Image patches are the basic processing units of vision Transformers, and it is convenient to operate the masking on patch-level that a patch is either fully visible or fully masked. For Swin Transformer, we consider equivalent patch sizes of different resolution stages, 4 × 4 ～ 32 × 32, and adopt 32 × 32 by default which is the patch size of the last stage. For ViT, we adopt 32×32 as the default masked patch size.Patch 对齐的随机掩码 我们首先介绍一种 patch 对齐的随机掩码策略。图像 patch 是视觉 Transformer的基本处理单元，在 patch 级别执行掩码操作较为方便，即一个 patch要么完全可见，要么完全被掩码。对于 Swin Transformer，我们考虑不同分辨率阶段的等效 patch 大小，4×4˙∼32×324 \times \dot { 4 } \sim 3 2 \times 3 24×4˙∼32×32并默认采用 32 × 32，即最后一个阶段的 patch大小。对于 ViT，我们采用 32×32 作为默认的被掩码 patch 大小。Other masking strategies We also try other masking strategies in previous works: 1) [38] introduces a central region masking strategy. We relax it to be randomly movable on the image. 2) [1] introduces a complex block-wise masking strategy. We try this mask strategy on two masked patch sizes of 16 × 16 and 32 × 32.其他掩码策略我们还尝试了先前工作中的其他掩码策略：1）[38]提出了一种中心区域掩码策略。我们将其放宽为可以在图像上随机移动。2)[1]提出了一种复杂的分块掩码策略。我们在两种被掩码 patch 尺寸16×16 和32×32上尝试了该掩码策略。Raw imageSquare (32)Block-wise (16)Random (4)Random (8)Random (16)Random (32)Figure 2: Figure 2. Illustration of masking area generated by different masking strategies using a same mask ratio of o.6: square masking [38], block-wise masking [1] apply on 16-sized patches, and our simple random masking strategy on different patch sizes (e.g., 4, 8, 16 and 32).图 2：图 2。使用相同掩码比例 0.6 时，不同掩码策略生成的掩码区域示意图：方形掩码[38]、应用于大小为16 的patch 的分块掩码[1]，以及我们在不同 patch 尺寸（例如 4、8、16 和 32)上的简单随机掩码策略。3.3. Prediction Head3.3. 预测头The prediction head can be of arbitrary form and capacity, as long as its input conforms with the encoder output and its output accomplishes the prediction target. Some early works follow auto-encoders to employ a heavy prediction head (decoder) [38]. In this paper, we show that the prediction head can be made extremely lightweight, as light as a linear layer. We also try heavier heads such as a 2-layer MLP, an inverse Swin-T, and an inverse Swin-B.预测头可以具有任意形式和容量，只要其输入与编码器输出相匹配，且其输出能够完成预测目标即可。一些早期工作遵循自动编码器，采用了较重的预测头（解码器)[38]。本文表明，预测头可以做得极其轻量，甚至轻至一个线性层。我们还尝试了更重的预测头，例如 2 层 MLP、逆向 Swin-T 和逆向 Swin-B。3.4. Prediction Targets3.4. 预测目标Raw pixel value regression The pixel values are continuous in the color space. A straight-forward option is to predict raw pixels of the masked area by regression. In general, vision architectures usually produce feature maps of downsampled resolution, e.g., 16× in ViT and 32× for most other architectures. To predict all pixel values at a full resolution of input images, we map each feature vector in feature map back to the original resolution, and let this vector take charge of the prediction of corresponding raw pixels.原始像素值回归颜色空间中的像素值是连续的。一个直接的选择是通过回归来预测被掩码区域的原始像素。通常，视觉架构会生成下采样分辨率的特征图，例如ViT中的16×以及大多数其他架构中的32×。为了预测输入图像完整分辨率下的所有像素值，我们将特征图中的每个特征向量映射回原始分辨率，并让该向量负责预测对应的原始像素。For example, on the 32× down-sampled feature maps produced by a Swin Transformer encoder, we apply a 1 × 1 convolution (linear) layer with output dimension of3˙07˙2=32×3˙2×3\mathbf { \dot { 3 } 0 \dot { 7 } 2 = 3 2 \times \dot { 3 } 2 } \times 33˙07˙2=32×3˙2×3to stand for the RGB values of 32 × 32 pixels. We also consider lower resolution targets by downsampling the original images by{ 32×,16×,8×,4×,2×}\{ 3 2 \times , 1 6 \times , 8 \times , 4 \times , 2 \times \}{32×,16×,8×,4×,2×}, respectively.例如，对于由 Swin Transformer 编码器生成的 32×下采样特征图，我们应用一个输出维度为3072=32×32×33 0 7 2 = 3 2 \times 3 2 \times 33072=32×32×3的1×11 \times 11×1卷积（线性）层，以表示32×323 2 \times 3 232×32个像素的RGB 值。我们还分别考虑通过将原始图像下采样{ 32×,16×,8×,4×,2×}\{ 3 2 \times , 1 6 \times , 8 \times , 4 \times , 2 \times \}{32×,16×,8×,4×,2×}得到的更低分辨率目标。Anℓ1−∣0SS\ell _ { 1 } { - } | _ { \mathrm { 0 S S } }ℓ1−∣0SSis employed on the masked pixels:在被掩码像素上采用ℓ1−∣0≤S\ell _ { 1 } - | _ { 0 \leq S }ℓ
SimMIM: a Simple Framework for Masked Image ModelingSimMIM：一种用于掩码图像建模的简单框架Zhenda Xie 1* Zheng Zhang 2* Yue Cao 2*Yutong Lin 3 Jianmin Bao 2 Zhuliang Yao 1 Qi Dai 2 Han Hu2*1 Tsinghua University 2 Microsoft Research Asia 3 Xi’an Jiaotong University¹ 清华大学²微软亚洲研究院⁸ 西安交通大学{t-zhxie,zhez,yuecao,t-yutonglin,jianmin.bao,t-zhuyao,qid,hanhu}@microsoft.comAbstract摘要This paper presents SimMIM, a simple framework for masked image modeling. We have simplified recently proposed relevant approaches, without the need for special designs, such as block-wise masking and tokenization via discrete VAE or clustering. To investigate what makes a masked image modeling task learn good representations, we systematically study the major components in our framework, and find that the simple designs of each component have revealed very strong representation learning performance: 1) random masking of the input image with a moderately large masked patch size (e.g., 32) makes a powerful pre-text task; 2) predicting RGB values of raw pixels by direct regression performs no worse than the patch classification approaches with complex designs; 3) the prediction head can be as light as a linear layer, with no worse performance than heavier ones. Using ViT-B, our approach achieves 83.8% top-1 fine-tuning accuracy on ImageNet-iK by pre-training also on this dataset, surpassing previous best approach by +o.6%. When applied to a larger model with about 650 million parameters, SwinV2-H, it achieves 87.1% top-1 accuracy on ImageNet-1K using only ImageNet-1K data. We also leverage this approach to address the datahungry issue faced by large-scale model training, that a 3B model (SwinV2-G) is successfully trained to achieve state-of-the-art accuracy on four representative vision benchmarks using 4o× less labelled data than that in previous practice (JFT-3B). The code is available at https://github.com/microsoft/SimMIM.本文提出了SimMIM，一个用于掩码图像建模的简单框架。我们简化了最近提出的相关方法，无需采用特殊设计，例如按块掩码以及通过离散VAE或聚类进行 tokenization。为了研究是什么使掩码图像建模任务能够学习到良好的表征，我们系统地研究了框架中的主要组件，并发现每个组件的简单设计都展现出非常强的表征学习性能：1)以适度较大的掩码 patch大小(例如32)对输入图像进行随机掩码，可构成一个强大的 pre-text 任务；2)通过直接回归预测原始像素的RGB 值，其表现不逊于采用复杂设计的patch 分类方法；3）预测头可以轻量至仅使用一个线性层，性能不逊于更重的预测头。使用 ViT-B，我们的方法仅在ImageNet-1K数据集上进行预训练，即可在ImageNet-1K上实现 83.8%的 top-1 Fine-tuning 准确率，比此前最佳方法高出+0.6%。当应用于一个拥有约 6.5 亿个 Parameter的更大模型SwinV2- H 时，仅使用ImageNet-1K 数据，即可在ImageNet-1K上实现 87.1%的 top-1 准确率。我们还利用该方法解决了大规模模型训练面临的数据饥渴问题：一个3B 模型(SwinV2-G)仅使用以往实践(JFT-3B)中1/40的标注数据，便成功训练并在四个具有代表性的视觉基准上取得了最先进的准确率。代码可在 https://github 获取。com/microsoft/SimMIM。1. Introduction1.引言“Masked signal modeling” is one such task that learns to create: masking a portion of input signals and trying to predict these masked signals. In NLP, following this phi-“掩码信号建模”便是这样一种任务，其学习方式为：遮蔽一部分输入信号，并尝试预测这些被遮蔽的信号。在NLP中，遵循这一理-Figure 1: Figure 1. An illustration of our simple framework for masked language modeling, named SimMIM. It predicts raw pixel values of the randomly masked patches by a lightweight one-layer head, and performs learning using a simple l1 loss.图1：图1. 我们提出的用于掩码语言建模的简单框架 SimMIM 示意图。它通过轻量级的单层头部预测随机遮蔽图块的原始像素值，并使用简单的l₁损失进行学习。losophy, self-supervised learning approaches built on the masked language modeling tasks have largely repainted the field [2,12,32], i.e., learning very large-scale language models by using huge amounts of unlabeled data has been shown to generalize well to a broad range of NLP applications.念，基于掩码语言建模任务的自监督学习方法在很大程度上重塑了该领域[2,12,32]，即，利用海量无标注数据学习超大规模语言模型已被证明能够很好地泛化到广泛的NLP 应用中。In computer vision, although there are pioneers leveraging this philosophy for self-supervised representation learning [13, 60, 61], in previous years, this line of work was almost buried by the contrastive learning approaches [8, 21, 51]. The different difficulties of applying this task to the language and visual domains can be explained by the differences between two modalities. One of the differences is that images exhibit stronger locality: pixels that are close to each other tend to be highly correlated [27], so the task can be done by duplicating close pixels rather than by semantic reasoning. Another difference is that visual signals are raw and low-level, while text tokens are human-generated high-level concepts. This raises a question of whether the prediction of low-level signals is useful for high-level visual recognition tasks. A third difference is that the visual signal is continuous, and the text token is discrete. It is unknown how classification-based masked language modeling approaches can be adapted to handle continuous visual signals well.在计算机视觉中，尽管已有先驱利用这一理念进行自监督表征学习[13，60，61]，但在过去几年中，这一研究方向几乎被对比学习方法[8,21，51]所掩盖。将该任务应用于语言和视觉领域时所面临的不同困难，可以通过两种模态之间的差异来解释。其中一个差异是，图像表现出更强的局部性：彼此接近的像素往往具有高度相关性[27]，因此该任务可以通过复制邻近像素来完成，而非依赖语义推理。另一个差异是，视觉信号是原始且低层级的，而文本Tokens是由人类生成的高层级概念。这引出了一个问题：对低层级信号的预测是否对高层级视觉识别任务有用。第三个差异是，视觉信号是连续的，而文本Token是离散的。尚不清楚基于分类的掩码语言建模方法如何能够适配，以良好地处理连续的视觉信号。Until recently, there have been trials that attempt to bridge modality gaps and resolve the obstacles, by introduc-直到最近，才出现了一些尝试通过引入来弥合模态差距并解决这些障碍，ing several special designs, for example, by converting continuous signals into color clusters [7], by patch tokenization using an additional network [1], or by a block-wise masking strategy to break short-range connections [1], etc. Through these special designs, the learned representations proved to be well transferable to several visual recognition tasks例如，通过将连续信号转换为颜色簇[7]、使用额外网络进行图块标记化[1]，或采用块级掩码策略以破坏短程连接[1]等多种特殊设计。通过这些特殊设计，所学习的表示被证明能够很好地迁移到多个视觉识别任务中。In contrast to requiring special complex designs, in this paper, we present a simple framework which aligns well with the nature of visual signals, as shown in Figure 1, and is able to learn similar or even better representations than previously more complex approaches: random masking of input image patches, using a linear layer to regress the raw pixel values of the masked area with an l1 loss. The key designs and insights behind this simple framework include:与需要特殊复杂设计的方法相比，本文提出了一个简单框架，该框架与视觉信号的本质高度契合，如图1所示，并且能够学习到与先前更复杂方法相当甚至更好的表示：对输入图像图块进行随机掩码，使用线性层以l1损失回归被掩码区域的原始像素值。该简单框架背后的关键设计与洞见包括：- Random masking is applied on image patches, which is simple and convenient for vision Transformers. For masked pixels, either larger patch size or higher masking ratio can result in a smaller chance of finding visible pixels that are close. For a large masking patch size of 32, the approach can achieve competitive performance in a wide range of masking ratios (10%-70%). For a small mask patch size of 8, the masking ratio needs to be as high as 8o% to perform well. Note that the preferred masking ratios are very different from that in the language domain, where a small masking ratio of o.15 is adopted as default. We hypothesize that different degrees of information redundancy in two modalities may lead to the different behaviors.在图像块上应用随机掩码，这对于视觉Transformer而言简单且方便。对于被掩码的像素，更大的图像块大小或更高的掩码比例都会降低找到邻近可见像素的概率。当掩码图像块大小较大(32)时，该方法可在较宽的掩码比例范围内(10%-70%)取得具有竞争力的性能。当掩码图像块大小较小(8)时，掩码比例需要高达80%才能获得良好性能。请注意，优选的掩码比例与语言领域中的情况截然不同，后者默认采用较小的0.15掩码比例。我们假设，两种模态中不同程度的信息冗余可能导致了这种不同的行为。- A raw pixel regression task is used. The regression task aligns well with the continuous nature of visual signals, which possesses ordering property. This simple task performs no worse than the classification approaches with classes specially defined by tokenization, clustering, or discretization采用原始像素回归任务。该回归任务与视觉信号的连续性及其所具有的有序性质高度契合。与通过分词、聚类或离散化专门定义类别的分类方法相比，这一简单任务的表现并不逊色。- An extremely lightweight prediction head (e.g., a linear layer) is adopted, which achieves similarly or slightly better transferring performance than that of heavier prediction heads (e.g., an inverse Swin-B). The use of an extremely lightweight prediction head brings a remarkable speedup in pre-training. In addition, we note that a broad range of target resolutions (e.g., 122 -962) perform competitive with the highest 1922. While heavier heads or higher resolutions generally result in greater generation capability, this greater capability does not necessarily benefit down-stream fine-tuning tasks.采用极轻量级的预测头（例如线性层），其迁移性能与更重型的预测头（例如逆向Swin-B）相当或略优。使用极轻量级预测头可显著加速预训练。此外，我们注意到，较宽范围的目标分辨率（例如12²- 96²）与最高的192²具有竞争力。尽管更重型的头或更高的分辨率通常会带来更强的生成能力，但这种更强的能力并不一定有利于下游 Fine-tuning 任务。Though simple, the proposed SimMIM approach is very effective for representation learning. Using ViT-B, it achieves 83.8% top-1 fine-tuning accuracy on ImageNet-1 K , surpassing previous best approach ([1]) by +0.6%. SimMIM has also shown to be scalable to larger models: with a SwinV2-H model (658M parameters) [33], it achieves 87.1% top-1 accuracy on ImageNet-1K classification, which is the highest number among methods that use ImageNet-1K data only. This result encourages the use of self-supervised learning to address the increasing datahungry problem caused by quickly rising model capacity. In fact, with the help of SimMIM, we successfully trained a SwinV2-G model with 3 billion parameters [33] using ~40× smaller data than that of Google’s JFT-3B dataset, and set new records on several representative benchmarks:84.0% top-1 accuracy on ImageNet-V2 classification [42], 63.1/54.4 box/mask mAP on COCO object detection [6,31], 59.9 mIoU on ADE2oK semantic segmentation [52, 63], and 86.8% top-1 accuracy on Kinetics-400 action recognition [28,35].尽管简单，所提出的 SimMIM 方法在表征学习方面非常有效。使用 ViT-B，它在ImageNet-1 K上实现了 83.8%的top-1 Fine-tuning 准确率，超过了此前最佳方法（[1]) +0.6%。SimMIM 还展现出可扩展至更大模型的能力：使用SwinV2-H 模型(658M Parameter)[33]，它在 ImageNet-1K 分类上实现了 87.1%的 top-1 准确率，这是仅使用ImageNet-1K数据的方法中最高的结果。这一结果鼓励使用自监督学习来应对因模型容量快速增长而导致的数据饥渴问题。事实上，借助SimMIM，我们成功地使用比 Google的JFT-3B 数据集小~40×的数据训练了一个拥有30亿Parameter 的 SwinV2-G 模型[33]，并在多个具有代表性的基准上创下新纪录：在ImageNet-V2 分类[42] 上达到84.0% top-1 准确率，在 COCO 目标检测 [6,31] 上达到 63.1/54.4 box/mask mAP，在 ADE20K 语义分割 [52, 63]上达到 59.9 mloU，以及在 Kinetics-400 动作识别 [28,35] 上达到 86.8%的 top-1 准确率。While in recent years we have witnessed an increasing overlap between NLP and computer vision in both basic modeling and learning algorithms, as well as in multi-modal applications, which aligns well with how human brains achieve general intelligence capabilities, we hope that our demonstration of “masked signal modeling" in computer vision can drive this trend a bit further and encourage deeper interaction of different AI fields.近年来，我们见证了NLP与计算机视觉之间在基础建模和学习算法以及多模态应用方面日益增多的交集，这与人类大脑实现通用智能能力的方式非常契合。我们希望，在计算机视觉中对“掩码信号建模”的展示能够进一步推动这一趋势，并促进不同 AI 领域之间更深入的交互。2. Related Work2. 相关工作Masked language modeling (MLM) Masked language modeling [12,32] and its auto-regressive variants [2] are the dominant self-supervised learning approaches in the field of natural language processing (NLP). Given visible tokens in a sentence or a sentence pair / triplet, the approaches learn representations by predicting invisible tokens of the input. This line of approaches has repainted the field since about 3 years ago [12], that it enables the learning of very large language models and generalizes well on broad language understanding and generation tasks by leveraging huge data.掩码语言建模(MLM) 掩码语言建模[12,32]及其自回归变体[2]是自然语言处理(NLP)领域中主流的自监督学习方法。给定一个句子或句子对/三元组中的可见 Tokens，这些方法通过预测输入中不可见的 Tokens 来学习表示。自约 3年前以来，这一类方法重塑了该领域[12]：它能够利用海量数据学习非常大的 Large Language Model，并能在广泛的语言理解和生成任务上实现良好的泛化。Masked image modeling (MIM) Masked image modeling [7, 13, 23, 38, 47] progressed in parallel with the MLM task in NLP but located in a non-mainstream position for a long time. The context encoder approach [38] is a pioneer work in this direction, which masks a rectangle area of the original images, and predicts the missing pixels. CPC [23,47] predicts patches via a verification task in each batch with a contrastive predictive coding loss. Recently, iGPT [7], ViT [15] and BEiT [1] recall this learning approach on the modern vision Transformers, and show strong potential in representation learning by introducing special designs on some components, such as clustering on pixels [7], prediction of mean color [15], and tokenization via an additional dVAE network with a block-wise masking strategy [1]. In contrary to these complex designs we present an extremely simple framework, SimMIM, which shows similar or even slightly better effectiveness.掩码图像建模(MIM) 掩码图像建模[7，13， 23， 38， 47] 与 NLP 中的 MLM 任务并行发展，但长期处于非主流位置。上下文编码器方法[38]是该方向的开创性工作，它对原始图像中的矩形区域进行掩码，并预测缺失的像素。CPC[23,47] 通过每个批次中的验证任务，利用对比预测编码损失来预测图像块。近年来，iGPT[7]、ViT[15] 和 BEiT[1] 在现代视觉Transformer上重新采用了这种学习方法，并通过在某些组件中引入特殊设计展现出强大的表征学习潜力，例如对像素进行聚类[7]、预测平均颜色[15]，以及通过额外的 dVAE 网络结合块级掩码策略进行标记化[1]。与这些复杂设计相反，我们提出了一个极其简单的框架SimMIM，其效果相当甚至略优。Reconstruction based methods are also related to our approach, particularly the auto-encoder approaches [24, 30, 37,41,49,50]. Similar as in our approach, they adopt a reconstruction task to recover the original signals. However, they are based on a different philosophy of visible signal reconstruction, other than the creation or prediction of invisible signals as in our approach. They thus progress in a very different path, by studying how to effectively regularize the task learning byproper regularization or architecture bottlenecks.基于重建的方法也与我们的方法相关，特别是自编码器方法[24，30，37,41,49,50]。与我们的方法类似，它们采用重建任务来恢复原始信号。然而，它们基于的是一种不同的可见信号重建理念，而非像我们的方法那样生成或预测不可见信号。因此，它们沿着一条截然不同的路径发展，通过研究如何借助恰当的正则化或架构瓶颈来有效地正则化任务学习。Image inpainting methods Beyond representation learning, masked image modeling is a classical computer vision problem, named image inpainting. This problem has been extensively studied in computer vision for a long time [39, 55, 56], aiming for improving the inpainting quality and without connecting to self-supervised representation learning. While we advocate image inpainting as a strong self-supervised pre-text task, we also find stronger inpainting capability does not necessarily leads to stronger finetuning performance on down-stream tasks图像修复方法除了表征学习之外，掩码图像建模还是一个经典的计算机视觉问题，称为图像修复。该问题长期以来已在计算机视觉领域得到广泛研究[39，55，56]，旨在提高修复质量，而未与自监督表征学习建立联系。尽管我们提倡将图像修复作为一种强大的自监督前置任务，但我们也发现，更强的修复能力并不一定会带来下游任务上更强的微调性能。Compressed sensing The approach in this paper is also related to compressed sensing [14], which affirms most of the data we acquire including image signals can be thrown away with almost no perceptual loss. Such claim is also partly supported by recent works of sparse inference [2o] that the recognition accuracy has very little drop after throwing a large portion of image features [25,43,54]. The observation in this paper goes further for the input signals, that with an extremely small portion of randomly selected input image patches as input, i.e., 10%, the inpainting task can still be learnt to produce good visual representations.压缩感知本文的方法也与压缩感知「14]相关，该理论指出，我们获取的大多数数据（包括图像信号）都可以在几乎不造成感知损失的情况下被丢弃。近期关于稀疏Inference的研究工作[20]也在一定程度上支持了这一主张：在丢弃大部分图像特征后，识别准确率仅有极小幅度的下降[25,43,54]。本文的观察进一步延伸至输入信号：仅使用极小一部分随机选择的输入图像块作为输入，即10%，仍然可以学习图像修复任务并产生良好的视觉表征。Other self-supervised learning approaches During the last two decades, there have been numerous pretext tasks to learn visual representation in a self-supervised way: grayscale image colorization [6o], jigsaw puzzle solving [36], split-brain auto-encoding [61], rotation prediction [18], learning to cluster [4]. Though very different from masked image modeling, some of them interestingly also follow a philosophy of predicting the invisible parts of signals, e.g., [6o,61] use one or two color channels as input to predict values of other channels. Another large portion of works lie in the contrastive learning approaches [3,8,16,19,21,51,53], which are the previous mainstream. We hope our work can encourage the study of masked language modeling as a pretext task for self-supervised visual representation learning.其他自监督学习方法在过去二十年中，已经出现了许多用于以自监督方式学习视觉表征的预训练任务：灰度图像着色[60]、拼图求解[36]、分脑自编码[61]、旋转预测[18]、学习聚类[4]。尽管与掩码图像建模非常不同，其中一些方法有趣地也遵循了预测信号不可见部分的理念，例如，[60,61]使用一个或两个颜色通道作为输入来预测其他通道的值。另一大类工作属于对比学习方法[3,8,16,19,21,51,53]，它们是此前的主流。我们希望我们的工作能够促进将掩码语言建模作为自监督视觉表征学习预训练任务的研究。3. Approach3. 方法3.1. A Masked Image Modeling Framework3.1. 掩码图像建模框架Our approach SimMIM learns representation through masked image modeling, which masks a portion of input image signals and predicts the original signals at masked area. The framework consists of 4 major components:我们的方法 SimMIM 通过掩码图像建模学习表征：它掩盖部分输入图像信号，并预测被掩盖区域中的原始信号。该框架由4 个主要组件组成：Masking strategy. Given an input image, this component designs how to select the area to mask, and how to implement masking of selected area. The transformed image after masking will be used as the input.掩码策略。给定一张输入图像，该组件设计如何选择要掩盖的区域，以及如何对所选区域实施掩码。掩码后的变换图像将作为输入。Encoder architecture. It extracts a latent feature representation for the masked image, which is then used to predict the original signals at the masked area. The learnt encoder is expected to be transferable to various vision tasks. In this paper, we mainly consider two typical vision Transformer architectures: a vanilla ViT [15] and Swin Transformer [34].编码器架构。它为掩码图像提取潜在特征表示，随后用于预测掩码区域的原始信号。所学习的编码器应能够迁移到各种视觉任务中。本文主要考虑两种典型的视觉 Transformer 架构：标准 ViT [15] 和 Swin Transformer[34]。Prediction head. The prediction head will be applied on the latent feature representation to produce one form of the original signals at the masked area.预测头。预测头将应用于潜在特征表示，以生成掩码区域中原始信号的一种形式。Prediction target. This component defines the form of original signals to predict. It can be either the raw pixel values or a transformation of the raw pixels. This component also defines the loss type, with typical options including the cross-entropy classification loss and the l1 or l2 regression losses.预测目标。该组件定义要预测的原始信号形式。它可以是原始像素值，也可以是原始像素的变换形式。该组件还定义损失类型，典型选项包括交叉熵分类损失以及l1 或l2 回归损失。In the following subsections, we will present typical options of each component. These options are then systematically studied. By combining simple designs of each component, we have been able to achieve strong representation learning performance.在以下小节中，我们将介绍每个组件的典型选项。随后将对这些选项进行系统研究。通过组合每个组件的简单设计，我们能够实现强大的表征学习性能。3.2. Masking Strategy3.2. 掩码策略For input transformation of masked area, we follow the NLP community [12,32] and BEiT [1] to use a learnable mask token vector to replace each masked patch. The token vector dimension is set the same as that of the other visible patch representation after patch embedding. For masking area selection, we study the following masking strategies (illustrated in Figure 2):对于掩码区域的输入变换，我们遵循 NLP 社区 [12,32] 和 BEiT [1] 的做法，使用一个可学习的 mask Token 向量来替换每个被掩码的 patch。该 Token 向量的维度设置为与经过 patch Embedding 后其他可见 patch 表示的维度相同。对于掩码区域选择，我们研究了以下掩码策略（如图2所示）：Patch-aligned random masking We first present a patch-aligned random masking strategy. Image patches are the basic processing units of vision Transformers, and it is convenient to operate the masking on patch-level that a patch is either fully visible or fully masked. For Swin Transformer, we consider equivalent patch sizes of different resolution stages, 4 × 4 ～ 32 × 32, and adopt 32 × 32 by default which is the patch size of the last stage. For ViT, we adopt 32×32 as the default masked patch size.Patch 对齐的随机掩码 我们首先介绍一种 patch 对齐的随机掩码策略。图像 patch 是视觉 Transformer的基本处理单元，在 patch 级别执行掩码操作较为方便，即一个 patch要么完全可见，要么完全被掩码。对于 Swin Transformer，我们考虑不同分辨率阶段的等效 patch 大小，4×4˙∼32×324 \times \dot { 4 } \sim 3 2 \times 3 24×4˙∼32×32并默认采用 32 × 32，即最后一个阶段的 patch大小。对于 ViT，我们采用 32×32 作为默认的被掩码 patch 大小。Other masking strategies We also try other masking strategies in previous works: 1) [38] introduces a central region masking strategy. We relax it to be randomly movable on the image. 2) [1] introduces a complex block-wise masking strategy. We try this mask strategy on two masked patch sizes of 16 × 16 and 32 × 32.其他掩码策略我们还尝试了先前工作中的其他掩码策略：1）[38]提出了一种中心区域掩码策略。我们将其放宽为可以在图像上随机移动。2)[1]提出了一种复杂的分块掩码策略。我们在两种被掩码 patch 尺寸16×16 和32×32上尝试了该掩码策略。Raw imageSquare (32)Block-wise (16)Random (4)Random (8)Random (16)Random (32)Figure 2: Figure 2. Illustration of masking area generated by different masking strategies using a same mask ratio of o.6: square masking [38], block-wise masking [1] apply on 16-sized patches, and our simple random masking strategy on different patch sizes (e.g., 4, 8, 16 and 32).图 2：图 2。使用相同掩码比例 0.6 时，不同掩码策略生成的掩码区域示意图：方形掩码[38]、应用于大小为16 的patch 的分块掩码[1]，以及我们在不同 patch 尺寸（例如 4、8、16 和 32)上的简单随机掩码策略。3.3. Prediction Head3.3. 预测头The prediction head can be of arbitrary form and capacity, as long as its input conforms with the encoder output and its output accomplishes the prediction target. Some early works follow auto-encoders to employ a heavy prediction head (decoder) [38]. In this paper, we show that the prediction head can be made extremely lightweight, as light as a linear layer. We also try heavier heads such as a 2-layer MLP, an inverse Swin-T, and an inverse Swin-B.预测头可以具有任意形式和容量，只要其输入与编码器输出相匹配，且其输出能够完成预测目标即可。一些早期工作遵循自动编码器，采用了较重的预测头（解码器)[38]。本文表明，预测头可以做得极其轻量，甚至轻至一个线性层。我们还尝试了更重的预测头，例如 2 层 MLP、逆向 Swin-T 和逆向 Swin-B。3.4. Prediction Targets3.4. 预测目标Raw pixel value regression The pixel values are continuous in the color space. A straight-forward option is to predict raw pixels of the masked area by regression. In general, vision architectures usually produce feature maps of downsampled resolution, e.g., 16× in ViT and 32× for most other architectures. To predict all pixel values at a full resolution of input images, we map each feature vector in feature map back to the original resolution, and let this vector take charge of the prediction of corresponding raw pixels.原始像素值回归颜色空间中的像素值是连续的。一个直接的选择是通过回归来预测被掩码区域的原始像素。通常，视觉架构会生成下采样分辨率的特征图，例如ViT中的16×以及大多数其他架构中的32×。为了预测输入图像完整分辨率下的所有像素值，我们将特征图中的每个特征向量映射回原始分辨率，并让该向量负责预测对应的原始像素。For example, on the 32× down-sampled feature maps produced by a Swin Transformer encoder, we apply a 1 × 1 convolution (linear) layer with output dimension of3˙07˙2=32×3˙2×3\mathbf { \dot { 3 } 0 \dot { 7 } 2 = 3 2 \times \dot { 3 } 2 } \times 33˙07˙2=32×3˙2×3to stand for the RGB values of 32 × 32 pixels. We also consider lower resolution targets by downsampling the original images by{ 32×,16×,8×,4×,2×}\{ 3 2 \times , 1 6 \times , 8 \times , 4 \times , 2 \times \}{32×,16×,8×,4×,2×}, respectively.例如，对于由 Swin Transformer 编码器生成的 32×下采样特征图，我们应用一个输出维度为3072=32×32×33 0 7 2 = 3 2 \times 3 2 \times 33072=32×32×3的1×11 \times 11×1卷积（线性）层，以表示32×323 2 \times 3 232×32个像素的RGB 值。我们还分别考虑通过将原始图像下采样{ 32×,16×,8×,4×,2×}\{ 3 2 \times , 1 6 \times , 8 \times , 4 \times , 2 \times \}{32×,16×,8×,4×,2×}得到的更低分辨率目标。Anℓ1−∣0SS\ell _ { 1 } { - } | _ { \mathrm { 0 S S } }ℓ1−∣0SSis employed on the masked pixels:在被掩码像素上采用ℓ1−∣0≤S\ell _ { 1 } - | _ { 0 \leq S }ℓ
