宿州银行网站建设,在线手机网站预览,代加工网,有深度的公司名字Clip 文章目录Clip前言一、原理1.1 摘要1.2 引言1.3 方法1.4 实验1.4.1 zero-shot Transfer1.4.2 PROMPT ENGINEERING AND ENSEMBLING1.5 局限性二、总结前言
阅读论文#xff1a; Learning Transferable Visual Models From Natural Language Supervision CLIP 论文逐段精读…Clip 文章目录Clip前言一、原理1.1 摘要1.2 引言1.3 方法1.4 实验1.4.1 zero-shot Transfer1.4.2 PROMPT ENGINEERING AND ENSEMBLING1.5 局限性二、总结前言
阅读论文 Learning Transferable Visual Models From Natural Language Supervision CLIP 论文逐段精读【论文精读】 Github: https://openai.com/research/clip https://github.com/OpenAI/CLIP 知乎 如何评价OpenAI最新的工作CLIP连接文本和图像zero shot效果堪比ResNet50 OpenAI发布CLIP模型快一年了盘点那些CLIP相关让人印象深刻的工作 一、原理
原理 一个batch中image encoder 可以是resnet也可以是visual transformers对应 text encoder 在矩阵中进行对比学习蓝色对角线上的是正例样本其余的都是负样本。推理的时候如何做到不需要imagenet的监督学习就可以做到监督信号的呢那是构造了prompt template, 原本的linear的1000个类构造成 a photo of a [object label]经过text encoder(pretrain的 encoder)得到的向量和 image 经过 image encoder的向量进行求cosine similarity.
针对prompt template的构造还有 prompt engineering和prompt ensmble两种方法比对学习需要大量的图片文本的数据集openai收集了4亿对的数据集进行预训练
效果迁移学习能力非常强zeroshot在视觉数据集上效果很好,尤其是ImageNet上的效果摆脱了categorical label的限制 The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on https://openai.com/research/clip 和NLP的结合CLIP学出来的视觉特征和语言描述的某些物体产生强烈的联系
有趣的应用 styleCLIP text 2 修改图片 CLIPDraw text 2 简笔画的生成抽象主义的 物体监测分割 open-vocabulary detector 视频检索clifs
1.1 摘要
imagenet 1000 类 CIFAR 10 CIFAR 100 目标监测 coco 80 语义分割 city scapes 19 视频 Kineitcs 400 想法 Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. 对应的任务设计成 We demonstrate that the simple pre-training task of redicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. 实验 在30个不同的CV datasets上做测试迁移的效果非常好 开源的代码只有推理的部分并没有预训练的部分 1.2 引言 The development of “text-to-text” as a standardized input-output interface (McCann et al., 2018;Radford et al., 2019; Raffel et al., 2019) has enabled task-agnostic architectures to zero-shot transfer to downstream datasets removing the need for specialized output heads or dataset specific customization 核心在于预训练的架构和下游任务无关这样就不需要监督信号学习一个和下游任务相关的分类头。 NLP的那套预训练的方法 These results suggest that the aggregate supervision accessible to modern pre-training methods within web-scale collections of text surpasses that of high-quality crowd-labeled NLP datasets 这个方法是很有效的希望用到CV的相关任务中来。
作者从1999年的相关论文讨论到2021年 主要跟2017年的Learning visual n-grams from web data 有了transformer、完形填空自监督的学习信号后有了VirTex\ICMLM\Con-VIRT,基于transformer做的后面的一些工作是想把一些弱的监督信号用起来 Instead, more narrowly scoped but well-targeted uses of weak supervision have improved performance. Mahajan et al. (2018) showed that predicting ImageNet-related hashtags on Instagram images is an effective pre-training task. When fine-tuned to ImageNet these pre-trained models increased accuracy by over 5% and improved the overall state of the art at the time. Kolesnikov et al. (2019) and Dosovitskiy et al. (2020) have also demonstrated large gains on a broader set of transfer benchmarks by pre-training models to predict the classes of the noisily labeled JFT-300M dataset. 好处是弱的监督信号的数据集大因此作者认为使用gold-labels是很有限的反而希望用上那些取之不尽用之不竭的文本即使用上了在模型层面还是很有局限性主要是用静态的softmax作为分类头缺乏zero-shot的能力。 Both works carefully design, and in the process limit, their supervision to 1000 and 18291 classes respectively. Natural language is able to express, and therefore supervise, a much wider set of visual concepts through its generality. Both approaches also use static softmax classifiers to perform prediction and lack a mechanism for dynamic outputs. This severely curtails their flexibility and limits their “zero-shot” capabilities. 之前的工作不行主要是数据集的规模和模型的规模都要上去。accelerator years所以作者团队先从数据集开始入手收集了4对的文本图片对。模型层面从resnet\efficient net\vision transformer(VIT Large),就提出了CLIP。单单视觉上的模型就用了8个最大和最小的模型容量差了100倍。
作者怒刷30个数据集看泛化性和迁移的效果。在做zero-shot之前呢作者去看了linear-probe为了进一步提供模型的学习能力直接把主干网络冻住训练最后一层的分类头发现全方面碾压之前的方法。
1.3 方法
用上了deep contextual representations, like bert就能利用上abundant source of supervision, 总结下来说文本监督信号帮助训练一个视觉模型是很有潜力前提是数据集量够大目前能用的数据集MS-COCO\Visual Genome,好归好但是数据量太少了JFT300M有3亿个样本、Instagram 有3.5billion。YFCC100标注质量太差了有人去清晰了下只身下15M了。
NLP那边的数据集来说和GPT2差不多的级别CV和JFT300m还多了一个亿。WIT数据集。
之前训练的模型也还只是在1000类别上就已经如此的耗时了更不用说是开放点视觉概念任务上了。作者提出 In the course of our efforts, we found training efficiency was key to successfully scaling natural language supervision and we selected our final pre-training method based on this metric 训练的效率视乎是训练自然语言监督信号的核心。
step1, similar to VirTex , jointly trained an image CNN and text transformer from scratch to predict the caption of an image, 结果很慢contrastive objectives can learn better representations than their equivalent predictive objective 比对学习目标比预测型的目标更加好学不仅如此推理的速度更加快快了4倍 橙色是将文本变成全局的特征而不是逐字逐句的特征再把约束放宽推理速度又更近一步。伪代码
# image_encoder - ResNet or Vision Transformer
# text_encoder - CBOW or Text Transformer
# I[n, h, w, c] - minibatch of aligned images [8, 224, 224, 3]
# T[n, l] - minibatch of aligned texts [8,512]
# W_i[d_i, d_e] - learned proj of image to embed
# W_t[d_t, d_e] - learned proj of text to embed
# t - learned temperature parameter# extract feature representations of each modality
I_f image_encoder(I) #[n, d_i]
T_f text_encoder(T) #[n, d_t]# joint multimodal embedding [n, d_e]
I_e l2_normalize(np.dot(I_f, W_i), axis1)
T_e l2_normalize(np.dot(T_f, W_t), axis1)# scaled pairwise cosine similarities [n, n]
logits np.dot(I_e, T_e.T) * np.exp(t)# symmetric loss function
# SimCLR 到 BYOL, 一直到最新的MOCO V3 DINO这些工作都是用对称式的目标函数
labels np.arange(n) 对角线上的元素
loss_i cross_entropy_loss(logits, labels, axis0)
loss_t cross_entropy_loss(logits, labels, axis1)
loss (loss_i loss_t)/2Figure 3. Numpy-like pseudocode for the core of an implementation of CLIP.训练细节:
数据集太大不会导致over-fitting的问题从头预训练没有加载imagesnet权重和文本的权重也没有用非线性层的映射在表示层和比对的embedding映射空间这里只用了linear层移除了只采样图片文本的一个句子的功能简化了图片数据增强的功能只采用裁剪这种方式对于比对学习中temperature parameter参数只是设置为可以学习的标量视觉部分模型可以选择ResNET还稍微做了一些修改 ,也可以选择visual transformersVIT, 文本部分只是使用的transformers . As a base size we use a 63M-parameter 12-layer 512-wide model with 8 attention heads. BPE\49152词表、76的最大长度并在模型的宽度和深度做了一些简单的尝试5个resnet(50-101-50x4-50x16-50x64), 3个vit(32-16-14)模型训练的是32epochsAdam优化器权重衰减、not gains, not biasescosine schedule的lr只在resnet50做了grid searches 一个epoches,32768的batch_size, 天啊混精度训练、 gradient checkpointing、 half-precision Adam statistics、half-precision stochastically rounded text encoder weights相似度的计算也是放在不同的GPU上
题外话openai热衷于GPT GPT系列、DALL-E、Image gpt 和 openai codex
1.4 实验
1.4.1 zero-shot Transfer Our focus on studying zero-shot transfer as an evaluation of task learning is inspired by work demonstrating task learning in the field of NLP 作者的核心就是使用一张图片分别问1000个句子之后做一个softmax, 就可以进行相对应的zero-shot了。 大幅度提升了效果 1.4.2 PROMPT ENGINEERING AND ENSEMBLING
主要的问题是词语的多义性polysemy When the name of a class is the only information provided to CLIP’s text encoder it is unable to differentiate which word sense is meant due to the lack of context 论文提出模板式 A photo of a {label} to be a good default that helps specify the text is about the content of the image 如果知道更多信息那效果会更好 For example on Oxford-IIIT Pets, using “A photo of a {label}, a type of pet.” to help provide context worked well. Likewise, on Food101 specifying a type of food and on FGVC Aircraft a type of aircraft helped too. 作者用了80个提示模板 https://github.com/openai/CLIP/blob/main/notebooks/Prompt_Engineering_for_ImageNet.ipynb 再做了27个数据集的实验
clip的zero-shot,以及linear probe就是冻住主干只训练最后一层的linear层linear probe是基线版本绿色是优于probe的蓝色是低的物品的分类效果会更加好更难的数据集纹理、物体的计数会更加抽象会更难难的任务可能需要few shot few shot的实验也做了横坐标是每个label使用的样本纵坐标是20个数据集中的平均准确率同时都是用的linear probeclip冻住的是图片的encoderbit是专门为迁移学习所做的当时最好的迁移学习的模型很强的baseline124的fewshot效果还没有多模态的zero-shot好说明文本的监督信号确实强
zeroshot\few shot都做完了接下来如果直接使用全量的监督信号的数据进行实验会如何 方法有两种 1、linear probe 2、finetune 作者只用第一种方式减少预训练对数据的影响看预训练的好坏。finetune太多参数可以调这样就不好比对效果。 横坐标是一张图经过的参数量纵坐标是准确率 作者再把clip和efficientnet做对比 Fitting a linear classifier on CLIP’s features outperforms using the Noisy Student EfficientNet-L2 on 21 out of 27 datasets. 冻住主干网络只训分类头全量的数据。
当数据有偏移的时候模型表现如何 和人类进行比较找了5个人来做实验 表格的体现出来的效果还是很好的 人类觉得难的模型也觉得难 做了去重的实验还是觉得clip的泛化性能好
1.5 局限性 Significant work is still needed to improve the task learning and transfer capabilities of CLIP. While scaling has so far steadily improved performance and suggests a route for continued improvement, we estimate around a 1000x increase in compute is required for zero-shot CLIP to reach overall state-of-the-art performance 扩大规模来弥补和stoa的差距不现实 CLIP also struggles with more abstract and systematic tasks such as counting the number of objects in an image. Finally for novel tasks which are unlikely to be included in CLIP’s pre-training dataset, such as classifying the distance to the nearest car in a photo, CLIP’s performance can be near random. We are confident that there are still many, many, tasks where CLIP’s zero-shot performance is near chance level. 更难的任务上确实不太行 However, CLIP only achieves 88% accuracy on the handwritten digits of MNIST 预训练的数据集和下游的数据分布如果是out of distribution也不太行
最好的是直接生成图片的标题这就是端到端的了而不是给你一个自然语言的监督信号做成一个生成式的模型。对比学习的函数和生成式的目标函数合在一起
对数据的利用并不高效。如何提高数据的利用效率自监督的方式和伪标签的方式
做实验过程总是以测试集为导向进行调参而不是真正的zeroshot
这选中的27个数据集也是有主观的偏见的如果有一个数据集是专门来做zeroshot的那就太好了
数据都是网上爬的没有经过过滤的会带有社会的偏见
在一些很难用语言描述的任务过程中如果你不提供训练样本的表现由于你few shot的效果 二、总结 We have investigated whether it is possible to transfer the success of task-agnostic web-scale pre-training in NLP to another domain. We find that adopting this formula results in similar behaviors emerging in the field of computer vision and discuss the social implications of this line of research. In order to optimize their training objective, CLIP models learn to perform a wide variety of tasks during pretraining. This task learning can then be leveraged via natural language prompting to enable zero-shot transfer to many existing datasets. At sufficient scale, the performance of this approach can be competitive with task-specific supervised models although there is still room for much improvement. 打破了固定标签的学习范式无监督的方式进行学习数据处理更方便模型也是方便推理更加方便。新意度100 有效性100 问题大小100分
