Beyond Masking: Demystifying Token-Based Pre-Training for Vision Transformers

Last update: Sep 27, 2022

Related tags

Text Data & NLP beyond_masking

Overview

beyond masking

Beyond Masking: Demystifying Token-Based Pre-Training for Vision Transformers

The code is coming

Figure 1: Pipeline of token-based pre-training.

Figure 2: The visualization of the proposed 5 tasks.

main results

All the results are pre-trained for 300 epochs using Vit-base as default.

	zoomed-in	zoomed-out	distorted	blurred	de-colorized
finetune	`82.7`	`82.5`	`82.1`	`81.8`	`81.4`

	zoomed-in (a)	mask (m)	(a)+(m)
finetune	`82.7`	`82.9`	`83.2`

We note that the integrated version dose not require extra computational cost.

Effiencicy

Figure 3: Efficiency of the integrated task.

Owner

Yunjie Tian

I am a Ph.D student in University of Chinese Academy of Sciences (UCAS), studying computer vision, especially NAS and self-supervised learning.

GitHub Repository

xFormers is a modular and field agnostic library to flexibly generate transformer architectures by interoperable and optimized building blocks.

Description xFormers is a modular and field agnostic library to flexibly generate transformer architectures by interoperable and optimized building bl

2.3k Jan 08, 2023

"Investigating the Limitations of Transformers with Simple Arithmetic Tasks", 2021

transformers-arithmetic This repository contains the code to reproduce the experiments from the paper: Nogueira, Jiang, Lin "Investigating the Limitat

33 Nov 16, 2022

Syntax-aware Multi-spans Generation for Reading Comprehension (TASLP 2022)

SyntaxGen Syntax-aware Multi-spans Generation for Reading Comprehension (TASLP 2022) In this repo, we upload all the scripts for this work. Due to siz

3 Jun 13, 2022

This repository contains the codes for LipGAN. LipGAN was published as a part of the paper titled "Towards Automatic Face-to-Face Translation".

LipGAN Generate realistic talking faces for any human speech and face identity. [Paper] | [Project Page] | [Demonstration Video] Important Update: A n

438 Dec 31, 2022

Simple Python library, distributed via binary wheels with few direct dependencies, for easily using wav2vec 2.0 models for speech recognition

Wav2Vec2 STT Python Beta Software Simple Python library, distributed via binary wheels with few direct dependencies, for easily using wav2vec 2.0 mode

22 Dec 29, 2022

Beyond Masking: Demystifying Token-Based Pre-Training for Vision Transformers

Related tags

Overview

beyond masking

main results

Effiencicy

Owner

Yunjie Tian

xFormers is a modular and field agnostic library to flexibly generate transformer architectures by interoperable and optimized building blocks.

"Investigating the Limitations of Transformers with Simple Arithmetic Tasks", 2021

Syntax-aware Multi-spans Generation for Reading Comprehension (TASLP 2022)

This repository contains the codes for LipGAN. LipGAN was published as a part of the paper titled "Towards Automatic Face-to-Face Translation".

Simple Python library, distributed via binary wheels with few direct dependencies, for easily using wav2vec 2.0 models for speech recognition

Longformer: The Long-Document Transformer

Multilingual word vectors in 78 languages

Lattice methods in TensorFlow

fastNLP: A Modularized and Extensible NLP Framework. Currently still in incubation.

Tensorflow Implementation of A Generative Flow for Text-to-Speech via Monotonic Alignment Search

Parrot is a paraphrase based utterance augmentation framework purpose built to accelerate training NLU models

CorNet Correlation Networks for Extreme Multi-label Text Classification

Code for paper "Role-oriented Network Embedding Based on Adversarial Learning between Higher-order and Local Features"

Code for EmBERT, a transformer model for embodied, language-guided visual task completion.

YACLC - Yet Another Chinese Learner Corpus

Predicting the usefulness of reviews given the review text and metadata surrounding the reviews.

Translation for Trilium Notes. Trilium Notes 中文版.

a chinese segment base on crf

🎐 a python library for doing approximate and phonetic matching of strings.

Harvis is designed to automate your C2 Infrastructure.