Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Last update: Dec 29, 2022

Related tags

Deep Learning wechsel

Overview

WECHSEL

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

arXiv: https://arxiv.org/abs/2112.06598

Models from the paper are available on the HuggingFace Hub:

Installation

We distribute a Python Package via PyPI:

pip install wechsel

Alternatively, clone the repository, install requirements.txt and run the code in wechsel/.

Example usage

Transferring English roberta-base to Swahili:

import torch
from transformers import AutoModel, AutoTokenizer
from datasets import load_dataset
from wechsel import WECHSEL, load_embeddings

source_tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model = AutoModel.from_pretrained("roberta-base")

target_tokenizer = source_tokenizer.train_new_from_iterator(
    load_dataset("oscar", "unshuffled_deduplicated_sw", split="train")["text"],
    vocab_size=len(source_tokenizer)
)

wechsel = WECHSEL(
    load_embeddings("en"),
    load_embeddings("sw"),
    bilingual_dictionary="swahili"
)

target_embeddings, info = wechsel.apply(
    source_tokenizer,
    target_tokenizer,
    model.get_input_embeddings().weight.detach().numpy(),
)

model.get_input_embeddings().weight.data = torch.from_numpy(target_embeddings)

# use `model` and `target_tokenizer` to continue training in Swahili!

Bilingual dictionaries

We distribute 3276 bilingual dictionaries from English to other languages for use with WECHSEL in dicts/.

Citation

Please cite WECHSEL as

@misc{minixhofer2021wechsel,
      title={WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models}, 
      author={Benjamin Minixhofer and Fabian Paischer and Navid Rekabsaz},
      year={2021},
      eprint={2112.06598},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Related tags

Overview

WECHSEL

Installation

Example usage

Bilingual dictionaries

Citation

Owner

Institute of Computational Perception

LSTM Neural Networks for Spectroscopic Studies of Type Ia Supernovae

3D position tracking for soccer players with multi-camera videos

Tools for the Cleveland State Human Motion and Control Lab

Tensorflow implementation of Swin Transformer model.

Official code repository for ICCV 2021 paper: Gravity-Aware Monocular 3D Human Object Reconstruction

ECLARE: Extreme Classification with Label Graph Correlations

3ds-Ghidra-Scripts - Ghidra scripts to help with 3ds reverse engineering

Flexible-Modal Face Anti-Spoofing: A Benchmark

Garbage Detection system which will detect objects based on whether it is plastic waste or plastics or just garbage.

In this project, we create and implement a deep learning library from scratch.

FastFace: Lightweight Face Detection Framework

Airborne magnetic data of the Osborne Mine and Lightning Creek sill complex, Australia

CAPITAL: Optimal Subgroup Identification via Constrained Policy Tree Search

ISBI 2022: Cross-level Contrastive Learning and Consistency Constraint for Semi-supervised Medical Image.

Open & Efficient for Framework for Aspect-based Sentiment Analysis

The project is an official implementation of our paper "3D Human Pose Estimation with Spatial and Temporal Transformers".

Self Governing Neural Networks (SGNN): the Projection Layer

N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting

[ICML 2020] Prediction-Guided Multi-Objective Reinforcement Learning for Continuous Robot Control

Deep motion transfer