BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

Human Detection - Pedestrian Detection using OpenCV Python

PyTorch implementation of the Deep SLDA method from our CVPRW-2020 paper "Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis"

Homepage of paper: Paint Transformer: Feed Forward Neural Painting with Stroke Prediction, ICCV 2021.

Configure SRX interfaces with Scrapli

PyTorch Implementation for "ForkGAN with SIngle Rainy NIght Images: Leveraging the RumiGAN to See into the Rainy Night"

EMNLP 2021 Findings' paper, SCICAP: Generating Captions for Scientific Figures

AQP is a modular pipeline built to enable the comparison and testing of different quality metric configurations.

MADT: Offline Pre-trained Multi-Agent Decision Transformer

Laplacian Score-regularized Concrete Autoencoders

An essential implementation of BYOL in PyTorch + PyTorch Lightning

Automatic meme generation model using Tensorflow Keras.

GANsformer: Generative Adversarial Transformers Drew A

Hyperparameter Optimization for TensorFlow, Keras and PyTorch

Pytorch implementation of YOLOX、PPYOLO、PPYOLOv2、FCOS an so on.

Mixed Transformer UNet for Medical Image Segmentation

The code of “Similarity Reasoning and Filtration for Image-Text Matching” [AAAI2021]

End-To-End Optimization of LiDAR Beam Configuration

Data-Uncertainty Guided Multi-Phase Learning for Semi-supervised Object Detection

Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.

Implementation of MA-Trace - a general-purpose multi-agent RL algorithm for cooperative environments.