Pytorch implementation of AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

Last update: Sep 16, 2022

Related tags

Deep Learning AngularGrad

Overview

AngularGrad Optimizer

This repository contains the oficial implementation for AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks in PyTorch.

AngularGrad reduces the zig-zag effect in the optimization trajectory. Fluctuations are significantly smoothed, tracing a more direct path towards the minimum of the cost function.

If you have questions or suggestions, please feel free to open an issue. Please cite as:

@article{roy2021angulargrad,
  title={AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks},
  author={S.K. Roy, M.E. Paoletti, J.M. Haut, S.R. Dubey, P. Kar, A. Plaza and B.B. Chaudhuri},
  journal={arXiv preprint arXiv:2105.10190},
  year={2021}
}

In PyTorch:

optimizer = cosangulargrad(params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0)
<or>
optimizer = tanangulargrad(params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0)
Example:
optimizer = cosangulargrad(net.parameters())

Experiments

Experiments in the paper:

Analitycal

cd analitycal/
python main.py

CIFAR-10/100

cd cifar/
python main.py --dataset <cifar10/cifar100> --model <r18/r34/.../vgg16/d121> --alg <adam/sgd/.../cosangulargrad/tanangulargrad> --lr <float>
Example:
python main.py --dataset cifar10 --model r50 --alg cosangulargrad --lr 1e-3

Mini-ImageNet:

cd mini-imagenet/
wget URL dataset
python main.py DATADIR --alg <adam/sgd/.../cosangulargrad/tanangulargrad> --lr <float>
Example:
python main.py ./split_mini/ --alg cosangulargrad --model r50 --lr 1e-3

Fine-Grained:

cd fine-grained/
wget URL datasets
python main.py DATADIR --dataset <dsetname> --alg <adam/sgd/.../cosangulargrad/tanangulargrad> --lr <float>
Example:
python main.py ./data/Car196/ --dataset cars --alg adam --lr 1e-3

Pytorch implementation of AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

Related tags

Overview

AngularGrad Optimizer

Experiments

Owner

mario

Code for "Universal inference meets random projections: a scalable test for log-concavity"

This is the unofficial code of Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes. which achieve state-of-the-art trade-off between accuracy and speed on cityscapes and camvid, without using inference acceleration and extra data

Pytorch implementation of NEGEV method. Paper: "Negative Evidence Matters in Interpretable Histology Image Classification".

Training Confidence-Calibrated Classifier for Detecting Out-of-Distribution Samples / ICLR 2018

EMNLP'2021: Simple Entity-centric Questions Challenge Dense Retrievers

Learning to Simulate Dynamic Environments with GameGAN (CVPR 2020)

Python interface for SmartRF Sniffer 2 Firmware

Python scripts for performing road segemtnation and car detection using the HybridNets multitask model in ONNX.

Keras like implementation of Deep Learning architectures from scratch using numpy.

Anchor-free Oriented Proposal Generator for Object Detection

Tooling for the Common Objects In 3D dataset.

ColossalAI-Benchmark - Performance benchmarking with ColossalAI

WiFi-based Multi-task Sensing

(ICCV 2021) ProHMR - Probabilistic Modeling for Human Mesh Recovery

The final project of "Applying AI to 2D Medical Imaging Data" of "AI for Healthcare" nanodegree - Udacity.

This is the code for the paper "Motion-Focused Contrastive Learning of Video Representations" (ICCV'21).

PointPillars inference with TensorRT

The final project of "Applying AI to EHR Data" of "AI for Healthcare" nanodegree - Udacity.

Train Scene Graph Generation for Visual Genome and GQA in PyTorch >= 1.2 with improved zero and few-shot generalization.

Framework for training options with different attention mechanism and using them to solve downstream tasks.