Localizing-Visual-Sounds-the-Hard-Way

Code and Dataset for "Localizing Visual Sounds the Hard Way".

The repo contains code and our pre-trained model.

Environment

Python 3.6.8
Pytorch 1.3.0

Flickr-SoundNet

We provide the pretrained model here.

To test the model, testing data and ground truth should be downloaded from learning to localize sound source.

Then run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --gt_path "path to ground truth" --testset "flickr"

VGG-Sound Source

We provide the pretrained model here.

To test the model, run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --testset "vggss"

(Note, some gt bounding boxes are updated recently, all results on VGG-SS cause a 2~3% difference on IoU.)

Both test data should be placed in the following structure.

data path
│
└───frames
│   │   image001.jpg
│   │   image002.jpg
│   │
└───audio
    │   audio011.wav
    │   audio012.wav

Citation

@InProceedings{Chen21,
              title        = "Localizing Visual Sounds the Hard Way",
              author       = "Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, Andrew Zisserman",
              booktitle    = "CVPR",
              year         = "2021"}

Localizing Visual Sounds the Hard Way

Related tags

Overview

Localizing-Visual-Sounds-the-Hard-Way

Environment

Flickr-SoundNet

VGG-Sound Source

Citation

Owner

Honglie Chen

This repository is the code of the paper Accelerating Deep Reinforcement Learning for Digital Twin Network Optimization with Evolutionary Strategies

Final project code: Implementing MAE with downscaled encoders and datasets, for ESE546 FA21 at University of Pennsylvania

EgoNN: Egocentric Neural Network for Point Cloud Based 6DoF Relocalization at the City Scale

Joint detection and tracking model named DEFT, or ``Detection Embeddings for Tracking.

Remote sensing change detection tool based on PaddlePaddle

Tensorboard for pytorch (and chainer, mxnet, numpy, ...)

[CVPR 2022 Oral] Crafting Better Contrastive Views for Siamese Representation Learning

E2e music remastering system - End-to-end Music Remastering System Using Self-supervised and Adversarial Training

ManipNet: Neural Manipulation Synthesis with a Hand-Object Spatial Representation - SIGGRAPH 2021

Train CNNs for the fruits360 data set in NTOU CS「Machine Vision」class.

Randomizes the warps in a stock pokeemerald repo.

YOLOX + ROS(1, 2) object detection package

OpenMMLab Text Detection, Recognition and Understanding Toolbox

Pun Detection and Location

A Unified Framework and Analysis for Structured Knowledge Grounding

Flax is a neural network ecosystem for JAX that is designed for flexibility.

Chess reinforcement learning by AlphaGo Zero methods.

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation

This is a Pytorch implementation of paper: DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

The repository contains source code and models to use PixelNet architecture used for various pixel-level tasks. More details can be accessed at .