POCO: Point Convolution for Surface Reconstruction

Related tags

Deep LearningPOCO
Overview

POCO: Point Convolution for Surface Reconstruction

by: Alexandre Boulch and Renaud Marlet

Abstract

Implicit neural networks have been successfully used for surface reconstruction from point clouds. However, many of them face scalability issues as they encode the isosurface function of a whole object or scene into a single latent vector. To overcome this limitation, a few approaches infer latent vectors on a coarse regular 3D grid or on 3D patches, and interpolate them to answer occupancy queries. In doing so, they loose the direct connection with the input points sampled on the surface of objects, and they attach information uniformly in space rather than where it matters the most, i.e., near the surface. Besides, relying on fixed patch sizes may require discretization tuning. To address these issues, we propose to use point cloud convolutions and compute latent vectors at each input point. We then perform a learning-based interpolation on nearest neighbors using inferred weights. Experiments on both object and scene datasets show that our approach significantly outperforms other methods on most classical metrics, producing finer details and better reconstructing thinner volumes.

Citation

TODO

Dependencies

Our code rely on Pytorch Geometric, which should be installed. Then, run:

python setup.py build_ext --inplace

Data

ShapeNet (Occupancy Network pre-processing)

We use the ShapeNet dataset as pre-processed by Occupancy Networks. Please refer to original repository for downloading the data.

It should be placed in the folder data/ShapeNet.

SyntheticRooms (Occupancy Network pre-processing)

We use the SyntheticRooms dataset as created by Occupancy Networks. Please refer to original repository for downloading the data.

It should be placed in the folder data/synthetic_room_dataset.

ABC / RealWorld / Famous and Thingi10k

They should be placed in the folders: data/3d_shapes_abc, data/3d_shapes_abc_training, data/3d_shapes_famous, data/3d_shapes_real_world and data/3d_shapes_thingi10k.

SceneNet

Download the SceneNet dataset.

Watertight models

We use the code from https://github.com/hjwdzh/Manifold for watertight model creation. Please download and compile it.

To create the watertight models, create a destination directory data/SceneNet, copy-paste the python script datasets/scenenet_watertight.py into the directory and run it with python scenenet_watertight.py. You will need to modify the raw_data_dir and the manifold_code_dir to match your data and code locations.

Creation of the point clouds

You can use the script datasets/scenenet_sample.py. You will also need to modify the paths.

Precomputed meshes and point clouds

For easy use of the code, we provide precomputed meshes, and point clouds SceneNet20, SceneNet100, SceneNet500 and SceneNet1000.

Location

The watertight models should be in data/SceneNet.

The points are located in data/SceneNet20, data/SceneNet100, data/SceneNet1000 and data/SceneNet500.

Training

# train on ShapeNet with 3k points, noise and without normals 
python train.py --config configs/config_shapenet.yaml 

# train on ShapeNet with 10k points, no noise and normals
python train.py --config configs/config_shapenet.yaml --normals True --random_noise 0 --experiment_name Normals

# train on ABC with 3k points
python train.py --config configs/config_abc.yaml --experiment_name 3k

# train on ABC with 10k points
python train.py --config configs/config_abc.yaml --experiment_name 10k --manifold_points 10000 --training_batch_size 8

Generation

ShapeNet

python generate.py --config results/ShapeNet_None_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_resolution_global 128

SyntheticRooms

python generate.py --config results/SyntheticRooms_None_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_resolution_global 256 --num_mesh 1

ABC, RealWorld, Famous and Thingi10k

# Model trained with 3k points
python generate.py --config results/ABC_3k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256

python generate.py --config results/ABC_3k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256 --manifold_points -1 --gen_subsample_manifold 3000 --gen_subsample_manifold_iter 10 --gen_descriptor gen_sub3k_iter10

# Model trained with 10k points
python generate.py --config results/ABC_10k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256

python generate.py --config results/ABC_10k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256 --manifold_points -1 --noise 0.0 --gen_subsample_manifold 10000 --gen_subsample_manifold_iter 10 --gen_descriptor gen_sub3k_iter10

For faster generation, one would want to use a lower marching cubes parameter --gen_resolution_global, e.g. 128.

DATASET_NAME should be replaced by ABCTest, ABCTestNoiseFree, ABCTestExtraNoise, RealWorld, FamousTest, FamousTestNoiseFree, FamousTestExtraNoisy, FamousTestSparse, FamousTestDense, Thingi10kTest, Thingi10kTestNoiseFree, Thingi10kTestExtraNoisy, Thingi10kTestSparse, Thingi10kTestDense

DATASET_DIR should be replaced by 3d_shapes_abc, 3d_shapes_famous, 3d_shapes_real_world, 3d_shapes_thingi10k

SceneNet

python generate.py --config results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_autoscale True --gen_autoscale_target 0.01 --gen_resolution_metric 0.01 --dataset_name DATASET_NAME --dataset_root data/DATASET_NAME --manifold_points -1

DATASET_NAME should be replaced by SceneNet20, SceneNet100, SceneNet500, SceneNet1000.

The --gen_autoscale_target parameter is the average distance between a point and its nearest neighbor in the training set, 0.01 for ShapeNet train set with 3000 points.

Evaluation

ShapeNet

python eval_meshes.py --gendir results/ShapeNet_None_FKAConv_InterpAttentionKHeadsNet_None/gen_ShapeNet_test_3000/ --meshdir meshes --dataset ShapeNet --split test --gtdir data/ShapeNet

SyntheticRooms

python eval_meshes.py --gendir results/SyntheticRooms_None_FKAConv_InterpAttentionKHeadsNet_None/gen_SyntheticRooms_test_10000/ --meshdir meshes --dataset SyntheticRooms --split test --gtdir data/synthetic_room_dataset

SceneNet

python eval_scenenet.py --gendir results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/gen_SceneNet20_test_allPts/ --meshdir meshes --gtdir data/SceneNet

ABC, Famous and Thingi10k

For evaluation on the ABCTest please run:

python eval_point2surf_datasets.py --gendir results/ABC_3k_FKAConv_InterpAttentionKHeadsNet_None/gen_ABCTest_test_3000 --meshdir meshes/04_pts/ --gtdir data/3d_shapes_abc/abc/

You can adapt the paths to evaluate on the different datasets and noise variants.

Pretrained models

We provide pre-trained models for FKAConv backbone.

ShapeNet 3k, noise, no normals

ShapeNet 3k, no noise, normals

SyntheticRooms 10k

ABC 3k

ABC 10k

Configuration

The code was used with Ubuntu, Python 3.7.10, Cuda 11.1 and Pytorch 1.8.1

Comments
  • Package Versions

    Package Versions

    Hi,

    I tried to reproduce your results, but I ran into a possible version mismatch between Pytorch and Pytorch_geometric.

    I created my environment with the following commands:

    conda create --name poco python=3.7.10
conda install pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.1 cudatoolkit=11.1 -c pytorch -c conda-forge
conda install -c conda-forge cython
conda install -c conda-forge tqdm 
conda install -c conda-forge scikit-image 
conda install -c open3d-admin open3d 
conda install -c conda-forge scikit-learn 
conda install -c conda-forge pyyaml 
conda install -c conda-forge addict 
conda install -c conda-forge pandas 
conda install -c conda-forge plyfile 
conda install -c conda-forge pytorch_geometric

    Compilation with python setup.py build_ext --inplace seems to work but python generate.py --config results/ABC_10k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/3d_shapes_abc/abc/ --gen_resolution_global 256 results in OSError: /home/perler/miniconda3/envs/poco/lib/python3.7/site-packages/torch_sparse/_version.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

    Installed versions are:

    (poco) [email protected]:~/repos/poco$ conda list pytorch
# packages in environment at /home/perler/miniconda3/envs/poco:
#
# Name                    Version                   Build  Channel
pytorch                   1.8.1           py3.7_cuda11.1_cudnn8.0.5_0    pytorch
pytorch-cpu               1.1.0               py3.7_cpu_0    pytorch
pytorch_geometric         2.0.3              pyh6c4a22f_0    conda-forge
pytorch_sparse            0.6.4            py37hcae2be3_0    conda-forge

    Again, the CPU-version... but that's a different issue.

    AFAIK, they added sparse tensors only recently to Pytorch, so the installed Pytorch-geometric version might be too new. Which version of Pytorch-geometric do I need?

    Can you please create a requiremtents.txt and/or environment.yaml?

    opened by ErlerPhilipp 13
  • Cython related error when building extensions

    Cython related error when building extensions

    Hi, I completely followed the instructions, using cuda11.1 and Ubuntu, but still having this issue gcc: error: /usr/POCO/build/temp.linux-x86_64-3.7/eval/src/utils/libkdtree/pykdtree/kdtree.o: No such file or directory gcc: error: /usr/POCO/build/temp.linux-x86_64-3.7/eval/src/utils/libkdtree/pykdtree/_kdtree_core.o: No such file or directory It's asking to link .o files before it is generated. But looks like required Cython 0.27.3 is not compatible with Python 3.7.10, which is required by this project. I'm wondering does anyone have a working combination of the version of the packages? Thank you!

    opened by zhaoyuanyuan2011 5
  • Download ABC / RealWorld / Famous and Thingi10k data

    Download ABC / RealWorld / Famous and Thingi10k data

    Hello,

    Thanks for the excellent paper and for releasing the code.

    I'm wondering where I can download the ABC, RealWorld, Famous, and Thingi10k data?

    Best and have a nice day, Mulin

    opened by MulinYu 2
  • Training on my own dataset

    Training on my own dataset

    I met some problem when training on my own dataset, which is processed as ShapeNet, containing pointclouds with normals, SDF points and occ values. However, there is an error when I try to train on it.

    File "/media/yangxilab/DiskB/sanren/3Drecon/POCOnasa/networks/backbone/fkaconv_network.py", line 193, in forward x0 = self.activation(self.bn0(self.cv0(x, pos, pos, data["ids00"]))) KeyError: 'ids00'

    I'm confusing that what data["ids00"] in x0 = self.activation(self.bn0(self.cv0(x, pos, pos, data["ids00"]))) refers to? And similarily, there are many unkown data like data["ids10"]. How could I get this data? Thanks in advance!

    opened by Green89757 2
  • Failed to run

    Failed to run "python setup.py build_ext --inplace"

    Hello,

    After installing Pytorch and Pytorch Geometric, running python setup.py build_ext --inplace gives the following error:

    gcc: error: /my/computer/path/to/POCO/build/temp.linux-x86_64-3.7/eval/src/utils/libkdtree/pykdtree/_kdtree_core.o: No such file or directory
error: command 'gcc' failed with exit status 1

    Any suggestions would be appreciated.

    opened by TixiaoShan 2
  • Bug during generating

    Bug during generating

    Hello again,

    When I try to generate the shapenet results with the command:

    python generate.py --config results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_resolution_global 128

    I got this error:

    **loading CUDA OK INFO:root:Creating the network INFO:root:InterpNet - Simple - K=64 INFO:root:Network -- backbone -- 12783956 parameters INFO:root:Network -- projection -- 6498 parameters INFO:root:Network -- Number of parameters 12790454 INFO:root:Getting the dataset INFO:root:Normals as features INFO:root:Dataset - ShapeNet- None INFO:root:Dataset - len 8751 INFO:root:InterpNet - Simple - K=64 INFO:root:Network -- backbone -- 12783956 parameters INFO:root:Network -- projection -- 6498 parameters

    0%| | 0/8751 [00:00<?, ?it/s] 0%| | 0/8751 [00:04<?, ?it/s] Traceback (most recent call last): File "generate.py", line 594, in main(config) File "generate.py", line 539, in main step=step File "generate.py", line 170, in export_mesh_and_refine_vertices_region_growing_v2 mesh = o3d.geometry.TriangleMesh(o3d_verts, o3d_faces) TypeError: init(): incompatible constructor arguments. The following argument types are supported: 1. open3d.open3d.geometry.TriangleMesh() 2. open3d.open3d.geometry.TriangleMesh(arg0: open3d.open3d.geometry.TriangleMesh)

    Invoked with: std::vectorEigen::Vector3d with 133788 elements. Use numpy.asarray() to access data., std::vectorEigen::Vector3i with 267574 elements. Use numpy.asarray() to access data.

    Did you forget to #include <pybind11/stl.h>? Or <pybind11/complex.h>, <pybind11/functional.h>, <pybind11/chrono.h>, etc. Some automatic conversions are optional and require extra headers to be included when compiling your pybind11 module. **

    Thanks in advance and have a nice day. Best. Mulin

    opened by MulinYu 1
  • It takes one hour to generate the mesh from 16k point clouds in ScanNet dataset

    It takes one hour to generate the mesh from 16k point clouds in ScanNet dataset

    Dear author,

    Thanks for sharing your interesting work!

    I use the provided script to generate the mesh from the ScanNet dataset, and I found that it takes a long time (e.g, nearly 1 hour from 16K point clouds input). I am wondering if it is a normal situation. BTW, I just use the following script to generate mesh; python generate.py --config results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_autoscale True --gen_autoscale_target 0.01 --gen_resolution_metric 0.01 --dataset_name SceneNet20 --dataset_root data/SceneNet20 --manifold_points -1

    Looking forward to your reply. Many thanks in advance.

    Best, Runsong

    opened by Runsong123 0
  • RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when running the code

    RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when running the code

    Hi! Thanks for your interesting and enlightening work on point cloud reconstruction tasks, and we are trying to reproduce your work. However, we encounter an error when running your code: Capture I am wondering how to fix this bug? Thanks for your timely response.

    opened by CUHKWilliam 3
Owner
valeo.ai
We are an international team based in Paris, conducting AI research for Valeo automotive applications, in collaboration with world-class academics.
valeo.ai
GitHub Repository
MOT-Tracking-by-Detection-Pipeline - For Tracking-by-Detection format MOT (Multi Object Tracking), is it a framework that separates Detection and Tracking processes?

MOT-Tracking-by-Detection-Pipeline Tracking-by-Detection形式のMOT(Multi Object Trac

KazuhitoTakahashi 41 Nov 23, 2022
Magic tool for managing internet connection in local network by @zalexdev

Megacut ✂️ A new powerful Python3 tool for managing internet on a local network Installation git clone https://github.com/stryker-project/megacut cd m

Stryker 12 Dec 15, 2022
Natural Posterior Network: Deep Bayesian Predictive Uncertainty for Exponential Family Distributions

Natural Posterior Network This repository provides the official implementation o

Oliver Borchert 54 Dec 06, 2022
RP-GAN: Stable GAN Training with Random Projections

RP-GAN: Stable GAN Training with Random Projections This repository contains a reference implementation of the algorithm described in the paper: Behna

Ayan Chakrabarti 20 Sep 18, 2021
Differential Privacy for Heterogeneous Federated Learning : Utility & Privacy tradeoffs

Differential Privacy for Heterogeneous Federated Learning : Utility & Privacy tradeoffs In this work, we propose an algorithm DP-SCAFFOLD(-warm), whic

19 Nov 10, 2022
Learning multiple gaits of quadruped robot using hierarchical reinforcement learning

Learning multiple gaits of quadruped robot using hierarchical reinforcement learning We propose a method to learn multiple gaits of quadruped robot us

Yunho Kim 17 Dec 11, 2022
Nest - A flexible tool for building and sharing deep learning modules

Nest - A flexible tool for building and sharing deep learning modules Nest is a flexible deep learning module manager, which aims at encouraging code

ZhouYanzhao 41 Oct 10, 2022
Implementation of Lie Transformer, Equivariant Self-Attention, in Pytorch

Lie Transformer - Pytorch (wip) Implementation of Lie Transformer, Equivariant Self-Attention, in Pytorch. Only the SE3 version will be present in thi

Phil Wang 78 Oct 26, 2022
BraTs-VNet - BraTS(Brain Tumour Segmentation) using V-Net

BraTS(Brain Tumour Segmentation) using V-Net This project is an approach to dete

Rituraj Dutta 7 Nov 27, 2022
Self-Supervised Image Denoising via Iterative Data Refinement

Self-Supervised Image Denoising via Iterative Data Refinement Yi Zhang1, Dasong Li1, Ka Lung Law2, Xiaogang Wang1, Hongwei Qin2, Hongsheng Li1 1CUHK-S

Zhang Yi 72 Jan 01, 2023
UniFormer - official implementation of UniFormer

UniFormer This repo is the official implementation of "Uniformer: Unified Transf

SenseTime X-Lab 573 Jan 04, 2023
Face recognition with trained classifiers for detecting objects using OpenCV

Face_Detector Face recognition with trained classifiers for detecting objects using OpenCV Libraries required to be installed using pip Command: cv2 n

Chumui Tripura 0 Oct 31, 2021
Source code for GNN-LSPE (Graph Neural Networks with Learnable Structural and Positional Representations)

Graph Neural Networks with Learnable Structural and Positional Representations Source code for the paper "Graph Neural Networks with Learnable Structu

Vijay Prakash Dwivedi 180 Dec 22, 2022
なりすまし検出(anti-spoof-mn3)のWebカメラ向けデモ

FaceDetection-Anti-Spoof-Demo なりすまし検出(anti-spoof-mn3)のWebカメラ向けデモです。 モデルはPINTO_model_zoo/191_anti-spoof-mn3からONNX形式のモデルを使用しています。 Requirement mediapipe

KazuhitoTakahashi 8 Nov 18, 2022
Convert game ISO and archives to CD CHD for emulation on Linux.

tochd Convert game ISO and archives to CD CHD for emulation. Author: Tuncay D. Source: https://github.com/thingsiplay/tochd Releases: https://github.c

Tuncay 20 Jan 02, 2023
Implements pytorch code for the Accelerated SGD algorithm.

AccSGD This is the code associated with Accelerated SGD algorithm used in the paper On the insufficiency of existing momentum schemes for Stochastic O

205 Jan 02, 2023
Keyword-BERT: Keyword-Attentive Deep Semantic Matching

project discription An implementation of the Keyword-BERT model mentioned in my paper Keyword-Attentive Deep Semantic Matching (Plz cite this github r

1 Nov 14, 2021
Predicting the duration of arrival delays for commercial flights.

Flight Delay Prediction Our objective is to predict arrival delays of commercial flights. According to the US Department of Transportation, about 21%

Jordan Silke 1 Jan 11, 2022
Deep Learning Theory

Deep Learning Theory 整理了一些深度学习的理论相关内容，持续更新。 Overview Recent advances in deep learning theory 总结了目前深度学习理论研究的六个方向的一些结果，概述型，没做深入探讨(2021)。 1.1 complexity

fq 103 Jan 04, 2023
A Parameter-free Deep Embedded Clustering Method for Single-cell RNA-seq Data

A Parameter-free Deep Embedded Clustering Method for Single-cell RNA-seq Data Overview Clustering analysis is widely utilized in single-cell RNA-seque

AI-Biomed @NSCC-gz 3 May 08, 2022