A simple python tool for explore your object detection dataset

Overview

Pyodi

Python Object Detection Insights

Documentation: https://gradiant.github.io/pyodi

Introduction

A simple tool for explore your object detection dataset. The goal of this library is to provide simple and intuitive visualizations from your dataset and automatically find the best parameters for generating a specific grid of anchors that can fit you data characteristics

Component Description
paint annotations paints COCO format annotations and predictions
ground-truth explore your dataset ground truth characteristics
evaluation evaluates AP and AR for between predictions and ground truth
train-config generation automatically generate anchors for your data
train-config evaluation evaluate the fitness between you data and your anchors
coco merge automatically merge COCO annotation files
coco split automatically split COCO annotation files in train and val subsets
crops split creates a new dataset by splitting images into crops and adapting the annotations file
crops merge translate COCO ground truth or COCO predictions crops split into original image coordinates

Installation

pip install pyodi

Usage

Pyodi includes different applications that can help you to extract the most from your dataset. You can download our TINY_COCO_ANIMAL dataset from the releases page in order to test the example commands. A classic flow could follow the following steps:

1. Annotation visualization

With pyodi paint_annotations you can easily visualize in a beautiful format your object detection dataset.

pyodi paint-annotations \
  $TINY_COCO_ANIMAL/annotations/train.json \
  $TINY_COCO_ANIMAL/sample_images \
  $TINY_COCO_ANIMAL/painted_images

COCO image with painted annotations

2. Ground truth exploration

It is very recommended to intensively explore your dataset before starting training. The analysis of your images and annotations will allow you to optimize aspects as the optimum image input size for your network or the shape distribution of the bounding boxes. You can use the ground_truth app for this task:

pyodi ground-truth $TINY_COCO_ANIMAL/annotations/train.json

Image shape distribution

Bbox distribution

Bbox center distribution

3. Train config generation

The design of anchors is critical for the performance of one-stage detectors. Pyodi can help you to automatically design a set of anchors that fit your data distribution.

pyodi train-config generation \
  $TINY_COCO_ANIMAL/annotations/train.json \
  --input-size [1280,720] \
  --n-ratios 3 --n-scales 3

Anchor clustering plot

4. Train config evaluation

Pyodi evaluation app has been designed with the aim of providing a simple tool to understand how well are your anchors matching your dataset. It automatically runs by default after executing train-config generation but it can also be run independently with:

pyodi train-config evaluation \
  $TINY_COCO_ANIMAL/annotations/train.json \
  $TINY_COCO_ANIMAL/resources/anchor_config.py \
  --input-size [1280,720]

Anchor overlap plot

Contributing

We appreciate all contributions to improve Pyodi. Please refer to Contributing guide for more info.

Comments
  • the results are not displayed or saved,

    the results are not displayed or saved,

    When I run pyodi ground-truth ../instances_train2017.json according to the tutorial, the results are not displayed or saved, and the code does not report errors. The following are the results:

    2021-08-27 21:57:18.223 | INFO     | pyodi.core.utils:coco_ground_truth_to_df:38 - Loading Ground Truth File
    2021-08-27 21:57:18.302 | INFO     | pyodi.core.utils:coco_ground_truth_to_df:48 - Converting COCO Ground Truth to pd.DataFrame
    2021-08-27 21:57:18.418 | INFO     | pyodi.plots.common:plot_scatter_with_histograms:61 - Plotting Scatter with Histograms
    2021-08-27 21:57:18.735 | INFO     | pyodi.plots.common:plot_scatter_with_histograms:61 - Plotting Scatter with Histograms
    
    
    bug 
    opened by cyh-123 5
  • add windows support by updating dependency

    add windows support by updating dependency

    @mmeendez8 @igonro @cgiraldo thanks a lot for this awesome package!

    pycocotools-fix package currently doesnt work on windows: https://github.com/junjuew/cocoapi/pull/4

    official pycocotools package is constantly updated by the facebook research team: https://pypi.org/project/pycocotools/

    by accepting this pr, pyodi package will be supported on windows.

    opened by fcakyon 5
  • Question about coco merge

    Question about coco merge

    I am attempting to merge 2 coco format file with pyodi and i cannot get it to work at all.

    The command I am running: pyodi coco merge result.json studio_test_val.json output.json

    The error I am getting: Traceback (most recent call last):

      File "/home/christ/miniconda3/envs/label-studio/bin/pyodi", line 5, in <module>
        from pyodi.cli import app
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pyodi/cli.py", line 5, in <module>
        from pyodi.apps.evaluation import evaluation
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pyodi/apps/evaluation.py", line 39, in <module>
        from pycocotools.cocoeval import COCOeval
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pycocotools/cocoeval.py", line 7, in <module>
        from . import mask as maskUtils
      File "/home/christ/miniconda3/envs/label-studio/lib/python3.9/site-packages/pycocotools/mask.py", line 3, in <module>
        import pycocotools._mask as _mask
      File "pycocotools/_mask.pyx", line 1, in init pycocotools._mask
    ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
    

    Please help me with this issue. Thank you.

    opened by ChrisTurnbull04 2
  • Update kaleido and plotly versions

    Update kaleido and plotly versions

    Fixes kaleido permission err:

    PermissionError: [Errno 13] Permission denied: '/home/mmendez/anaconda3/envs/pyodi/lib/python3.7/site-packages/kaleido-0.1.0-py3.7-linux-x86_64.egg/kaleido/executable/kaleido'

    bug 
    opened by mmeendez8 2
  • Fix precommit issue between black and flake

    Fix precommit issue between black and flake

    Code to reproduce:

        image_0[
            int(image_0_bbox_top) : int(image_0_bbox_top + image_0_bbox_height),
            int(image_0_bbox_left) : int(image_0_bbox_left + image_0_bbox_width),
        ] = 0
    

    Output:

    flake8...................................................................Failed
    - hook id: flake8
    - exit code: 1
    
    tests/apps/test_paint_annotations.py:137:30: E203 whitespace before ':'
    

    Black is adding extra whitespaces that do not match with flake8 config

    bug CI/CD 
    opened by mmeendez8 2
  • [apps.coco_merge] `image_id` not being properly handled

    [apps.coco_merge] `image_id` not being properly handled

    The way image_id is currently updated only works when all the jsons contain image_ids filling all the range [0, N).

    If any of the jsons start it's image_id at other number or some image_id is missing from the range, the results are incorrect.

    bug 
    opened by daavoo 2
  • Add versioneer

    Add versioneer

    Use https://github.com/warner/python-versioneer for version control.

    Follow setup: https://jacobtomlinson.dev/posts/2020/versioning-and-formatting-your-python-code/

    enhancement CI/CD 
    opened by daavoo 2
  • [apps.paint_annotations] Extend coloring options

    [apps.paint_annotations] Extend coloring options

    Currently, when using pyodi paint-annotations each of the bounding boxes is painted with a random color:

    https://github.com/pyodi/pyodi/blob/master/pyodi/apps/paint_annotations.py#L83

    It would be a desirable feature to make this behaviour optional and allow the user to specify different coloring options. For example:

    • Constant color. Use same color for all annotations.
    • Color by key (property inside each annotation dictionary; i.e. category_id). Use same color for annotations sharing the same key.

    Example content of annotation:

    {
        "image_id": 0,
        "bbox": [704.2313842773438, 441.23284912109375, 15.76861572265625, 16.6729736328125],
        "score": 0.7878580093383789, 
        "category_id": 1
    }
    
    enhancement good first issue 
    opened by daavoo 2
  • Change histograms colors

    Change histograms colors

    Right now the histograms have the same color, so the appearance of the legend is a little bit weird.

    It would be nice if both histograms have different colors. The colors should be also different from the default plotly color sequence (More info: https://plotly.com/python/discrete-color/#color-sequences-in-plotly-express)

    enhancement 
    opened by igonro 2
  • Updated numpy version.

    Updated numpy version.

    Input:

    pip install pyodi
    pyodi coco merge 1.json 2.json output.json
    

    Output:

    ValueError: numpy.ndarray size changed, may indicate binary incompatibility
    

    When you run this code, the error will be fixed.

    pip install numpy==1.23.5
    
    opened by kadirnar 1
  • Question about Train Config Generation App.

    Question about Train Config Generation App.

    I want to get the anchor setting of my dataset, the example you showed is as follows: pyodi train-config generation \ $TINY_COCO_ANIMAL/annotations/train.json \ --input-size [1280,720] \ --n-ratios 3 --n-scales 3

    However, for Faster R-CNN, the size of input image is not fixed, in this case, how should I set the ' --input-size'?

    opened by missyoudaisy 0
  • Question: how precisely we can design anchors with pyodi visualisations

    Question: how precisely we can design anchors with pyodi visualisations

    The readme states Pyodi can help you to automatically design a set of anchors that fit your data distribution can you provide an example or reference that describes this process, and how the visualisations relate? Many thanks

    opened by robmarkcole 2
  • About clustering

    About clustering

    Excuse me, I want to know how clustering is done. I don't see the call of core/clustering.py in the place where the plot is drawn? Did I understand it wrong?

    question 
    opened by cyh-123 1
  • [apps.ground_truth] Add Image & Bounding Box combined plots.

    [apps.ground_truth] Add Image & Bounding Box combined plots.

    The current plots analyze the images and bounding boxes separately. There is some important insights missing that need to combine both sources of information. For example:

    • Total Area occupied by bounding boxes in each image.
    • Number of bounding boxes per image.
    • Mean distance between bounding box centroids per image.
    enhancement 
    opened by daavoo 0
Releases(v0.0.8)
  • v0.0.8(Dec 19, 2022)

    What's Changed

    • Feature/issue 174 update setup packages version by @jorgenusan in https://github.com/Gradiant/pyodi/pull/176
    • Bump to 0.0.8 by @jorgenusan in https://github.com/Gradiant/pyodi/pull/181

    New Contributors

    • @jorgenusan made their first contribution in https://github.com/Gradiant/pyodi/pull/176

    Full Changelog: https://github.com/Gradiant/pyodi/compare/v0.0.7...v0.0.8

    Source code(tar.gz)
    Source code(zip)
  • v0.0.7(Dec 16, 2021)

    What's Changed

    • Remove logger.catch decorator by @mmeendez8 in https://github.com/Gradiant/pyodi/pull/166
    • Feature: make JSON indentation configurable when merging COCO files by @philenius in https://github.com/Gradiant/pyodi/pull/171

    New Contributors

    • @philenius made their first contribution in https://github.com/Gradiant/pyodi/pull/171

    Full Changelog: https://github.com/Gradiant/pyodi/compare/v0.0.6...v0.0.7

    Source code(tar.gz)
    Source code(zip)
  • v0.0.6(Oct 29, 2021)

    What's Changed

    • Update kaleido and plotly versions by @mmeendez8 in https://github.com/Gradiant/pyodi/pull/163
    • Fix broken release link in README by @mmeendez8 in https://github.com/Gradiant/pyodi/pull/168
    • Fix macOS install by @daavoo in https://github.com/Gradiant/pyodi/pull/170

    Full Changelog: https://github.com/Gradiant/pyodi/compare/v0.0.5...v0.0.6

    Source code(tar.gz)
    Source code(zip)
  • v0.0.5(Aug 26, 2021)

    • Readme add pypi badge (#154)
    • Bugfix crops split fix minor bugs in crops split (#156)
    • Add windows to ci (#159)
    • update pycocotools dependency (#157)
    • [core nms] remove ensemble boxes dependency
    Source code(tar.gz)
    Source code(zip)
  • v0.0.4(Jul 5, 2021)

  • v0.0.3(Jul 2, 2021)

    • Call train-config evaluation from train-config generation app #147
    • Refactor load_ground_truth #149
    • [apps.paint_annotations] Add argument to paint only first n images #151
    Source code(tar.gz)
    Source code(zip)
Owner
GRADIANT - Centro Tecnolóxico de Telecomunicacións de Galicia
GRADIANT - Centro Tecnolóxico de Telecomunicacións de Galicia
🐍PyNode Next allows you to easily create beautiful graph visualisations and animations

PyNode Next A complete rewrite of PyNode for the modern era. Up to five times faster than the original PyNode. PyNode Next allows you to easily create

ehne 3 Feb 12, 2022
Graphical visualizer for spectralyze by Lauchmelder23

spectralyze visualizer Graphical visualizer for spectralyze by Lauchmelder23 Install Install matplotlib and ffmpeg. Put ffmpeg.exe in same folder as v

Matthew 1 Dec 21, 2021
Simple Python interface for Graphviz

Simple Python interface for Graphviz

Sebastian Bank 1.3k Dec 26, 2022
GD-UltraHack - A Mod Menu for Geometry Dash. Specifically a MegahackV5 clone in Python. Only for Windows

GD UltraHack: The Mod Menu that Nobody asked for. This is a mod menu for the gam

zeo 1 Jan 05, 2022
Visualize the bitcoin blockchain from your local node

Project Overview A new feature in Bitcoin Core 0.20 allows users to dump the state of the blockchain (the UTXO set) using the command dumptxoutset. I'

18 Sep 11, 2022
Data visualization electromagnetic spectrum

Datenvisualisierung-Elektromagnetischen-Spektrum Anhand des Moduls matplotlib sollen die Daten des elektromagnetischen Spektrums dargestellt werden. D

Pulsar 1 Sep 01, 2022
🎨 Python Echarts Plotting Library

pyecharts Python ❤️ ECharts = pyecharts English README 📣 简介 Apache ECharts (incubating) 是一个由百度开源的数据可视化,凭借着良好的交互性,精巧的图表设计,得到了众多开发者的认可。而 Python 是一门富有表达

pyecharts 13.1k Jan 03, 2023
Glue is a python project to link visualizations of scientific datasets across many files.

Glue Glue is a python project to link visualizations of scientific datasets across many files. Click on the image for a quick demo: Features Interacti

675 Dec 09, 2022
A little word cloud generator in Python

Linux macOS Windows PyPI word_cloud A little word cloud generator in Python. Read more about it on the blog post or the website. The code is tested ag

Andreas Mueller 9.2k Dec 30, 2022
Compute and visualise incidence (reworking of the original incidence package)

incidence2 incidence2 is an R package that implements functions and classes to compute, handle and visualise incidence from linelist data. It refocuss

15 Nov 22, 2022
A Python Library for Self Organizing Map (SOM)

SOMPY A Python Library for Self Organizing Map (SOM) As much as possible, the structure of SOM is similar to somtoolbox in Matlab. It has the followin

Vahid Moosavi 497 Dec 29, 2022
A concise grammar of interactive graphics, built on Vega.

Vega-Lite Vega-Lite provides a higher-level grammar for visual analysis that generates complete Vega specifications. You can find more details, docume

Vega 4k Jan 08, 2023
🐞 📊 Ladybug extension to generate 2D charts

ladybug-charts Ladybug extension to generate 2D charts. Installation pip install ladybug-charts QuickStart import ladybug_charts API Documentation Loc

Ladybug Tools 3 Dec 30, 2022
A Scheil-Gulliver simulation tool using pycalphad.

scheil A Scheil-Gulliver simulation tool using pycalphad. import matplotlib.pyplot as plt from pycalphad import Database, variables as v from scheil i

pycalphad 6 Dec 10, 2021
Plotting library for IPython/Jupyter notebooks

bqplot 2-D plotting library for Project Jupyter Introduction bqplot is a 2-D visualization system for Jupyter, based on the constructs of the Grammar

3.4k Dec 30, 2022
Open-questions - Open questions for Bellingcat technical contributors

Open questions for Bellingcat technical contributors These are difficult, long-term projects that would contribute to open source investigations at Be

Bellingcat 234 Dec 31, 2022
Yata is a fast, simple and easy Data Visulaization tool, running on python dash

Yata is a fast, simple and easy Data Visulaization tool, running on python dash. The main goal of Yata is to provide a easy way for persons with little programming knowledge to visualize their data e

Cybercreek 3 Jun 28, 2021
Cartopy - a cartographic python library with matplotlib support

Cartopy is a Python package designed to make drawing maps for data analysis and visualisation easy. Table of contents Overview Get in touch License an

1.2k Jan 01, 2023
A python package for animating plots build on matplotlib.

animatplot A python package for making interactive as well as animated plots with matplotlib. Requires Python = 3.5 Matplotlib = 2.2 (because slider

Tyler Makaro 394 Dec 18, 2022
Python & Julia port of codes in excellent R books

X4DS This repo is a collection of Python & Julia port of codes in the following excellent R books: An Introduction to Statistical Learning (ISLR) Stat

Gitony 5 Jun 21, 2022