Visual disk-usage analyser for docker images

Related tags

DevOps Toolspythondockerdisk-usage
Overview

whaler

PyPI versions PyPI versions

What? A command-line tool for visually investigating the disk usage of docker images

Why? Large images are slow to move and expensive to store. They cost developer productivity by lengthening devops tasks and often contain unnecessary data

Who is this for? Primarily for engineers working with images containing Python packages.

User Stories

This tool should allow you to answer questions such as:

  1. Which file types are occupying the most disk space?
  2. Which are my largest Python packages?
  3. What are my unknown causes of high disk usage?

Quick start

pip install whaler

Run against a local directory

➜ whaler .venv
Running bash -c cd .venv && du -a -k
Done. Serving output at http://localhost:8000 (ctrl+c to exit)
Running python3 -m http.server 8000 --directory=_whaler/html

Run against a docker image

The tool will pull the image first if it is not present.

whaler --image='hl:latest' /
Running docker run --rm --entrypoint=du --workdir=/ hl:latest -a -k
Ignoring what seems to be non-fatal error(s):
du: cannot access './proc/1/task/1/fd/4': No such file or directory
du: cannot access './proc/1/task/1/fdinfo/4': No such file or directory
du: cannot access './proc/1/fd/3': No such file or directory
du: cannot access './proc/1/fdinfo/3': No such file or directory


Done. Serving output at http://localhost:8000 (ctrl+c to exit)
Running python3 -m http.server 8000 --directory=_whaler/html

HTML Report

Play with a hosted demo

Limitations

  1. Platform: whaler uses du to gather disk usage data. It must be present in your docker image
  2. Scale: I have tested the web UI with up to 500,000 file system nodes with du output of up to ~100MB.

Alternatives/Complements to this tool:

  1. Whaler can tell you what is taking up space in the final layer of your Docker image, but you may have intermediate layers which are contributing to the image size. For diving through the layers, use dive
    • Related: read up on multi-stage builds to understand how to mitigate the problem of intermediate layers bloating your image.
  2. For investigating disk usage in non-docker directories, Disk Inventory X is a great tool on OS X which I have based whaler on.
  3. GoogleContainerTools/distroless for base images not containing standard linux distro contents

See also

  1. a good blog post on making small python images

Developing

See .github/workflows/test.yml for the development platform and setup.

For UI, see whaler-ui

You might also like...
Lima is an alternative to using Docker Desktop on your Mac.
Lima is an alternative to using Docker Desktop on your Mac.

lima-xbar-plugin Table of Contents Description Installation Dependencies Lima is an alternative to using Docker Desktop on your Mac. Description This

Joe Block 68 Dec 22, 2022
Build Netbox as a Docker container

netbox-docker The Github repository houses the components needed to build Netbox as a Docker container. Images are built using this code and are relea

Farshad Nick 1 Dec 18, 2021
Some automation scripts to setup a deployable development database server (with docker).

Postgres-Docker Database Initializer This is a simple automation script that will create a Docker Postgres database with a custom username, password,

Pysogge 1 Nov 11, 2021
A job launching library for docker, EC2, GCP, etc.

doodad A library for packaging dependencies and launching scripts (with a focus on python) on different platforms using Docker. Currently supported pl

Justin Fu 55 Aug 27, 2022
A repository containing a short tutorial for Docker (with Python).
A repository containing a short tutorial for Docker (with Python).

Docker Tutorial for IFT 6758 Lab In this repository, we examine the advtanges of virtualization, what Docker is and how we can deploy simple programs

Arka Mukherjee 0 Dec 14, 2021
Bitnami Docker Image for Python using snapshots for the system packages repositories

Python Snapshot packaged by Bitnami What is Python Snapshot? Python is a programming language that lets you work quickly and integrate systems more ef

Bitnami 1 Jan 13, 2022
Hatch plugin for Docker containers

hatch-containers CI/CD Package Meta This provides a plugin for Hatch that allows

Ofek Lev 11 Dec 30, 2022
Create pinned requirements.txt inside a Docker image using pip-tools
Create pinned requirements.txt inside a Docker image using pip-tools

Pin your Python dependencies! pin-requirements.py is a script that lets you pin your Python dependencies inside a Docker container. Pinning your depen

null 4 Aug 18, 2022
This repository contains useful docker-swarm-tools.

docker-swarm-tools This repository contains useful docker-swarm-tools. swarm-guardian This Docker image is intended to be used in a multihost docker e

NeuroForge GmbH & Co. KG 4 Jan 12, 2022
Comments
  • Bounding box on selected node barely visible

    Bounding box on selected node barely visible

    This is due to the canvas not being fully re-drawn upon selection.

    The solution is probably to ensure the canvas can be re-drawn quickly and creating a fresh render every time

    bug 
    opened by alex-treebeard 0
Releases(0.1.2)
Owner
Treebeard Technologies
Helping with Data Infra
Treebeard Technologies
GitHub Repository https://treebeardtech.github.io/whaler/
A Habitica Integration with Github Workflows.

Habitica-Workflow A Habitica Integration with Github Workflows. How To Use? Fork (and Star) this repository. Set environment variable in Settings - S

Priate 2 Dec 20, 2021
SSH to WebSockets Bridge

wssh wssh is a SSH to WebSockets Bridge that lets you invoke a remote shell using nothing but HTTP. The client connecting to wssh doesn't need to spea

Andrea Luzzardi 1.3k Dec 25, 2022
Python job scheduling for humans.

schedule Python job scheduling for humans. Run Python functions (or any other callable) periodically using a friendly syntax. A simple to use API for

Dan Bader 10.4k Jan 02, 2023
Knock your images before these make you painful.

image-knocker Knock your images before these make you painful. Background One day, I had run my deep learning model training program and got off work

Yonghye Kwon 9 Jul 25, 2022
A charmed operator for running PGbouncer on kubernetes.

operator-template Description TODO: Describe your charm in a few paragraphs of Markdown Usage TODO: Provide high-level usage, such as required config

Canonical 1 Dec 01, 2022
A cron monitoring tool written in Python & Django

Healthchecks Healthchecks is a cron job monitoring service. It listens for HTTP requests and email messages ("pings") from your cron jobs and schedule

Healthchecks 5.8k Jan 02, 2023
DAMPP (gui) is a Python based program to run simple webservers using MySQL, Php, Apache and PhpMyAdmin inside of Docker containers.

DAMPP (gui) is a Python based program to run simple webservers using MySQL, Php, Apache and PhpMyAdmin inside of Docker containers.

Sehan Weerasekara 1 Feb 19, 2022
Tools and Docker images to make a fast Ruby on Rails development environment

Tools and Docker images to make a fast Ruby on Rails development environment. With the production templates, moving from development to production will be seamless.

1 Nov 13, 2022
Bugbane - Application security tools for CI/CD pipeline

BugBane Набор утилит для аудита безопасности приложений. Основные принципы и осо

GardaTech 20 Dec 09, 2022
Honcho: a python clone of Foreman. For managing Procfile-based applications.

___ ___ ___ ___ ___ ___ /\__\ /\ \ /\__\ /\ \ /\__\ /\

Nick Stenning 1.5k Jan 03, 2023
A Simple script to hunt unused Kubernetes resources.

K8SPurger A Simple script to hunt unused Kubernetes resources. Release History Release 0.3 Added Ingress Added Services Account Adding RoleBindding Re

Yogesh Kunjir 202 Nov 19, 2022
MagTape is a Policy-as-Code tool for Kubernetes that allows for evaluating Kubernetes resources against a set of defined policies to inform and enforce best practice configurations.

MagTape is a Policy-as-Code tool for Kubernetes that allows for evaluating Kubernetes resources against a set of defined policies to inform and enforce best practice configurations. MagTape includes

T-Mobile 143 Dec 27, 2022
Django-Kubernetes - Learn how to deploy a docker-based Django application into a Kubernetes cluster into production on DigitalOcean

Django & Kubernetes Learn how to deploy a production-ready Django application in

Coding For Entrepreneurs 100 Jan 01, 2023
Deploying a production-ready Django project using Nginx and Gunicorn

django-nginx-gunicorn This project is for deploying a production-ready Django project using Nginx and Gunicorn. Running a local server of Django is no

Arash Sayareh 8 Jul 03, 2022
Automate SSH in python easily!

RedExpect RedExpect makes automating remote machines over SSH very easy to do and is very fast in doing exactly what you ask of it. Based on ssh2-pyth

Red_M 19 Dec 17, 2022
Tiny Git is a simplified version of Git with only the basic functionalities to gain better understanding of git internals.

Tiny Git is a simplified version of Git with only the basic functionalities to gain better understanding of git internals. Implemented Functi

Ahmed Ayman 2 Oct 15, 2021
This project shows how to serve an TF based image classification model as a web service with TFServing, Docker, and Kubernetes(GKE).

Deploying ML models with CPU based TFServing, Docker, and Kubernetes By: Chansung Park and Sayak Paul This project shows how to serve a TensorFlow ima

Chansung Park 104 Dec 28, 2022
Bash-based Python-venv convenience wrapper

venvrc Bash-based Python-venv convenience wrapper. Demo Install Copy venvrc file to ~/.venvrc, and add the following line to your ~/.bashrc file: # so

1 Dec 29, 2022
Daemon to ban hosts that cause multiple authentication errors

__ _ _ ___ _ / _|__ _(_) |_ ) |__ __ _ _ _ | _/ _` | | |/ /| '_ \/ _` | ' \

Fail2Ban 7.8k Jan 09, 2023
A honey token manager and alert system for AWS.

SpaceSiren SpaceSiren is a honey token manager and alert system for AWS. With this fully serverless application, you can create and manage honey token

287 Nov 09, 2022