ioztat is a storage load analysis tool for OpenZFS

Related tags

Miscellaneousioztat
Overview

ioztat

ioztat is a storage load analysis tool for OpenZFS. It provides iostat-like statistics at an individual dataset/zvol level.

The statistics offered are read and write operations per second, read and write throughput per second, and the average size (in kilobytes) of read and write operations issued in the current reporting interval. Viewing these statistics at the individual dataset level allows system administrators to identify storage "hot spots" in larger multi-tenant systems--particularly those with many VMs or containers operating essentially independent workloads.

This sample output shows activity which has taken place in the most recent second, on a the ssd zpool of a ZFS virtualization host:

[email protected]:~# ioztat -y -c1 ssd
dataset                                         w/s      wMB/s        r/s      rMB/s   wareq-sz   rareq-sz
ssd                                            0.00       0.00       0.00       0.00       0.00       0.00
   images                                      0.00       0.00       0.00       0.00       0.00       0.00
      DC1                                     17.96       0.10       3.99       0.05       5.66      12.29
      DC2                                     21.95       0.14       0.00       0.00       6.59       0.00
      QB                                       2.00       0.01       0.00       0.00       7.17       0.00
      SAP-TC                                   2.99       0.02       0.00       0.00       6.83       0.00
      SAP4-WIN2019                             2.99       0.05       0.00       0.00      17.07       0.00
      nagios                                   0.00       0.00       0.00       0.00       0.00       0.00
      qemu                                     0.00       0.00       0.00       0.00       0.00       0.00
         autostart                             0.00       0.00       0.00       0.00       0.00       0.00
   iso                                         0.00       0.00       0.00       0.00       0.00       0.00
   unsnapped                                   0.00       0.00       0.00       0.00       0.00       0.00
      rp9                                      0.00       0.00       0.00       0.00       0.00       0.00

For the most part, ioztat behaves the same way that the system standard iostat tool does, with similar arguments.

usage: ioztat [-h] [-s SORT] [-i INTERVAL] [-c COUNT] [-y] [-b]
              pool [pool ...]

iostat for ZFS datasets

positional arguments:
  pool         ZFS pool

optional arguments:
  -h, --help   show this help message and exit
  -s SORT      Sort by: name / wps / wMBps / rps / rMBps
  -i INTERVAL  Time between each report
  -c COUNT     Number of reports generated
  -y           Skip the initial "summary" report
  -b           Use binary (power-of-two) prefixes

The only required argument is the name of at least one zpool to monitor. Without any other arguments, ioztat first prints a summary record showing activity per dataset since the most recent system boot, then prints a new record showing the most recent activity once per second. The -i argument can be used to change the report interval, and the -c argument can be used to limit ioztat to a certain number of intervals before exiting.

For those who wish a continually-updated, easy to read summary of pool activity, watch -n1 ioztat poolname -c1 -y will suit nicely.

Comments
  • Rework formatting

    Rework formatting

    This makes ioztat output more closely resemble zpool iostat:

                                  operations      bandwidth         opsize
    dataset                       read   write    read   write    read   write
    --------------------------  ------  ------  ------  ------  ------  ------
    rpool
       ROOT
          default                   23       0   62.6K       0    2.7K     759
       cvs                           0       0    1.6K     892   24.2K  108.4K
       home                          0       0       0       0       0       0
          freaky                     3       2   29.7K    8.3K   10.2K    3.9K
    

    This also adds -e for exact values (-p in zpool iostat, which we're currently using elsewhere) and -H for scripted mode that drops the header and separates fields by tabs.

    The intent is for this to eventually fix #23, probably by adjusting the space for dataset.

    opened by Freaky 12
  • Painting the bikeshed

    Painting the bikeshed

    Should bandwidth be changed to throughput? This is more correct, but deviates from zpool-iostat(8).

    Should we default to binary mode, since that's all zpool-iostat(8) supports?

    Should we default to a single iteration? This is what iostat(1) and zpool-iostat(8) both do. They also accept interval and count as positional arguments.

    Is + as an ellipsis for truncated dataset names reasonable, or should it be ... despite the additional space it uses?

    opened by Freaky 8
  • Overwrite old reports in terminal by default

    Overwrite old reports in terminal by default

    I thought that this script put a little too much into the terminal. This PR makes it so that only the most recent report is visible, which can be disabled with the new -n argument.

    opened by Aluminite 4
  • No warning if zfs version too low (was: doesn't find datasets on Debian Buster, zfs 0.7.12-2)

    No warning if zfs version too low (was: doesn't find datasets on Debian Buster, zfs 0.7.12-2)

    ioztat -Hey as suggested by the man page doesn't return anything.

    ioztat -SIn rpool, also suggested by the man page, gives "dataset not found: 'rpool'" (rpool of course exists).

    Is there a way to further debug this? Am I missing a dependency?

    opened by steffen-AEI 3
  • Make numeric precision variable

    Make numeric precision variable

    Matching ZFS number formatting behaviour, use between 0 and 2 decimal places according to available space, and further reduce the column size to again match zpool-iostat and make more room for dataset names.

    This includes the careful rounding behaviour of the previous implementation, which I believe ZFS lacks.

    opened by Freaky 3
  • Fix -y when not combined with -c

    Fix -y when not combined with -c

    Fix an error when args.count is None. While I'm here also fix a count of zero that was ignored due to zero being falsey.

    Simplify the logic slightly by changing skipsummary from a bool to a generic skip count.

    opened by Freaky 3
  • Output is insensitive to terminal width

    Output is insensitive to terminal width

    The current output width is 106 columns, which by happy coincidence is precisely the width of my usual terminals.

    This coincidence seems unlikely to be universal, and some sensitivity to this would seem to be in order.

    opened by Freaky 3
  • Fix and simplify overwrite mode

    Fix and simplify overwrite mode

    This fixes overwrite mode by replacing the line counting and iterated cursor repositioning with a clear screen and placing the cursor at 1,1 - exactly what GNU watch does.

    I've also done some tidying, hoisting some iteration and filtering logic into dedicated generator functions which helps to further simplify the main loop.

    opened by Freaky 3
  • python3 shebang--does this work in FreeBSD?

    python3 shebang--does this work in FreeBSD?

    @Freaky, I discovered after merging your patches today that ioztat does not work on the Python2.7 that my Ubuntu workstation at the home office provides with "env python". I had to update the shebang to "env python3" to get it to work again.

    This should be fine for Ubuntu, but does it work in FreeBSD-land as well? I don't know if there's a python3 executable in your environment, or whether there is only python and you get what you get.

    opened by jimsalterjrs 3
  • Format name with all intermediates for disambiguation

    Format name with all intermediates for disambiguation

    Fixes #5

    This replaces the "just-the-rightmost-dataset-name-component" logic by "components-of-the-name-unique-after-the-previous-one". Output with default options looks like

    dataset                                         w/s      wMB/s        r/s      rMB/s   wareq-sz   rareq-sz
    zroot                                          0.00       0.00       0.00       0.00       0.00       0.00
       ROOT/default                                0.00       0.00       3.88       0.00       0.00       0.16
       tmp                                         0.00       0.00       0.00       0.00       0.00       0.00
    

    (e.g. here instead of default being indented two, and ROOT being invisible, we get the dataset-name-components that distinguish it from zroot, the previously-printed name)

    opened by adriaandegroot 3
  • Unmounted datasets lead to confusing gaps in output

    Unmounted datasets lead to confusing gaps in output

    Stats for unmounted datasets aren't exported, so you can end up with incomplete hierarchies in ioztat output. For example here rpool and rpool/ROOT aren't mounted, leaving rpool/ROOT/default without the path leading up to it:

    dataset                                         w/s      wMB/s        r/s      rMB/s   wareq-sz   rareq-sz
          default                                  0.00       0.00       0.97       0.00       0.00       0.05
    

    Similarly, rpool/usr/local/poudriere/ports/default misses the mountpoint-less poudriere/ports segments:

       usr                                         0.00       0.00       0.97       0.00       0.00       0.01
          local                                    0.00       0.00       0.00       0.00       0.00       0.00
                   default                         0.00       0.00       0.00       0.00       0.00       0.00
    
    opened by Freaky 3
  • using dataclasses

    using dataclasses

    Hi,

    I started of my own version of this, but found yours basically the last moment … My name of choice was objset, but the poll is finished I guess.

    I used dataclasses - ported here. And updated the fields to match current OpenZFS git.

    opened by commonism 7
  • no activity reported for zfs replication

    no activity reported for zfs replication

    is it expected that ioztat does not seem to report activity for zfs replication ?

    if this is not a bug but by design, maybe it's worth mentioning in the

    opened by devZer0 0
  • Packages

    Packages

    Produce .deb/.rpm/etc packages. sanoid is already doing this. I'm currently using sanoid package in latest ubuntu release.

    Getting these included will take some time but is worthwhile I think.

    enhancement help wanted 
    opened by morganchristiansson 2
Releases(v2.0.1)
  • v2.0.1(Mar 4, 2022)

  • v2.0.0(Mar 3, 2022)

    Added

    • An iostat(8) man page
    • CHANGELOG.md, this Changelog
    • -e and -H flags for use in scripting [#26]
    • -I flag for totals since last update instead of per-second
    • -o flag to overwrite prior reports [#9] [#20]
    • -P and -p flags to override dataset name display [#19]
    • -S flag for including child dataset statistics in parents [#32]
    • -T d|u flag for adding a timestamp to each report [#25]
    • -V, --version flags [#13] [#14]
    • -x flag for extended statistics, including unlink counts under -xx [#33]
    • interval and count positional arguments [#31]

    Changed

    • Shebang line is now /usr/bin/env python3 [#18]
    • Formatting has changed from resembling GNU iostat(8) to ZFS zpool-iostat(8) [#26]
    • Header is now printed periodically on a tty unless -N is specified [#26]
    • Display is clamped to the terminal width by truncating dataset names if necessary [#26]
    • Sort field names have been changed, with fallbacks for compatibility [#26]
    • Sort options are now case-insensitive [#26]
    • count now defaults to 1 unless an interval is specified [#31]
    • dataset is now an optional argument [#20]
    • Binary (1024-based) formatting is now default, with new -D flag for decimal
    • Average I/O sizes are now hidden beyind -x flag by default to reduce clutter [#33]
    • Exit with an error if a requested dataset does not exist or is not mounted

    Removed

    • -b flag. Binary mode is now the default to match other iostat tools.

    Fixed

    • count and interval must now be positive [#31]
    • WIFSIGNALED() status is now propagated properly to caller
    • Sleep interval now adjusts to compensate for runtime
    • Unhandled BrokenPipeError exception on SIGPIPE
    • Unhandled FileNotFoundError exception on Linux if a dataset is destroyed while enumerating files in /proc
    • Unhandled CalledProcessError exception on FreeBSD if a requested pool does not exist when using sysctl(8) fallback
    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Jan 20, 2022)

    Thanks to the efforts of @Freaky and @adriaandegroot, ioztat has new features, improved output readability, and now works on FreeBSD as well as Linux!

    New features include initial selection at the dataset level rather than pool level, optional suppression of zero-activity datasets with -z, and optional suppression of recursion into child datasets with -n.

    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Jan 12, 2022)

    The first release of ioztat builds on efforts from the Reddit r/zfs community, including:

    u/55rzs (initial creation) u/d1722825 (substantial refactoring and cleanup) u/mercenary_sysadmin (addition of the -y flag to allow for easy use with the GNU watch command)

    Enjoy!

    Source code(tar.gz)
    Source code(zip)
Owner
Jim Salter
Jim Salter
Simple plug-and-play installer for users who want to LineageOS from stock firmware, or from another custom ROM.

LineageOS for the Teracube 2e Simple plug-and-play installer for users who want to LineageOS from stock firmware, or from another custom ROM. Dependen

Gagan Malvi 5 Mar 31, 2022
Write complicated anonymous functions other than lambdas in Python.

lambdex allows you to write multi-line anonymous function expression (called a lambdex) in an idiomatic manner.

Xie Jingyi 71 May 19, 2022
Python scripts to interact with Upper Deck ePack online trading card platform

This script should connect to the Upper Deck ePack API using your browser cookies and download a list of your current collection and save it as a CSV.

Adrian Kent 1 Nov 22, 2021
Generate Gaussian 09 input files for the rotamers of an input compound.

Rotapy Purpose Generate Gaussian 09 input files for the rotamers of an input compound. Distance to the axis of rotation remains constant throughout th

1 Jul 16, 2021
A program made in PYTHON🐍 that automatically performs data insertions into a POSTGRES database 🐘 , using as base a .CSV file 📁 , useful in mass data insertions

A program made in PYTHON🐍 that automatically performs data insertions into a POSTGRES database 🐘 , using as base a .CSV file 📁 , useful in mass data insertions.

Davi Galdino 1 Oct 17, 2022
Traductor de webs desde consola usando el servicio de Google Traductor.

proxiGG Traductor de webs desde consola usando el servicio de Google Traductor. Se adjunta el código fuente para Python3 y un binario compilado en C p

@as_informatico 2 Oct 20, 2021
pyToledo is a Python library to interact with the common virtual learning environment for the Association KU Leuven (Toledo).

pyToledo pyToledo is a Python library to interact with the common virtual learning environment for the Association KU Leuven a.k.a Toledo. Motivation

Daan Vervacke 5 Jan 03, 2022
Python bindings for `ign-msgs` and `ign-transport`

Python Ignition This project aims to provide Python bindings for ignition-msgs and ignition-transport. It is a work in progress... C++ and Python libr

Rhys Mainwaring 3 Nov 08, 2022
Collection of functions for working with interlaced content in VapourSynth.

vsfieldkit Collection of functions for working with interlaced content in VapourSynth. It does not have any hard dependencies outside of VapourSynth.

Justin Turner Arthur 11 May 27, 2022
Windows symbol tables for Volatility 3

Windows Symbol Tables for Volatility 3 This repository is the Windows Symbol Table storage for Volatility 3. How to Use $ git clone https://github.com

JPCERT Coordination Center 31 Dec 25, 2022
IOP Support for Python (Experimental)

TAGS Experimental IOP Framework for Python WARNING: Currently, this project has NO EXCEPTION HANDLING. USE AT YOUR OWN RISK! I. Introduction to Interf

1 Oct 22, 2021
A simple and convenient build-and-run system for C and C++.

smake Smake is a simple and convenient build-and-run system for C and C++ projects. Why make another build system? CMake and GNU Make are great build

Venkataram Edavamadathil Sivaram 18 Nov 13, 2022
Simple utlity for sniffing decrypted HTTP/HTTPS traffic on a jailbroken iOS device into an HAR format.

Description iOS devices contain a hidden feature for sniffing decrypted HTTP/HTTPS traffic from all processes using the CFNetwork framework into an HA

83 Dec 25, 2022
Scripts for BGC analysis in large MAGs and results of their application to soil metagenomes within Chernevaya Taiga RSF-funded project

Scripts for BGC analysis in large MAGs and results of their application to soil metagenomes within Chernevaya Taiga RSF-funded project

1 Dec 06, 2021
Project for viewing the cheapest flight deals from Netherlands to other countries.

Flight_Deals_AMS Project for viewing the cheapest flight deals from Netherlands to other countries.

2 Dec 17, 2022
List Less Than Ten with python

List Less Than Ten with python

PyLaboratory 0 Feb 07, 2022
this is a basic python project that I made using python

this is a basic python project that I made using python. This project is only for practice because my python skills are still newbie.

Elvira Firmansyah 2 Dec 14, 2022
This is the DBMS Project done in 5th sem of B.E CS.

Student-Result-Management-System This is the DBMS Project done in 5th sem of B.E CS. You need to install SQlite DB Browser in your pc or laptop to ope

Vivek kulkarni 1 Jan 14, 2022
The code behind sqlfmt.com, a web UI for sqlfmt

The code behind sqlfmt.com, a web UI for sqlfmt

Ted Conbeer 2 Dec 14, 2022
Palestra sobre desenvolvimento seguro de imagens e containers para a DockerCon 2021 sala Brasil

Segurança de imagens e containers direto na pipeline Palestra sobre desenvolvimento seguro de imagens e containers para a DockerCon 2021 sala Brasil.

Fernando Guisso 10 May 19, 2022