I can generate a Flamegraph using Brendan Gregg's Flamegraph perl script as shown in the documentation for Hatchet. https://www.speedscope.app/ is a nice visualizer for Flamegraphs that is suppose to support the folded stacks format, but it does not recognize my file (the one that Flamegraph.pl reads fine). I was just wondering if anyone has successfully used https://www.speedscope.app/ with output from hatchet or not (in which case, maybe it is something specific to file file).

This PR is created in response to https://github.com/hatchet/hatchet/issues/443.

This PR adds a new GitHub Action that essentially performs automated regression testing on PyPI releases. It will install each considered version of Hatchet under each considered version of Python, checkout that Hatchet version's release branch, and perform the version's unit tests.

The following Hatchet versions are currently considered:

• v1.2.0 (omitted, missing writers module)
• v1.3.0 (omitted, missing writers module)
• v1.3.1a0 (omitted, missing writers module)
• 2022.1.0 (omitted, missing writers module)
• 2022.1.1 (omitted, missing writers module)
• 2022.2.0 (omitted, missing writers module)

Similarly, the following versions of Python are currently considered:

• 2.7 (omitted, missing version in docker)
• 3.5 (omitted, missing version in docker)
• 3.6
• 3.7
• 3.8
• 3.9

Before merging, the following tasks must be done:

• [X] ~Replace the workflow_dispatch (i.e., manual) trigger with the commented out schedule trigger in pip_unit_tester.yaml~ Superseded by a task in a later comment
• [x] Change the "Install Hatchet" step to install llnl-hatchet instead of hatchet. This will be changed once the llnl-hatchet package goes live on PyPI

This PR allows us to avoid this issue with the setup-python Action used in Hatchet's CI.

To do so, it simply changes the OS image for the CI from ubuntu-latest to ubuntu-20.04. When the linked issue is resolved, we can switch back to ubuntu-latest.

• Added to_dict and to_json readers to the graphframe. Added a from_json reader.
• Added tests to verify that these readers and writers work in addition to a thicket generated json file to verify backwards compatibility.
• Added json files for tests.

In the 2022.1.0 release, running the install.sh script produces a ModuleNotFoundError for the bs4 package (BeautifulSoup). The import of this package is in hatchet/vis/static_fixer.py.

@slabasan do we want to include BeautifulSoup as a dependency of Hatchet?

Work in Progress

Added:

1. Object Oriented Refactor of tree code
2. Redesign of "collapsed" nodes
3. Additional legend
4. Menu Bar
5. Improved interface for mass pruning

Note: Merge after PR #26

This PR adds the generate_exclusive_columns function to calculate exclusive metrics from inclusive metrics. It does this by calculating the sum of the inclusive metric for each node's children and then subtracting that from the node's inclusive metric. It will only attempt to calculate exclusive metrics in certain situations, namely:

• The inclusive metric name ends in "(inc)", but there is not an exclusive metric with the same name, minus the "(inc)"
• There is an inclusive metric without the "(inc)" suffix

This might not be ideal. However, Hatchet currently provides no mechanism internally for explicitly correlating exclusive and inclusive metrics. So, until such functionality is added, this PR must use some solution based on metric names to determine what to calculate. When the internal mechanism for recording inclusive and exclusive metrics is updated, this function will be updated to use that new feature.

This PR builds off of #18, so it will be marked as a Draft until that PR is merged.

This is a small PR to fix a bug in GraphFrame.update_inclusive_columns that causes existing values in GraphFrame.inc_columns to be dropped.

As an example, consider a GraphFrame with the following metrics:

• exc_metrics: ["time"]
• inc_metrics: ["foo"]

Currently, after calling update_inclusive_columns, inc_metrics will no longer contain "foo". Instead, inc_metrics will simply be ["time (inc)"].

This PR will extend inc_metrics instead of overriding. So, in the above example, inc_metrics will now be ["foo", "time (inc)"].

This PR implements a new function called unify_ensemble that takes a list of GraphFrame objects with equal graphs and returns a new GraphFrame containing the data of all the inputs. In the output data, a new DataFrame column, called dataset, is added that informs the user which GraphFrame that row came from. If the dataset attribute of the GraphFrame (explained below) is set, that value will be used for the corresponding rows in the output. Otherwise, the string "gframe_#" is used, with "#" being replaced by the index of the GraphFrame in the input list.

To help link output data to input data, this PR also adds a new dataset attribute to the GraphFrame class and a graphframe_reader decorator to help set this attribute. The dataset attribute is meant to be a string that labels the GraphFrame. For most readers, this attribute will be set automatically by the graphframe_reader decorator. This decorator is meant to be applied to from_X static methods in the GraphFrame class. This decorator does 3 things:

1. Runs the from_X function it decorates
2. If the from_X function did not set the dataset attribute and the first argument to from_X is a string, this first argument will be considered a path to the read data, and it will be used to set dataset
3. Returns the (potentially) modified GraphFrame produced by from_X

Tiny update to the install script to remove build artifacts before rebuilding Cython modules. Especially useful when switching between major versions of Python.

When creating parent nodes, we need to handle the case that the parent might be a root node. Previously, the recursive _create_parent calls were being made on root nodes, and we incorrectly tried to index into the grandparent callpath tuple, even though it was empty. This ends the recursion if we encounter an empty callpath tuple.

Summary

Currently, the Object-based dialect and String-based dialect of the Query Language cannot handle GraphFrames containing a DataFrame with a multi-index (e.g., when you have rank and thread info).

This PR adds support for that type of data to the Object-based Dialect and String-based Dialect. This support comes in the form of a new multi_index_mode argument to the ObjectQuery constructor, the StringQuery constructor, the parse_string_dialect function, and the GraphFrame.filter function. This argument can have one of three values:

• "off" (default): query will be applied under the assumption that the DataFrame does not have a MultiIndex (i.e., the currently behavior of the QL)
• "all": when applying a predicate to a particular node's data in the DataFrame, all rows associated with the node must satisfy the predicate
• "any": when applying a predicate to a particular node's data in the DataFrame, at least one row associate with the node must satisfy the predicate

The implementation of these three modes is performed within the ObjectQuery and StringQuery classes. In these classes, the translation of predicates from dialects to the "base" syntax (represented by the Query class) will differ depending on the value of multi_index_mode. Since the implementation of this functionality is in ObjectQuery and StringQuery, the multi_index_mode arguments to parse_string_dialect and GraphFrame.filter are simply passed through to the correct class.

Finally, one important thing to note is that this functionality is ONLY implemented for new-style queries (as defined in PR #72). Old-style queries (e.g., using the QueryMatcher class) do not support this behavior.

What's Left to Do?

In short, all that's left in this PR is unit testing. I still need to implement tests in test/query.py and confirm that everything is working correctly.

Summary

This PR refactors the Query Language (QL) to prepare it for use in Thicket, improve its overall extensibility, and make its terminology more in line with that of the QL paper.

First and foremost, the QL is no longer contained within a single file. Now, all code for the QL is contained in the new query directory. This directory contains the following files:

• __init__.py: contains re-exports for everything in the QL so it can all be imported with from hatchet.query import ... (same as before)
• engine.py: contains a class containing the algorithm for applying queries to GraphFrames
• errors.py: contains any errors the QL may raise
• query.py: contains the class representing the base QL syntax and compound queries (i.e., classes for operations like "and," "or," "xor," and "not"
• object_dialect.py: contains the class representing the Object-based dialect
• string_dialect.py: contains the class representing the String-based dialect
• compat.py: contains various classes that ensure (deprecated) backwards compatibility with earlier versions of Hatchet

In this PR, queries are represented by one of 3 classes:

• Query: represents the base syntax for the QL
• StringQuery: represents the String-based dialect. This class extends Query and implements the conversion from String-based dialect to base syntax
• ObjectQuery: represents the Object-based dialect. This class extends Query and implements the conversion from Object-based dialect to base syntax

Additionally, as before, there are classes to allow queries to combined via set operations. All of these classes extend the CompoundQuery class. These classes are:

• ConjunctionQuery: combines the results of a set of queries through set conjunction (equivalent to logical AND)
• DisjunctionQuery: combines the results of a set of queries through set disjunction (equivalent to logical OR)
• ExclusiveDisjunctionQuery: combines the results of a set of queries through exclusive set disjunction (equivalent to logical XOR)
• NegationQuery: inverts the results of a query (equivalent to logical NOT)

As before, these "compound queries" can easily be created from the 3 main query classes using the &, |, ^, and ~ operators.

New in this PR, the algorithm for applying queries to GraphFrames has been separated from query composition. The algorithm is now contained within the new QueryEngine class.

Finally, all the old QL classes and functions have been reimplemented to be thin wrappers around the classes mentioned above. As a result, this PR should ensure full backwards compatibility with old QL code. However, if this PR is merged, all "old-style" query code should be considered deprecated.

What's left to do

All the implementation has been completed for this PR. Additionally, all existing unit tests that do not involve query composition are passing, which validates my claims about backwards compatibility. All that's left to do before this PR can be merged is:

• [x] Move the existing QL unit tests into a new file (e.g., query_compat.py)
• [x] Create a new QL unit tests file for "new-style" queries
• [x] Move query construction unit tests into the new file and refactor as needed
• [x] Add tests (based on the old ones) to confirm that new-style queries are working as intended

The CCT visualization now supports auto-updating. If the user places a "?" in front of the input variable name passed as an argument to the visualization it will reload automatically when that variable updates anywhere in the notebook. A second argument is added for the automatic return of selection based and snapshot based queries.

Original functionality is maintained.

Example Syntax:

%cct ?gf ?queries

The data stored in queries is an dictionary comprised of two fields:

{
    tree_state: <string> query describing the current state/shape of the tree,
    selection: <string> query describing the currently selected subtree
}

Update basic tutorial to walk through hatchet-tutorial github: https://llnl-hatchet.readthedocs.io/en/latest/basic_tutorial.html#installing-hatchet-and-tutorial-setup