Core API¶

Evaluation API¶

class ftpvl.evaluation.Evaluation(df: pandas.core.frame.DataFrame, eval_id: int = None)¶

A collection of test results from a single evaluation of a piece of software on one or more test cases.

Parameters:	df (pd.DataFrame) – The dataframe that contains the test results of the given evaluation, rows for each test case and columns for each recorded metric eval_id (int, optional) – The ID number of the evaluation, by default None

get_copy() → ftpvl.evaluation.Evaluation¶: Returns a deep copy of the Evaluation instance

get_df() → pandas.core.frame.DataFrame¶: Returns a copy of the Pandas DataFrame that represents the evaluation

get_eval_id() → Optional[int]¶: Returns the ID number of the evaluation if specified, otherwise None

process(pipeline: List[Processor]) → Evaluation¶

Executes each processor in the pipeline and returns a new Evaluation.

Parameters:	pipeline (a list of Processors to process the Evaluation in order) –
Returns:	an Evaluation instance that was processed by the pipeline

Fetchers API¶

Fetchers are responsible for ingesting and standardizing data for future processing.

HydraFetcher¶

class ftpvl.fetchers.HydraFetcher(project: str, jobset: str, eval_num: int = 0, absolute_eval_num: bool = False, mapping: dict = None, hydra_clock_names: list = None)¶

Represents a downloader and preprocessor of test results from hydra.vtr.tools.

Parameters:

project (str) – The project name to use when fetching from Hydra
jobset (str) – The jobset name to use when fetching from Hydra
eval_num (int, optional) – An integer that specifies the evaluation to download. Functionality differs depending on whether absolute_eval_num is True, by default 0
absolute_eval_num (bool, optional) – Flag that specifies if the eval_num is an absolute identifier instead of a relative identifier. If True, the fetcher will find an evaluation with the exact ID in eval_num. If False, eval_num should be a non-negative integer with 0 being the latest evaluation and 1 being the second latest evaluation, etc. By default False.
mapping (dict, optional) – A dictionary mapping input column names to output column names, if needed for remapping, by default None
hydra_clock_names (list, optional) – An optional ordered list of strings used in finding the actual frequency for each build result, by default None

get_evaluation() → ftpvl.evaluation.Evaluation¶: Returns an Evaluation that represents the fetched data.

JSONFetcher¶

class ftpvl.fetchers.JSONFetcher(path: str, mapping: dict = None)¶

Represents a loader and preprocessor of test results from a JSON file.

Parameters:	path (str) – A string file path pointing to the dataframe JSON file. mapping (dict, optional) – An optional dictionary mapping input column names to output column names., by default None

get_evaluation() → ftpvl.evaluation.Evaluation¶: Returns an Evaluation that represents the fetched data.

Processors API¶

Processors transform Evaluations to be more useful when visualized.

class ftpvl.processors.AddNormalizedColumn(groupby: str, input_col_name: str, output_col_name: str, direction: ftpvl.processors.Direction = <Direction.MAXIMIZE: 1>)¶

Processor that groups rows by a column, finds the best value of the specified column, and adds a new column with the normalized values of the row compared to the best value.

The best value is specified by the direction parameter. If the direction is 1, then the best value is the maximum. If direction is -1, then the best value is the minimum.

Parameters:	groupby (str) – the column to group by input_col_name (str) – the input column to normalize output_col_name (str) – the column to write the normalized values to direction (Direction) – specifies how to find the ‘best’ value to normalize against. By default MAXIMIZE, all values will be compared to the max value of the input column.

class ftpvl.processors.CleanDuplicates(duplicate_col_names: List[str], sort_col_names: List[str] = None, reverse_sort: bool = False)¶

Processor that outputs a new dataframe that removes duplicate rows. Optionally can sort the dataframe before removing duplicates.

Two rows are considered duplicates if and only if all values specified in duplicate_col_names matches.

By default, the first instance of a duplicate is retained, and all others are removed. You can optionally specify columns to sort by and which way to sort, which provides fine-grained control over which rows are removed.

Parameters:	duplicate_col_names (List[str]) – column names to use when finding duplicates sort_col_names (List[str], optional) – column names to sort by, by default None reverse_sort (bool, optional) – sort in ascending order, by default False

class ftpvl.processors.ExpandColumn(input_col_name: str, output_col_names: List[str], mapping: Dict[str, List[str]])¶

Processor that turns one column into more than one column by mapping values of a column to multiple values.

Parameters:

input_col_name (str) – the column name to map from
output_col_names (List[str]) – the column names to map to
mapping (Dict[str, List[str]]) – a dictionary mapping a value to a list of values. The processor will look at the value at input_col_name, use it as the key to index into mapping, and get the corresponding list of strings that will used as the values for the new metrics with names in output_col_names.

class ftpvl.processors.MinusOne¶

Processor that returns the input Evaluation by subtracting one from every data value.

This processor is useful for testing the functionality of processors on Evaluations.

class ftpvl.processors.Normalize(normalize_direction: Dict[str, ftpvl.processors.Direction])¶

Processor that normalizes all specified values in an Evaluation by column around zero.

All normalized values are between 0 and 1, with 0.5 being the baseline. Therefore, a value of zero is mapped to 0.5, positive values are mapped to values > 0.5, and negative values are mapped to values < 0.5.

This is useful in creating styles for evaluations that have already performed calculations to compare multiple evaluations. For example, you can subtract one evaluation from another, then apply this processor before styling.

Parameters:	normalize_direction (Dict[str, Direction]) – a dictionary mapping column names to the optimization direction of the column. Used to determine if increases or decreases to baseline are perceived to be ‘better’.

class ftpvl.processors.NormalizeAround(normalize_direction: Dict[str, ftpvl.processors.Direction], group_by: str, idx_name: str, idx_value: str)¶

Processor that normalizes all specified values in an Evaluation around a baseline that is chosen based on a specified index name and value.

All normalized values are between 0 and 1, with 0.5 being the baseline. This is useful in creating styles that show relative differences between each row and the baseline.

Parameters:

normalize_direction (Dict[str, Direction]) – a dictionary mapping column names to the optimization direction of the column. Used to determine if increases or decreases to baseline are perceived to be ‘better’.
group_by (str) – the column name used to group results before finding the baseline of the group and normalizing
idx_name (str) – the name of the index used to find the baseline result. The baseline result will become the baseline which all other grouped results will be normalized by
idx_value (str) – the value of the baseline result at idx_name

class ftpvl.processors.Reindex(reindex_names: List[str])¶

Processor that reassigns current columns as indices for improved visualization.

Reindexing is useful for grouping similar results in the final visualization, since each row is identified by a (usually-unique) value in the index. Indices are always the first columns, which improves readability of the final visualization.

Parameters:	reindex_names (List[str]) – A list of column names to reindex

class ftpvl.processors.SortIndex(sort_names: List[str])¶

Processor that sorts an evaluation by indices.

Parameters:	sort_names (List[str]) – a list of index names to sort by

class ftpvl.processors.StandardizeTypes(types: dict)¶

Processor that casts metrics in an Evaluation to the specified type.

The type of each metric in an Evaluation is inferred after fetching. This processor accepts a dictionary of types and casts the Evaluation to those types.

Parameters:	types (dict) – A mapping from column names to types

class ftpvl.processors.RelativeDiff(a: ftpvl.evaluation.Evaluation)¶

Processor that outputs the relative difference between evaluation A and B.

All numeric metrics will be compared, and all others will not be included in the output. B is compared to A, where the output is greater than 0 if B is greater than A, and less than 0 otherwise.

The calculation performed is (B - A) / A, where B is the evaluation that this processor is being applied to and A is the evaluation passed as a parameter.

Parameters:	a (Evaluation) – The evaluation to use when comparing against the Evaluation that is being processed. Corresponds to evaluation A in the description.

Examples

>>> a = Evaluation(pd.DataFrame(
... data=[
...     {"x": 1, "y": 5},
...     {"x": 4, "y": 10}
... ]))
>>> b = Evaluation(pd.DataFrame(
... data=[
...     {"x": 2, "y": 20},
...     {"x": 2, "y": 2}
... ]))
>>> b.process([RelativeDiff(a)]).get_df()
     x    y
0  1.0  3.0
1 -0.5 -0.8

class ftpvl.processors.FilterByIndex(index_name: str, index_value: Any)¶

Processor that filters an Evaluation by matching a specified index value after indexing.

This is best used in a processing pipeline after the Reindex processor. For filtering an evaluation based on metrics (which is not an index), use the FilterByMetric processor.

Parameters:	index_name (str) – the name of the index to use when filtering index_value (Any) – the value to compare with

Examples

>>> a = Evaluation(pd.DataFrame(
... data=[
...     {"x": 1, "y": 5},
...     {"x": 4, "y": 10}
... ],
... index=pd.Index(["a", "b"], name="key")))
>>> a.process([FilterByIndex("key", "a")]).get_df()
    x    y
key
a   1    5

class ftpvl.processors.Aggregate(func: Callable[[pandas.core.series.Series], Union[int, float]])¶

Processor that allows you to aggregate all the numeric fields of an Evaluation using a specified function.

This acts as a superclass for specific aggregator implementations, such as GeomeanAggregate. It can also be used for custom aggregations, by supplying an aggregator function to the constructor.

Parameters:	func (Callable[[pd.Series], Union[int, float]]) – a function that takes a Pandas Series and aggregates it into a single number, possibly a NaN value

Examples

>>> a = Evaluation(pd.DataFrame(
... data=[
...     {"x": 1, "y": 5},
...     {"x": 4, "y": 10}
... ]))
>>> a.process([Aggregate(lambda x: x.sum())]).get_df()
    x    y
0   5    15

class ftpvl.processors.GeomeanAggregate¶

Processor that aggregates an entire Evaluation by finding the geometric mean of each numeric metric.

Subclass of Aggregate class.

Examples

>>> a = Evaluation(pd.DataFrame(
... data=[
...     {"x": 1, "y": 8},
...     {"x": 4, "y": 8}
... ]))
>>> a.process([GeomeanAggregate()).get_df()
    x    y
0   2    8

class ftpvl.processors.CompareToFirst(normalize_direction: Dict[str, ftpvl.processors.Direction], suffix: str = '.relative')¶

Processor that compares numeric rows in an evaluation to the first row by adding columns that specify the relative difference between the first row and all other rows.

You can specify the direction that improvements should be outputted. For example, a change from 100 to 50 may be a 2x change if the objective is minimization, while it may be a 0.5x change if the objective is maximization.

Parameters:	normalize_direction (Dict[str, Direction]) – a dictionary mapping column names to the optimization direction of the column. Used to determine if increases or decreases to baseline are perceived to be ‘better’. suffix (str) – the suffix to use when creating new columns that contain the relative comparison to the first row, by default “.relative”

Examples

>>> a = Evaluation(pd.DataFrame(
... data=[
...     {"x": 1, "y": 8},
...     {"x": 4, "y": 8}
... ]))
>>> direction = {"x": Direction.MAXIMIZE, "y": Direction.MAXIMIZE}
>>> a.process([CompareToFirst(direction, suffix=".diff")).get_df()
    x   x.diff  y   y.diff
0   1   1.00    8   1.0
1   4   4.00    8   1.0

>>> a = Evaluation(pd.DataFrame(
... data=[
...     {"x": 1, "y": 8},
...     {"x": 4, "y": 8}
... ]))
>>> direction = {"x": Direction.MINIMIZE, "y": Direction.MINIMIZE}
>>> a.process([CompareToFirst(direction, suffix=".diff")).get_df()
    x   x.diff  y   y.diff
0   1   1.00    8   1.0
1   4   0.25    8   1.0

Styles API¶

Styles are special processors that transform an evaluation into CSS styles.

ColorMapStyle¶

class ftpvl.styles.ColorMapStyle(cmap: matplotlib.colors.Colormap)¶

Represents a processor that uses a matplotlib Colormap to create a styled evaluation where the background of a cell is specified by the value in the cell evaluated by the colormap.

Values in the input are expected to be either a float between 0 and 1 (inclusive) or a empty string.

Parameters:	cmap (Colormap) – the colormap to use when transforming input Evaluation to CSS styles.

Visualizers API¶

Visualizers are used for displaying the results to the user in an IPython notebook.

class ftpvl.visualizers.DebugVisualizer(evaluation: ftpvl.evaluation.Evaluation, version_info: bool = False, custom_styles: List[dict] = None, column_order: List[str] = None)¶

Represents a visualizer that will print the given Evaluation, possibly with version information.

Useful for debugging.

Parameters:

evaluation (Evaluation) – the Evaluation to display
version_info (bool, optional) – Flag to display version information from the build results in the final visualization, by default False
custom_styles (List[dict], optional) – Specify additional styling for the final visualzation. See formatting here: https://pandas.pydata.org/pandas-docs/stable/user_guide/style.html#Table-styles, by default None
column_order (List[str], optional) – Specify the columns and ordering in the final visualization. Overrides version_info, so you must specify version columns in addition. Defaults to None, which will set the column order to a preset useful for VtR.

get_visualization() :: returns a displayable visualizaton, can be displayed by calling display() in an interactive Python environment

get_visualization()¶: Returns a displayable object which can be displayed by calling display() in an interactive Python environment.

class ftpvl.visualizers.SingleTableVisualizer(evaluation: ftpvl.evaluation.Evaluation, style_eval: ftpvl.evaluation.Evaluation, version_info: bool = False, custom_styles: List[dict] = None, column_order: List[str] = None)¶

Represents a visualizer for a styled single table, possibly with version information.

Parameters:

evaluation (Evaluation) – the Evaluation with values to display
style_eval (Evaluation) – the Evaluation to use for styling. Should be processed using a Style, all values are valid CSS strings or empty.
version_info (bool, optional) – Flag to display version information from the build results in the final visualization, by default False
custom_styles (List[dict], optional) – Specify additional styling for the final visualzation. See formatting here: https://pandas.pydata.org/pandas-docs/stable/user_guide/style.html#Table-styles, by default None
column_order (List[str], optional) – Specify the columns and ordering in the final visualization. Overrides version_info, so you must specify version columns in addition. Defaults to None, which will set the column order to a preset useful for VtR.

get_visualization() :: returns a displayable visualizaton, can be displayed by calling display() in an interactive Python environment

get_visualization()¶: Returns a displayable object which can be displayed by calling display() in an interactive Python environment.

Enums¶

Direction¶

class ftpvl.processors.Direction¶

Represents the optimization direction for certain test metrics. For example, runtime is usually minimized, while frequency is maximized.

MAXIMIZE = 1¶: Indicates that the corresponding metric is optimized by maximizing the value

MINIMIZE = -1¶: Indicates that the corresponding metric is optimized by minimizing the value