Dataframe

This module requires an optional dependency. See Installation for details.

Dataframe operations

Tasks related to moving data in or out of Tamr using pandas.DataFrames

tamr_toolbox.data_io.dataframe.flatten(df, *, delimiter='|', columns=None, force=False)

Converts DataFrame columns with list type to strings and returns a copy of the DataFrame with this change. Tamr often produces datasets with columns of list type, which often are more easily consumed as single-valued columns.

Parameters

• df (DataFrame) – DataFrame from Tamr dataset

• delimiter (str) – string to use as delimiter for concatenating lists

• columns (Optional[List[str]]) – optional, list of columns to flatten

• force (bool) – if True, will force non-string inner types to string

Return type

DataFrame

Returns

flattened DataFrame

tamr_toolbox.data_io.dataframe.from_dataset(dataset, *, columns=None, flatten_delimiter=None, flatten_columns=None, force_flatten=False, nrows=None, allow_dataset_refresh=False)

Creates a DataFrame from a Tamr Dataset

Parameters

• dataset (Dataset) – Tamr Dataset object

• columns (Optional[List[str]]) – optional, ordered list of columns to keep

• flatten_delimiter (Optional[str]) – if set, flatten list types to strings by concatenating with this delimiter

• flatten_columns (Optional[List[str]]) – optional, list of columns to flatten

• force_flatten (bool) – if False, arrays with inner types other than string will not be flattened. if True, will force all inner types to strings when flattening values. if True, flatten_delimiter must be specified.

• nrows (Optional[int]) – number of rows to read. default None will read all rows

• allow_dataset_refresh (bool) – if True, allows running a job to refresh dataset to make streamable

Return type

DataFrame

Returns

DataFrame

Raises

ValueError – if columns or flatten_columns contain columns that are not present in dataset

tamr_toolbox.data_io.dataframe.profile(df)

Computes profile statistics from an input DataFrame, and returns statistics in another DataFrame. Intended to be used for validation checks on a DataFrame before upserting records to a Tamr Dataset

Parameters

df (DataFrame) – DataFrame

Return type

DataFrame

Returns

DataFrame with profile statistics

tamr_toolbox.data_io.dataframe.validate(df, *, raise_error=True, require_present_columns=None, require_unique_columns=None, require_nonnull_columns=None, custom_checks=())

Performs validation checks on a DataFrame. Returns a dict of columns that fail each check, and optionally returns an error. Intended to be used on a DataFrame prior to upserting records into a Tamr dataset.

Parameters

• df (DataFrame) – DataFrame

• raise_error (bool) – if True, will raise a ValueError on failing checks. if False, will print Warning and return a dict

• require_present_columns (Optional[List[str]]) – list of columns that are checked to be present

• require_unique_columns (Optional[List[str]]) – list of columns that are checked to have all unique values, e.g. a primary key column

• require_nonnull_columns (Optional[List[str]]) – list of columns that are checked to have all non-null values

• custom_checks (Iterable[Tuple[Callable[[Any], bool], List[str]]]) – collection of tuples each containing a custom function and list of columns, on which to apply it

Return type

ValidationCheck

Returns

ValidationCheck object, with bool for whether all checks passed and dict of failing columns

Raises

ValueError – if raise_error is set True, and any checks fail