Filesystem

Bash

Tasks related to managing unix-level operations like file and directory management

tamr_toolbox.filesystem.bash.create_directories(directories, *, require_absolute_path=True, failure_if_exists=False)

Creates directories recursively from a list of target directories if not already created

Parameters

• directories (List[Union[Path, str]]) – List of directories to create.

• require_absolute_path (bool) – Boolean flag indicating whether to require absolute paths.

• failure_if_exists (bool) – Boolean flag requiring failure if attempt to create a directory that already exists.

Return type

None

Returns

None

tamr_toolbox.filesystem.bash.remove_directories(directories, *, require_absolute_path=True, failure_if_not_exists=False, allow_recursive_deletes=False)

Removes directories from a list of target directories if created. The directory must be empty

Parameters

• directories (List[Union[Path, str]]) – List of directories to remove

• require_absolute_path (bool) – Boolean flag indicating whether to require absolute paths.

• failure_if_not_exists (bool) – Boolean flag requiring failure if attempt to remove a directory that does not exist.

• allow_recursive_deletes (bool) – Boolean flag indicating whether to allow recursive deletion.

Return type

None

Returns

None

tamr_toolbox.filesystem.bash.delete_old_files(data_path, *, num_days_to_keep, exclude_paths=None)

Find all files in the directory data_path and its subdirectory which have not been modified for a specified number of days and delete them except if their subdirectory is specified in the folders to exclude

Parameters

• data_path (Union[Path, str]) – path to the root folder from where to search for files

• num_days_to_keep (int) – number of days to keep files for

• exclude_paths (Optional[List[Union[Path, str]]]) – list of path to exclude from the search

Return type

List[str]

Returns

A list of deleted files

Raises

• ValueError – if num_days_to_keep is less than 0

• FileNotFoundError – if data_path doesn’t exist

Cloud

tamr_toolbox.filesystem.cloud.gcs_upload(cloud_client, *, source_filepath, destination_filepath, bucket_name)

Upload data to a Google storage bucket :type source_filepath: str :param source_filepath: file path of source file being uploaded :type destination_filepath: str :param destination_filepath: path to google bucket :type cloud_client: Client :param cloud_client: google storage client with user credentials :type bucket_name: str :param bucket_name: name of google bucket

tamr_toolbox.filesystem.cloud.gcs_download(cloud_client, *, source_filepath, destination_filepath, bucket_name)

Download data to a Google storage bucket :type source_filepath: str :param source_filepath: Path to file being downloaded on Google :type destination_filepath: str :param destination_filepath: Destination filepath for file being downloaded :type cloud_client: Client :param cloud_client: google storage client with user credentials :type bucket_name: str :param bucket_name: name of google bucket being downloaded from

tamr_toolbox.filesystem.cloud.s3_upload(cloud_client, *, source_filepath, destination_filepath, bucket_name)

Upload data to Amazon AWS bucket :type source_filepath: str :param source_filepath: file path of source file being uploaded :type destination_filepath: str :param destination_filepath: path to AWS bucket :type cloud_client: S3Client :param cloud_client: AWS client with user credentials :type bucket_name: str :param bucket_name: name of AWS bucket :param downloads copy of file being uploaded to filepath:

tamr_toolbox.filesystem.cloud.s3_download(cloud_client, *, source_filepath, destination_filepath, bucket_name)

Download data from an Amazon AWS bucket :type source_filepath: str :param source_filepath: Path to file being downloaded on AWS :type destination_filepath: str :param destination_filepath: Destination filepath for file being downloaded :type cloud_client: S3Client :param cloud_client: AWS client with user credentials :type bucket_name: str :param bucket_name: name of AWS bucket being downloaded from