Filesystem¶
Bash¶
Tasks related to managing unix-level operations like file and directory management
-
tamr_toolbox.filesystem.bash.
create_directories
(directories, *, require_absolute_path=True, failure_if_exists=False)[source]¶ Creates directories recursively from a list of target directories if not already created
- Parameters
- Return type
- Returns
None
-
tamr_toolbox.filesystem.bash.
remove_directories
(directories, *, require_absolute_path=True, failure_if_not_exists=False, allow_recursive_deletes=False)[source]¶ Removes directories from a list of target directories if created. The directory must be empty
- Parameters
directories (
List
[Union
[Path
,str
]]) – List of directories to removerequire_absolute_path (
bool
) – Boolean flag indicating whether to require absolute paths.failure_if_not_exists (
bool
) – Boolean flag requiring failure if attempt to remove a directory that does not exist.allow_recursive_deletes (
bool
) – Boolean flag indicating whether to allow recursive deletion.
- Return type
- Returns
None
-
tamr_toolbox.filesystem.bash.
delete_old_files
(data_path, *, num_days_to_keep, exclude_paths=None)[source]¶ Find all files in the directory data_path and its subdirectory which have not been modified for a specified number of days and delete them except if their subdirectory is specified in the folders to exclude
- Parameters
- Return type
- Returns
A list of deleted files
- Raises
ValueError – if num_days_to_keep is less than 0
FileNotFoundError – if data_path doesn’t exist
Cloud¶
-
tamr_toolbox.filesystem.cloud.
gcs_upload
(cloud_client, *, source_filepath, destination_filepath, bucket_name)[source]¶ Upload data to a Google storage bucket :type source_filepath:
str
:param source_filepath: file path of source file being uploaded :type destination_filepath:str
:param destination_filepath: path to google bucket :type cloud_client:Client
:param cloud_client: google storage client with user credentials :type bucket_name:str
:param bucket_name: name of google bucket
-
tamr_toolbox.filesystem.cloud.
gcs_download
(cloud_client, *, source_filepath, destination_filepath, bucket_name)[source]¶ Download data to a Google storage bucket :type source_filepath:
str
:param source_filepath: Path to file being downloaded on Google :type destination_filepath:str
:param destination_filepath: Destination filepath for file being downloaded :type cloud_client:Client
:param cloud_client: google storage client with user credentials :type bucket_name:str
:param bucket_name: name of google bucket being downloaded from
-
tamr_toolbox.filesystem.cloud.
s3_upload
(cloud_client, *, source_filepath, destination_filepath, bucket_name)[source]¶ Upload data to Amazon AWS bucket :type source_filepath:
str
:param source_filepath: file path of source file being uploaded :type destination_filepath:str
:param destination_filepath: path to AWS bucket :type cloud_client:S3Client
:param cloud_client: AWS client with user credentials :type bucket_name:str
:param bucket_name: name of AWS bucket :param downloads copy of file being uploaded to filepath:
-
tamr_toolbox.filesystem.cloud.
s3_download
(cloud_client, *, source_filepath, destination_filepath, bucket_name)[source]¶ Download data from an Amazon AWS bucket :type source_filepath:
str
:param source_filepath: Path to file being downloaded on AWS :type destination_filepath:str
:param destination_filepath: Destination filepath for file being downloaded :type cloud_client:S3Client
:param cloud_client: AWS client with user credentials :type bucket_name:str
:param bucket_name: name of AWS bucket being downloaded from