barman.cloud module#

class barman.cloud.BackupFileInfo(oid=None, base=None, path=None, compression=None)View on GitHub #

Bases: object

__init__(oid=None, base=None, path=None, compression=None)View on GitHub #

class barman.cloud.CloudBackup(server_name, cloud_interface, postgres, backup_name=None)View on GitHub #

Bases: object

Abstract base class for taking cloud backups of PostgreSQL servers.

This class handles the coordination of the physical backup copy with the PostgreSQL server via the PostgreSQL low-level backup API. This is handled by the _coordinate_backup method.

Concrete classes will need to implement the following abstract methods which are called during the _coordinate_backup method:

_take_backup _upload_backup_label _finalise_copy _add_stats_to_backup_info

Implementations must also implement the public backup method which should carry out any prepartion and invoke _coordinate_backup.

__init__(server_name, cloud_interface, postgres, backup_name=None)View on GitHub #

Parameters:

server_name (str) – The name of the server being backed up.
cloud_interface (CloudInterface) – The CloudInterface for interacting with the cloud object store.
postgres (barman.postgres.PostgreSQLConnection|None) – A connection to the PostgreSQL instance being backed up.
backup_name (str|None) – A friendly name which can be used to reference this backup in the future.

_abc_impl = <_abc._abc_data object>#

abstract _add_stats_to_backup_info()View on GitHub #: Add statistics about the backup to self.backup_info.

_check_postgres_version()View on GitHub #: Verify we are running against a supported PostgreSQL version.

_coordinate_backup()View on GitHub #: Coordinate taking the backup with the PostgreSQL server.

_create_restore_point()View on GitHub #: Create a restore point named after this backup.

abstract _finalise_copy()View on GitHub #: Perform any finalisation required to complete the copy of backup data.

_get_backup_info(server_name)View on GitHub #: Create and return the backup_info for this CloudBackup.

_log_end_of_backup()View on GitHub #: Write log lines indicating end of backup.

_start_backup()View on GitHub #: Start the backup via the PostgreSQL backup API.

_stop_backup()View on GitHub #: Stop the backup via the PostgreSQL backup API.

abstract _take_backup()View on GitHub #

Perform the actions necessary to create the backup.

This method must be called between pg_backup_start and pg_backup_stop which is guaranteed to happen if the _coordinate_backup method is used.

_upload_backup_info()View on GitHub #: Upload the backup_info for this CloudBackup.

abstract _upload_backup_label()View on GitHub #: Upload the backup label to cloud storage.

abstract backup()View on GitHub #

External interface for performing a cloud backup of the postgres server.

When providing an implementation of this method, concrete classes must set self.backup_info before coordinating the backup. Implementations should call self._coordinate_backup to carry out the backup process.

handle_backup_errors(action, exc, backup_info)View on GitHub #

Mark the backup as failed and exit

Parameters:

action (str) – the upload phase that has failed
exc (BaseException) – the exception that caused the failure
backup_info (barman.infofile.BackupInfo) – the backup info file

class barman.cloud.CloudBackupCatalog(cloud_interface, server_name)View on GitHub #

Bases: KeepManagerMixinCloud

Cloud storage backup catalog

__init__(cloud_interface, server_name)View on GitHub #

Object responsible for retrieving backup catalog from cloud storage

Parameters:

cloud_interface (CloudInterface) – The interface to use to upload the backup
server_name (str) – The name of the server as configured in Barman

_abc_impl = <_abc._abc_data object>#

_get_backup_info_from_name(backup_name)View on GitHub #

Get the backup metadata for the named backup.

Parameters:: backup_name (str) – The name of the backup for which the backup metadata should be retrieved
Return BackupInfo|None:: The backup metadata for the named backup

get_backup_files(backup_info, allow_missing=False)View on GitHub #

Get the list of expected files part of a backup

Parameters:

backup_info (BackupInfo) – the backup information
allow_missing (bool) – True if missing backup files are allowed, False otherwise. A value of False will cause a SystemExit to be raised if any files expected due to the backup_info content cannot be found.

Return type:

dict[int, BackupFileInfo]

get_backup_info(backup_id)View on GitHub #

Load a BackupInfo from cloud storage

Parameters:: backup_id (str) – The backup id to load
Return type:: BackupInfo

get_backup_list()View on GitHub #

Retrieve the list of available backup from cloud storage

Return type:: Dict[str,BackupInfo]

get_wal_paths()View on GitHub #: Retrieve a dict of WAL paths keyed by the WAL name from cloud storage

get_wal_prefixes()View on GitHub #: Return only the common prefixes under the wals prefix.

parse_backup_id(backup_id)View on GitHub #

Parse a backup identifier and return the matching backup ID. If the identifier is a backup ID it is returned, otherwise it is assumed to be a name.

Parameters:: backup_id (str) – The backup identifier to be parsed
Return str:: The matching backup ID for the supplied identifier

remove_backup_from_cache(backup_id)View on GitHub #: Remove backup with backup_id from the cached list. This is intended for cases where we want to update the state without firing lots of requests at the bucket.

remove_wal_from_cache(wal_name)View on GitHub #: Remove named wal from the cached list. This is intended for cases where we want to update the state without firing lots of requests at the bucket.

class barman.cloud.CloudBackupSnapshot(server_name, cloud_interface, snapshot_interface, postgres, snapshot_instance, snapshot_disks, backup_name=None)View on GitHub #

Bases: CloudBackup

A cloud backup client using disk snapshots to create the backup.

__init__(server_name, cloud_interface, snapshot_interface, postgres, snapshot_instance, snapshot_disks, backup_name=None)View on GitHub #

Create the backup client for snapshot backups

Parameters:

server_name (str) – The name of the server as configured in Barman
cloud_interface (CloudInterface) – The interface to use to upload the backup
snapshot_interface (SnapshotInterface) – The interface to use for creating a backup using snapshots
postgres (barman.postgres.PostgreSQLConnection|None) – A connection to the PostgreSQL instance being backed up.
snapshot_instance (str) – The name of the VM instance to which the disks to be backed up are attached.
snapshot_disks (list[str]) – A list containing the names of the disks for which snapshots should be taken at backup time.
backup_name (str|None) – A friendly name which can be used to reference this backup in the future.

_abc_impl = <_abc._abc_data object>#

_add_stats_to_backup_info()View on GitHub #: Add statistics about the backup to self.backup_info.

_check_backup_preconditions()View on GitHub #

Perform additional checks for snapshot backups, specifically:

check that the VM instance for which snapshots should be taken exists

check that the expected disks are attached to that instance

check that the attached disks are mounted on the filesystem

Raises a BackupPreconditionException if any of the checks fail.

_finalise_copy()View on GitHub #

Perform any finalisation required to complete the copy of backup data.

This is a no-op for snapshot backups.

_take_backup()View on GitHub #: Make a backup by creating snapshots of the specified disks.

_upload_backup_label()View on GitHub #

Upload the backup label to cloud storage.

Snapshot backups just upload the backup label as a single object rather than adding it to a tar archive.

backup()View on GitHub #: Take a backup by creating snapshots of the specified disks.

class barman.cloud.CloudBackupUploader(server_name, cloud_interface, max_archive_size, postgres, compression=None, backup_name=None, min_chunk_size=None, max_bandwidth=None)View on GitHub #

Bases: CloudBackup

Uploads backups from a PostgreSQL server to cloud object storage.

__init__(server_name, cloud_interface, max_archive_size, postgres, compression=None, backup_name=None, min_chunk_size=None, max_bandwidth=None)View on GitHub #

Base constructor.

Parameters:

server_name (str) – The name of the server as configured in Barman
cloud_interface (CloudInterface) – The interface to use to upload the backup
max_archive_size (int) – the maximum size of an uploading archive
postgres (barman.postgres.PostgreSQLConnection|None) – A connection to the PostgreSQL instance being backed up.
compression (str) – Compression algorithm to use
backup_name (str|None) – A friendly name which can be used to reference this backup in the future.
min_chunk_size (int) – the minimum size of a single upload part
max_bandwidth (int) – the maximum amount of data per second that should be uploaded during the backup

_abc_impl = <_abc._abc_data object>#

_add_stats_to_backup_info()View on GitHub #: Adds statistics from the upload controller to the backup_info.

_backup_config_files(controller, backup_info)View on GitHub #

Perform the backup of any external config files.

Parameters:

controller (barman.cloud.CloudUploadController) – upload controller
backup_info (barman.infofile.BackupInfo) – backup information

_backup_data_files(controller, backup_info, pgdata_dir, server_major_version)View on GitHub #

Perform the actual copy of the data files uploading it to cloud storage.

First, it copies one tablespace at a time, then the PGDATA directory, then pg_control.

Bandwidth limitation, according to configuration, is applied in the process.

Parameters:

controller (barman.cloud.CloudUploadController) – upload controller
backup_info (barman.infofile.BackupInfo) – backup information
pgdata_dir (str) – Path to pgdata directory
server_major_version (str) – Major version of the postgres server being backed up

_create_upload_controller(backup_id)View on GitHub #

Create an upload controller from the specified backup_id

Parameters:: backup_id (str) – The backup identifier
Return type:: CloudUploadController
Returns:: The upload controller

_finalise_copy()View on GitHub #: Close the upload controller, forcing the flush of any buffered uploads.

_get_tablespace_location(tablespace)View on GitHub #

Return the on-disk location of the supplied tablespace.

This will usually just be the location of the tablespace however subclasses which run against Barman server will need to override this method.

Parameters:: tablespace (infofile.Tablespace) – The tablespace whose location should be returned.
Return type:: str
Returns:: The path of the supplied tablespace.

property _pgdata_dirView on GitHub #: The location of the PGDATA directory to be backed up.

_take_backup()View on GitHub #: Make a backup by copying PGDATA, tablespaces and config to cloud storage.

_upload_backup_label()View on GitHub #

Upload the backup label to cloud storage.

Upload is via the upload controller so that the backup label is added to the data tarball.

backup()View on GitHub #: Upload a Backup to cloud storage directly from a live PostgreSQL server.

class barman.cloud.CloudBackupUploaderBarman(server_name, cloud_interface, max_archive_size, backup_dir, backup_id, compression=None, min_chunk_size=None, max_bandwidth=None)View on GitHub #

Bases: CloudBackupUploader

A cloud storage upload client for a preexisting backup on the Barman server.

__init__(server_name, cloud_interface, max_archive_size, backup_dir, backup_id, compression=None, min_chunk_size=None, max_bandwidth=None)View on GitHub #

Create the cloud storage upload client for a backup in the specified location with the specified backup_id.

Parameters:

server_name (str) – The name of the server as configured in Barman
cloud_interface (CloudInterface) – The interface to use to upload the backup
max_archive_size (int) – the maximum size of an uploading archive
backup_dir (str) – Path to the directory containing the backup to be uploaded
backup_id (str) – The id of the backup to upload
compression (str) – Compression algorithm to use
min_chunk_size (int) – the minimum size of a single upload part
max_bandwidth (int) – the maximum amount of data per second that should be uploaded during the backup

_abc_impl = <_abc._abc_data object>#

_get_tablespace_location(tablespace)View on GitHub #

Return the on-disk location of the supplied tablespace.

Combines the backup_dir and the tablespace OID to determine the location of the tablespace on the Barman server.

Parameters:: tablespace (infofile.Tablespace) – The tablespace whose location should be returned.
Return type:: str
Returns:: The path of the supplied tablespace.

property _pgdata_dirView on GitHub #: The location of the PGDATA directory to be backed up.

_take_backup()View on GitHub #: Make a backup by copying PGDATA and tablespaces to cloud storage.

backup()View on GitHub #

Upload a Backup to cloud storage

This deviates from other CloudBackup classes because it does not make use of the self._coordinate_backup function. This is because there is no need to coordinate the backup with a live PostgreSQL server, create a restore point or upload the backup label independently of the backup (it will already be in the base backup directoery).

handle_backup_errors(action, exc)View on GitHub #

Log that the backup upload has failed and exit

This differs from the function in the superclass because it does not update the backup.info metadata (this must be left untouched since it relates to the original backup made with Barman).

Parameters:

action (str) – the upload phase that has failed
exc (BaseException) – the exception that caused the failure

class barman.cloud.CloudInterface(url, jobs=2, tags=None, delete_batch_size=None)View on GitHub #

Bases: object

Abstract base class which provides the interface between barman and cloud storage providers.

Support for individual cloud providers should be implemented by inheriting from this class and providing implementations for the abstract methods.

This class provides generic boilerplate for the asynchronous and parallel upload of objects to cloud providers which support multipart uploads. These uploads are carried out by worker processes which are spawned by _ensure_async and consume upload jobs from a queue. The public async_upload_part and async_complete_multipart_upload methods add jobs to this queue. When the worker processes consume the jobs they execute the synchronous counterparts to the async_* methods (_upload_part and _complete_multipart_upload) which must be implemented in CloudInterface sub-classes.

Additional boilerplate for creating buckets and streaming objects as tar files is also provided.

abstract property MAX_ARCHIVE_SIZEView on GitHub #

Maximum size in bytes of a single file in cloud storage.

Type:: int

abstract property MAX_CHUNKS_PER_FILEView on GitHub #

Maximum number of chunks allowed in a single file in cloud storage. The exact definition of chunk depends on the cloud provider, for example in AWS S3 a chunk would be one part in a multipart upload. In Azure a chunk would be a single block of a block blob.

Type:: int

abstract property MAX_DELETE_BATCH_SIZEView on GitHub #

The maximum number of objects which can be deleted in a single batch.

Type:: int

abstract property MIN_CHUNK_SIZEView on GitHub #

Minimum size in bytes of a single chunk.

Type:: int

__init__(url, jobs=2, tags=None, delete_batch_size=None)View on GitHub #

Base constructor

Parameters:

url (str) – url for the cloud storage resource
jobs (int) – How many sub-processes to use for asynchronous uploading, defaults to 2.
tags (List[tuple]) – List of tags as k,v tuples to be added to all uploaded objects
delete_batch_size (int|None) – the maximum number of objects to be deleted in a single request

_abc_impl = <_abc._abc_data object>#

_abort()View on GitHub #: Abort all the operations

abstract _abort_multipart_upload(upload_metadata, key)View on GitHub #

Abort a certain multipart upload

The implementation of this method should clean up any dangling resources left by the incomplete upload.

Parameters:

upload_metadata (dict) – Provider-specific metadata for this upload e.g. the multipart upload handle in AWS S3
key (str) – The key to use in the cloud service

abstract _check_bucket_existence()View on GitHub #

Check cloud storage for the target bucket

Returns:: True if the bucket exists, False otherwise
Return type:: bool

abstract _complete_multipart_upload(upload_metadata, key, parts_metadata)View on GitHub #

Finish a certain multipart upload

Parameters:

upload_metadata (dict) – Provider-specific metadata for this upload e.g. the multipart upload handle in AWS S3
key (str) – The key to use in the cloud service
parts_metadata (List[dict]) – The list of metadata for the parts composing the multipart upload. Each part is guaranteed to provide a PartNumber and may optionally contain additional metadata returned by the cloud provider such as ETags.

abstract _create_bucket()View on GitHub #: Create the bucket in cloud storage

abstract _delete_objects_batch(paths)View on GitHub #

Delete a single batch of objects

Parameters:: paths (List[str])

_ensure_async()View on GitHub #: Ensure that the asynchronous execution infrastructure is up and the worker process is running

_handle_async_errors()View on GitHub #

If an upload error has been discovered, stop the upload process, stop all the workers and raise an exception

Returns:

abstract _reinit_session()View on GitHub #: Reinitialises any resources used to maintain a session with a cloud provider. This is called by child processes in order to avoid any potential race conditions around re-using the same session as the parent process.

_retrieve_results()View on GitHub #: Receive the results from workers and update the local parts DB, making sure that each part list is sorted by part number

abstract _upload_part(upload_metadata, key, body, part_number)View on GitHub #

Upload a part into this multipart upload and return a dict of part metadata. The part metadata must contain the key “PartNumber” and can optionally contain any other metadata available (for example the ETag returned by S3).

The part metadata will included in a list of metadata for all parts of the upload which is passed to the _complete_multipart_upload method.

Parameters:

upload_metadata (dict) – Provider-specific metadata for this upload e.g. the multipart upload handle in AWS S3
key (str) – The key to use in the cloud service
body (object) – A stream-like object to upload
part_number (int) – Part number, starting from 1

Returns:

The part metadata

Return type:

dict[str, None|str]

_worker_process_execute_job(task, process_number)View on GitHub #

Exec a single task

Parameters:

task (Dict) – task to execute
process_number (int) – the process number, used in the logging

output :return:

_worker_process_main(process_number)View on GitHub #

Repeatedly grab a task from the queue and execute it, until a task containing “None” is grabbed, indicating that the process must stop.

Parameters:: process_number (int) – the process number, used in the logging

output

async_complete_multipart_upload(upload_metadata, key, parts_count)View on GitHub #

Asynchronously finish a certain multipart upload. This method grant that the final call to the cloud storage will happen after all the already scheduled parts have been uploaded.

Parameters:

upload_metadata (dict) – Provider-specific metadata for this upload e.g. the multipart upload handle in AWS S3
key (str) – The key to use in the cloud service
parts_count (int) – Number of parts

async_upload_part(upload_metadata, key, body, part_number)View on GitHub #

Asynchronously upload a part into a multipart upload

Parameters:

upload_metadata (dict) – Provider-specific metadata for this upload e.g. the multipart upload handle in AWS S3
key (str) – The key to use in the cloud service
body (any) – A stream-like object to upload
part_number (int) – Part number, starting from 1

close()View on GitHub #: Wait for all the asynchronous operations to be done

abstract create_multipart_upload(key)View on GitHub #

Create a new multipart upload and return any metadata returned by the cloud provider.

This metadata is treated as an opaque blob by CloudInterface and will be passed into the _upload_part, _complete_multipart_upload and _abort_multipart_upload methods.

The implementations of these methods will need to handle this metadata in the way expected by the cloud provider.

Some cloud services do not require multipart uploads to be explicitly created. In such cases the implementation can be a no-op which just returns None.

Parameters:: key – The key to use in the cloud service
Returns:: The multipart upload metadata
Return type:: dict[str, str]|None

delete_objects(paths)View on GitHub #

Delete the objects at the specified paths

Deletes the objects defined by the supplied list of paths in batches specified by either batch_size or MAX_DELETE_BATCH_SIZE, whichever is lowest.

Parameters:: paths (List[str])

abstract delete_under_prefix(prefix)View on GitHub #

Delete all objects under the specified prefix.

Parameters:: prefix (str) – The object key prefix under which all objects should be deleted.

abstract download_file(key, dest_path, decompress)View on GitHub #

Download a file from cloud storage

Parameters:

key (str) – The key identifying the file to download
dest_path (str) – Where to put the destination file
decompress (str|None) – Compression scheme to use for decompression

extract_tar(key, dst)View on GitHub #

Extract a tar archive from cloud to the local directory

Parameters:

key (str) – The key identifying the tar archive
dst (str) – Path of the directory into which the tar archive should be extracted

abstract get_prefixes(prefix)View on GitHub #

Return only the common prefixes under the supplied prefix.

Parameters:: prefix (str) – The object key prefix under which the common prefixes will be found.
Return type:: Iterator[str]
Returns:: A list of unique prefixes immediately under the supplied prefix.

abstract list_bucket(prefix='', delimiter='/')View on GitHub #

List bucket content in a directory manner

Parameters:

prefix (str)
delimiter (str)

Returns:

List of objects and dirs right under the prefix

Return type:

List[str]

abstract remote_open(key, decompressor=None)View on GitHub #

Open a remote object in cloud storage and returns a readable stream

Parameters:

key (str) – The key identifying the object to open
decompressor (barman.clients.cloud_compression.ChunkedCompressor) – A ChunkedCompressor object which will be used to decompress chunks of bytes as they are read from the stream

Returns:

A file-like object from which the stream can be read or None if the key does not exist

setup_bucket()View on GitHub #: Search for the target bucket. Create it if not exists

abstract test_connectivity()View on GitHub #

Test that the cloud provider is reachable

Returns:: True if the cloud provider is reachable, False otherwise
Return type:: bool

abstract upload_fileobj(fileobj, key, override_tags=None)View on GitHub #

Synchronously upload the content of a file-like object to a cloud key

Parameters:

IOBase (fileobj) – File-like object to upload
key (str) – The key to identify the uploaded object
override_tags (List[tuple]) – List of k,v tuples which should override any tags already defined in the cloud interface

wait_for_multipart_upload(key)View on GitHub #

Wait for a multipart upload to be completed and return the result

Parameters:: key (str) – The key to use in the cloud service

exception barman.cloud.CloudProviderErrorView on GitHub #

Bases: BarmanException

This exception is raised when we get an error in the response from the cloud provider

class barman.cloud.CloudSnapshotInterfaceView on GitHub #

Bases: object

Defines a common interface for handling cloud snapshots.

_abc_impl = <_abc._abc_data object>#

_required_config_for_backup = ('snapshot_disks', 'snapshot_instance')#

_required_config_for_restore = ('snapshot_recovery_instance',)#

abstract delete_snapshot_backup(backup_info)View on GitHub #

Delete all snapshots for the supplied backup.

Parameters:: backup_info (barman.infofile.LocalBackupInfo) – Backup information.

abstract get_attached_volumes(instance_name, disks=None, fail_on_missing=True)View on GitHub #

Returns metadata for the volumes attached to this instance.

Queries the cloud provider for metadata relating to the volumes attached to the named instance and returns a dict of VolumeMetadata objects, keyed by disk name.

If the optional disks parameter is supplied then this method must return metadata for the disks in the supplied list only. A SnapshotBackupException must be raised if any of the supplied disks are not found to be attached to the instance.

If the optional disks parameter is supplied then this method returns metadata for the disks in the supplied list only. If fail_on_missing is set to True then a SnapshotBackupException is raised if any of the supplied disks are not found to be attached to the instance.

If the disks parameter is not supplied then this method must return a VolumeMetadata for all disks attached to this instance.

Parameters:

instance_name (str) – The name of the VM instance to which the disks to be backed up are attached.
disks (list[str]|None) – A list containing the names of disks to be backed up.
fail_on_missing (bool) – Fail with a SnapshotBackupException if any specified disks are not attached to the instance.

Return type:

dict[str, VolumeMetadata]

Returns:

A dict of VolumeMetadata objects representing each volume attached to the instance, keyed by volume identifier.

abstract instance_exists(instance_name)View on GitHub #

Determine whether the named instance exists.

Parameters:: instance_name (str) – The name of the VM instance to which the disks to be backed up are attached.
Return type:: bool
Returns:: True if the named instance exists, False otherwise.

abstract take_snapshot_backup(backup_info, instance_name, volumes)View on GitHub #

Take a snapshot backup for the named instance.

Implementations of this method must do the following:

Create a snapshot of the disk.

Set the snapshots_info field of the backup_info to a SnapshotsInfo implementation which contains the snapshot metadata required both by Barman and any third party tooling which needs to recover the snapshots.

Parameters:

backup_info (barman.infofile.LocalBackupInfo) – Backup information.
instance_name (str) – The name of the VM instance to which the disks to be backed up are attached.
volumes (dict[str,barman.cloud.VolumeMetadata]) – Metadata for the volumes to be backed up.

classmethod validate_backup_config(config)View on GitHub #

Additional validation for backup options.

Raises a ConfigurationException if any required options are missing.

Parameters:: config (argparse.Namespace) – The backup options provided at the command line.

classmethod validate_restore_config(config)View on GitHub #

Additional validation for restore options.

Raises a ConfigurationException if any required options are missing.

Parameters:: config (argparse.Namespace) – The backup options provided at the command line.

class barman.cloud.CloudTarUploader(cloud_interface, key, chunk_size, compression=None, max_bandwidth=None)View on GitHub #

Bases: object

__init__(cloud_interface, key, chunk_size, compression=None, max_bandwidth=None)View on GitHub #

A tar archive that resides on cloud storage

Parameters:

cloud_interface (CloudInterface) – cloud interface instance
key (str) – path inside the bucket
compression (str) – required compression
chunk_size (int) – the upload chunk size
max_bandwidth (int) – the maximum amount of data per second that should be uploaded by this tar uploader

_buffer(buffering=-1, encoding=None, newline=None, *, suffix='.part', prefix='barman-upload-', dir=None, delete=False, errors=None)#

Create and return a temporary file. Arguments: ‘prefix’, ‘suffix’, ‘dir’ – as for mkstemp. ‘mode’ – the mode argument to io.open (default “w+b”). ‘buffering’ – the buffer size argument to io.open (default -1). ‘encoding’ – the encoding argument to io.open (default None) ‘newline’ – the newline argument to io.open (default None) ‘delete’ – whether the file is deleted on close (default True). ‘errors’ – the errors argument to io.open (default None) The file is created as mkstemp() would do it.

Returns an object with a file-like interface; the name of the file is accessible as its ‘name’ attribute. The file will be automatically deleted when it is closed unless the ‘delete’ argument is set to False.

On POSIX, NamedTemporaryFiles cannot be automatically deleted if the creating process is terminated abruptly with a SIGKILL signal. Windows can delete the file even in this case.

_throttle_upload(part_size)View on GitHub #

Throttles the upload according to the value of self.max_bandwidth.

Waits until enough time has passed since the last upload that a new part can be uploaded without exceeding self.max_bandwidth. If sufficient time has already passed then this function will return without waiting.

Parameters:: part_size (int) – Size in bytes of the part which is to be uplaoded.

close()View on GitHub #

flush()View on GitHub #

write(buf)View on GitHub #

class barman.cloud.CloudUploadController(cloud_interface, key_prefix, max_archive_size, compression, min_chunk_size=None, max_bandwidth=None)View on GitHub #

Bases: object

__init__(cloud_interface, key_prefix, max_archive_size, compression, min_chunk_size=None, max_bandwidth=None)View on GitHub #

Create a new controller that upload the backup in cloud storage

Parameters:

cloud_interface (CloudInterface) – cloud interface instance
key_prefix (str|None) – path inside the bucket
max_archive_size (int) – the maximum size of an archive
compression (str|None) – required compression
min_chunk_size (int|None) – the minimum size of a single upload part
max_bandwidth (int|None) – the maximum amount of data per second that should be uploaded during the backup

_build_dest_name(name, count=0)View on GitHub #: Get the destination tar name :param str name: the name prefix :param int count: the part count :rtype: str

_get_tar(name)View on GitHub #: Get a named tar file from cloud storage. Subsequent call with the same name return the same name :param str name: tar name :rtype: tarfile.TarFile

add_file(label, src, dst, path, optional=False)View on GitHub #

add_fileobj(label, fileobj, dst, path, mode=None, uid=None, gid=None)View on GitHub #

close()View on GitHub #

copy_end_time#: Copy end time

copy_start_time#: Copy start time

statistics()View on GitHub #

Return statistics about the CloudUploadController object.

Return type:: dict

upload_directory(label, src, dst, exclude=None, include=None)View on GitHub #

upload_stats#: Already finished uploads list

exception barman.cloud.CloudUploadingErrorView on GitHub #

Bases: BarmanException

This exception is raised when there are upload errors

class barman.cloud.DecompressingStreamingIO(streaming_response, decompressor)View on GitHub #

Bases: RawIOBase

Provide an IOBase interface which decompresses streaming cloud responses.

This is intended to wrap azure_blob_storage.StreamingBlobIO and aws_s3.StreamingBodyIO objects, transparently decompressing chunks while continuing to expose them via the read method of the IOBase interface.

This allows TarFile to stream the uncompressed data directly from the cloud provider responses without requiring it to know anything about the compression.

COMPRESSED_CHUNK_SIZE = 65536#

__init__(streaming_response, decompressor)View on GitHub #

Create a new DecompressingStreamingIO object.

A DecompressingStreamingIO object will be created which reads compressed bytes from streaming_response and decompresses them with the supplied decompressor.

Parameters:

streaming_response (RawIOBase) – A file-like object which provides the data in the response streamed from the cloud provider.
barman.clients.cloud_compression.ChunkedCompressor – A ChunkedCompressor object which provides a decompress(bytes) method to return the decompressed bytes.

_abc_impl = <_abc._abc_data object>#

_read_from_uncompressed_buffer(n)View on GitHub #

Read up to n bytes from the local buffer of uncompressed data.

Removes up to n bytes from the local buffer and returns them. If n is greater than the length of the buffer then the entire buffer content is returned and the buffer is emptied.

Parameters:: n (int) – The number of bytes to read
Returns:: The bytes read from the local buffer
Return type:: bytes

read(n=-1)View on GitHub #

Read up to n bytes of uncompressed data from the wrapped IOBase.

Bytes are initially read from the local buffer of uncompressed data. If more bytes are required then chunks of COMPRESSED_CHUNK_SIZE are read from the wrapped IOBase and decompressed in memory until >= n uncompressed bytes have been read. n bytes are then returned with any remaining bytes being stored in the local buffer for future requests.

Parameters:: n (int) – The number of uncompressed bytes required
Returns:: Up to n uncompressed bytes from the wrapped IOBase
Return type:: bytes

class barman.cloud.FileUploadStatistics(*args, **kwargs)View on GitHub #

Bases: dict

__init__(*args, **kwargs)View on GitHub #

set_part_end_time(part_number, end_time)View on GitHub #

set_part_start_time(part_number, start_time)View on GitHub #

class barman.cloud.SnapshotMetadata(mount_options=None, mount_point=None)View on GitHub #

Bases: object

Represents metadata for a single snapshot.

This class holds the snapshot metadata common to all snapshot providers. Currently this is the mount_options and the mount_point of the source disk for the snapshot at the time of the backup.

The identifier and device properties are part of the public interface used within Barman so that the calling code can access the snapshot identifier and device path without having to worry about how these are composed from the snapshot metadata for each cloud provider.

Specializations of this class must:

Add their provider-specific fields to _provider_fields.

Implement the identifier abstract property so that it returns a value which can identify the snapshot via the cloud provider API. An example would be the snapshot short name in GCP.

Implement the device abstract property so that it returns a full device path to the location at which the source disk was attached to the compute instance.

__init__(mount_options=None, mount_point=None)View on GitHub #

Constructor accepts properties generic to all snapshot providers.

Parameters:

mount_options (str) – The mount options used for the source disk at the time of the backup.
mount_point (str) – The mount point of the source disk at the time of the backup.

_provider_fields = ()#

classmethod from_dict(info)View on GitHub #

Create a new SnapshotMetadata object from the raw metadata dict.

This function will set the generic fields supported by SnapshotMetadata before iterating through fields listed in cls._provider_fields. This means subclasses do not need to override this method, they just need to add their fields to their own _provider_fields.

Parameters:: info (dict[str,str]) – The raw snapshot metadata.
Return type:: SnapshotMetadata

abstract property identifierView on GitHub #

An identifier which can reference the snapshot via the cloud provider.

Subclasses must ensure this returns a string which can be used by Barman to reference the snapshot when interacting with the cloud provider API.

Return type:: str
Returns:: A snapshot identifier.

to_dict()View on GitHub #

Seralize this SnapshotMetadata object as a raw dict.

This function will create a dict with the generic fields supported by SnapshotMetadata before iterating through fields listed in self._provider_fields and adding them to a special provider field. As long as they add their provider-specific fields to _provider_fields then subclasses do not need to override this method.

Return type:: dict
Returns:: A dict containing the metadata for this snapshot.

class barman.cloud.SnapshotsInfo(snapshots=None)View on GitHub #

Bases: object

Represents the snapshots_info field of backup metadata stored in BackupInfo.

This class holds the metadata for a snapshot backup which is common to all snapshot providers. This is the list of SnapshotMetadata objects representing the individual snapshots.

Specializations of this class must:

Add their provider-specific fields to _provider_fields.

Set their _snapshot_metadata_cls property to the required specialization of SnapshotMetadata.

Set the provider property to the required value.

__init__(snapshots=None)View on GitHub #

Constructor saves the list of snapshots if it is provided.

Parameters:: snapshots (list[SnapshotMetadata]) – A list of metadata objects for each snapshot.

_provider_fields = ()#

_snapshot_metadata_clsView on GitHub #: alias of SnapshotMetadata

classmethod from_dict(info)View on GitHub #

Create a new SnapshotsInfo object from the raw metadata dict.

This function will iterate through fields listed in cls._provider_fields and add them to the instantiated object. It will then create a new SnapshotMetadata object (of the type specified in cls._snapshot_metadata_cls) for each snapshot in the raw dict.

Subclasses do not need to override this method, they just need to add their fields to their own _provider_fields and override _snapshot_metadata_cls.

Parameters:: info (dict) – The raw snapshots_info dict.
Return type:: SnapshotsInfo
Returns:: The SnapshotsInfo object representing the raw dict.

to_dict()View on GitHub #

Seralize this SnapshotMetadata object as a raw dict.

This function will create a dict with the generic fields supported by SnapshotMetadata before iterating through fields listed in self._provider_fields and adding them to a special provider_info field. The SnapshotMetadata objects in self.snapshots are serialized into the dict via their own to_dict function.

As long as they add their provider-specific fields to _provider_fields then subclasses do not need to override this method.

Return type:: dict
Returns:: A dict containing the metadata for this snapshot.

class barman.cloud.TarFileIgnoringTruncate(name=None, mode='r', fileobj=None, format=None, tarinfo=None, dereference=None, ignore_zeros=None, encoding=None, errors='surrogateescape', pax_headers=None, debug=None, errorlevel=None, copybufsize=None)View on GitHub #

Bases: TarFile

Custom TarFile class that ignore truncated or vanished files.

addfile(tarinfo, fileobj=None)View on GitHub #

Add the provided fileobj to the tar ignoring truncated or vanished files.

This method completely replaces TarFile.addfile()

format = 2#

class barman.cloud.VolumeMetadataView on GitHub #

Bases: object

Represents metadata for a single volume attached to a cloud VM.

The main purpose of this class is to allow calling code to determine the mount point and mount options for an attached volume without needing to know the details of how these are determined for a specific cloud provider.

Implementations must therefore:

Store metadata obtained from the cloud provider which can be used to resolve this volume to an attached and mounted volume on the instance. This will typically be a device name or something which can be resolved to a device name.
Provide an implementation of resolve_mounted_volume which executes commands on the cloud VM via a supplied UnixLocalCommand object in order to set the _mount_point and _mount_options properties.

If the volume was cloned from a snapshot then the source snapshot identifier must also be stored in this class so that calling code can determine if/how/where a volume cloned from a given snapshot is mounted.

__init__()View on GitHub #

property mount_optionsView on GitHub #

The mount options with which this device is currently mounted.

This must be resolved using metadata obtained from the cloud provider which describes how the volume is attached to the VM.

property mount_pointView on GitHub #

The mount point at which this volume is currently mounted.

This must be resolved using metadata obtained from the cloud provider which describes how the volume is attached to the VM.

abstract resolve_mounted_volume(cmd)View on GitHub #

Resolve the mount point and mount options using shell commands.

This method must use cmd together with any additional private properties available in the provider-specific implementation in order to resolve the mount point and mount options for this volume.

Parameters:: cmd (UnixLocalCommand) – Wrapper for local/remote commands on the instance to which this volume is attached.

abstract property source_snapshotView on GitHub #

The source snapshot from which this volume was cloned.

Return type:: str|None
Returns:: A snapshot identifier.

barman.cloud.configure_logging(config)View on GitHub #: Get a nicer output from the Python logging package

barman.cloud.copyfileobj_pad_truncate(src, dst, length=None)View on GitHub #: Copy length bytes from fileobj src to fileobj dst. If length is None, copy the entire content. This method is used by the TarFileIgnoringTruncate.addfile().