barman.wal_archiver module#

class barman.wal_archiver.CloudWalStorageStrategy(backup_manager, server)View on GitHub #

Bases: WalStorageStrategy

WAL storage strategy for cloud storage.

__init__(backup_manager, server)View on GitHub #

Constructor.

Parameters:

backup_manager (barman.backup_manager.BackupManager) – The backup manager
server (barman.server.Server) – The server

_abc_impl = <_abc._abc_data object>#

_build_wal_object_key(wal_name, compression)View on GitHub #

Build the full cloud object key for a WAL file.

Parameters:

wal_name (str) – the WAL file name (e.g. 000000010000000000000001)
compression (str|None) – the compression algorithm name used for the WAL

Returns:

the full cloud object key, including compression extension if applicable

Return type:

str

_check_duplicate(wal_info, object_key)View on GitHub #

Check if the cloud object object_key is identical to the source file.

If the WAL was compressed, the decompress parameter of download_file is used to decompress the cloud object before comparison.

Parameters:

wal_info (WalFileInfo) – the WAL file info object
object_key (str) – the cloud storage object key

Raises:

DuplicateWalFile – if the destination file exists and is different from the source file
MatchingDuplicateWalFile – if the destination file exists and is identical to the source file

_get_compression_extension(compression)View on GitHub #

Return the file extension for the given compression algorithm.

The compression value must be either None or a key present in compression_registry whose class defines an EXTENSION attribute. Passing an unrecognised algorithm or one without EXTENSION (e.g. pigz, custom) will raise KeyError or AttributeError respectively.

Parameters:: compression (str|None) – the compression algorithm name
Returns:: the file extension (e.g. “.gz”) or “” if no compression
Return type:: str

delete(wals_to_delete)View on GitHub #

Delete WAL files according to the configured destination.

Parameters:: wals_to_delete (dict[str, list[WalFileInfo]]) – A dictionary where key is the WAL directory name and value is a list of wal_info objects representing the WALs to be deleted in that directory.
Return list[str]:: a list of deleted WAL names.

exists(wal_full_path)View on GitHub #

get_full_path(wal_name)View on GitHub #

Construct the full cloud object key for a given WAL file name.

Note

This method uses the current compression configuration to determine the file extension. If the compression config has changed since the WAL was originally stored, this method will return a path that does not match the actual cloud object. Callers that need to handle WALs stored under a previous compression config should use bucket listing instead.

Parameters:: wal_name (str) – the WAL file name
Returns:: the full cloud object key
Return type:: str

save(compressor, encryption, wal_info, **kwargs)View on GitHub #

Effectively persist a WAL file according to the configured destination.

If a compressor is provided, the WAL file is compressed in-memory before upload and the cloud key includes the compression extension. The wal_info object is updated with the compression type and compressed size so that xlogdb records accurate metadata.

Parameters:

compressor – an InternalCompressor instance, or None if no compression is desired
encryption (None|Encryption) – the encryptor for the file (if any)
wal_info (WalFileInfo) – the WAL file is being processed
kwargs –
additional parameters for the storage strategy, if any:
- ”skip_delete”: if True, the source file will not be deleted after a successful upload.

Raises:

CompressionException – if compressor is not an InternalCompressor instance

Note

Only InternalCompressor subclasses (gzip, bzip2, xz, zstd, lz4, snappy) are supported for cloud WAL storage. pigz and custom are not supported because they rely on external processes and cannot compress in-memory.

Note

Encryption is not yet supported for cloud WAL storage. The encryption parameter is kept for interface compatibility.

class barman.wal_archiver.FileWalArchiver(backup_manager)View on GitHub #

Bases: WalArchiver

Manager of file-based WAL archiving operations (aka ‘log shipping’).

__init__(backup_manager)View on GitHub #

Base class init method.

Parameters:

backup_manager – The backup manager
name – The name of this archiver

Returns:

_abc_impl = <_abc._abc_data object>#

check(check_strategy)View on GitHub #

Perform additional checks for FileWalArchiver - invoked by server.check_postgres

Parameters:: check_strategy (CheckStrategy) – the strategy for the management of the results of the various checks

fetch_remote_status()View on GitHub #

Returns the status of the FileWalArchiver.

This method does not raise any exception in case of errors, but set the missing values to None in the resulting dictionary.

Return type:: dict[str, None|str]

get_next_batch()View on GitHub #

Returns the next batch of WAL files that have been archived through a PostgreSQL’s ‘archive_command’ (in the ‘incoming’ directory)

Returns:: WalArchiverQueue: list of WAL files

status()View on GitHub #: Set additional status info - invoked by Server.status()

class barman.wal_archiver.LocalWalStorageStrategy(backup_manager, server)View on GitHub #

Bases: WalStorageStrategy

WAL storage strategy for local filesystem storage.

Note

For now, this class is also responsible for encrypting and compressing files before storing them, but in the future this responsibility should be moved elsewhere, desirably in a shared component, so that strategies’ only responsibility is to actually persist files.

__init__(backup_manager, server)View on GitHub #

Constructor.

Parameters:

backup_manager (barman.backup_manager.BackupManager) – The backup manager
server (barman.server.Server) – The server

_abc_impl = <_abc._abc_data object>#

_check_duplicate(src_file, dst_file, wal_info)View on GitHub #

Check if the destination WAL file already exists in local storage, and if so, whether it is identical to the source file.

Parameters:

src_file (str) – the source WAL file path
dst_file (str) – the destination WAL file path
wal_info (WalFileInfo) – the WAL file info object

Raises:

DuplicateWalFile – if the destination file exists and is different from the source file
MatchingDuplicateWalFile – if the destination file exists and is identical to the source file

_compress_file(compressor, src_file, dst_dir, wal_info)View on GitHub #

Compress src_file to a “temp” file inside dst_dir and updates wal_info.

Note

The temporary compressed file will have the same name as the source file, with a “.compressed” suffix appended. It’s created on the assumption that the caller will later remove the file or rename/move it to its final destination.

Parameters:

compressor – the compressor to use
src_file (str) – the file to compress
dst_dir (str) – the directory where the compressed file will be created
wal_info (WalFileInfo) – the WAL file info object

Returns:

the path to the compressed temporary file

_copy_stats(src_file, dst_file, wal_info)View on GitHub #

Copy stats from src_file to dst_file and updates its wal_info.

This is used preserve the metadata of the original file after compression or encryption, updating only the size in wal_info accordingly.

Parameters:

src_file (str) – the source WAL file path
dst_file (str) – the destination WAL file path
wal_info (WalFileInfo) – the WAL file info object

_delete_wal_directory(wal_dir, wal_list)View on GitHub #

_delete_wal_file(wal_info)View on GitHub #

Perform the actual deletion of the WAL file from local storage.

Parameters:: wal_info (WalFileInfo) – the WAL file info object

_encrypt_file(encryption, src_file, dst_dir, wal_info)View on GitHub #

Encrypt src_file to a “temp” file inside dst_dir and updates wal_info.

Note

The temporary encrypted file will have the same name as the source file, with a “.gpg” suffix appended. It’s created on the assumption that the caller will later remove the file or rename/move it to its final destination.

Parameters:

encryption – the encryptor to use
src_file (str) – the file to encrypt
dst_dir (str) – the directory where the encrypted file will be created
wal_info (WalFileInfo) – the WAL file info object

Returns:

the path to the encrypted temporary file

_fsync_contents(src_dir, dst_dir, dst_file)View on GitHub #

Fsync the contents of source and destination directories and the destination file.

Parameters:

src_dir (str) – the source directory path
dst_dir (str) – the destination directory path
dst_file (str) – the destination file path

_remove_intermediary_files()View on GitHub #: Remove any intermediary files created during the archival process

_rename_or_copy_file(src_file, dst_file)View on GitHub #

Rename or copy src_file to dst_file.

Rename is attempted first, and if it fails (because the source and destination are on different filesystems), a copy is performed.

Parameters:

src_file (str) – the source file path
dst_file (str) – the destination file path

delete(wals_to_delete)View on GitHub #

Delete WAL files according to the configured destination.

Parameters:: wals_to_delete (dict[str, list[WalFileInfo]]) – A dictionary where key is the WAL directory name and value is a list of wal_info objects representing the WALs to be deleted in that directory.
Return list[str]:: a list of deleted WAL names.

exists(wal_full_path)View on GitHub #

get_full_path(wal_name)View on GitHub #

save(compressor, encryption, wal_info, **kwargs)View on GitHub #

Effectively persist a WAL file according to the configured destination.

Compression and encryption are applied, if requested.

Parameters:

compressor – the compressor for the file (if any)
encryption (None|Encryption) – the encryptor for the file (if any)
wal_info (WalFileInfo) – the WAL file is being processed
kwargs – additional parameters for the storage strategy, if any

Raises:

DuplicateWalFile – if the destination file exists and is different from the source file
MatchingDuplicateWalFile – if the destination file exists and is identical to the source file

class barman.wal_archiver.StreamingWalArchiver(backup_manager)View on GitHub #

Bases: WalArchiver

Object used for the management of streaming WAL archive operation.

__init__(backup_manager)View on GitHub #

Base class init method.

Parameters:

backup_manager – The backup manager
name – The name of this archiver

Returns:

_abc_impl = <_abc._abc_data object>#

_is_synchronous()View on GitHub #

Check if receive-wal process is eligible for synchronous replication

The receive-wal process is eligible for synchronous replication if synchronous_standby_names is configured and contains the value of streaming_archiver_name

Return type:: bool

_reset_streaming_status(postgres_status, streaming_status)View on GitHub #: Reset the status of receive-wal by removing the .partial file that is marking the current position and creating one that is current with the PostgreSQL insert location

_truncate_partial_file_if_needed(xlog_segment_size)View on GitHub #

Truncate .partial WAL file if size is not 0 or xlog_segment_size

Parameters:: xlog_segment_size (int)

check(check_strategy)View on GitHub #

Perform additional checks for StreamingWalArchiver - invoked by server.check_postgres

Parameters:: check_strategy (CheckStrategy) – the strategy for the management of the results of the various checks

fetch_remote_status()View on GitHub #

Execute checks for replication-based wal archiving

This method does not raise any exception in case of errors, but set the missing values to None in the resulting dictionary.

Return type:: dict[str, None|str]

get_next_batch()View on GitHub #

Returns the next batch of WAL files that have been archived via streaming replication (in the ‘streaming’ directory)

This method always leaves one file in the “streaming” directory, because the ‘pg_receivexlog’ process needs at least one file to detect the current streaming position after a restart.

Returns:: WalArchiverQueue: list of WAL files

receive_wal(reset=False)View on GitHub #

Creates a PgReceiveXlog object and issues the pg_receivexlog command for a specific server

Parameters:: reset (bool) – When set reset the status of receive-wal
Raises:: ArchiverFailure – when something goes wrong

status()View on GitHub #: Set additional status info - invoked by Server.status()

class barman.wal_archiver.WalArchiver(backup_manager, name)View on GitHub #

Bases: RemoteStatusMixin

Base class for WAL archiver objects

__init__(backup_manager, name)View on GitHub #

Base class init method.

Parameters:

backup_manager – The backup manager
name – The name of this archiver

Returns:

_abc_impl = <_abc._abc_data object>#

archive(verbose=True)View on GitHub #

Archive WAL files, discarding duplicates or those that are not valid.

Parameters:: verbose (boolean) – Flag for verbose output

abstractmethod check(check_strategy)View on GitHub #

Perform specific checks for the archiver - invoked by server.check_postgres

Parameters:: check_strategy (CheckStrategy) – the strategy for the management of the results of the various checks

abstractmethod get_next_batch()View on GitHub #

Return a WalArchiverQueue containing the WAL files to be archived.

Return type:: WalArchiverQueue

receive_wal(reset=False)View on GitHub #

Manage reception of WAL files. Does nothing by default. Some archiver classes, like the StreamingWalArchiver, have a full implementation.

Parameters:: reset (bool) – When set, resets the status of receive-wal
Raises:: ArchiverFailure – when something goes wrong

abstractmethod status()View on GitHub #: Set additional status info - invoked by Server.status()

static summarise_error_files(error_files)View on GitHub #

Summarise a error files list

Parameters:: error_files (list[str]) – Error files list to summarise
Return str:: A summary, None if there are no error files

class barman.wal_archiver.WalArchiverQueue(items, errors=None, skip=None, batch_size=0, total_size=0)View on GitHub #

Bases: list

__init__(items, errors=None, skip=None, batch_size=0, total_size=0)View on GitHub #

A WalArchiverQueue is a list of WalFileInfo which has two extra attribute list:

errors: containing a list of unrecognized files
skip: containing a list of skipped files.

It also stores batch run size information in case it is requested by configuration, in order to limit the number of WAL files that are processed in a single run of the archive-wal command.

Note

This class was originally designed to hold all the WAL files pending to be archived, and to process up to a certain number of files (the batch size). However, it is now used to hold only the files that are being processed in a single run of the archive-wal command. That change was made to avoid the overhead of having to create lots of WalFileInfo objects, even if only a subset of them would be used by the archiver in a given batch. We might want to rework or remove this class in the future, as it doesn’t seem to add much value to the archiving process anymore.

Parameters:

items – iterable from which initialize the list
errors – an optional list of unrecognized files
skip – an optional list of skipped files
batch_size – size of the current batch run (0=unlimited)
total_size – the total number of WAL files available for archiving.

class barman.wal_archiver.WalStorageStrategy(backup_manager, server)View on GitHub #

Bases: object

Abstract base class for WAL storage strategies.

WAL storage strategies are used to effectively store WAL files in the connfigured destination, be it local filesystem or cloud storage.

__init__(backup_manager, server)View on GitHub #

Constructor.

Parameters:

backup_manager (barman.backup_manager.BackupManager) – The backup manager
server (barman.server.Server) – The server

_abc_impl = <_abc._abc_data object>#

_run_post_archive_scripts(wal_info, dest_file, error)View on GitHub #

Run the post-archive scripts.

Parameters:

wal_info (WalFileInfo) – the WAL file info object
dest_file (str) – the destination WAL file path
error (Exception|None) – the exception raised during the archival process, if any

_run_post_delete_wal_scripts(wal_info, error=None)View on GitHub #

Run the post-delete hook-scripts, if any, on the given WAL.

Parameters:

wal_info (barman.infofile.WalFileInfo) – WAL to run the script on.
error (None|str) – error message in case a failure happened.

_run_pre_archive_scripts(wal_info, src_file)View on GitHub #

Run the pre-archive scripts.

Parameters:

wal_info (WalFileInfo) – the WAL file info object
src_file (str) – the source WAL file path

_run_pre_delete_wal_scripts(wal_info)View on GitHub #

Run the pre-delete hook-scripts, if any, on the given WAL.

Parameters:: wal_info (barman.infofile.WalFileInfo) – WAL to run the script on.

abstractmethod delete(wals_to_delete)View on GitHub #

Delete WAL files according to the configured destination.

Parameters:: wals_to_delete (dict[str, list[WalFileInfo]]) – A dictionary where key is the WAL directory name and value is a list of wal_info objects representing the WALs to be deleted in that directory.
Return list[str]:: a list of deleted WAL names.

abstractmethod save(compressor, encryption, wal_info, **kwargs)View on GitHub #

Effectively persist a WAL file according to the configured destination.

Parameters:

compressor – the compressor for the file (if any)
encryption (None|Encryption) – the encryptor for the file (if any)
wal_info (WalFileInfo) – the WAL file is being processed
kwargs – additional parameters for the storage strategy, if any

barman.wal_archiver module#

This Page