The cback3-amazons3-sync command
Prev | Chapter 4. Command Line Tools | Next

## The cback3-amazons3-sync command

### Introduction

The cback3-amazons3-sync tool is used for synchronizing entire directories of files up to an Amazon S3 cloud storage bucket, outside of the normal Cedar Backup process.

This might be a good option for some types of data, as long as you understand the limitations around retrieving previous versions of objects that get modified or deleted as part of a sync. S3 does support versioning, but it won't be quite as easy to get at those previous versions as with an explicit incremental backup like cback3 provides. Cedar Backup does not provide any tooling that would help you retrieve previous versions.

The underlying functionality relies on the AWS CLI toolset. Before you use this extension, you need to set up your Amazon S3 account and configure AWS CLI as detailed in Amazons's setup guide. The aws command will be executed as the same user that is executing the cback3-amazons3-sync command, so make sure you configure it as the proper user. (This is different than the amazons3 extension, which is designed to execute as root and switches over to the configured backup user to execute AWS CLI commands.)

### Syntax

The cback3-amazons3-sync command has the following syntax:

     Usage: cback3-amazons3-sync [switches] sourceDir s3bucketUrl

     Cedar Backup Amazon S3 sync tool.

     This Cedar Backup utility synchronizes a local directory to an Amazon S3
     bucket.  After the sync is complete, a validation step is taken.  An
     error is reported if the contents of the bucket do not match the
     source directory, or if the indicated size for any file differs.
     This tool is a wrapper over the AWS CLI command-line tool.

     The following arguments are required:

       sourceDir            The local source directory on disk (must exist)
       s3BucketUrl          The URL to the target Amazon S3 bucket

     The following switches are accepted:

       -h, --help           Display this usage/help listing
       -V, --version        Display version information
       -b, --verbose        Print verbose output as well as logging to disk
       -q, --quiet          Run quietly (display no output to the screen)
       -l, --logfile        Path to logfile (default: /var/log/cback3.log)
       -o, --owner          Logfile ownership, user:group (default: root:adm)
       -m, --mode           Octal logfile permissions mode (default: 640)
       -O, --output         Record some sub-command (i.e. aws) output to the log
       -d, --debug          Write debugging information to the log (implies --output)
       -s, --stack          Dump Python stack trace instead of swallowing exceptions
       -D, --diagnostics    Print runtime diagnostics to the screen and exit
       -v, --verifyOnly     Only verify the S3 bucket contents, do not make changes
       -w, --ignoreWarnings Ignore warnings about problematic filename encodings

     Typical usage would be something like:

       cback3-amazons3-sync /home/myuser s3://example.com-backup/myuser

     This will sync the contents of /home/myuser into the indicated bucket.

### Switches

`-h`, `--help`

  Display usage/help listing.

`-V`, `--version`

  Display version information.

`-b`, `--verbose`

  Print verbose output to the screen as well writing to the logfile. When this option is enabled, most information that would normally be written to the logfile will also be written to the screen.

`-q`, `--quiet`

  Run quietly (display no output to the screen).

`-l`, `--logfile`

  Specify the path to an alternate logfile. The default logfile file is `/var/log/cback3.log`.

`-o`, `--owner`

  Specify the ownership of the logfile, in the form `user:group`. The default ownership is `root:adm`, to match the Debian standard for most logfiles. This value will only be used when creating a new logfile. If the logfile already exists when the cback3-amazons3-sync command is executed, it will retain its existing ownership and mode. Only user and group names may be used, not numeric uid and gid values.

`-m`, `--mode`

  Specify the permissions for the logfile, using the numeric mode as in chmod(1). The default mode is `0640` (`-rw-r-----`). This value will only be used when creating a new logfile. If the logfile already exists when the cback3-amazons3-sync command is executed, it will retain its existing ownership and mode.

`-O`, `--output`

  Record some sub-command output to the logfile. When this option is enabled, all output from system commands will be logged. This might be useful for debugging or just for reference.

`-d`, `--debug`

  Write debugging information to the logfile. This option produces a high volume of output, and would generally only be needed when debugging a problem. This option implies the `--output` option, as well.

`-s`, `--stack`

  Dump a Python stack trace instead of swallowing exceptions. This forces Cedar Backup to dump the entire Python stack trace associated with an error, rather than just propagating last message it received back up to the user interface. Under some circumstances, this is useful information to include along with a bug report.

`-D`, `--diagnostics`

  Display runtime diagnostic information and then exit. This diagnostic information is often useful when filing a bug report.

`-v`, `--verifyOnly`

  Only verify the S3 bucket contents against the directory on disk. Do not make any changes to the S3 bucket or transfer any files. This is intended as a quick check to see whether the sync is up-to-date.

  Although no files are transferred, the tool will still execute the source filename encoding check, discussed below along with `--ignoreWarnings`.

`-w`, `--ignoreWarnings`

  The AWS CLI S3 sync process is very picky about filename encoding. Files that the Linux filesystem handles with no problems can cause problems in S3 if the filename cannot be encoded properly in your configured locale. As of this writing, filenames like this will cause the sync process to abort without transferring all files as expected.

  To avoid confusion, the cback3-amazons3-sync tries to guess which files in the source directory will cause problems, and refuses to execute the AWS CLI S3 sync if any problematic files exist. If you'd rather proceed anyway, use `--ignoreWarnings`.

  If problematic files are found, then you have basically two options: either correct your locale (i.e. if you have set `LANG=C`) or rename the file so it can be encoded properly in your locale. The error messages will tell you the expected encoding (from your locale) and the actual detected encoding for the filename.

Prev | Up | Next
The cback3 command | Home | The cback3-span command