[s3ql] Naive expectations about s3qlcp snapshots?

[Rabid Mutant]

Installed S3QL version 2.28.

I use an S3 backend for storing data, with hourly backups. This has worked
well for several years.

On June 24 this year I started creating snapshots, and the S3 costs seem to
have linearly increased since then (see chart below), with the bulk of the
new costs being APS2-EarlyDelete-SIA and APS2-Retrieval-SIA.

Since the data is relatively static (very little deleted, slow additions),
my naive expectation was that with de-duping etc, there would be little or
no impact. Yet I am seeing quite remarkable growth.

I should note that the total storage in that time has gone from 19GB to
31GB, so not a huge increase.

The basic hourly process is:

- mount the S3 file system
- rsync -avr /my/local/data /my/s3/mountpoint
- unmount

(this syncs local data with the 'data' dir in the S3 mount point).

The basic daily operation is:

- mount the S3 file system
- s3qlcp /my/s3/mountpoint/data /my/s3/mountpoint/snapshots/<daily-dir>
- rsync -avr --delete /my/local/data /my/s3/mountpoint
- unmount

The additions in this process are the daily rsync with "--delete" and the
s3qlcp; I do not expect the --delete is deleting many files based on log
files of the operation.

Have I been hopelessly naive here? Or is there something I am missing in
how this should work? What I had expected is that the snapshots would add
little or no overhead.


[image: S3 costs.PNG]


I have now disabled the daily snapshots and will see if the costs reduce,
but any help or insights would be appreciated.

[Rabid Mutant]

Ahhhh....maybe this is relevant....I also do a daily s3ql_verify....would I
be correct in assuming that daily snapshot verification costs the same as
verification of the main data? That might explain the growing retrieval
costs, but not really the "early deletion" growth.