[s3ql] Understanding AWS billing with S3QL

[h***@gmail.com]

Hey guys,

We have been using/testing s3ql for around a year now. And our s3 bill is
usually around the same each month. We backup 4 servers that total about
200GB using CDP/R1soft. I think we keep 7 days of delta's.

As you may know. S3 inbound is actually free, it's only outbound that you
get charged. Here is a look at my bill.

Loading Image...

Typically, if your saving files to s3, its free. And the first 1gb data
transfer out is also free. At first thought, you might assume that backing
up, would only require inbound.

Now, I can only speculate that the costs that we are incurring, is the
backup software pulling deltas/blocks from s3, so that it can merge them
with the original backup file. Maybe it has to do this in memory?

Why else would we be charged for OUTBOUND file charges/perGB.

Now that I have fully speculated, can someone tell me exactly, what
function of s3ql/CDP is causing outbound data transfer?

What recommendations if any, do you have to limit the costs?

Thanks,

Sysadmin

[Nikolaus Rath]

Post by h***@gmail.com
Typically, if your saving files to s3, its free. And the first 1gb data
transfer out is also free. At first thought, you might assume that backing
up, would only require inbound.
Now, I can only speculate that the costs that we are incurring, is the
backup software pulling deltas/blocks from s3, so that it can merge them
with the original backup file. Maybe it has to do this in memory?

No, the comparison is only using checksums.

Post by h***@gmail.com
Why else would we be charged for OUTBOUND file charges/perGB.

Are you always mounting on the same computer using the same --cachedir?
If not, S3QL will always have to download an up to date copy of the
metadata.

If not, my best guess is that some other program is reading data from
the mounted filesystem (maybe some indexer?).

Does AWS give you more detailed logs? It would be interesting to know
which objects are being downloaded.

Best,
-Nikolaus