[s3ql] Optimizing deduplication

[c***@iu.edu]

Hi, I am interested in learning a bit more about the architecture,
specifically what is the order of operations between; splitting data into
chunks, compression, deduplication, and encryption?

I have a large amount of data that is very similar, but each file is only
somewhat compressible. My concern is that compression might have a negative
effect on the potential for efficient deduplication... is this possible?
Also, is the compression done separately on each chunk, or on a per-file
basis? Finally, will smaller chunks (max-file-size) produce better
deduplication (just at the expense of more network operations)? For what
it's worth, my data is mostly 3D numerical arrays where each file is a few
GB.

Thanks so much. Chris

[Nikolaus Rath]

Post by c***@iu.edu
Hi, I am interested in learning a bit more about the architecture,
specifically what is the order of operations between; splitting data into
chunks, compression, deduplication, and encryption?

1. Split
2. Deduplication
3. Compression
4. Encryption

Post by c***@iu.edu
I have a large amount of data that is very similar, but each file is only
somewhat compressible. My concern is that compression might have a negative
effect on the potential for efficient deduplication... is this
possible?

Not unless you do the splitting into blocks after the compression.

Post by c***@iu.edu
Also, is the compression done separately on each chunk, or on a per-file
basis?

Per chunk.

Post by c***@iu.edu
Finally, will smaller chunks (max-file-size) produce better
deduplication (just at the expense of more network operations)?

Whether it will result in more de-duplication depends on your data, but
it will certainly not result in less. It will, however, also increase
the size of your metadata DB.

Best,
-Nikolaus