[s3ql] OVH swift storage.. s3ql.backends.s3c.HTTPError: 413 Request Entity Too Large error

[d***@gmail.com]

Hi,

I have a large-ish S3QL filesystem stored on OVH Swift Storage.. it was
created using S3QL 2.22, and has been upgraded over time, so it is now
using S3QL 2.29

Every so often for various reasons the mount has crashed (ovh network
issues, space issues on my server, etc).. but running a fsck.s3ql on the
system to resolve the issues, and remounting and continuing use has been OK.

But a few days ago the mount went offline with the error: *s3ql.backends.s3c.HTTPError:
413 Request Entity Too Large*

I started an fsck.s3ql - and it failed to complete with the (not
unexpected!) same error. *s3ql.backends.s3c.HTTPError: 413 Request Entity
Too Large*

I am now running* fsck.f3ql* with --debug in the hope it will allow me to
zoom in on the problem object or objects, as the swift CLI seems quite slow
for digging down unto this information by listing object sizes, I'll also
open a request with OVH support if they have changed any limits on their
swift object server recently.

Are there any other commands I can run that will help drill down to find
and resolve the issue? s3ql_verify might just run for a long time and hit
the same problem?

The file system was created with the default --max-obj-size so I cant see
this being an issue? when OVH swift storage allows much much larger objects
according to the capabilities command.

more details below......

*Details from mount.log of the mount error:*


2018-08-08 14:47:30.960 18159:Metadata-Upload-Thread
s3ql.metadata.upload_metadata: Compressing and uploading metadata...
2018-08-08 15:41:03.977 18159:Metadata-Upload-Thread root.excepthook:
Uncaught top-level exception:
Traceback (most recent call last):
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/mount.py",
line 64, in run_with_except_hook
run_old(*args, **kw)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/mount.py",
line 659, in run
upload_metadata(backend, fh, self.param)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/metadata.py",
line 322, in upload_metadata
metadata=param, is_compressed=True)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/common.py",
line 108, in wrapped
return method(*a, **kw)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/common.py",
line 340, in perform_write
return fn(fh)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/comprenc.py",
line 536, in __exit__
self.close()
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/comprenc.py",
line 530, in close
self.fh.close()
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/common.py",
line 108, in wrapped
return method(*a, **kw)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/s3c.py",
line 948, in close
headers=self.headers, body=self.fh)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/swift.py",
line 268, in _do_request
raise HTTPError(resp.status, resp.reason, resp.headers)
s3ql.backends.s3c.HTTPError: 413 Request Entity Too Large
2018-08-08 15:41:40.912 18159:MainThread s3ql.mount.unmount: Unmounting
file system...
2018-08-08 15:41:41.312 18159:MainThread root.excepthook: Uncaught
top-level exception:
Traceback (most recent call last):
File "/usr/local/bin/mount.s3ql", line 11, in <module>
load_entry_point('s3ql==2.29', 'console_scripts', 'mount.s3ql')()
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/mount.py",
line 222, in main
raise RuntimeError('Received signal %d, terminating' % (ret,))
RuntimeError: Received signal 15, terminating

*Details from fsck.log of the fsck error...*

2018-08-11 13:32:24.138 2396:MainThread s3ql.metadata.dump_metadata:
..contents..
2018-08-11 13:48:30.702 2396:MainThread s3ql.metadata.dump_metadata:
..ext_attributes..
2018-08-11 13:48:30.710 2396:MainThread s3ql.metadata.upload_metadata:
Compressing and uploading metadata...
2018-08-11 14:40:03.096 2396:MainThread root.excepthook: Uncaught top-level
exception:
Traceback (most recent call last):
File "/usr/local/bin/fsck.s3ql", line 11, in <module>
load_entry_point('s3ql==2.29', 'console_scripts', 'fsck.s3ql')()
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/fsck.py",
line 1343, in main
dump_and_upload_metadata(backend, db, param)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/metadata.py",
line 313, in dump_and_upload_metadata
upload_metadata(backend, fh, param)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/metadata.py",
line 322, in upload_metadata
metadata=param, is_compressed=True)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/common.py",
line 108, in wrapped
return method(*a, **kw)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/common.py",
line 340, in perform_write
return fn(fh)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/comprenc.py",
line 536, in __exit__
self.close()
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/comprenc.py",
line 530, in close
self.fh.close()
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/common.py",
line 108, in wrapped
return method(*a, **kw)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/s3c.py",
line 948, in close
headers=self.headers, body=self.fh)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/swift.py",
line 268, in _do_request
raise HTTPError(resp.status, resp.reason, resp.headers)
s3ql.backends.s3c.HTTPError: 413 Request Entity Too Large

*Output of s3qlstat from a recent run of my backup script*

Directory entries: 439577485
Inodes: 439576371
Data blocks: 3933555
Total data size: 50.9 TB
After de-duplication: 1.62 TB (3.17% of total)
After compression: 812 GiB (1.56% of total, 49.04% of de-duplicated)
Database size: 52.0 GiB (uncompressed)
Cache size: 55.4 GiB, 338820 entries
Cache size (dirty): 0 bytes, 0 entries
Queued object removals: 0

*Output of "swift capabilities"*

Core: swift
Options:
account_autocreate: True
account_listing_limit: 10000
allow_account_management: True
container_listing_limit: 10000
extra_header_count: 0
max_account_name_length: 256
max_container_name_length: 256
max_file_size: 5368709122
max_header_size: 8192
max_meta_count: 90
max_meta_name_length: 128
max_meta_overall_size: 4096
max_meta_value_length: 256
max_object_name_length: 1024
policies: [{u'default': True, u'diskfile': u'egg:swift#replication.fs',
u'name': u'PCS', u'aliases': u'PCS'}, {u'diskfile':
u'egg:swift#erasure_coding.fs', u'name': u'PCA', u'aliases': u'PCA'}]
strict_cors_mode: True
valid_api_versions: [u'v1', u'v1.0']
version: 2.15.1.dev65
Additional middleware: account_quotas
Additional middleware: bulk_delete
Options:
max_deletes_per_request: 10000
max_failed_deletes: 1000
Additional middleware: bulk_upload
Options:
max_containers_per_extraction: 10000
max_failed_extractions: 1000
Additional middleware: cname_lookup
Options:
lookup_depth: 1
Additional middleware: container_quotas
Additional middleware: container_sync
Options:
realms: {u'OVH_PUBLIC_CLOUD': {u'clusters': {u'WAW1': {}, u'SERCO-DIAS1':
{}, u'GRA1': {}, u'GRA3': {}, u'GRA5': {}, u'UK1': {}, u'SBG5': {},
u'SBG3': {}, u'SBG1': {}, u'BHS3': {}, u'DE1': {}, u'BHS1': {u'current':
True}}}}
Additional middleware: crossdomain
Additional middleware: domain_remap
Options:
default_reseller_prefix: None
Additional middleware: formpost
Additional middleware: slo
Options:
max_manifest_segments: 1000
max_manifest_size: 2097152
min_segment_size: 1
Additional middleware: staticweb
Additional middleware: tempurl
Options:
incoming_allow_headers: []
incoming_remove_headers: [u'x-timestamp']
methods: [u'GET', u'HEAD', u'PUT', u'POST', u'DELETE']
outgoing_allow_headers: [u'x-object-meta-public-*']
outgoing_remove_headers: [u'x-object-meta-*']

[Nikolaus Rath]

Post by d***@gmail.com
413 Request Entity Too Large*

[...]

Post by d***@gmail.com
s3ql.metadata.upload_metadata: Compressing and uploading metadata...
raise HTTPError(resp.status, resp.reason, resp.headers)
s3ql.backends.s3c.HTTPError: 413 Request Entity Too Large

Most likely your metadata object has exceeded the maximum size allowed
by the server (this means that s3ql_verify will not show the problem,
because it does not upload any metadata).

Unfortunately there is currently no workaround for this. See https://bitbucket.org/nikratio/s3ql/issues/266/support-metadata-5-gb


Best,
-Nikolaus

[d***@gmail.com]

Post by Nikolaus Rath

Post by d***@gmail.com
413 Request Entity Too Large*

[...]

Post by d***@gmail.com
s3ql.metadata.upload_metadata: Compressing and uploading metadata...
raise HTTPError(resp.status, resp.reason, resp.headers)
s3ql.backends.s3c.HTTPError: 413 Request Entity Too Large

Most likely your metadata object has exceeded the maximum size allowed
by the server (this means that s3ql_verify will not show the problem,
because it does not upload any metadata).
Unfortunately there is currently no workaround for this. See
https://bitbucket.org/nikratio/s3ql/issues/266/support-metadata-5-gb

Is there a way to force a mount of the existing data using local metadata
so I can trim down the data until the meta data is smaller - I was keeping
a lot of extra copies "because I could" - completely unaware I was running
into a brick wall - or use the temp. mount to re-factor this into a set of
smaller S3QL filesystems instead of one larger one?
So it seems my fsck.s3ql working with the local metadata succeeds.. it just
fails when it uploads it to OVH..

[d***@gmail.com]

Post by d***@gmail.com

Post by Nikolaus Rath

Post by d***@gmail.com
413 Request Entity Too Large*

[...]

Post by d***@gmail.com
s3ql.metadata.upload_metadata: Compressing and uploading metadata...
raise HTTPError(resp.status, resp.reason, resp.headers)
s3ql.backends.s3c.HTTPError: 413 Request Entity Too Large

Most likely your metadata object has exceeded the maximum size allowed
by the server (this means that s3ql_verify will not show the problem,
because it does not upload any metadata).
Unfortunately there is currently no workaround for this. See
https://bitbucket.org/nikratio/s3ql/issues/266/support-metadata-5-gb

Is there a way to force a mount of the existing data using local metadata
so I can trim down the data until the meta data is smaller - I was keeping
a lot of extra copies "because I could" - completely unaware I was running
into a brick wall - or use the temp. mount to re-factor this into a set of
smaller S3QL filesystems instead of one larger one?
So it seems my fsck.s3ql working with the local metadata succeeds.. it
just fails when it uploads it to OVH..

Or a way to copy this data to a storage method (any storage method!)
without the 5G metadata limit (sshfs? s3? something?) so it can be
accessed? I'm assuming clone_fs.py as it is written will use the stored
metadata.

[Nikolaus Rath]

Post by d***@gmail.com

Post by Nikolaus Rath

Post by d***@gmail.com
413 Request Entity Too Large*

[...]

Post by d***@gmail.com
s3ql.metadata.upload_metadata: Compressing and uploading metadata...
raise HTTPError(resp.status, resp.reason, resp.headers)
s3ql.backends.s3c.HTTPError: 413 Request Entity Too Large

Most likely your metadata object has exceeded the maximum size allowed
by the server (this means that s3ql_verify will not show the problem,
because it does not upload any metadata).
Unfortunately there is currently no workaround for this. See
https://bitbucket.org/nikratio/s3ql/issues/266/support-metadata-5-gb

Is there a way to force a mount of the existing data using local metadata
so I can trim down the data until the meta data is smaller - I was keeping
a lot of extra copies "because I could" - completely unaware I was running
into a brick wall - or use the temp. mount to re-factor this into a set of
smaller S3QL filesystems instead of one larger one?
So it seems my fsck.s3ql working with the local metadata succeeds.. it just
fails when it uploads it to OVH..

Try the following (dangerous) patch. It will make fsck.s3ql skip the
actual data upload. If you then run mount.s3ql with the same cache
directory, it should use the local copy and you can trim it down:

diff --git a/src/s3ql/fsck.py b/src/s3ql/fsck.py
--- a/src/s3ql/fsck.py
+++ b/src/s3ql/fsck.py
@@ -1340,7 +1340,7 @@
param['last_fsck'] = time.time()
param['last-modified'] = time.time()

- dump_and_upload_metadata(backend, db, param)
+ #dump_and_upload_metadata(backend, db, param)
save_params(cachepath, param)

log.info('Cleaning up local metadata...')



HTH,
-Nikolaus

[d***@gmail.com]

On Sunday, August 12, 2018 at 12:26:09 AM UTC+10, Nikolaus Rath wrote:
<.. snip...>

Post by Nikolaus Rath
Try the following (dangerous) patch. It will make fsck.s3ql skip the
actual data upload. If you then run mount.s3ql with the same cache
diff --git a/src/s3ql/fsck.py b/src/s3ql/fsck.py
--- a/src/s3ql/fsck.py
+++ b/src/s3ql/fsck.py
@@ -1340,7 +1340,7 @@
param['last_fsck'] = time.time()
param['last-modified'] = time.time()
- dump_and_upload_metadata(backend, db, param)
+ #dump_and_upload_metadata(backend, db, param)
save_params(cachepath, param)
log.info('Cleaning up local metadata...')

This was enough to get me online again.. then a temporary mount with option
"--metadata-upload-interval 2629800" just to do some scripted trimming of
older increments with s3qlrm..
then a successful unmount with where I was happy to see:

s3ql.metadata.upload_metadata: Wrote 4.33 GiB of compressed metadata.

Now I have some instances of "clone_fs.py" running so I can break this down
into 3 separate filesystems to spread the load (and metadata) .. and backed
out the dangerous, but useful patch to fsck..

Thanks!

[d***@gmail.com]

.. or thinking this from the other side.. the last successful upload of
metadata is under the limit, so if I do an fsck / mount or clone using only
the remote data I should get it accessible so that I can take required
evasive action (move to S3 or nfs? break down into X smaller mounts?)