[s3ql] Corruption upgrading S3 backend from rev 23 to 24

[Xomex]

Upgrading the backend with "s3qladm upgrade" from rev 23 to 24 stopped
about 78% through
with a CorruptedObjectError. Repeated attempts stops around the same place.
What can I do to repair and complete the revision bump? Hopefully all is
not lost?
X.

The end of the Upgrade log is as follows:
...


































*2018-06-24 13:16:55.057 546703:Thread-12 s3ql.backends.s3c._send_request:
sending HEAD /xomex-s3ql-new/s3ql_data_1364942018-06-24 13:16:55.057
546703:MainThread root.excepthook: Uncaught top-level exception:Traceback
(most recent call last): File "/usr/bin/s3qladm", line 11, in <module>
load_entry_point('s3ql==2.28', 'console_scripts', 's3qladm')() File
"/usr/lib64/python3.6/site-packages/s3ql-2.28-py3.6-linux-x86_64.egg/s3ql/adm.py",
line 94, in main return upgrade(options) File
"/usr/lib64/python3.6/site-packages/s3ql-2.28-py3.6-linux-x86_64.egg/s3ql/common.py",
line 436, in wrapper return fn(*a, **kw) File
"/usr/lib64/python3.6/site-packages/s3ql-2.28-py3.6-linux-x86_64.egg/s3ql/adm.py",
line 300, in upgrade update_obj_metadata(backend, backend_factory, db,
options) File
"/usr/lib64/python3.6/site-packages/s3ql-2.28-py3.6-linux-x86_64.egg/s3ql/adm.py",
line 351, in update_obj_metadata t.join_and_raise() File
"/usr/lib64/python3.6/site-packages/s3ql-2.28-py3.6-linux-x86_64.egg/s3ql/common.py",
line 390, in join_and_raise raise EmbeddedException(exc_info,
self.name)s3ql.common.EmbeddedException: caused by an exception in thread
Thread-5.Original/inner traceback (most recent call last): Traceback (most
recent call last): File
"/usr/lib64/python3.6/site-packages/s3ql-2.28-py3.6-linux-x86_64.egg/s3ql/common.py",
line 368, in run self.run_protected() File
"/usr/lib64/python3.6/site-packages/s3ql-2.28-py3.6-linux-x86_64.egg/s3ql/common.py",
line 420, in run_protected self.target(*self.args, **self.kwargs) File
"/usr/lib64/python3.6/site-packages/s3ql-2.28-py3.6-linux-x86_64.egg/s3ql/adm.py",
line 389, in upgrade_loop meta = backend.lookup(key) File
"/usr/lib64/python3.6/site-packages/s3ql-2.28-py3.6-linux-x86_64.egg/s3ql/backends/comprenc.py",
line 72, in lookup meta_raw = self.backend.lookup(key) File
"/usr/lib64/python3.6/site-packages/s3ql-2.28-py3.6-linux-x86_64.egg/s3ql/backends/common.py",
line 108, in wrapped return method(*a, **kw) File
"/usr/lib64/python3.6/site-packages/s3ql-2.28-py3.6-linux-x86_64.egg/s3ql/backends/s3c.py",
line 316, in lookup return self._extractmeta(resp, key) File
"/usr/lib64/python3.6/site-packages/s3ql-2.28-py3.6-linux-x86_64.egg/s3ql/backends/s3c.py",
line 761, in _extractmeta raise CorruptedObjectError('Invalid metadata
format: %s' % format_)s3ql.backends.common.CorruptedObjectError: Invalid
metadata format: raw*

[Nikolaus Rath]

Post by Xomex
Upgrading the backend with "s3qladm upgrade" from rev 23 to 24 stopped
about 78% through
with a CorruptedObjectError. Repeated attempts stops around the same place.
What can I do to repair and complete the revision bump? Hopefully all is
not lost?
X.

[...]

You could try to just skip over the corrupted objects - but note that
this will only allow the upgrade to proceed, you will still have
corrupted data in your filesystem.

In general, S3QL relies heavily on the backend maintaining data
integrity. If your backend isn't reliable, you will run into problems
regularly.

Try this patch:

diff --git a/src/s3ql/adm.py b/src/s3ql/adm.py
--- a/src/s3ql/adm.py
+++ b/src/s3ql/adm.py
@@ -9,7 +9,7 @@
from .logging import logging, QuietError, setup_logging
from . import CURRENT_FS_REV, REV_VER_MAP
from .backends.comprenc import ComprencBackend
-from .backends.common import NoSuchObject
+from .backends.common import NoSuchObject, CorruptedObjectEror
from .database import Connection
from .common import (get_seq_no, is_mounted, get_backend, load_params,
save_params, handle_on_return, get_backend_factory,
@@ -394,6 +394,9 @@
meta = backend.lookup(key)
except NoSuchObject:
continue
+ except CorruptedObjectEror:
+ log.warning("Object %s is corrupted, skipping!", key)
+ continue

if meta['needs_reupload']:
del meta['needs_reupload']


Best,
-Nikolaus

[Xomex]

Post by Nikolaus Rath
diff --git a/src/s3ql/adm.py b/src/s3ql/adm.py
--- a/src/s3ql/adm.py
+++ b/src/s3ql/adm.py
@@ -9,7 +9,7 @@
from .logging import logging, QuietError, setup_logging
from . import CURRENT_FS_REV, REV_VER_MAP
from .backends.comprenc import ComprencBackend
-from .backends.common import NoSuchObject
+from .backends.common import NoSuchObject, CorruptedObjectEror
from .database import Connection
from .common import (get_seq_no, is_mounted, get_backend, load_params,
save_params, handle_on_return, get_backend_factory,
@@ -394,6 +394,9 @@
meta = backend.lookup(key)
continue
+ log.warning("Object %s is corrupted, skipping!", key)
+ continue
del meta['needs_reupload']

Fantastic service Nikolaus. The patch works great thankyou! (Once I
corrected the bad spelling of "CorruptedObjectEror" to
"CorruptedObjectError").
Following the filesystem upgrade I'm Just cleaning up the mess now with
fsck.s3ql and s3ql_verify. Looks like there wa sonly one corrupted object.
Regards, X.