Ticket #11999 (confirmed Bug)

Opened 5 years ago

Last modified 3 years ago

Webdav upload might lead to database inconsistencies

Reported by: mjm Owned by:
Priority: major Milestone: 4.x
Component: General Version: 4.1
Keywords: webdav Cc:

Description (last modified by kleist) (diff)

I have used webdav to upload a bunch of files to a Plone 4.1rc3 site. During one of the uploads some timeouts occured. Sometime afterwards I cleared and rebuilt the catalog but got the error below. Which is not very helpful in finding out what the offending file is.

------
2011-07-10T17:38:33 ERROR ZODB.Connection Couldn't load state for 0x0f8328
Traceback (most recent call last):
  File "/web/python/ZopeB/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZODB/Connection.py", line 860, in setstate
    self._setstate(obj)
  File "/web/python/ZopeB/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZODB/Connection.py", line 922, in _setstate
    obj._p_blob_committed = self._storage.loadBlob(obj._p_oid, serial)
  File "/web/python/ZopeB/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZODB/blob.py", line 644, in loadBlob
    raise POSKeyError("No blob file", oid, serial)
POSKeyError: 'No blob file'
------
2011-07-10T17:38:33 ERROR Zope.SiteErrorLog 1310312313.220.348911732611 http://vm20.anywise.net:8010/vraagbaak/portal_catalog/manage_catalogRebuild
Traceback (innermost last):
  Module ZPublisher.Publish, line 126, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 46, in call_object
  Module Products.CMFPlone.CatalogTool, line 462, in manage_catalogRebuild
  Module plone.app.discussion.patches, line 47, in patchedClearFindAndRebuild
  Module OFS.FindSupport, line 239, in ZopeFindAndApply
  Module OFS.FindSupport, line 239, in ZopeFindAndApply
  Module OFS.FindSupport, line 239, in ZopeFindAndApply
  Module OFS.FindSupport, line 239, in ZopeFindAndApply
  Module OFS.FindSupport, line 239, in ZopeFindAndApply
  Module OFS.FindSupport, line 239, in ZopeFindAndApply
  Module OFS.FindSupport, line 227, in ZopeFindAndApply
  Module plone.app.discussion.patches, line 25, in indexObject
  Module Products.Archetypes.CatalogMultiplex, line 38, in indexObject
  Module Products.CMFPlone.CatalogTool, line 390, in catalog_object
  Module Products.ZCatalog.ZCatalog, line 476, in catalog_object
  Module Products.ZCatalog.Catalog, line 321, in catalogObject
  Module Products.ZCatalog.Catalog, line 271, in updateMetadata
  Module Products.ZCatalog.Catalog, line 395, in recordify
  Module plone.indexer.wrapper, line 59, in __getattr__
  Module plone.indexer.delegate, line 16, in __call__
  Module Products.CMFPlone.CatalogTool, line 221, in getObjSize
  Module Products.ATContentTypes.content.base, line 198, in get_size
  Module plone.app.blob.field, line 273, in get_size
  Module plone.app.blob.field, line 85, in get_size
  Module plone.app.blob.utils, line 52, in openBlob
  Module ZODB.Connection, line 860, in setstate
  Module ZODB.Connection, line 922, in _setstate
  Module ZODB.blob, line 644, in loadBlob
POSKeyError: 'No blob file'

Change History

comment:1 Changed 5 years ago by hannosch

  • Summary changed from Clear and rebuild catalog fails if blob file not found to Webdav upload might lead to database inconsistencies
  • Milestone changed from 4.1 to 4.x

Clarified the title of the ticket. The problem you had here is apparently a problem during the webdav upload, which in turn lead to an inconsistent database.

No matter what part of your database is inconsistent (POSKeyError) most code will fail. Trying to work around an inconsistent database in any application level code is a bad strategy. What you (we) need to fix is the root cause. And yes, blobs are no different from other objects stored in the database and not less important in any way - they just happen to be stored in a different storage but that doesn't make them optional for anything, like a catalog rebuild.

comment:2 Changed 5 years ago by mjm

OK, but then another ticket is in order: make it feasible for an administrator to recover from this problem. Currently the site is broken, because I cannot recatalog. AFAICT there is no way to find out the offending content item. The error message does not contain this information.

comment:3 Changed 5 years ago by kleist

  • Keywords webdav added
  • Component changed from Unknown to Infrastructure
  • Description modified (diff)

Just because I'm curious: Which WebDAV client did you use?

[Wrapped traceback in triple curly brackets.]

comment:4 Changed 5 years ago by mjm

WebDav client: Cyberduck on OS X.

comment:5 Changed 4 years ago by kleist

  • Status changed from new to confirmed
  • Version set to 4.1
  • Component changed from Infrastructure to General

comment:6 Changed 3 years ago by dturvene

I recently upgraded from 3.3.2 to 4.2.1 and started seeing this error. I suspected it was for AT objects for files and images.

I fixed it using the following procedure:

I put a pdb breakpoint before the exception in ZODB.blob.BlobStorageMixin.loadBlob and ran plone in the foreground. I then ran ZMI rebuild the catalog, which hits the breakpoint. In pdb (I LOVE pdb) I moved up a couple stack frames where the object instance is defined and printed out the info, then deleted it from the ZODB. Then I continue running until the breakpoint again until it completes the reindexing.

This is a good hack when there are only a small number of bad blob objects.

I'm unclear how the blob data (e.g. an image) is suppose to be migrated out of Data.fs and into blobstorage. I agree with mjm that there should be an admin tool to find blobstorage problems. I'm unclear how difficult it will be to fix an issue like this other than delete the entire instance and recreate it in 4.X

Note: See TracTickets for help on using tickets.