Ticket #7822 (closed PLIP: fixed)
Make standard file content types use ZODB BLOB support
Reported by: | limi | Owned by: | witsch |
---|---|---|---|
Priority: | major | Milestone: | 4.0 |
Component: | General | Version: | 4.0 |
Keywords: | focusarea | Cc: | nateaune, klm, interra, grahamperrin, forsberg, lucie, garbas, csenger, esteele, plip-advisories@… |
Description (last modified by witsch) (diff)
Copied in part from PLIP #154 in the roadmap:
Large file handling
It is possible to configure Zope to work with very large files, but the out-of-the-box story is not terribly great. It should be obvious how to configure Plone so that it can handle large volumes of MS Office, PDF or media files, for example.
- Proposed by
- Martin Aspeli
- Seconded by
- Martijn Pieters
- Proposal type
- Architecture
- Repository branch
- plone.app.blob, plone.app.imaging
Motivation
In many ways, Plone is well-suited to document management and the management of files in general. Tools such as ExternalEditor and Enfold Desktop make this even more true. However, due to the way the ZODB works, large files can be problematic to work with — if you're not careful, your ZODB could balloon because each change to a file revisions the whole object.
There are solutions to this problem, which usually involve storing some content outside the ZODB. However, the out-of-the-box story in Plone isn't good enough. It needs to be clear how to set up a site to support large files, and as far as possible this should work transparently whether enabled or disabled.
Proposal
ZODB has integrated BLOB ("binary large object") support starting from version 3.8. BLOBs present a way to transparently store binary content outside of the usual file storage (i.e. the Data.fs), but still as part of the ZODB itself. BLOB objects behave like regular files, but are entirely managed by the ZODB, meaning that the developer doesn't need to care about their file names, transaction-awareness, historic revisions etc. Furthermore, accessing such objects is far more efficient as they don't have to be loaded into memory as part of another object anymore. This significantly lowers memory requirements and thereby frees up RAM as well as the ZODB cache for other data.
This PLIP therefore proposes to integrate and use ZODB BLOB support in Plone. The existing "File" and "Image" content types should be replaced by compatible, blow-aware versions. Any new such content created will use ZODB BLOBs to store the actual payload data. Existing object can be either left untouched, or else converted to the new types via provided in-place migrations.
The key points of such an integration are:
- Transparency
- The provided optimisations should be as transparent as possible and work seamlessly with existing products and tools.
- Compatibility
- The new replacement types should aim for full backward-compatibility with ATContentTypes' "File" and "Image" types.
- Performance
- Dealing with large files is typically a performance problem. Loading a 1Gb video file into memory every now and then is not acceptable!
- Ease of set-up
- It should be easy and obvious how large file optimisations are enabled, and what implications any configuration changes have.
Implementation
BLOBs are merely an additional feature of the ZODB and not used in any way out-of-the-box. The integrational package plone.app.blob uses them to provide drop-in replacements for ATContentTypes' "File" and "Image" content types. plone.app.imaging is a supplemental package paving the way for using BLOBs to store image scales as well (apart from making the scale sizes configurable through-the-web).
plone.app.blob provides a blob-aware rewritten version of Archetypes' ObjectField class as well a base content type using that field. So called "sub-typing" is then used to mimick the behaviour of the existing "File" and "Image" types. The application of marker interfaces as well as archetypes.schemaextender are used for this. At first this seems to complicate matters, but having only one implementation and content type for all binary data such as PDFs, audio or video content will make it much easier to provide add-on packages like the Plone4Artists suite or the various image gallery enhancements.
GenericSetup profiles can be used to enable these types to be the default when creating new "File" or "Image" content. The profiles also move the old types, so that they are still available and existing content keeps working.
Separate in-place migration facilities are provided to convert existing object to make use of the more efficient replacement types. The Products.contentmigration package is used here, but not needed when no migration is required or after it has been completed.
Both packages have extensive tests and provide ways to ensure backward compatibility by running tests from other packages such as Products.CMFPlone and Products.ATContentTypes with the blob-aware replacement types enabled.
Deliverables
- An add-on package for Plone versions 3.0 and later, providing blob support for existing sites
- Integration into Plone 4.0 so that the drop-in replacement types are used by default (essentially this will only hook up the aforementioned package and make the necessary test adjustments)
- Migration facilities for existing "File" and "Image" content
- Hooks for 3rd-party products to use the blob-aware field and base type in order to add additional sub-types and migrations
- Documentation for users, site integrators as well as developers including setup, migration, backup strategies as well as best practices of how to use the integration layer with custom types
Risks
- Compatibility issues may arise from using the (mostly rewritten) replacement types and there are already a number of known issues. However, those will be addressed before a final version of plone.app.blob and integration into Plone 4.0. Also, the package has already proven to work in several production sites, some of which using BLOB support for more than a year.
- The standard ZODB setup would not solely use a file-storage anymore, i.e. the common Data.fs, but also include a blob-storage, which consists of a directory hierarchy with files for each BLOB and its revisions. This might be confusing for users when it comes to backing up their Zope-related data. Recommended backup strategies should be documented to resolve this issue.
- Some existing content might not be "migratable" due to unforeseen issues. This shouldn't be a problem, however, as any existing content will remain functional, migrated or not.
- Using archetypes.schemaextender and marker interfaces introduces a new level of indirection when implementing content types. It also adds performance issues due to the additional and generally more expensive schema lookups. This can be solved by also shipping with archetypes.schematuning, which has been proposed as PLIP 9376. plone.app.blob doesn't change the schema after creation, so schema invalidation is not required here.
- ZODB's BLOB support might still be too fresh and potentially contain bugs. On the other hand, the ZODB 3.8 series has seen two final releases and blob support has been successfully used in real-life projects as well as other Plone/Zope add-ons.
Progress
- The integration package, plone.app.blob is in a usable state and being successfully used in several production sites.
- There are a number of remaining test failures regarding "Image" content. All backward-compatibility tests for "File" content currently pass.
- Also, there are a number of pending issues and sensible enhancements (see the issue tracker at http://plone.org/products/plone.app.blob/issues/)
- Several beta releases have been made, but a final release will need to address the aforementioned issues.
- In-place migration of existing content has been heavily tested, both via integration tests and in "real-life" test runs.
- Migrating a relatively large site with a 16 GB Data.fs including about 7,000 "File" content items showed significantly lower memory requirements and improved performance. The Data.fs went down to 2.5 GB and memory usage dropped from 8-9 GB to 3 GB with the same ZODB cache size. The migration itself took little more than one hour.
- Compatibility tests with Plone 3.x and 4.0 are provided via a buildbot ( http://blobot.zitc.de/). The latter currently still use ZODB 3.8.3 as newer versions, i.e. 3.9.x, break the necessary setup.
Participants
- Andreas Zeidler (IRC nickname: <witsch>)
Change History
comment:3 follow-up: ↓ 6 Changed 8 years ago by garbas
i'd like to help you here to hopefully speed up the proccess. as you probably know sunew better, do you maybe know when his patch will be availiable online?
comment:4 follow-up: ↓ 7 Changed 8 years ago by nateaune
- Cc nateaune added
I'm very excited about this one because the previous incarnation of large file solutions (ExternalStorage, PloneExFile, FileSystemStorage, BlobFile) have all tried to solve the problem at the Archetypes layer, whereas the problem should really be solved at the Zope layer.
For the first time, we have native blob support in the ZODB, and Andi's plone.app.blob products brings this blob support to Plone. http://plone.org/products/plone.app.blob
Andi - you should probably update the PLIP #154 to reflect the work you have done and intend to do to implement large file support in Plone. http://plone.org/products/plone/roadmap/154
comment:6 in reply to: ↑ 3 Changed 8 years ago by witsch
Replying to garbas:
i'd like to help you here to hopefully speed up the proccess. as you probably know sunew better, do you maybe know when his patch will be availiable online?
unfortunately no. he's dropped me a mail some time ago telling me he was about to commit it the other day and/or give me svn access to his repository. i haven't heard anything since, but didn't have time to follow-up either... guess he's probably busy with other things, too :)
but don't worry, it's very likely i'll have enough time to kick-start things starting next week...
comment:7 in reply to: ↑ 4 Changed 8 years ago by witsch
Replying to nateaune:
Andi - you should probably update the PLIP #154 to reflect the work you have done and intend to do to implement large file support in Plone. http://plone.org/products/plone/roadmap/154
yes, that's a good idea (and has already been lurking on my 2do list for a while ;))...
comment:8 Changed 8 years ago by witsch
- Description modified (diff)
"crossposting" my mail to the developers list as this ticket's description...
comment:10 follow-up: ↓ 11 Changed 8 years ago by interra
- Description modified (diff)
restoring accidental description vandalism on my part, sorry for extra e-mail notification...
comment:11 in reply to: ↑ 10 Changed 8 years ago by witsch
Replying to interra:
restoring accidental description vandalism on my part, sorry for extra e-mail notification...
:)
comment:17 Changed 7 years ago by witsch
quoted from #6805:
fyi, while the new beta fixes a number of issues, it's also a bit of a "snapshot" release i wanted to get out because of two reasons: firstly issue 7 was starting to cause serious trouble in a couple of installations — the last release was clearly showing how outdated it had become.
in addition, i won't be available for about two weeks. i'll continue to work on the package afterwards (as it'll be added to a production deployment soon), but didn't want to hold back the recent progress before that break. there'll also be a hopefully comprehensive blog post soon, but for now there's something far more important to do... :)
comment:18 follow-up: ↓ 19 Changed 7 years ago by limi
- Type changed from Feature Request to PLIP
- Milestone changed from Ongoing to 4.0
I think this is a great summary ticket that can be used as a PLIP for 4.0, as long as someone is willing to shepherd it.
comment:19 in reply to: ↑ 18 Changed 7 years ago by witsch
Replying to limi:
I think this is a great summary ticket that can be used as a PLIP for 4.0, as long as someone is willing to shepherd it.
i (still) am, even though i didn't manage to update the ticket and the status regarding the blob integration for much too long. development has been picked up recently again, though, and i think the package is generally in pretty good shape to be ready for plone 4.0 in time...
ps: thanks for another "herding effort", limi ;)
comment:22 Changed 7 years ago by erikrose
- Owner witsch deleted
Clearing Owner field of 4.0 PLIPs so we can use it to mean "implementor". (Many of these owners were automatically assigned from choosing a Component that had a default owner.)
comment:24 Changed 7 years ago by davisagli
- Status changed from assigned to new
- Owner set to witsch
It's unclear to me whether this PLIP proposes to replace ATFile and ATImage with the blob-based equivalent in Plone 4, or keep the existing ATFile and ATImage OOTB but ship with plone.app.blob so that it's easier to switch to using blobs. I think we probably want the former, though of course that means we need to take more care in making sure the migration goes smoothly.
It is very important that producing good documentation for how to work with blobs be included as part of the integration of this PLIP, both for sysadmins and product developers.
Two risks I noted while looking through the existing plone.app.blob package:
- The migration from non-blob ATFiles and ATImages depends on Products.contentmigration, so we would be introducing a dependency on that as well.
- One of the tests notes that versioning is not yet supported for blob-based types.
This PLIP doesn't include a discussion of risks. Are there other things that we should be aware of?
I'm re-assigning this to witsch as he indicated in http://dev.plone.org/plone/ticket/7822#comment:19 that he's willing to shepherd this.
My FWT vote: +1.
comment:25 Changed 7 years ago by rossp
My FWT vote is +1 as long as a discussion of risks is added. I think it's very important to have a exhaustive discussion of the risks. The BLOB story is so important that I expect any risks discussed will be acceptable I just want to make sure that they're anticipated and documented.
One of the risks might be confusion in the community as to when to use Products.Archetype.Field.FileField/ImageField and when to use plone.app.blob.field.BlobField.
comment:26 Changed 7 years ago by MatthewWilkes
FWT Vote: +1
With the caveat that this is likely a risky thing for migrations, and will need a LOT of testing in the implementation to give me enough confidence.
comment:27 Changed 7 years ago by witsch
- Status changed from new to assigned
just a quick note for now: the "risks" section is indeed missing here, which is because this was originally a PSPS "focus area" ticket. i've never properly updated it, though, so limi took the short-cut and simply turned it into a PLIP (instead of writing a new one :)). i shall update it soon to reflect the proper PLIP structure, though...
comment:28 Changed 7 years ago by raphael
FWT vote: +1
for trying it that is. Whether the solution is going to be acceptable remains to be seen in light of the risks mentioned above.
Go Andi, go ;-)
comment:29 follow-up: ↓ 31 Changed 7 years ago by calvinhp
FWT Vote: +1 my only worry is the exclusion of the risks in the PLIP. Specifically, the p4a.subtyper package doesn't appear to have a clear maintainer at this point.
comment:30 follow-up: ↓ 32 Changed 7 years ago by klm
i consider some kind of ZODB large file accommodations to be necessary, to reduce awkward logistics and efficiencies in handling blob-bloated Data.fs, as well as to mitigate inefficiencies in ZServer's in uploading and downloading large files. for that reason, i've experimented with plone.app.blob. while i think that ZODB BLOB support is quite promising, i encountered some shortcomings along the way about which i have reservations. i think they're important, to be weighed as risks if in fact they are still present. (it's been a while since i confirmed they exist, and had a build to test them.)
- copy/paste - blob-based objects were not susceptible to zope copy/paste operations. see
- zope3 bug 240381
- [ http://mail.zope.org/pipermail/zodb-dev/2008-October/012247.html
- Blobs, Copies, and Exports] - note the "I'd like to avoid copies though." from christian thieune, the main author of zope's blob support. there may be a good reason for avoiding copies, but that has significant operational implications for blob-based objects.
- Blobs, Copies, and Exports (Zope 3) - the initial message in the thread
- i'm not clear whether export/import works for blob objects, though i think it does. a message in the thread mentioned above refers to the pitfalls of a particular technique that jeffrey shell was using to export.
- i'm not clear whether export/import and copy of a Plone Site containing blobs works, and suspect it doesn't (but it may not work/be supported without blobs, as well).
- large file acceleration is somewhat important - having ZServer in the way adds high overhead. ZODB BLOBs thus far do not offer any savings in this regard. ZODB BLOB support does offer elimination of unnecessary overhead in communicating blobs that have been obtained by a ZEO client to the ZEO storage server - Blobs Zero Copy support. while valuable, configuration for it also adds some complexity to the buildout.
i'm trying to be clear here that i am unsure about the status of these issues, and that my purpose in mentioning them is to assess whether or not the issues still apply, and then decide for those that do what the respective costs versus benefits are.
comment:31 in reply to: ↑ 29 Changed 7 years ago by witsch
Replying to calvinhp:
FWT Vote: +1 my only worry is the exclusion of the risks in the PLIP.
again, the risks section is (still) missing, since limi turned this PSPS ticket into a PLIP... ;) see my comment above... i'm gonna add one once i've managed to get a few other things out of the way and things are back to (halfway) normal... :)
Specifically, the p4a.subtyper package doesn't appear to have a clear maintainer at this point.
not to worry — it's not used by plone.app.blob.
comment:32 in reply to: ↑ 30 ; follow-up: ↓ 33 Changed 7 years ago by witsch
Replying to klm:
i'm trying to be clear here that i am unsure about the status of these issues, and that my purpose in mentioning them is to assess whether or not the issues still apply, and then decide for those that do what the respective costs versus benefits are.
thanks for that list, ken. i very much appreciate that you keep collecting all of those issues and threads so persistently — that's extremely helpful. unfortunately i cannot answer right now, but i will do so soon! please bear with me for a little bit longer...
comment:33 in reply to: ↑ 32 ; follow-up: ↓ 35 Changed 7 years ago by klm
Replying to witsch:
Replying to klm:
i'm trying to be clear here that i am unsure about the status of these issues, and that my purpose in mentioning them is to assess whether or not the issues still apply, and then decide for those that do what the respective costs versus benefits are.
thanks for that list, ken. i very much appreciate that you keep collecting all of those issues and threads so persistently — that's extremely helpful. unfortunately i cannot answer right now, but i will do so soon! please bear with me for a little bit longer...
no worries, andi! i don't know the development schedules, but i'm not in a rush. i appreciate all you've done, and don't mean to increase the pressure or load - just trying to factor in what might be pitfalls to address when the time comes to decide about adoption. hopefully, by having the issues expressed deliberately, the burden of scoping them out won't fall only on you, and people won't complicate the decisions by jumping to conclusions.
comment:34 follow-ups: ↓ 36 ↓ 37 Changed 7 years ago by erikrose
I'm curious how the "subtyping" is implemented; if this PLIP proposes adding a whole subtyping framework of some kind to Plone 4, then that's worth a consideration of the capability/complexity tradeoffs. I'd also like to see the risks explicitly listed before we vote on the final implementation.
I think work should continue on this, hopefully for Plone 4, so my FWT vote is +1.
comment:35 in reply to: ↑ 33 Changed 7 years ago by witsch
Replying to klm:
[...] just trying to factor in what might be pitfalls to address when the time comes to decide about adoption.
i know you are, and like i said, much appreciate it... :)
comment:36 in reply to: ↑ 34 Changed 7 years ago by witsch
Replying to erikrose:
I'm curious how the "subtyping" is implemented;
it's using the same idea: dynamically adding and removing marker interfaces (using alsoProvides and noLongerProvides). however, most of p4a.subtyper is about adding a UI for letting the user pick a particular subtype. this wasn't needed here, and the helpers to add & remove interfaces were only a few lines anyway...
if this PLIP proposes adding a whole subtyping framework of some kind to Plone 4, then that's worth a consideration of the capability/complexity tradeoffs.
i think calling it a framework might be a bit too much. like i said, all it does is to add some interfaces, and that's currently only done when creating new instances, too. however, these interfaces are then used to modify the content items' schematas using archetypes.schemaextender, thereby introducing that dependency.
this is because the idea of the whole blob approach was to ultimately only have one "binary" content type, which can then expose different "behaviours" depending on the actual content. iow, pdfs, pictures, cd rom images, videos, mp3s, installers etc would not need special content types anymore (making migration and other things painful), but merely have a different representation (think flash video players etc). and modifying the schema is actually part of this, since for example the primary fields of "File" and "Image" content items need different widgets, validators and other settings...
comment:37 in reply to: ↑ 34 Changed 7 years ago by witsch
Replying to erikrose:
I think work should continue on this, hopefully for Plone 4, so my FWT vote is +1.
hmm, perhaps i should mention again here, that plone.app.blob tries hard to be backwards-compatible with the existing ATFile and ATImage types. i've been running all tests from CMFPlone, ATContentTypes as well as Archetypes with the new blob-based types dropped in. they've been passing for "File" content for quite some time now while support for "Image" content still has a few issues.
of course, passing tests alone don't say too much as our test base doesn't cover everything, esp not ttw editing and such. otoh, i think it's a good indicator that the package can already be used in production (i.e. the "File" type), and a number of people have also successfully done so...
comment:38 Changed 7 years ago by esteele
Approved by FWT vote.
comment:39 Changed 7 years ago by witsch
(In [28898]) add review buildout for blob support (refs #7822)
comment:40 Changed 7 years ago by witsch
- Description modified (diff)
update ticket to have a more PLIP-like structure & reflect the current status
comment:42 follow-up: ↓ 61 Changed 7 years ago by alecm
comment:43 in reply to: ↑ description Changed 7 years ago by witsch
- Using archetypes.schemaextender and marker interfaces introduces a new level of indirection when implementing content types. It also adds performance issues due to the additional and generally more expensive schema lookups.
said performance issues should be mostly fixed after r11344@archetypes. the slowdown for serving small images as reported in alec's review is now almost gone: the request rate for a 600+ byte file when up from about 80 to a little more than 100 again — pending further investigation the rest is likely due to the extra overhead of fetching two zodb objects and opening/reading the blob file...
This can be solved by also shipping with archetypes.schematuning, which has been proposed as PLIP 9376.
adding archetypes.schematuning to the mix now actually make the above benchmark run ever so slightly slower — 100 vs 101 req/s...
comment:44 follow-up: ↓ 62 Changed 7 years ago by optilude
comment:45 Changed 7 years ago by witsch
(In [29755]) use archetypes.schemaextender trunk in order to verify if the recent performance fix brings blob support back on par with vanilla plone 4 (refs #7822)
comment:46 Changed 7 years ago by witsch
comment:47 follow-up: ↓ 53 Changed 7 years ago by esteele
Your PLIP has been reviewed by the Framework team. Feel free to discuss any suggested changes either here in the PLIP ticket or on the mailing lists. Final deadline for this PLIP is set for September 30.
comment:48 Changed 7 years ago by witsch
(In [29857]) actually, r29755 should have gone here:
use archetypes.schemaextender trunk in order to verify if the recent performance fix brings blob support back on par with vanilla plone 4 (refs #7822)
comment:49 Changed 7 years ago by witsch
comment:50 Changed 7 years ago by witsch
comment:51 Changed 7 years ago by witsch
comment:52 Changed 7 years ago by witsch
(In [30019]) also apply the updated monkey patch for cgi.FieldStorage (in ZPublisher.HTTPRequest) to Zope 2.12 as it still only has the first version (refs #7822)
comment:53 in reply to: ↑ 47 ; follow-up: ↓ 54 Changed 7 years ago by witsch
Replying to esteele:
[...] Feel free to discuss any suggested changes either here in the PLIP ticket or on the mailing lists. Final deadline for this PLIP is set for September 30.
as of r30019 all tests pass on Plone 4.0 — i'll comment the review notes & polish things some more tomorrow (as i'm practically falling asleep now :))
comment:54 in reply to: ↑ 53 Changed 7 years ago by witsch
Replying to witsch:
as of r30019 all tests pass on Plone 4.0 — [...]
ah, i should mention that i had to downgrade to zodb 3.8.3 for now as 3.9 doesn't seem to allow wrapping a DemoStorage inside a BlobStorage anymore (as the test setup does). however, things do also work with 3.9 when testing ttw...
comment:55 Changed 7 years ago by witsch
(In [30024]) add tests against plone 4.0 to the buildbot (refs #7822)
comment:56 Changed 6 years ago by witsch
(In [30048]) update history wrt the test fixes for plone 4 (refs #7822)
comment:57 Changed 6 years ago by witsch
- Description modified (diff)
updated the status wrt tests on plone 4
comment:58 Changed 6 years ago by witsch
(In [30057]) hook up plone.app.blob & apply the patch needed to run blob-enabled tests (refs #7822)
comment:59 Changed 6 years ago by witsch
comment:60 Changed 6 years ago by witsch
(In [30063]) add bbb tests against plone 4.0 to the buildbot (refs #7822)
comment:61 in reply to: ↑ 42 Changed 6 years ago by witsch
Replying to alecm:
quoting parts of alec's review to be able to provide comments inline:
Users of the product seem to speak favorably of it.
very nicely put! :)
1) The implementer has failed to include a document describing the
PLIP along with the buildout.
my bad. i caught up just now, see r30062.
2) plone.app.blob tests do not pass under Plone 4.0 + Zope 2.12,
though they appear to be well designed and offer reasonable coverage.
they pass now, albeit i had to downgrade to ZODB 3.8.3 for the time being (please see r29981).
3) The Plone default profile has not been updated to install the BLOB
content types by default.
fixed in r30057.
4) The BLOB Image type install profile does not appear in the Quick
Installer.
quick-installer only supports exposes one profile per package. as image support is not quite ready yet for general consumption, and since i'm trying to provide a stable add-in for Plone 3.x (i.e. without branching things for Plone 4) the "image replacement" profile isn't applied by default yet. once image support is ready, the "default" profile will start depending on it.
for now you'll either have to apply the profile manually via portal_setup for Plone 3.x — for testing/reviewing in Plone 4.0 it should get applied by default (again see r30057)
5) Two BLOB File replacement install profiles appear in portal_setup
one of those is actually registered for ITestCasePloneSiteRoot, but still exposed by quick-installer. i'm not sure if this is intended behaviour, but the distinction should be clearer as of r29885.
6) The tests (which again don't pass) depend on
collective.monkeypatcher, but the buildout does not.
fixed in r29685, thanks martin!
7) The product relies on monkeypatching (ZODB.Blob and
ZPublisher.HTTPRequest). These patches should be integrated into the Zope core if we are going to include this product.
the patch for setstate_wrapper can go (as the comments state). i've already made the necessary fix in ZODB, which was subsequently released in 3.8.1.
the patches for HTTPRequest actually originated in Zope 2.11. however, the original fix doesn't work on windows, and the working version from plone.app.blob hasn't been pushed upstream yet. as this is a bug fix, though, i don't see what would prevent it from being accepted for 2.12.1 (also see r19124 for more info).
8) The product depends on plone.app.imaging which monkeypatches
Archetypes.Field.ImageField, this patch should be integrated if possible.
yes, it should. i've added it to the 2dos.
Issues and Concerns
1) Performance: When serving a small image (600 B) I can get ~108
req/sec with ATFile, but only ~80 req/sec with a BLOB File.
this was mostly due to a subtle performance issue in archetypes.schemaextender, which has been fixed in r11344 (in the archetypes repo). with that fix such small files are served at about 100 req/sec, leaving a performance penalty of less than 10%.
imho, it's rather unlikely that many small files (i.e. <1k) will be stored in actual "File" or "Image" content, but some generated image scales might be this small, of course. possible solutions to this are to either store image scales as attributes like before, which can be achieved by simply disabling the extra adapter provided by plone.app.blob, turning the same adapter into a local adapter which can then be enabled separately, or else by only storing the payload data on blobs from a certain size, e.g. 16k.
the last option would probably be best in all cases, but i'm not sure the extra effort (and added complexity) is really worth it. therefore i will conduct some more benchmarks and provide the results for the FWT to decide.
2) If versioning is enabled for Files, reverting to another version
removes the file content.
the versioning issues are indeed the ones that worry me most — i've looked into resolving them before, but gave up at some point. i hope that there will be a chance to look into this with alec and/or other people in budapest. until then versioning will have to remain disabled for blob-based content, which luckily also is the default in Plone.
3) The package depends on archetypes.schemaextender, which is a
worthwhile add-on package in wide use. However, the package introduces a significant potential decrease in performance (see PLIP #9376 and reviews).
please see above. the fix in r11344 (in the archetypes repo) resolves the performance issues in archetypes.schemaextender almost completely. with the fix only the actually added fields will have a performance impact, which is — according to my profiling and benchmarks — mostly neglectable, though. please note that this should also render PLIP #9376 mostly obsolete.
4) Image scales appear to be generated on the fly and not stored
anywhere.
this was fixed in r29953. i will provide updated benchmark results.
With some work to properly integrate this package into Plone 4.0 and some further discussion of the potential problems listed above, I would strongly consider recommending this PLIP for inclusion in Plone.
all remaining issues i consider to be critical before releasing 1.0 (to be included into Plone 4.0) are now listed in the review notes. there are still a number of bug fixes required, especially the one regarding versioning, but most of the current test failures in Plone and ATContentTypes are due to the new approach used for image scales (i.e. the traversal adapter vs attributes).
comment:62 in reply to: ↑ 44 Changed 6 years ago by witsch
Replying to optilude:
again, quoting parts of martin's review to be able to provide comments inline:
- We should make BLOB files and images the default for non-migrated sites.
as already mentioned, this is the case as of r30057. new content will use blobs, old content will remain untouched and functional, and can be updated using the provided migration views.
- The "plone.app.blob: Add Blob" permission needs to be granted to Owner, Manager, Contributor by default. It's not entirely clear how this relates to the "ATContentTypes: Add Image" and "ATContentTypes: Add File" permissions. I think ideally, we shouldn't have special permissions for blobs, but if we do, we need to document how these three permissions relate.
this was in fact a bug, that has been fixed in the meantime (see r29875). the new types should use the original permissions, i.e. "ATContentTypes: Add File" etc, in order to be fully compatible with existing sites. unfortunately, the bug was nicely hidden away due to the way test layers are stacked up.
- The "w/o blobs" types appear in the "types" control panel. They shouldn't by default.
i'm not sure if the FTIs need to be present for existing, unmigrated content. if not, i can make sure they're not exposed anymore. hmm, unless i've misread this so far and it is the plone control panel you're referring to here? in this case, yes, they should not be visible.
- I installed plone.app.blob, and turned on automatic versioning. I then uploaded a file. It suceeded, but I got this in the logs:
Exception OSError: (2, 'No such file or directory', '/var/folders/CS/CSVVQU2TGaiQYSKA+4-NNE+++TI/-Tmp-/tmpsUGy_l') in <bound method _TemporaryFileWrapper.del of <closed file '<fdopen>', mode 'w+b' at 0x32f1de0>> ignored
the fixed version of the monkey patch wasn't properly applied in Zope 2.12. this was fixed in r30019 — the message shouldn't appear anymore.
- I uploaded an image, then used the transform tab to rotate it 90 degrees. This caused an error:
this is a known issue and will have to be addressed (see ticket #8506). the fix should be rather straight-forward, though — i think i have a patch someone sent me for this.
- For migrated sites, we should make it opt-in to migrate to BLOB types.
this is already the case for existing content. or do you mean using blobs for new content should also be opt-in (for existing sites)?
comment:63 Changed 6 years ago by witsch
i will soon continue to address the remaining issues, but the plip should be ready for a second round of reviews.
comment:64 Changed 6 years ago by alecm
comment:65 Changed 6 years ago by rossp
FWT vote, +1 for merge
comment:66 Changed 6 years ago by esteele
This PLIP has been accepted for merging into Plone 4.0
The final vote was: Alec Mitchell +1 David Glick +1 Erik Rose +1 Laurence Rowe +1 Matthew Wilkes - Ross Patterson +1
Please merge your branches into the Plone 4.0 head by end-of-day Friday Oct 16. If you need assistance with merging, please contact me.
We'll be assigning a documentation ticket to this PLIP shortly. Please assist the docs team in documenting the changes and new features that this PLIP introduces.
comment:67 Changed 6 years ago by esteele
Please assist the doc team in creating/updating documentation relating to this PLIP. See #9598.
comment:69 Changed 6 years ago by witsch
comment:70 Changed 6 years ago by witsch
(In [30541]) merge branch for plip7822 into 4.0 (refs #7822)
comment:71 Changed 6 years ago by witsch
(In [30544]) merge buildout changes for plip7822 (refs #7822)
comment:72 Changed 6 years ago by witsch
comment:73 Changed 6 years ago by witsch
- Status changed from assigned to closed
- Resolution set to fixed
the PLIP has been merged, and even though there are still things left to wrap up & polish this ticket is done as far as the PLIP is concerned...
comment:74 Changed 6 years ago by witsch
(In [30549]) triggered by the insight that the blob test setup doesn't work with plone 4.0 and zodb 3.9, because it is messing with things too much when zodb 3.9's DemoStorage actually already is a BlobStorage as well, i thought i should try if the custom_zodb.py is still needed at all... (refs #7822)
comment:75 Changed 6 years ago by witsch
comment:76 Changed 6 years ago by witsch
comment:77 Changed 6 years ago by witsch
comment:78 Changed 6 years ago by witsch
(In [30597]) fix the missing dependency after not applying the blob gs profile anymore (refs #7822)
comment:79 Changed 6 years ago by witsch
(In [30615]) upgrade to ZODB 3.9.2 — the fix in 3.9.1 sounds like something we want (refs #7822)
comment:80 Changed 6 years ago by witsch
comment:81 Changed 6 years ago by witsch
comment:86 Changed 6 years ago by dukebody
comment:87 Changed 6 years ago by dukebody
Sorry for the message above. Wrong bug number. :S
comment:88 Changed 6 years ago by grahamperrin
#9553 is Large file handling.
comment:90 Changed 6 years ago by witsch
comment:92 Changed 4 years ago by grahamperrin
- Status changed from closed to confirmed
- Version set to 4.1
For completeness:
Plone 4 highlights: Massively improved handling of large files & media (2010-08)
Postscript … huh? I didn't consciously do anything to change the status or version. And in editing this comment, I see no way to revert those changes to the ticket.
happily accepting... :)