Ticket #9316 (closed PLIP: fixed)
Unify folder implementations
Reported by: | smcmahon | Owned by: | witsch |
---|---|---|---|
Priority: | n/a | Milestone: | 4.0 |
Component: | General | Version: | |
Keywords: | Cc: | plip-advisories@…, dukebody |
Description (last modified by witsch) (diff)
Copied from PLIP #191 in the roadmap:
Unify folder implementations
We currently have "Folder" and "Large Plone Folder" implementations.
There should be only one.
- Proposed by
- Martin Aspeli
- Seconded by
- Andreas Zeidler
- Proposal type
- Architecture
- Repository branch
- plone.folder, plone.app.folder
Motivation
Shipping with two folder types is unnecessary for several reasons:
- It forces the user to make an a-priori choice about the number of objects they plan to put into a folder
- We ship with the "Large" type disabled by default to avoid UI confusion
- We don't have a proper search-based UI for large folders anyway
Also the standard Folder type stores attributes, and has a single list _objects tuple which keeps the list of objects and order. This is prone to ConflictErrors and is slower. In simple benchmarks, a BTree-based folder performs orders of magnitude faster than a basic folder.
Proposal
Have a single folder implementation.
- The internal storage is BTrees
- It still supports ordering, by storing a separate sort order list/tree
- It has at least two views - one search-based for "large" folders, one batch-based for "small" folders. This is either just a "display" menu choice, or a choice in the object's schema. Explicit sorting may be turned off for "large" folders.
Implementation
The package plone.folder in the Plone SVN provides a base implementation of a folder base class, which is not Archetypes specific, based on BTreeFolder2, but adding ordering. The exact ordering implementation is left up to an adapter, with a default providing explicit ordering. This allows other implementations, such as auto-sorting based on some key.
The diagram below shows the folderish base- and mix-in classes used in OFS, CMF, Archetypes and Plone. Count 'em:
The new base class from plone.folder, i.e. OrderedBTreeFolderBase, is used by plone.app.folder to provide two folderish classes, one targeted to Archetypes (BaseBTreeFolder) as well as one for ATContentTypes (ATFolder). Both add relatively little extra code and setup in order to make them fully compatible with the original, to be replaced classes.
The package also provides a GenericSetup profile replacing the standard "Folder" content type with the new one. In-place migration will convert the internal data-structures when upgrading to use the new folders. Such migration code (including thorough tests) already exists in a project-specific package, and just needs to be moved into plone.folder itself. The migration runs relatively fast — in several performance tests about 13,000 folders holding some 200,000 items in total were migrated in about 5 minutes.
The Container type in plone.app.content does not use BTrees, nor does it support ordering. Rather than changing this class and providing migration, we could add a new type, e.g. called OrderedContainer or just Folder (to signify closer resemblance to the standard folder behaviour) that mixes in the new class instead of PortalFolder.
In addition, plone.folder also provides an ordering adapter, which only considers certain content to be orderable (implemented via marker interfaces). This allows for ordering of content that requires it, for example navigational items, without any added performance penalties if those share a folder with other, non-orderable content in large quantities.
Deliverables
- New folderish base classes that are BTree-based and support large content sets as well as ordering.
- Improved Products.ATContentTypes.content.ATFolder and Products.ATContentTypes.content.ATBTreeFolder versions which use BTrees and supports ordering.
- New class plone.app.content.container.Folder (?) that uses BTrees and supports ordering.
- The ability to switch to a folder view/behaviour that's optimised for "large" content sets.
- In-place Migration for all existing sites.
Risks
- There may be migration issues involved for derived folderish types with additional data-structures regarding containment and/or ordering.
- This vision makes the existing BaseBTreeFolder in Archetypes orderable (though in some UI configurations you may not see the ordering, to prevent the user invoking slow operations), which may be unexpected.
Progress
Complete:
- Create a base folder implementation - OrderedBTreeFolderBase
- Hook this into BaseBTreeFolder in Archetypes
- Fix CMFPlone so that it detects the order support based on a Zope 3 interface rather than a Zope 2 one
- Test that order support works on the new folder
- Update FTIs and portal_type's so that the default "Folder" portal_type is a btree-based folder
- Enable next/previous navigation for this folder type
- Enable photo-album support for this folder type
- Enable archiving support for this folder type
- Enable ordering policy to be specified by adapter
Releases:
- 1.0b3 of plone.folder has been released in May and is in Dexterity.
- 1.0a1 of plone.app.folder has been released in May in order to factor out all code specific to ATContentTypes
- Both releases have been heavily tested in a large-scale Jarn project and will go into production next week.
To-do - ATContentTypes:
- Merge in-place migration to new folder types
- Write migrations for FTI and portal_type changes
- Ensure compatibility with Plone 4.0, i.e. Python 2.6 & Zope 2.12
To-do - plone.app.content:
- Create new Folder base class, using the new mixin, with tests
Benchmark results
The following two graphs show some simple benchmarks using the various types of folders (regular = ATFolder, large = ATBTreeFolder, ordered = ATBTreeFolder w/ order support) with 50 and 500 items, respectively. The tests conducted where creating content, retrieving a batch of items from the folder, getting all items from the folder as well as random accessing the items. The benchmark tests used can be found in the tests/ folder.
A more realistic scenario was provided by some benchmarks based on JMeter (available in jmeter):
The graphs show the time it took to create a folder with 500 news items both using a standard folder and the new btree-based folder with order support. Also, and that's probably the most interesting figure so far, the time for repeatedly browsing the "folder contents" view is decreased to slightly less than 75% by using the btree folder.
Another interesting thing to note is that the "content creation" tests show that btree-based folders are faster than the currently used implementation (from ATFolder) even for very small numbers of contained objects. The tests still run faster when adding 500, 50, 10, 5, 2 or even just a single object to the folder.
Participants
- Martin Aspeli (IRC nickname: <optilude>)
- Andreas Zeidler (IRC nickname: <witsch>)
- Martijn Pieters (IRC nickname: <MJ>)
- Alec Mitchell (IRC nickname: <alecm>)
Change History
comment:1 Changed 7 years ago by witsch
- Status changed from new to assigned
- Owner set to witsch
- Description modified (diff)
comment:2 follow-up: ↓ 4 Changed 7 years ago by pupq
+1. And if it could sneak in a body field so that default pages can be a far-less-required feature, people will really, really love these new folders. That's the most common thing people ask about folders -- "can I add some text above the listing of the content items?"
comment:4 in reply to: ↑ 2 Changed 7 years ago by witsch
Replying to pupq:
+1. And if it could sneak in a body field so that default pages can be a far-less-required feature, people will really, really love these new folders. That's the most common thing people ask about folders -- "can I add some text above the listing of the content items?"
while i can certainly see your point, i don't think this can be part of this PLIP. it'll require discussion and an official proposal — sneaking things in is not what we want to do...
comment:5 follow-up: ↓ 7 Changed 7 years ago by erikrose
- Owner witsch deleted
Clearing Owner field of 4.0 PLIPs so we can use it to mean "implementor". (Many of these owners were automatically assigned from choosing a Component that had a default owner.)
comment:6 follow-up: ↓ 8 Changed 7 years ago by alecm
- Owner set to witsch
- Status changed from assigned to new
+1
This is important stuff, plus it makes Plone faster. Not sure how I became a participant though.
comment:7 in reply to: ↑ 5 Changed 7 years ago by witsch
Replying to erikrose:
Clearing Owner field of 4.0 PLIPs so we can use it to mean "implementor". (Many of these owners were automatically assigned from choosing a Component that had a default owner.)
in this case i think the owner was already assigned correctly... ;)
comment:8 in reply to: ↑ 6 Changed 7 years ago by witsch
Replying to alecm:
Not sure how I became a participant though.
:) i guess martin added you to the list when the PLIP was originally put up. interestingly, however, it was only created after we started the package back at the copenhagen performance sprint. so perhaps he wanted you to participate, and afair you did so in the discussion, no?
anyway, feel free to remove yourself from the list if you don't want to be blamed for the bugs later on... ;)
comment:9 Changed 7 years ago by witsch
- Status changed from new to assigned
- Cc optilude, mj, alecm, witsch added
adding the participants and taking it on (if somebody else wants to be responsible, please tell me! :))
comment:10 Changed 7 years ago by smcmahon
- Cc plip-advisories@… added; optilude, mj, alecm, witsch removed
comment:12 Changed 7 years ago by MatthewWilkes
FWT Vote: +1
comment:13 Changed 7 years ago by rossp
FWT vote: +1
comment:14 Changed 7 years ago by davisagli
FWT vote: +1.
Please include documentation for product developers on how to migrate existing subclasses of ATFolder.
What's the size of a BTree-based folder pickle vs. the current ATFolder?
comment:15 Changed 7 years ago by calvinhp
FWT Vote: +1
comment:16 Changed 7 years ago by raphael
FWT Vote: +1
comment:17 Changed 7 years ago by esteele
comment:18 Changed 7 years ago by esteele
Approved by FWT vote.
comment:19 Changed 7 years ago by witsch
- Description modified (diff)
more formatting fixes to finally match the state of the original plip
comment:20 Changed 7 years ago by witsch
- Description modified (diff)
update ticket to reflect the current status
comment:21 Changed 7 years ago by witsch
- Description modified (diff)
currently there are 0 test failures when running tests against Products.CMFPlone, Products.ATContentTypes and Products.Archetypes as of Plone 3.3rc5. the same tests still have to be conducted against Plone 4.0, though...
comment:22 Changed 7 years ago by witsch
- Description modified (diff)
this is not ReSt, so unicode characters can be included directly... yay! :)
comment:23 Changed 7 years ago by witsch
(In [29060]) add plone.app.folder as a dependency to Plone & update the FTI for "Folder" content to use the new implementation (refs #9316)
comment:24 Changed 7 years ago by witsch
comment:25 follow-up: ↓ 29 Changed 7 years ago by davisagli
comment:26 Changed 7 years ago by esteele
Your PLIP has passed the Framework team's initial review. Feel free to discuss any suggested changes either here in the PLIP ticket or on the mailing lists. Final deadline for this PLIP is set for September 30.
comment:27 Changed 6 years ago by witsch
(In [30089]) fix the mess left behind after the too late at night r29060: of course the product can remain as ATContentTypes after the original ATFolder was replaced with the btree-based implementation anyway. also loading plone.app.folder's zcml isn't necessary as the content type does still live in ATCT and the extra profile isn't needed either. so the only thing left is to load zcml for plone.folder.
this allows to actually run tests against in plip buildout again. (refs #9316)
comment:28 Changed 6 years ago by witsch
(In [30097]) partly apply current set of test patches for CMFPlone, resolving the test failures (refs #9316)
comment:29 in reply to: ↑ 25 ; follow-up: ↓ 30 Changed 6 years ago by witsch
Replying to davisagli:
quoting parts of david's review to be able to provide comments inline:
Notes and observations
- The original PLIP discussed the creation of a special view optimized for viewing folders with many items. I presume this was dropped due to the separate PLIP for revamping the collection UI? That's probably fine.
well, it wasn't really dropped because of that other PLIP, to be honest. it was just that when martin & i started working on this back in copenhagen, i never thought of a UI part, and i only read PLIP 191 again after steve had already used it as the base for this one... :)
iow, it was never my intention to provide an update folder_contents view for this, even though we have of course discussed the implications and possible solutions for quite a bit. we've also created a slightly adjusted version in one of our projects, but that's tailored towards the partial ordering adapter.
in any case, i won't have time to come up and implement such a view as part of this work. and i also think that it needs more discussion first. sean upton's work (as part of this year's gsoc) seems promising, and i'd very much like to see a PLIP for this for 4.1 or 4.2. it won't necessarily me who's driving this, though... :)
- The original PLIP also discussed some migration code for converting existing the internal data structures of existing folders. Does this code exist where where it can be reviewed?
not yet, it still lives inside a migration package of the project it was developed for. i will merge it into plone.app.folder next thing, though.
It's crucial that we get this right, of course, and that it can be made to work for custom subclasses of ATFolder as well. Assuming this concern is addressed, it looks like custom subclasses will simply pick up the new mixin and "just work" as long as they aren't overriding the old ordering implementation.
right. the migration has proven to work for our project: some 15+k folders were migrated a while ago and are now used in production by several thousand users every (week)day. so far we haven't seen any problems with neither the migration nor the new folder implementation. however, i should mention that this project is using the partial ordering adapter instead of the default one.
- The extension profile in plone.app.folder appears to merely duplicate the change that was made in CMFPlone's Folder.xml.
right, and the change to the default Folder.xml was bad anyway, see r30089. the same commit also keeps the extra profile from being loaded, fixing this issue. you are right, though, the profile is indeed still needed for plone 3.x.
- bin/alltests shows erroring tests in a bunch of packages, mostly with a traceback and error similar to the following:
this was also fixed by r30089 — the problem was that the factory (in plone.app.folder) hadn't been properly initialized yet by the time PTC was trying to create a folder during site setup. as ATFolder itself is now patched (by this PLIP, as opposed to the stand-alone plone.app.folder for 3.x), the factory can simply remain to be the old one.
ValueError: Product factory for Folder was invalid
on a side node, i only realized last night that i left the PLIP in a completely unfinished state the night i was trying to wrap it up for the first round of reviews. i found a yet uncommitted change and also remembered having seen that error before. however, i also remember constantly dozing off over the keyboard that night — no real surprise at 3am, i guess...
at some point i think i gave up on the above mentioned test setup problem, deciding i would look into it in the morning. however, the funny thing to me is that apparently i was so tired that i didn't even remember something still wasn't quite working the next day. judging from my time tracking i must have thought everything was fine and never looked back... so again, sorry about the mess. :)
There are also a lot of test issues in Archetypes.
right, there are indeed a number of test issues left. i've just updated the branches and checked a couple of packages:
- Product.Archetypes currently has 5 failures, 9 errors (these are very likely all because the sample type SimpleBTreeFolder can for some reason not be adapted to IOrdering anymore)
- Product.ATContenTypes has 1 failure in traversal.txt
- Product.CMFPlone has 0 failures
- plone.folder has 0 failures (needs to be run using bin/instance test -s plone.folder for some reason)
- plone.app.folder (which also needs to be run using bin/instance test ...) has 0 failures, but a problem with importing test_integration.py (i suspect a circular import or something)
i will investigate these asap, but none of them should be show-stoppers. the same tests all pass on plone 3.x, and i'm pretty sure the failures are only triggered because of the integrational changes in Archetypes and ATContentTypes.
I assume that these are mostly due to the switch to Plone 4 when this package was developed for Plone 3, but come on, these really need to be fixed for this PLIP to be considered seriously!
you're right of course. sorry about turning this in without properly finishing up first. but like i said, i thought i had... %)
- I did not have time to verify the benchmarks included in the PLIP overview, but don't have reason to doubt them.
actually, it's pretty easy to run them — see the comments in benchmark.py. the jmeter tests will be trickier to set up, of course.
- Any sense of how this affects pickle size and transaction size for operations on different sizes of folders?
no, i haven't checked yet. would you have any suggestions on what to test here in particular?
comment:30 in reply to: ↑ 29 Changed 6 years ago by witsch
Replying to witsch:
Replying to davisagli:
- I did not have time to verify the benchmarks included in the PLIP overview, but don't have reason to doubt them.
actually, it's pretty easy to run them — see the comments in benchmark.py. the jmeter tests will be trickier to set up, of course.
well, except that i just noticed they're currently broken, because the old folder has no FTI anymore. i'll look into that, but not before the other issues are fixed...
comment:31 Changed 6 years ago by witsch
comment:32 Changed 6 years ago by witsch
comment:33 Changed 6 years ago by witsch
(In [30120]) fix (most of) the migration tests — plip #9316 changes the original ATFolder class to use an implementation based on btrees, so it can no longer be used for testing the migration. hence a dummy content type much like the old class is needed... this leaves us with 1 failure and 1 error (refs #9316)
comment:34 Changed 6 years ago by witsch
comment:35 Changed 6 years ago by witsch
comment:36 follow-ups: ↓ 37 ↓ 38 Changed 6 years ago by davisagli
The test situation is looking much better. I'm still seeing the issues you mentioned in the ATContentTypes and Archetypes tests, as well as some in the CMFEditions tests. I think it's probably okay for these to be put off and fixed as part of the merge, as long as it gets done. ;)
The migration looks good...shouldn't we have this get called from an upgrade step in plone.app.upgrade though?
+1 for merge.
comment:37 in reply to: ↑ 36 Changed 6 years ago by witsch
Replying to davisagli:
I'm still seeing the issues you mentioned in the ATContentTypes and Archetypes tests,
yes, i didn't have time to look into them yet.
as well as some in the CMFEditions tests.
these errors like
Error in test test07_prepareObjectWithReferences (Products.CMFEditions.tests.test_ArchivistTool.TestArchivistToolZStorage) Traceback (most recent call last): File ".../lib/python2.6/unittest.py", line 279, in run testMethod() File ".../Products.CMFEditions/Products/CMFEditions/tests/test_ArchivistTool.py", line 247, in test07_prepareObjectWithReferences for sub in cloneValues: File ".../Products/ZCatalog/Lazy.py", line 158, in __getitem__ data.append(func(s[ind])) File ".../plone.folder/src/plone/folder/ordered.py", line 44, in _getOb return super(OrderedBTreeFolderBase, self)._getOb(id, default) File ".../Products/BTreeFolder2/BTreeFolder2.py", line 221, in _getOb return ob.__of__(self) AttributeError: __of__
also occur in plone 4.0 and 3.3 with the following patch
-
src/Products.CMFEditions/Products/CMFEditions/tests/test_ArchivistTool.py
47 47 self.portal.acl_users.userFolderAddUser('reviewer', 'reviewer', 48 48 ['Manager'], '') 49 49 self.portal.invokeFactory('Document', 'doc') 50 self.portal.invokeFactory('Folder', 'fol') 50 self.portal.portal_types.getTypeInfo('Large Plone Folder').global_allow = True 51 self.portal.invokeFactory('Large Plone Folder', 'fol') 51 52 self.portal.fol.invokeFactory('Document', 'doc1_inside') 52 53 self.portal.fol.invokeFactory('Document', 'doc2_inside') 53 54 self.portal.fol.invokeFactory('Document', 'doc3_outside')
iow, btree folders don't seem to be compatible with versioning atm. my guess is that perhaps nobody really tried to use this combination before, especially when taking into consideration that "large folders" have so far been mostly neglected in plone...
i'll try to have a look into this, even though it's strictly not a related problem. however, if we want to migrate to btree folders, the issue must be addressed, of course...
I think it's probably okay for these to be put off and fixed as part of the merge, as long as it gets done. ;)
yes, it needs to get done, of course, but i'm also planning on it... :)
The migration looks good...shouldn't we have this get called from an upgrade step in plone.app.upgrade though?
we should — i'll add it to the list...
comment:38 in reply to: ↑ 36 Changed 6 years ago by witsch
Replying to davisagli:
+1 for merge.
hmm, i suppose i never "officially" stated that this was ready for a second round of reviews, which is probably also why it's missing from the list of to be voted on PLIPs...
iow, could it please be added? :)
comment:39 Changed 6 years ago by rossp
FWT vote: +1 for merge
comment:40 Changed 6 years ago by esteele
This PLIP has been accepted for merging into Plone 4.0
The final vote was: Alec Mitchell +1 David Glick +1 Erik Rose +1 Laurence Rowe +1 Matthew Wilkes - Ross Patterson +1
Please merge your branches into the Plone 4.0 head by end-of-day Friday Oct 16. If you need assistance with merging, please contact me.
We'll be assigning a documentation ticket to this PLIP shortly. Please assist the docs team in documenting the changes and new features that this PLIP introduces.
comment:41 Changed 6 years ago by esteele
Please assist the doc team in creating/updating documentation relating to this PLIP. See #9615.
comment:42 Changed 6 years ago by witsch
comment:43 Changed 6 years ago by witsch
(In [30537]) merge branch for plip9316 into 4.0@30536 (refs #9316)
comment:44 Changed 6 years ago by witsch
(In [30538]) merge buildout changes for plip9316 (refs #9316)
comment:45 Changed 6 years ago by witsch
- Status changed from assigned to closed
- Resolution set to fixed
the PLIP has been merged, and even though there are still things left to wrap up & polish this ticket can be closed...
comment:46 Changed 6 years ago by witsch
comment:47 Changed 6 years ago by witsch
(In [30898]) add results of running the benchmark tests with different numbers of content items (5, 50, 500, 5000) and a helper script for converting them into something mainly intended to be imported into a spreadsheet app, but also slightly more human-readable (refs #9316)
to run the benchmarks yourself, you might use it like so::
$ bin/instance test -s plone.app.folder --tests-pattern=benchmarks 2>&1 | benchmarks/convert.py
comment:48 Changed 6 years ago by witsch
comment:49 Changed 6 years ago by davisagli
comment:50 Changed 6 years ago by witsch
comment:51 Changed 6 years ago by witsch
comment:52 Changed 6 years ago by witsch
comment:53 Changed 6 years ago by witsch
comment:54 Changed 6 years ago by witsch
comment:55 Changed 6 years ago by witsch
comment:56 Changed 6 years ago by witsch
comment:57 Changed 6 years ago by witsch
comment:58 Changed 6 years ago by witsch
comment:59 Changed 6 years ago by witsch
(In [34078]) use our own zcml for testing & also register the partial ordering adapter now that they get along with each other (refs #9316)
comment:60 Changed 6 years ago by witsch
comment:61 Changed 6 years ago by witsch
comment:62 Changed 6 years ago by witsch
(In [34156]) pull in the "nogopip" changes from plone.app.folder, which remove the getObjPositionInParent catalog index while keeping the possibility to sort by folder position (refs #9316)
comment:63 Changed 6 years ago by witsch
comment:64 Changed 6 years ago by witsch
comment:65 Changed 6 years ago by witsch
comment:66 Changed 6 years ago by witsch
comment:67 Changed 6 years ago by witsch
(In [34432]) partly revert r26681 & r26682, thereby removing support for setting __parent__ and __name__ for content providing IContained as it can cause severe performance issues when used on Zope 2.x (refs #9316)
many thanks steve & limi for copying this over in time before the deadline!! :) i've just fixed some of the remaining formatting glitches and will update the status soon...