Ticket #9283 (closed PLIP: wontfix)

Opened 7 years ago

Last modified 6 years ago

A more lightweight backend for collections

Reported by: elvix Owned by: elvix
Priority: minor Milestone: 4.1
Component: General Version:
Keywords: Cc: regebro@…, plip-advisories@…

Description (last modified by elvix) (diff)

Proposer: Geir Bækholt

Seconder: Laurence Rowe

Motivation

Collections in Plone are the most powerful tool content editors and site managers have to construct navigation and site sections. They do, however create somewhat of a mess in their current form.

For some reason, the criteria (i.e the internal implementation details) of collections are implemented as Archetypes content types. This has a number of unfortunate side-effects:

  • Performance. Content objects take a long time to save and are heavyweight objects. Using many of these for internal structure causes large overhead.
  • UI pain. These internal types pop up in a lot of lists and preference screens designed for managing content. It is currently necessary for developers to maintain lots of exceptions and filters and double lists to not show these non-types to users.
  • developer simplicity. It needs to be simpler and faster and more flexible for developers to expand or add to the existing types of criteria.

Assumptions

This work will not touch existing ATTopic, but replace it with an alternative implementation.

Proposal & Implementation

Reimplement the criteria in collections as simpler objects: Either simple custom classes, dexterity types or simply python simple types like dictionaries. The details have to be worked out as a team that can discuss the pros and cons of each solution.

The new type should be used as the base for listing-type tiles in the future Plone 5.

Deliverables

  • replacement product for ATTopic that uses a simpler, more lightweight backend for storing its query.
  • migrations (optional?) from ATTopic to new-collections
  • tests
  • i18n

Risks

  • Migrations from existing collections will have to be written.
  • A new user-interface will be needed (#9295)
  • Pluggability for adding new indexes and new types of criteria
  • New developer documentation on how to extend Plone with new indexes and criteria

Participants

Geir Bækholt, Laurence Rowe, Jarn

Progress

Thinking and discussion. Jarn is planning to arrange a sprint to get work done on the topic of search and collections.

Change History

comment:1 Changed 7 years ago by alecm

This is, IMHO, a bit too light on implementation/migration details to be accepted as a PLIP for 4.0. There are a lot of potential pitfalls in doing this, and they need to be addressed as well. I'd suggest making a "lightweight collections" add-on product which could become the default for 5.0.

comment:2 Changed 7 years ago by elvix

There are plans for at least one sprint addressing this issue. If needed we can do more technical docs now up front. I'll talk to Laurence tomorrow.

Plone 5 should not have collections as a type at all anymore, but a listing/collection tile, of course. ;)

comment:3 Changed 7 years ago by alecm

Some technical documentation would be nice. However, I think it's critical that the PLIP directly address the risks involved in migrating existing Collections, supporting custom criteria types, maintaining backwards compatibility or providing a migration path for custom sub-classes of ATTopic, ....

comment:4 Changed 7 years ago by elvix

  • Description modified (diff)

comment:5 Changed 7 years ago by elvix

  • Description modified (diff)

Alec,

I think this would be a replacement for ATTopic, not an improvement of the existing product. I'd prefer to build this as an independent product and,let ATTopic itself and existing subclasses of ATTopic stay as they are, and just deactivate the existing ATTopic as part of migrations.

We do, however, need to provide migrations for default ATTopic to this new replacement.

The technical details on the migration cannot be written until the technical implementation for the storage has been decided. I'd like to limit migrations to default criteria.

As for backward compatibility: IMO ATTopic and new-style-collections can coexist in a site if needed.

comment:6 Changed 7 years ago by alecm

Great, I think that's exactly the way to go with this sort of thing. I'd even suggest making the migration an optional step (since it's likely to be trouble prone), and simply disabling creation of new ATTopics by default.

The new PLIP along with the info in this comment (which I'd suggest including within PLIP text) is very promising. My one remaining concern is if it makes sense to do this fundamental redesign as part of the core, when we are supposedly getting rid of the collection type entirely in the next major revision. If this new type is likely to be used as the basis for the collection replacement for 5.0, then that concern vanishes. Thanks for the update.

comment:7 Changed 7 years ago by elvix

I think having the migrations optional is a good idea.

Sorry if this was not clear, but i'd like this to be the first step on what will be collection/list tiles in Plone 5. It will be impossible to build tiles on top of what we have (ATTopic) today.

comment:8 follow-up: ↓ 10 Changed 7 years ago by alecm

Excellent. You get a big +1 from me then. When's the sprint going to be?

comment:9 Changed 7 years ago by pupq

People are definitely scared by those weird content types in portal_types! Getting this to z3 just for that would be a win.

This is a huge +1, though, for a more critical reason: the collections UI needs some simple, low-hanging love. Getting this stuff in z3 widgets, where we do some of these things, will really help make collections more useful.

P.S. The most commonly requested feature for collections: "Is there a way to not show certain kinds of things". Having some sort of "last-minute-filter-out" or such would be very useful.

comment:10 in reply to: ↑ 8 Changed 7 years ago by elvix

Replying to alecm:

Excellent. You get a big +1 from me then. When's the sprint going to be?

We'll try to start with a small sprint this summer (july?) then hopefully a larger one not too long after.

comment:11 Changed 7 years ago by elvix

  • Description modified (diff)

updating description to be more in line with the guidelines.

comment:12 Changed 7 years ago by regebro

So we should change the milestone to 5.0, in fact?

comment:13 Changed 7 years ago by regebro

  • Cc regebro@… added

comment:14 Changed 7 years ago by erikrose

Clearing Owner field of 4.0 PLIPs so we can use it to mean "implementor". (Many of these owners were automatically assigned from choosing a Component that had a default owner.)

comment:15 Changed 7 years ago by smcmahon

  • Cc plip-advisories@… added

comment:16 Changed 7 years ago by MatthewWilkes

Hmm, we need this, but doing it as an external product is the sensible way. I'll probably say that we ship this as a by-default uninstalled product, or perhaps uninstalled for migrations and installed for new sites. Even if this PLIP fails, it would be a very useful 3rd party product.

FWT Vote: +0

comment:17 Changed 7 years ago by rossp

FWT vote -1. Very much want this, but this is perfect for starting and vetting as an add-on.

comment:18 Changed 7 years ago by rossp

Maybe this is a candidate for 4.x once vetted as an add-on?

comment:19 Changed 7 years ago by rossp

I want to clarify why I think this needs vetting as an add-on before inclusion into core. The current collection implementation is in wide-use in the wild and replacing it wholesale without giving users, integrators and add-on developers time to reconcile with a new implementation will be too disruptive even for a major release. Also, collections are tricky and I expect there to be unexpected interactions that can be resolved during vetting as an add-on.

comment:20 Changed 7 years ago by davisagli

FWT vote: +1. This would be a great improvement if done well. And I am happy with the discussion so far on how to transition from ATTopic, as well as excited with the prospect of creating something that can also be the basis for a query results tile in Plone 5. I am skeptical that this can come to maturity within the ambitious timeframe of Plone 4. But feel free to prove me wrong on that. :) And I think it would be a possible candidate for phasing in in later 4.x releases.

comment:21 Changed 7 years ago by raphael

FWT vote: -1 for Plone 4.0 (too risky)

But please don't let that stop you; I certainly consider this in scope for Plone 5 or an add-on to Plone 4.x

comment:22 Changed 7 years ago by calvinhp

FWT Vote: +1 this is an area that has needed much improvement, but I'd like to see ATTopics and the new replacements ones be able to coexist with the option for the user to choose to migrate them from a control panel.

comment:23 follow-up: ↓ 24 Changed 7 years ago by erikrose

  • How much do products currently depend on Collections API that would change? How popular are custom criteria in the wild? These could hurt hands-free migration, so I like Calvin's idea of a manual, optional migration.
  • I'd encourage the implementors to work closely with Hanno on this to make sure any API changes change once: not now and then again in 5. It sounds like they already have this in mind.
  • If we're going to do this in 4.x, my guts tell me it should be in 4.0; from a site manager's point of view, it seems a scary change for a .x release.
  • Can we get a UI (any UI) for reordering Subfolders along with this? I've had that request and personally done the crazy ZMI manual-bubble-sort workarounds for it.

As others point out, this PLIP is light on detail, so I'll be light on my opinion. FWT vote +1 so we can see what comes of it. We can always take a step back during reviews if it doesn't work out. (My ulterior motive is that I want the better Collections UI that depends on this!)

comment:24 in reply to: ↑ 23 Changed 7 years ago by elvix

Replying to erikrose:

  • How much do products currently depend on Collections API that would change? How popular are custom criteria in the wild? These could hurt hands-free migration, so I like Calvin's idea of a manual, optional migration.

I don't know about others but : We (Jarn) commonly add criteria registrations for new indexes. We almost never write custom criteria types. I think the latter is very uncommon.

  • I'd encourage the implementors to work closely with Hanno on this to make sure any API changes change once: not now and then again in 5. It sounds like they already have this in mind.

We share offices with Hanno, and would never do this without consulting him. 

  • Can we get a UI (any UI) for reordering Subfolders along with this? I've had that request and personally done the crazy ZMI manual-bubble-sort workarounds for it.

What is reordering subfolders and how does this relate to collections? Subfolders in collections? Shouldn't they use the same ui as other subfolders do? I don't think this PLIP will supply any such UI.

comment:25 Changed 7 years ago by esteele

Approved by FWT vote.

comment:26 Changed 7 years ago by esteele

  • Owner set to elvix

comment:27 Changed 7 years ago by elvix

  • Status changed from new to assigned

comment:29 Changed 7 years ago by alecm

(In [29638]) Add review for PLIP #9283 and #9295 (refs #9283, refs #9295)

comment:30 Changed 7 years ago by esteele

  • Status changed from assigned to closed
  • Resolution set to wontfix

The Framework Team has agreed that there is not enough completed work to properly review for inclusion into 4.0. Closing, with the hopes that we can revisit this idea in a later release.

comment:31 Changed 7 years ago by elvix

I agree completely with this decision. We won't be able to make this anything near stable and solid in time for Plone 4.

comment:32 Changed 6 years ago by limi

  • Status changed from closed to reopened
  • Resolution wontfix deleted
  • Milestone changed from 4.0 to 4.1

Moving to 4.1 at Geir's request.

comment:33 Changed 6 years ago by elvix

  • Status changed from reopened to closed
  • Resolution set to wontfix

This PLIP is superceded by #10902

comment:34 Changed 4 years ago by davisagli

  • Component changed from Infrastructure to General
Note: See TracTickets for help on using tickets.