Ticket #10359 (closed PLIP: fixed)

Opened 6 years ago

Last modified 2 hours ago

Convert control panels to use z3c.form

Reported by: hannosch Owned by:
Priority: minor Milestone: 5.0
Component: General Version:
Keywords: Cc: plip-advisories@…, amleczko

Description (last modified by timo) (diff)

Proposer: Hanno Schlichting
Seconder: Timo Stollenwerk

Abstract

Convert all formlib based control panel screens in plone.app.controlpanel to use z3c.form instead. Converting control panels currently not using formlib to z3c.form might be inside the scope of this PLIP, but only actual implementation will show.

Motivation

Our standard control panels, as found in plone.app.controlpanel are implemented using zope.formlib / zope.app.form. While formlib has been helpful to have an Archetypes independent form framework, it has proven to be too inflexible in real customer projects. Learning from the short-comings of formlib, z3c.form has been written and has meanwhile seen a major adoption in the community - both inside the Plone and the wider Zope communities.

The Zope 2 integration of zope.formlib has been moved to a separate package called five.formlib for Zope 2.12. In Zope 2.13 this package isn't shipped anymore. This reflects the dwindling mindshare that zope.formlib has seen. For Plone 5.x this could allow Plone to no longer ship with formlib at all.

Assumptions

This PLIP is dependent on PLIP #9473 - the general readiness of the Plone - z3c.form integration. It assumes that customizations of the Plone control panels are infrequent and changes to the internals of the control panels aren't seen as breaking public API's of Plone.

Proposal & Implementation

The aim of this PLIP is to re-implement all formlib based control panels inside plone.app.controlpanel on top of z3c.form. A new base class for add-on control panels will be provided, so add-ons can opt-in to the new approach. The old formlib based base classes would be deprecated and removed in Plone 5.0.

The implementation will use the plone.app.z3cform, plone.autoform and plone.app.registry packages. It will not use plone.directives.form and thus avoid a dependency on Grok.

Deliverables

  • A branch of plone.app.controlpanel with the new implementation
  • Documentation for writing your add-on control panel based on the new classes

The localization strings should be kept as close to the current code as possible, so no major new translation effort will be required. The actual UI should also stay very similar to the current state, so no major end-user documentation changes will be needed.

Risks

This PLIP doesn't consider the risks for including z3c.form itself. Specific risks for this PLIP are seen as minor. Only for the unlikely case that someone has customized the control panels or has code based on one of the specific panels breakage is to be expected. The only known add-on doing this is LinguaPlone, which extends the language control panel. It would be part of this PLIP to ensure LinguaPlone continues to work.

Participants

Timo Stollenwerk <tisto>

Progress

Change History

comment:1 Changed 6 years ago by davisagli

I am currently doing some work on adjusting the slots within the main form macros template in plone.z3cform and plone.app.z3cform, to better match the sorts of overrides that are commonly necessary -- see  http://groups.google.com/group/dexterity-development/msg/f4bb45493002bd3e ...so now would be a great time to do enough initial exploration to find out what slots you need in order to support a template for control panel forms, in case there are any other changes we need to make.

comment:2 Changed 6 years ago by ldr

  • Milestone changed from 4.x to 4.1

comment:3 Changed 6 years ago by jonstahl

I think this is a fantastic idea. I do have one friendly amendment: I think it would be smart (esp. if we intend to deprecate formlib in Plone 5) to include within the scope of this PLIP a short document that explains to beginning/intermediate product authors how to convert an existing add-on products control panel configlet to z3c.form. That will help our add-on products make the leap when it is time.

comment:4 Changed 6 years ago by hannosch

  • Status changed from new to assigned

comment:5 follow-up: ↓ 6 Changed 6 years ago by ldr

Would it make sense to move to plone.app.registry for setting storage as well? - it makes writing simple control panels ridiculously easy and simplifies the "how do I retrieve a setting" story. Backwards compatibility will be the biggest difficulty there, but might be possible with properties that get/set to the registry.

comment:6 in reply to: ↑ 5 Changed 6 years ago by hannosch

Replying to ldr:

Would it make sense to move to plone.app.registry for setting storage as well? - it makes writing simple control panels ridiculously easy and simplifies the "how do I retrieve a setting" story. Backwards compatibility will be the biggest difficulty there, but might be possible with properties that get/set to the registry.

I think it makes perfect sense to convert from portal_properties and custom tools to plone.app.registry in general. But I won't sign up or promise to work on this for 4.1. My next weeks are busy enough as it is, so I cannot afford any feature creep ;)

comment:7 Changed 6 years ago by esteele

Your PLIP has been accepted for consideration for Plone 4.1.

Framework Team voting on this PLIP was: Alec +1 Craig +1 Elizabeth +1 Laurence +1 Martijn +1 Matthew +1 Rob +1 Ross +1

The initial implementation deadline for your PLIP is October 1st, 2010. The Framework Team would certainly appreciate you finishing beforehand so that they may begin evaluating it as soon as possible. Announce its readiness here once your implementation is ready for review.

comment:8 Changed 6 years ago by cah190

  • Cc plip-advisories@… added

comment:9 Changed 5 years ago by hannosch

  • Milestone changed from 4.1 to Future

My Zope 2.13 PLIP is costing me too much time. I'll have to defer this one to Plone 4.2. Less review work for the fwt :)

comment:10 Changed 5 years ago by hannosch

  • Status changed from assigned to new
  • Owner hannosch deleted

comment:11 Changed 5 years ago by rossp

  • Status changed from new to closed
  • Resolution set to wontfix

PLEASE READ THIS AND RE-OPEN VALID PLIPS!

As we launch the new PLIP process we'd like to see which PLIPs:

  • are still appropriate/needed
  • still have owners/proposers/champions
  • still have available implementers

If this PLIP should still be considered for future releases of Plone please do re-open this ticket and assign an appropriate milestone. If it should be considered for the next release of Plone, use the 4.2 milestone. Also be sure to update the PLIP description, requester, owner, etc. and include a comment detailing recent progress and new plans. We will use all these details in the new continuous PLIP process.

comment:12 Changed 4 years ago by timo

  • Status changed from closed to reopened
  • Version set to 4.1
  • Resolution wontfix deleted
  • Description modified (diff)

comment:13 Changed 4 years ago by timo

  • Version 4.1 deleted
  • Description modified (diff)

comment:14 Changed 4 years ago by timo

I started a branch to work on this PLIP and I'm willing to implement it for Plone 4.3 (or whatever version the FWT think is appropriate):

 https://github.com/plone/plone.app.controlpanel/tree/plip10359-z3cform

I already replaced the editing control panel with a plone.autoform-based implementation as a proof-of-concept and moved most of the tests to use plone.app.testing.

I will investigate if it is possible to use plone.app.registry (within the scope of this PLIP) and amend the PLIP accordingly. As Laurence already pointed out, the biggest problem is backwards compability.

comment:15 Changed 4 years ago by davisagli

  • Component changed from Infrastructure to General

comment:16 Changed 4 years ago by timo

  • Description modified (diff)

comment:17 Changed 4 years ago by timo

Status Update:

I moved four control panels (editing, mail, navigation, security) to use plone.app.registry for the forms and the settings storage.

To keep backwards compatibility the new p.a.registry settings are synced with the old storage (portal_properties, MailHost, etc.). The sync from the old settings to the new ones is currently done by monkey patching portal_properties and MailHost:

 https://github.com/plone/plone.app.controlpanel/blob/plip10359-z3cform/plone/app/controlpanel/bbb/patches.py

For the final implementation we probably want these components to fire up events that can be used to do the sync.

Before I proceed, I would like somebody from the FWT (or an experienced Plone dev) to review the implementation of those four control panels to make sure I'm on the right path.

comment:18 Changed 3 years ago by timo

I talked to hannosch at the Plone conference about that PLIP and he said that the control panel code could be considered internal code that does not need to be backward compatible. An upgrade step would be sufficient.

Of course we would have to change some code in other packages that get/set controlpanel settings (which we have to do anyway). I guess this would make the implementation of this PLIP easier and the implementation more robust. I also could finish that PLIP a lot sooner.

comment:19 Changed 3 years ago by eleddy

Agreed. When migrating the cpy files out of skins we are converting to z3c.form as well. This is a great PLIP that wil be perfect for timing with that.

comment:20 Changed 3 years ago by esteele

For what it's worth, cpy-holder Products.PasswordResetTool gets a mention for merging/removal in #13283.

comment:21 Changed 3 years ago by amleczko

  • Cc amleczko added

comment:22 Changed 3 years ago by amleczko

I have checked your work and you are on the right path.

I agree with hannosch that the package is an internal code that does not need to be backward compatible. If we will have an upgrade step - it should be OK - p.a.controlpanel properties should be modified by GenericSetup profiles (in most of the cases).

I also agree with jonstahl that a short document that explains to beginning/intermediate product authors how to convert an existing add-on products control panel configlet to z3c.form would be a great to have.

One thing that I've noticed is the jenkins job failing -  https://jenkins.plone.org/me/my-views/view/All/job/plone-4.3-plips/PLIP=plip10359-z3cform-controlpanel.cfg,label=Master/lastBuild/console

comment:23 Changed 3 years ago by timo

Just a quick update on the work that I did at the beer and wine sprint in Munich:

  • All control panels have been migrated to either plone.autoform or plone.app.registry (including the users/groups control panels).
  • There are currently some test failures in the filter control panel (which means it is not 100% ready yet).
  • There are some z3c.form widgets missing and some cleanup is necessary.
  • I still have to write an upgrade step to for the GS profiles.
  • We should replace calls to the old portal tools API with p.a.registry API calls and deprecate the old methods.

There is already a tutorial that I wrote some time ago about how to create a control panel with p.a.registy:

 http://plone.org/documentation/kb/how-to-create-a-plone-control-panel-with-plone.app.registry

Is this already sufficient? I could write another tutorial about how to extend an existing control panel, but I don't think this use case is pretty common and it would be basically the same as  http://packages.python.org/plone.app.discussion/howtos/howto_extend_the_comment_form.html.

The Jenkins job is still failing because there are still failing tests.

I would like to know which version we target for this PLIP. If this is considered to go into the 4.x series I would like to know as soon as possible, because it would mean that I have to reserve some time for it.

comment:24 Changed 3 years ago by davisagli

I think it can go in 4.4 if the old properties are still available via some backwards-compatibility layer; otherwise 5.0

comment:25 Changed 3 years ago by rossp

I started a review but ran into import errors when run under latest coredev 4.3:

 https://github.com/plone/buildout.coredev/commit/01c1a7b766f7bdd5d318cce229ecf83ed96c897a

comment:26 Changed 3 years ago by amleczko

After your last plips/plip10359-z3cform-controlpanel.cfg update I get 2 tests failures:

 https://github.com/plone/buildout.coredev/blob/4.3/plips/reviews/plip10359-review-amleczko.rst

comment:27 Changed 3 years ago by timo

Yes. I saw that. I don't have time to look into it during the week. Will find to find some time on the weekend. I will get to you guys as soon as possible.

comment:28 Changed 3 years ago by rossp

Great, we'll resume our reviews when you get back to us.

comment:29 Changed 3 years ago by timo

I'm currently working on getting  http://dev.plone.org/ticket/12344 into Plone 4.4 and I would like to focus on that for now. I think the other PLIP is currently more important and also easier to integrate into the 4.x branch.

I will continue to work on this PLIP as soon as the other one has been merged. If there is enough time left, I will try to get this one into 4.4 as well, if not I'm happy to go for 5.0.

@amleczko @rossp: Thanks for the review so far, I will keep you posted.

comment:30 Changed 3 years ago by esteele

  • Milestone changed from Future to 5.0

The Framework Team has decided to move on to Plone 5. Updating milestones accordingly.

comment:31 Changed 3 years ago by timo

Since this PLIP will now hopefully become part of Plone 5, I would like to re-evaluate some of the assumptions and design decisions I made during the implementation.

1) My approach to sync the portal properties (old-way) with the new p.a.registry-based settings (new way) should be simplified by just ignoring the old properties in Plone 5 and write a migration step. This will reduce the complexity of the code and more important remove the likelihood of strange side effects (in case the two settings are out of sync). What I need to do for this is I have to find all occurences of code that amends the portal properties and change it. The question is: how and when do we mark those as deprecated? Is it sufficient to do this in a Plone 4.3.x release? When should I start with that? After the PLIP has been fully merged?

2) So far I used zope.interface to define the control panels:

 https://github.com/plone/plone.app.controlpanel/blob/plip10359-z3cform/plone/app/controlpanel/profiles/default/registry.xml

This works fine, the default values just need to be defined at the interface, otherwise they are all None, which isn't what we want. Though, the old way allowed integrators/developer to look at the gs profile and just copy and override certain values (e.g. go to profiles/default/propertiestool copy & paste it into your profile and amend say sitemapDepth).

If we use zope.interface, this isn't possible because the registry.xml looks like the example above. To solve this we could either define the values in registry.xml directly (see  https://pypi.python.org/pypi/plone.app.registry#creating-records) or we do this in Products.CMFPlone (this approach would not make sense if we merge p.a.contenttypes into P.CMFPlone later).

If we plan to take the first option (zope.interface) we should make sure that it is well documented and always up-to-date how to override those values. It is always easy to amend settings TTW and then export the registry.xml though.

comment:32 Changed 3 years ago by amleczko

My 5 cents:

  1. I think the migration step is enough. If it's about timeline I would vote for deprecating portal properties in 4.3.x release.
  2. If it's about default values I'm definitively for storing them in registry.xml

comment:33 Changed 2 years ago by timo

Cathedral Sprint Update / Request for review:

1) I moved the site and the search control panel from plone.app.controlpanel to Products.CMFPlone.

The two control panels now use z3c.form/plone.app.registry form. Plone now uses those settings (I had to adapt PloneTool in CMFPlone and plone.app.layout for this). Please review my changes (even though they have been merged, which was necessary because the new Bareceloneta theme required those changes).

 https://github.com/plone/Products.CMFPlone/tree/master/Products/CMFPlone/controlpanel

2) I'm not 100% happy with the settings lookup (search_settings = registry.forInterface(ISearchSchema, prefix="plone") and I would like to get rid of that.

 https://github.com/plone/Products.CMFPlone/blob/master/Products/CMFPlone/controlpanel/README.rst

3) So far I'm still using interfaces in registry.xml and the registry.xml is located in dependencies:

 https://github.com/plone/Products.CMFPlone/blob/master/Products/CMFPlone/profiles/dependencies/registry.xml#L3

If we want to make it easy for developers/intergrators, I think we should move the registry.xml to the default profile (or have a separate settings profile) and use XML, so integrators can just copy paste this file. It would also be nice if the plone.app.registry control panel would have an import/export TTW functionality to make this even easier.

davisagli's recommendation on IRC: "tisto_: we could fix it by making CMFPlone add the portal_registry tool itself rather than calling the plone.app.registry profile as a dependency to do so. and make sure the registry import step depends on the toolset one"

4) Two tests are commented out for now:

 https://github.com/plone/Products.CMFPlone/blob/master/Products/CMFPlone/controlpanel/tests/test_controlpanel_browser_site.py#L77

    @unittest.skip("XXX: TODO! We have to patch CMFDefault for this.")
    def test_site_title_can_be_looked_up_by_portal_title(self):
        self.browser.open(
            "%s/@@site-controlpanel" % self.portal_url)
        self.browser.getControl('Site title').value = u"My Site"
        self.browser.getControl('Save').click()

        self.assertEqual(self.portal.title, u'My Site')
        self.assertEqual(self.portal.Title(), u'My Site')

    @unittest.skip("XXX: TODO! Not implemented yet.")
    def test_site_description_can_be_looked_up_by_plone_portal_state(self):
        self.browser.open(
            "%s/@@site-controlpanel" % self.portal_url)
        self.browser.getControl('Site description').value = \
            u"This is a Plone site."
        self.browser.getControl('Save').click()

        portal_state = getMultiAdapter(
            (self.portal, self.request),
            name=u'plone_portal_state'
        )
        self.assertEqual(
            portal_state.portal_description(), u'This is a Plone site.')

Question: Do we want to support those use cases?

5) Future: I will continue to work on moving single control panels to z3c.form and ask the FWT to review them separately. I hope I will be able to move all control panels before the final release. Though, this shouldn't be a blocker for a release.

comment:34 Changed 2 years ago by davisagli

I looked this over.

I agree that the setting lookup is awkward, though it's as much because of needing to import 3 different things as because of needing to specify the prefix. Maybe we should use get_registry_record from plone.api ( http://developer.plone.org/reference_manuals/external/plone.api/portal.html#get-plone-app-registry-record).

Why does fixing test_site_title_can_be_looked_up_by_portal_title require patching CMFDefault? Can't we override the Title method on Products.CMFPlone.Portal.PloneSite?

I don't see any code for migrating the old properties to registry records when a site is upgraded to Plone 5. Do you plan to work on that?

We also need documentation: a table of what the settings are and what they do, and also what the old property was to help developers who are upgrading.

comment:36 Changed 14 months ago by khink

comment:37 Changed 14 months ago by khink

The 2 issues above have to do with moving the code to Products.CMFPlone.

"We want to migrate all control panel forms and related code from plone.app.controlpanel to CMFPlone. Control panels were already migrated to z3c.form in branch "plip10359-z3cform":  https://github.com/plone/plone.app.controlpanel/tree/plip10359-z3cform/plone/app/controlpanel . Code needs to be moved from this branch to CMFPlone." ( https://titanpad.com/0fyCHL1o6P)

comment:38 Changed 12 months ago by timo

PLIP has been merged. All control panels have been moved to p.a.registry/p.autoform.

comment:39 Changed 2 hours ago by bslash

  • Status changed from reopened to closed
  • Resolution set to fixed
Note: See TracTickets for help on using tickets.