Ticket #8808 (closed PLIP: fixed)

Opened 7 years ago

Last modified 6 years ago

Require Python 2.5 or 2.6, Zope 2.12, and CMF 2.2 for Plone 4.0

Reported by: hannosch Owned by: davisagli
Priority: major Milestone: 4.0
Component: General Version:
Keywords: Cc: rossp, csenger, plip-advisories@…, grahamperrin@…

Description (last modified by davisagli) (diff)

Overview

Plone 3.x today only has support for a rather old Python 2.4 version. Python has meanwhile seen new 2.5 and 2.6 releases. Recently a new backwards incompatible version of Python named Python 3 has been released.

In the meantime, Zope 2.12 has reached beta 2 status. Zope 2 supports Python 2.5 and Python 2.6 but not Python 2.4 officially (though it probably still works), as discussed in its release notes:

Zope 2 has supported and required Python 2.4 since its 2.9 release in summer
2006. Later versions of Python have so far been unsupported by Zope 2. This
version of Zope 2 adds support for both Python 2.5 and 2.6 at the same time. As
Python 2.4 is no longer maintained itself, it is no longer officially supported
by this Zope 2 version. There is however no code in Zope 2 yet which requires
Python 2.5, so applications built on top of Zope 2 should still continue to run
with Python 2.4.

In order to keep up with our programming language and application server foundation, we should support Python 2.5 and 2.6 for Plone 4. This will allow us to use the 4.x series as a transition period to update our codebase to be more easily ported to a Python 3.x version. Some more reasoning can be found at:  http://blog.hannosch.eu/2008/12/plone-versus-python-3.html

Compatibility with Python 2.5 and 2.6 requires upgrading to Zope 2.12, as it is the first Zope 2 release including the necessary changes to support these Python versions. Zope 2.12 in turn requires (the not-yet-released) CMF 2.2, largely due to the fact that Zope 2.12 has removed support for Zope 2-style Interfaces, and CMF 2.2 will be the first release of CMF adjusted not to use them. As a result, we need to tackle all three upgrades at once.

Benefits

While a project like this is inherently risky, it does bring some nice benefits.

For developers:

  • Python 2.5 and 2.6 has a number of new features.
  • Zope 2.12 is fully eggified, removing the need for fake eggs.
  • Zope 2.12 removes the need for everything to subclass acquisition or be acquisition-wrapped in order to locate the acl_users and provide security. Instead only maintaining an object's parent attribute is required.
  • Zope 2.12 includes ZODB3 3.9
  • Zope 2.12 object managers now support a more Pythonic syntax.
  • CMF 2.2 adds several nice extension points which are used by projects like, for example, Dexterity

For integrators:

  • Python 2.5 and 2.6 have much improved memory management, from what I hear, so overall memory usage is lower.
  • Python 2.5 and 2.6 have a C-based version of the profile module, which opens the possibility for someone to create a profiling tool that is a little more snappy.
  • Python 2.5 and 2.6 are more likely to be included with OS distributions, so the need for compiling a special Python to run Zope/Plone may decrease.
  • Python 2.5 and 2.6 are more likely to be supported by new Python package development efforts.

Risks

  • There may be some Python libraries that work with Python 2.4 but not Python 2.5 and 2.6. However, I think we need to balance the desire not to break anything that works now with the desire to not be stuck on a Python release which is not being targeted by new package development. I've already seen one or two packages that are not compatible with Python 2.4. In addition, it's worth noting that as noted in the Zope 2.12 release notes, Python 2.4 may still work even though it is no longer officially supported.
  • The upgrade process for people with existing Plone sites must be well-documented. For users of the Unified Installer, we can make it so they only have to run the new installer. For users of other buildouts, one can simply re-bootstrap and run buildout with the new Python, after installing it.
  • Zope 2.12 has not reached a final release yet, and CMF 2.2 has not even had preliminary releases (though it is also the least risky of the changes discussed here).

Progress

See the lengthy comment by David below regarding his progress on a set of branches that are compatible with Python 2.6, Zope 2.12, and CMF 2.2.

Change History

comment:1 Changed 7 years ago by hannosch

  • Owner set to hannosch

comment:2 Changed 7 years ago by erikrose

Hooray! +100!

comment:3 Changed 7 years ago by rossp

  • Cc rossp added

comment:4 Changed 7 years ago by esteele

  • Milestone changed from Trunk to 4.0

Stealing this for Plone 4. ;)

comment:5 Changed 7 years ago by davisagli

  • Owner changed from hannosch to davisagli
  • Priority changed from minor to major
  • Description modified (diff)
  • Summary changed from Require Python 2.6 for Plone 4.0 to Require Python 2.6, Zope 2.12, and CMF 2.2 for Plone 4.0

From my e-mail to the framework team:

As some of you may have noticed, I've been working on a branch that can hopefully become the basis of Plone 4, including support for Python 2.6, Zope 2.12 (which is the first Zope 2 supporting python 2.6 and also the first official eggified Zope2) and CMF 2.2 (which will be the first CMF release that doesn't rely on Zope 2-style interfaces, which were removed in Zope 2.12...and also adds a few handy extension points). My basic approach is to start with the packages that make up Plone 3.3, and then backport the changes from trunk that seem non-risky. So far I haven't gotten through the entire history of CMFPlone trunk, nor through all of the rest of the packages, but I have at least gotten far enough to start up Zope and successfully add a Plone site.

Some of the changelogs that are relevant here, FYI: Zope 2.12 release notes:  http://docs.zope.org/zope2/releases/2.12/WHATSNEW.html Zope 2.12 changelog:  http://docs.zope.org/zope2/releases/2.12/CHANGES.html CMFCore 2.2 changelog:  http://svn.zope.org/Products.CMFCore/trunk/Products/CMFCore/CHANGES.txt?view=auto CMFDefault 2.2 changelog:  http://svn.zope.org/Products.CMFDefault/trunk/Products/CMFDefault/CHANGES.txt?view=auto

As I've worked my way along, I've found myself making a number of judgment calls on where to draw the line between bugfixes and changes that are significant enough to warrant discussion. In case anyone cares to object, here are a few things I opted to include that I thought might be controversial:

  • some optimizations coming out of the 2 performance sprints
  • switching action icons to use icon_expr instead of portal_actionicons (hannosch) — see also  http://dev.plone.org/old/plone/ticket/8801
  • switch to png icons (wiggy, dannyb)
  • conversion of migration step registration to be based on GS upgrade steps (hannosch) - see also  http://dev.plone.org/old/plone/ticket/8802
  • c22120 and following - removal of the globalized variables for performance reasons (malthe, hannosch)
  • c22987 - remove unused Favorite content type (hannosch)

There were also a number of things I decided not to merge immediately until we can have some more discussion or finish some needed tasks. Some of these may not actually be desired until 5.0... I would really appreciate feedback here especially.

  • c20188, c20190, c23197 — removal of PloneFolder (hannosch) — need to double check whether this requires a migration for any persistent objects, and write that if necessary
  • c20337, c20338, c22979, c22981, c22988 - switch the control panel to be based on normal actions in a new action category, rather than using a special control panel tool (hannosch) — see also  http://dev.plone.org/old/plone/ticket/8804
  • c20365, c20368, c20369 — moved fullscreen css to a separate file (spliter) — needs a migration
  • c22831, c22832, c22891, c22951 — remove external editor (hannosch) — see  http://dev.plone.org/old/plone/ticket/8592
  • c22847 — remove wicked from core (hannosch) — see  http://dev.plone.org/old/plone/ticket/8593 (but note that as of this week Carsten Senger has been doing some maintenance...yay!)
  • c22969 — removal of AT reference graphviz visualization (hannosch) — needs a migration to update actions for existing types
  • c23193 — turning default_error_message into a browser view (hannosch) — seems like a decent idea to me, but as implemented here it removes the redirector and content search features
  • c23198, c23199 — removal of portal_interface tool (hannosch) — if we remove this then we lose the way to check for marker interfaces from action expressions, for instance. also, it's missing a migration
  • c23226, c23233 — moving migrations to plone.app.upgrade and using all GS without the migration tool (hannosch) — -0 from me...I think moving these to a separate package may actually make it harder to keep migrations in sync with the Plone release they belong too, simply from the standpoint of reviewing svn history
  • c23335 — Changed workflow actor variable from user/getUserName to user/getId (hannosch) — needs a migration
  • c23337 — Removed strangely implemented users search from the users folder (hannosch) — I didn't look carefully enough yet to see whether this is a good idea or not :)
  • c23343 - Removed initial content from default installation (hannosch) — This probably does belong in a profile in a separate package that is included by the standard installers. But creating that hasn't been done yet.
  • Consolidating the skin directories into just plone_browser, plone_resources, and plone_images....plus moving 3rd party javascripts to plone.app.javascript. — I'm -0 on this. Keeping the old layout would make it easier to merge things from the 3.x branch; switching to the new layout would make it easier to merge from trunk. Keeping the old layout has the advantage of not obsoleting a lot of existing documentation, and not changing something that we think there's a good chance we'll change again in 5.0 when we hopefully have a better unified Zope 3 vs. CMF skinning story.
  • Archetypes c8430 — return unicode encoding by default (hannosch) — My charset and encoding fu is weak. Is this what we want?

Some more general comments on risks / outstanding concerns:

  • I need to finish going through the changesets that I haven't looked at yet.
  • There are a number of failing tests in CMFPlone that need to be investigated, and I haven't even tried to run the tests in most of the other packages yet.
  • The current situation with regard to PlacelessTranslationService needs to be reviewed to make sure that our language negotiator and product i18n dirs are still getting registered properly following simplification/removal of Five's Zope2 i18n integration layer. Hanno said he could probably take a look at this.
  • I switched back to Zope 2.12.0b1 for now because with Zope 2.12.0b2 there is an issue preventing the Unauthorized error when you first try to access the ZMI from getting propagated into an HTTP authentication request. I haven't had time to dig into this yet although I did check a non-Plone Zope 2.12.0b2 and it didn't seem to be an issue there.
  • We need to make plone.recipe.zope2instance work with an eggified zope. Currently it doesn't know where to find the mkzopeinstance script or other skeleton configuration. I made a quick hacky branch to make my dev buildout work, but this needs some more careful work.
  • I commented out references to KSS rather than trying to make the adjustments needed to make it work, since there's been some discussion of removing KSS from Plone core and because I don't have access to the KSS repository.
  • CMF 2.2 has not yet been released, and Zope 2.12 is still in beta. So we'll need to do some coordination in terms of release schedules.

If you want to try this out / contribute, the mr.developer-based buildout is at  http://svn.plone.org/svn/plone/buildouts/plone-coredev/branches/4.0/davisagli-zope212-cmf22-compatibility.cfg ...if you find a changeset that still needs to be backported, please feel free to do so as long as you make sure the svn mergeinfo remains accurate.

comment:6 follow-up: ↓ 8 Changed 7 years ago by pupq

From our integrators perspective, this seems +0 or +1. People ask about Python 2.5/2.6, but the need for it is obviously more for developers than integrators (who, generally, speaking, aren't tracking new Pythons quite as much).

Would this *require* Python 2.6? Would 2.5/2.4 be acceptable?

comment:7 Changed 7 years ago by csenger

  • Cc csenger added

comment:8 in reply to: ↑ 6 ; follow-up: ↓ 9 Changed 7 years ago by esteele

Replying to pupq:

From our integrators perspective, this seems +0 or +1. People ask about Python 2.5/2.6, but the need for it is obviously more for developers than integrators (who, generally, speaking, aren't tracking new Pythons quite as much).

Would this *require* Python 2.6? Would 2.5/2.4 be acceptable?

I think it's really important that we write up exactly how everyone benefits from these upgrades. For the developers, it's a no-brainer. I get the feeling that to an average integrator though, installing a new Python seems a larger hurdle than just updating Plone would be.

comment:9 in reply to: ↑ 8 ; follow-up: ↓ 10 Changed 7 years ago by rossp

Replying to esteele:

Replying to pupq:

From our integrators perspective, this seems +0 or +1. People ask about Python 2.5/2.6, but the need for it is obviously more for developers than integrators (who, generally, speaking, aren't tracking new Pythons quite as much).

Would this *require* Python 2.6? Would 2.5/2.4 be acceptable?

I think it's really important that we write up exactly how everyone benefits from these upgrades. For the developers, it's a no-brainer. I get the feeling that to an average integrator though, installing a new Python seems a larger hurdle than just updating Plone would be.

For my part, I *am* concerned about the risk proposed by this PLIP because it seems like one of the riskiest. It's also likely to be one of the most distuptive PLIPs for add-on developers and maintainers. So far, however, I'm +1 on it because of the number of times I've heard concern expressed in the wild that Plone runs out out-dated software. Please don't respond to that on technical merits because that's not where those who express that concern are coming from. This is a perceptual problem and while indulging perceptual problems can be problematic, in this case it's a big enough perceptual problems and there are also large technical benefits.

comment:10 in reply to: ↑ 9 Changed 7 years ago by rossp

Replying to rossp:

Replying to esteele:

Replying to pupq:

For my part, I *am* concerned about the risk proposed by this PLIP because it seems like one of the riskiest. It's also likely to be one of the most distuptive PLIPs for add-on developers and maintainers. So far, however, I'm +1 on it because of the number of times I've heard concern expressed in the wild that Plone runs out out-dated software. Please don't respond to that on technical merits because that's not where those who express that concern are coming from. This is a perceptual problem and while indulging perceptual problems can be problematic, in this case it's a big enough perceptual problems and there are also large technical benefits.

I just realized A) the number of typos in my comment :) and B) that I was replying to something other than what Eric said. I definitely second the suggestion that we thoroughly investigate and document the existing deployment upgrde story for this PLIP.

I still think the perceptual problem is my primary reason for supporting this PLIP, even if it was not really in reply. :)

comment:11 Changed 7 years ago by pupq

The vast majority of ordinary integrators use the universal installer--which gives you Python. And they don't use that Python for anything else.

So, even if our answer is "re-run the universal installer and copy your buildout.cfg over and re-run buildout", that's a reasonable solution for 98%.

Now, of course, they don't benefit from the new Python, either.

Would it, in fact, be required to use Py2.6? Or just optional? If it's required, are there almost all of the typical add-ons (database adapters, LDAP libraries, etc, for it?) *That's* more the challenge than "installing new Python".

comment:12 Changed 7 years ago by davisagli

  • Description modified (diff)
  • Summary changed from Require Python 2.6, Zope 2.12, and CMF 2.2 for Plone 4.0 to Require Python 2.5 or 2.6, Zope 2.12, and CMF 2.2 for Plone 4.0

I updated the description to include some more information on benefits and compatibility/risks, based on the discussion so far.

comment:13 Changed 7 years ago by erikrose

  • Owner davisagli deleted

Clearing Owner field of 4.0 PLIPs so we can use it to mean "implementor". (Many of these owners were automatically assigned from choosing a Component that had a default owner.)

comment:14 Changed 7 years ago by smcmahon

  • Cc plip-advisories@… added

comment:15 Changed 7 years ago by davisagli

  • Owner set to davisagli

I abstain from the FWT vote on this since I'm the primary implementer (or backporter, anyway).

comment:16 Changed 7 years ago by rossp

My FWT vote is +1.

I would like to see some thought put in to try and anticipate some of the different packaging/OS issues that might surprise users/integrators in ways they might not be accustomed to troubleshooting. For example, might some users with non-installer buildouts have a OS python package installed for 2.4 but not for 2.5 or 2.6. Some might then get confused when just re-bootstrapping their with the new python and re-running the buildout doesn't work. I'm sure we'll discover other, similar cases and putting a little effort into a "If foo happens, check bar and try qux" FAQ could yield significant value in terms of user perceptions and experience.

comment:17 Changed 7 years ago by MatthewWilkes

FWT Vote: +1

comment:18 Changed 7 years ago by raphael

FWT vote: +1

This is a "must have" IMHO

comment:19 Changed 7 years ago by calvinhp

FWT Vote: +1

comment:20 Changed 7 years ago by erikrose

FWT vote: +1. I just spent the weekend writing in 2.6, and it was oh-so-nice. :-) I want the better memory management for production, the with statement, and the repaired exception syntax. Bring it on.

comment:21 Changed 7 years ago by esteele

Approved by FWT vote.

comment:22 Changed 7 years ago by davisagli

  • Status changed from new to assigned

Update:

  • As of last Tuesday, all the backporting work so far has been merged and is now the official Plone 4 branch. Bugs remain, no doubt, and should be filed in trac against the 4.0 milestone.
  • At this point, most tests are passing, even while using the same package as 3.3 for quite a few more minor packages. Packages with remaining test issues: CMFPlone (1 failure), kss.core (1 failure, 1 error), plone.app.customerize (2 failures), plone.app.layout (3 failures), plone.app.linkintegrity (25 failures), wicked (3 failures, 43 errors but that's b/c I haven't run the tests correctly yet), CMFEditions (6 failures, 1 error), CMFPlacefulWorkflow (1 error), PluggableAuthService (16 errors), SecureMailHost (5 errors but this will be removed by another PLIP). (These numbers may be a bit out of date.) Please feel free to help me track down these issues. Some packages will require a new release even though their tests are all passing already, in order to clean up deprecation warnings.
  • It would be helpful if someone could start trying out some common add-on products and start to document the ways in which they break, so we can document commonly needed changes and/or correct things to avoid them.
  • As of today, the branch now has PlonePAS from trunk, which includes some refactoring from Hanno to remove the group and member tool implementations from CMFPlone (the functionality has been merged into the versions of those tools in PlonePAS...essentially it's just collapsing some pointless class inheritance) and remove our dependency on GroupUserFolder.
  • I'm having second thoughts about the decision to remove most global template variables. We definitely should make Plone itself not depend on them being available, which is now the case, but I'm not sure it's worth the add-on product breakages this will cause. Hanno, you've told me before that the global_defines were a performance bottleneck...do you have any numbers?
  • Andreas Jung has proposed the following release schedule for Zope 2.12: Beta 3: July 14th, RC 1: August 6th, Final: September 4th. The main thing that could block this is the final ZODB 3.9 release.
  • Tres Seaver said CMF 2.2 can probably be released whenever Zope 2.12 is.

comment:23 Changed 7 years ago by grahamperrin

  • Cc grahamperrin@… added

comment:24 Changed 7 years ago by davisagli

I'm not sure about the best way to tackle review for this PLIP, given the unusual breadth of changes that are incorporated. I would appreciate feedback from the framework team on what information would be most helpful to facilitate that review. I will at least write up some basic pointers tomorrow.

comment:25 Changed 7 years ago by optilude

(In [29679]) Review refs #8808

comment:26 Changed 6 years ago by esteele

Please assist the doc team in creating/updating documentation relating to this PLIP. See #9620.

comment:27 Changed 6 years ago by esteele

  • Status changed from assigned to closed
  • Resolution set to fixed

comment:28 Changed 4 years ago by davisagli

  • Component changed from Infrastructure to General
Note: See TracTickets for help on using tickets.