Ticket #8814 (closed PLIP: fixed)

Opened 7 years ago

Last modified 6 years ago

Replace SecureMailHost with a standard Zope mailhost

Reported by: hannosch Owned by: alecm
Priority: n/a Milestone: 4.0
Component: General Version:
Keywords: Cc: rossp, plip-advisories@…

Description (last modified by alecm) (diff)

Proposed by: Alexander Limi
Seconded by: Alec Mitchell

Overview

Plone 2.1 introduced its own mailhost implementation to support TLS encryption. Recent Zope versions (2.11+) include a zope.sendmail based mailhost, which provides equivalent functionality.

We should remove SecureMailHost and replace it with a standard Zope2 mailhost.

Motivation

SecureMailHost is used exclusively by Plone, provides a unique API, and is not compatible with Zope 3 mail API and even some Zope 2 applications. Because the limited additional features of SecureMailHost are now available in the standard Zope mailer, it no longer makes sense to maintain our own legacy MailHost implementation.

Additionally, zope.sendmail is extensible and allows features like asynchronous queued mail sending (similar to MailDropHost), which are not directly possible with SecureMailHost.

Risks

  1. The persistent SecureMailHost and any mail related settings in existing sites will need to be migrated.
  2. Some existing sites may have alternative MailHost implementations installed. We should not attempt to migrate these automatically; However, we will need to document the process of migrating common MailHost replacements (e.g. MaildDropHost).
  3. 3rd party products which make explicit use of the unique SecureMailHost API will need to be updated to use the standard MailHost or zope.sendmail API. This will require migration documentation aimed at product developers. Such cases are, hopefully, somewhat rare. It may make sense to check for such issues in common 3rd party products which use mail facilities, and alert their authors before final release.
  4. This feature requires an updated Zope. In the very unlikely event that Plone 4.0 is not able to ship with Zope >= 2.11, this change will not be possible.

Progress

Much of the work has already been completed on Plone trunk by the improbably productive Hanno Schlichting. He summarizes:

SecureMailHost has been replaced by a Zope2 mailhost and Plone has been adjusted 
to use the standard API. The controlpanel screens have been updated and tested 
and the GS handlers been reviewed for compatibility.

The primary remaining development task is developing an automated migration of existing SecureMailHost instances and their settings to the new Zope mailhost. Documentation for updating 3rd party code and for migrating non-SecureMailHost sites also needs to be produced.

Participants

Alec Mitchell
Hanno Schlichting (whose work is already completed)

Change History

comment:1 Changed 7 years ago by hannosch

  • Owner set to hannosch

comment:2 Changed 7 years ago by hannosch

  • Description modified (diff)

comment:3 Changed 7 years ago by rossp

  • Cc rossp added

comment:4 Changed 7 years ago by hannosch

  • Description modified (diff)

comment:5 Changed 7 years ago by hannosch

  • Component changed from Mail to Infrastructure

comment:6 Changed 7 years ago by limi

  • Priority changed from minor to n/a
  • Owner changed from hannosch to alecm
  • Description modified (diff)
  • Milestone changed from 5.0 to 4.0

Alec Mitchell volunteered to take this on for 4.0:

On Sun, Jun 21, 2009 at 5:56 PM, Alec Mitchell wrote:

I'm willing to work on this. Is there already a PLIP for it that we can shift to Plone 4.0?

Assigning it to 4.0, I assume Alec will expand the PLIP when he wakes up in a few hours. :)

comment:7 Changed 7 years ago by alecm

  • Description modified (diff)

comment:8 Changed 7 years ago by hannosch

Things to watch out for: the secureSend and send methods have the "message to (mto)" and "message from (mfrom)" arguments in a different order as the second and third argument. Makes it easy to overlook or forget to adjust those.

One point I didn't test as much or check sufficiently is the handling of different kind of mail text formats. There might be differences in handling mime, plain text, some Python email.Something objects and so on and naturally charsets and encodings are always some more fun. There's a bit of an opportunity for better test coverage here ;)

comment:9 Changed 7 years ago by erikrose

  • Owner alecm deleted

Clearing Owner field of 4.0 PLIPs so we can use it to mean "implementor". (Many of these owners were automatically assigned from choosing a Component that had a default owner.)

comment:10 Changed 7 years ago by alecm

  • Owner set to alecm

comment:11 Changed 7 years ago by smcmahon

  • Cc plip-advisories@… added

comment:12 Changed 7 years ago by davisagli

FWT vote: +1

comment:13 Changed 7 years ago by MatthewWilkes

FWT Vote: +1

comment:14 Changed 7 years ago by rossp

FWT vote: +1

comment:15 Changed 7 years ago by alecm

Abstaining from my PLIP

comment:16 Changed 7 years ago by raphael

FWT vote: +1

comment:17 Changed 7 years ago by calvinhp

FWT Vote: +1

comment:18 Changed 7 years ago by erikrose

Does SecureMailHost presently fire off DeprecationWarnings? If not, I think we should before ripping it out.

FWT vote: -1 until some deprecation warnings are added in some (possibly 3.x) release.

comment:19 Changed 7 years ago by alecm

I think the only method we'd have to deprecate is "secureSend". I'll probably end up patching that into our new mailhost along with a deprecation warning. The patched method could then be removed in a later release. There are also some convenience regexps and similar in SMH that developers may be importing and relying on. We may want to consider providing some BBB imports for those, since they can be quite convenient.

comment:20 Changed 7 years ago by esteele

Approved by FWT vote.

comment:21 Changed 7 years ago by davisagli

Hanno removal of SecureMailHost on trunk is in r24623, FWIW.

comment:22 Changed 7 years ago by alecm

  • Status changed from new to assigned

comment:23 Changed 7 years ago by alecm

(In [28664]) Branch for PLIP #8814 SecureMailHost removal implementation (refs #8814)

comment:24 Changed 7 years ago by alecm

(In [28690]) Begin PLIP #8814 implementation by merging r24622:24623 from trunk and removing secureSend references (refs #8814)

comment:25 Changed 7 years ago by alecm

(In [28712]) Branch for PLIP 8814 implementation, would do this on 2.0 branch but MailHost is currently broken in numerous ways. refs #8814

comment:26 Changed 7 years ago by alecm

(In [28718]) Add SMH BBB monkey patches (refs #8814)

comment:27 Changed 7 years ago by alecm

(In [28719]) Add PLIP 8814 buildout config and documentation (refs #8814)

comment:28 Changed 7 years ago by alecm

(In [28919]) Branch plone.app.upgrade for PLIP 8814 (refs #8814)

comment:29 Changed 7 years ago by alecm

(In [28921]) Branch plone.app.upgrade for PLIP 8814 (refs #8814)

comment:30 Changed 7 years ago by alecm

(In [28925]) Add migrations and migration tests for PLIP 8814 (refs #8814)

comment:31 Changed 7 years ago by alecm

(In [28926]) Update PLIP #8814 buildout with new branches and document status. (refs #8814)

comment:32 Changed 7 years ago by alecm

(In [28976]) PLIP #8814 is ready for review (refs #8814)

comment:33 Changed 7 years ago by davisagli

(In [29351]) add FWT review for PLIP 8814 (refs #8814)

comment:34 Changed 7 years ago by alecm

(In [29380]) Add toolset migration step for MailHost (refs #8814)

comment:35 Changed 7 years ago by alecm

David,

Could you send some info about the original MailHost settings you migrated? Was the original set to support SSL/TLS connections, what was the port? The new mailer will use TLS automatically if the server supports it, so perhaps the migration should not bother to set the force TLS flag based on the original value. TLS seems to be working here with gmail using port 25.

comment:36 Changed 7 years ago by davisagli

"Disable TLS" was off in the old MailHost (so TLS was on, I guess), and it was using localhost port 25. I can send you the Data.fs; but it's really just a stock unmodified Plone 2.5 install.

comment:37 Changed 7 years ago by alecm

OK, it seems the TLS semantics for SecureMailHost were different than I had surmised. It's probably incorrect to assume that if "Disable TLS" is off, that "Force TLS" should be on. I'll just ignore that flag in the migration and keep the "Force TLS" flag unset by default, since there seems to be no way to actually "Disable TLS" in MailHost. Nobody really wants to disable TLS anyway I'd guess, and MailHost seems to do a good job of figuring out whether it can use it or not on its own.

comment:38 Changed 7 years ago by alecm

I've fixed the TLS issue and added the toolset registry migration. Unfortunately, tests are failing now due to updates on Zope trunk. I don't see a way to pin a repository to a particular version with mr.developer, but r102819 of Zope trunk works. Hopefully, the changes can be merged soon.

comment:39 Changed 7 years ago by rossp

I updated Zope2 to r102819 but still couldn't get the load tests to run. See "Installing coreloadtests" in  http://weblion.psu.edu/static/loadtesting/plone4.0/plip8814-replace-securemailhost.log. I verified after the error that Zope2 was in fact at the right revision.

comment:40 Changed 7 years ago by alecm

(In [29745]) Update plip buildout and description to reflect changes merged into Zope 2.12 (refs #8814)

comment:41 Changed 7 years ago by esteele

Your PLIP has been reviewed by the Framework team. Feel free to discuss any suggested changes either here in the PLIP ticket or on the mailing lists. Final deadline for this PLIP is set for September 30.

comment:42 Changed 7 years ago by alecm

(In [30021]) The MailHost would prefer an empty string to None. refs #8814

comment:43 Changed 6 years ago by alecm

(In [30025]) Update documentation. This PLIP is ready for final review/merge. refs #8814

comment:44 Changed 6 years ago by davisagli

(In [30034]) update review, refs #8814

comment:45 Changed 6 years ago by rossp

FWT vote: +1 for merge

comment:46 Changed 6 years ago by esteele

This PLIP has been accepted for merging into Plone 4.0

The final vote was:

Alec Mitchell - David Glick 1 Erik Rose 1 Laurence Rowe 1 Matthew Wilkes - Ross Patterson 1

Please merge your branches into the Plone 4.0 head by end-of-day Friday Oct 16. If you need assistance with merging, please contact me.

We'll be assigning a documentation ticket to this PLIP shortly. Please assist the docs team in documenting the changes and new features that this PLIP introduces.

comment:47 Changed 6 years ago by esteele

Please assist the doc team in creating/updating documentation relating to this PLIP. See #9600.

comment:48 Changed 6 years ago by alecm

(In [30546]) Merge r28921:30021 from plip8814-mailhost branch (refs #8814)

comment:49 Changed 6 years ago by alecm

Oops, that should have been r28712:28918 for plone.app.contentrules

comment:50 Changed 6 years ago by alecm

(In [30547]) Merge r28921:30021 from plip8814-mailhost branch (refs #8814)

comment:51 Changed 6 years ago by alecm

(In [30548]) Merged r28664:29350 from plip8814-mailhost branch (refs #8814)

comment:52 Changed 6 years ago by alecm

This PLIP is now fully merged.

comment:53 Changed 6 years ago by esteele

  • Status changed from assigned to closed
  • Resolution set to fixed

comment:54 Changed 6 years ago by witsch

(In [30913]) fix mailhost migration — portal_setup gets passed to migration handler, not the portal object itself (refs #8814)

comment:55 Changed 4 years ago by davisagli

  • Component changed from Infrastructure to General
Note: See TracTickets for help on using tickets.