Ticket #10778 (closed PLIP: fixed)

Opened 6 years ago

Last modified 5 years ago

Standalone UUID implementation

Reported by: optilude Owned by: optilude
Priority: minor Milestone: 4.1
Component: General Version:
Keywords: Cc: plip-advisories@…

Description (last modified by optilude) (diff)

Motivation

Many applications require some notion of a universally unique ID. For example:

  • Archetypes/Kupu-style link-by-uid
  • References to content from non-content items, e.g. tiles/portlets
  • Disambiguation of content, e.g. in export/import scenarios
  • Effective tokenisation of content for vocabularies and selection widgets

Archetypes has a solution its UID() attribute, but the implementation is AT-specific and rather heavyweight, requiring the interaction of three separate catalogues (portal_catalog, reference_catalog, uid_catalog).

zope.intid/five.intid provide more generic integer ids, but these are not universally unique and so not stable across sites.

CMFUId is a similar, CMF-specific solution.

This PLIP proposes a lightweight, simple UUID mechanism that is independent of any content implementation. The aims are to have:

  • A standards-based UUID implementation, with
  • a simple, intuitive API, and
  • standard utility views

Assumptions

Plone wants to support content not exclusively built with Archetypes.

Proposal & Implementation

  • A small package, plone.uuid, with minimal dependencies
  • A package plone.app.uuid for Zope 2/Plone specific stuff
  • An adapter interface, IUUID, to obtain a UUID for a content item.
  • UUIDs are generated using the standard library uuid module
  • A marker interface, 'IAttributeUUID', which lets types opt into an on-create event handler, which allocates a UUID and stores it
  • The default implementation stores UUIDs in an attribute on the content object (an annotation would be less efficient, since it'd be in a separate persistent object)
  • Some utility views are registered:
    • @@uuid, a view on any IUUIDAware item returning a UUID
    • @@redirect-to-uuid, uses sub-path traversal to look up a UUID, returning a 304 response
  • Archetypes is updated to support the IUUID protocol and generate UUIDs in the same way for new content (only). No migration will be required. The UID() method will become a shim for IUUID(self). Code should be updated to use IUUID instead if it needs to be compatible with non-AT content. It's hoped that this can make it possible for AT content to make simple references to non-AT content

Deliverables

  • The plone.uuid and plone.app.uuid packages
  • Updates to Products.Archetypes
  • Minor changes to Products.CMFPlone
  • Documentation

Risks

Participants

Martin Aspeli, and hopefully others :-)

Progress

  • PLIP submitted
  • PLIP implemented
  • PLIP reviewed

Change History

comment:1 Changed 6 years ago by davisagli

Overall this looks good. While working on Dexterity we keep coming across new features for which a true UUID (as opposed to a site-specific intid like five.intid provides) would be useful. And the details reflect the discussion that occurred regarding UUIDs on the Dexterity mailing list following the Tahoe sprint.

Are you explicitly excluding from the scope of this PLIP the task of updating existing features that are based on Archetypes UIDs and five.intid ids to use the new UUID implementation instead? If so, have you at least done any thinking regarding a timeline for when or in what way that could happen?

comment:2 Changed 6 years ago by optilude

Hi David,

I think changing Archetypes' UID() method, reference engine and catalogues needs to be out of scope for this PLIP. It'd simply be too invasive and too likely to break existing assumptions. However, the "new" UUID implementation should work with AT as well.

I'm not sure how many other things use intids internally. z3c.relationfield does, but for performance that's probably the right thing. I imagine that any import/export mechanism for references (e.g. transmogrifier pipeline sections) would need UUIDs, though.

I'd like to consider updating TinyMCE to use the "new" link-by-uid here, if feasible. It'd need Rob's help, though.

Martin

comment:3 follow-up: ↓ 5 Changed 6 years ago by davisagli

I'm actually thinking of submitting a PLIP to factor the link-by-uid and captioning transforms out of kupu and tinymce into a separate package. If I do that I'll coordinate with you and consider supporting the new UUIDs.

comment:4 Changed 6 years ago by optilude

Hi David,

That makes sense: They don't belong in Kupu, for sure, and shouldn't be tied to AT.

Martin

comment:5 in reply to: ↑ 3 Changed 6 years ago by hannosch

Replying to davisagli:

I'm actually thinking of submitting a PLIP to factor the link-by-uid and captioning transforms out of kupu and tinymce into a separate package. If I do that I'll coordinate with you and consider supporting the new UUIDs.

Note that the "link-by-uid" feature is LinguaPlone / translation aware. If you submit a PLIP for this, I'd be happy to help out with that part and make sure it continues to work.

comment:6 Changed 6 years ago by hannosch

+1 on the general idea. I think it makes a good deal of sense to introduce this new mechanism. We can later try to update some of the existing consumers of AT's UID() to the new approach. Separate PLIP's for CMFEditions / CMFUid and plone.app.iterate come to mind - but these are outside the scope of the basic work here.

comment:7 Changed 6 years ago by optilude

I've just checked in plone.uuid, which implements most of this.

Still to come:

  • Indexing (via plone.indexer) and catalogue index
  • The @@redirect-to-uuid view
  • The @@resolve-uuid view
  • Marking of Products.CMFCore.DynamicType.DynamicType with IAttributeUUID

The first three all have a dependency on the catalog. I think these either need to live in Products.CMFPlone, or in David's proposed "link-to-uid" package. The alternative would be a new plone.app.uuid package.

comment:8 Changed 6 years ago by optilude

See: r38386

comment:9 Changed 6 years ago by optilude

(In [38408]) Initial checkin. Basic implementation is working. Refs #10778

comment:10 Changed 6 years ago by optilude

As per the checkin above: I've made a plone.app.uuid that integrates plone.uuid into the catalog.

It contains the @@redirect-to-uuid view. I'm not quite convinced anymore that @@resolve-uuid makes sense as a view. I have added some utility methods instead, including a uuidToURL() that'd be useful in building a link-by-uid style transform.

That means at least the basic implementation of this PLIP (sans migration for existing content and integration into Products.CMFPlone) is more or less done.

comment:11 Changed 6 years ago by davisagli

  • Owner set to optilude

comment:12 Changed 6 years ago by ldr

We need to ensure that AT's UID() method returns the same UUID as @@uuid, that means paying attention to the migration and AT's uid assignment for new types.

comment:13 Changed 6 years ago by optilude

Hi Laurence,

Why do we need to make sure AT's UID() returns the same UUID as @@uuid?

For this PLIP at this point in time, I think it's too risky to change the AT internals. My intention was for the AT UID/uid_catalog/reference_catalog implementation to remain untouched for now, and perhaps then submit a "migration" PLIP for 4.2. Do you think this is a bad idea?

Martin

comment:14 Changed 6 years ago by esteele

Your PLIP has been accepted for consideration for Plone 4.1.

Framework Team voting on this PLIP was: Alec +1 Craig +0 Elizabeth +0 Laurence +1 Martijn +.5 Matthew +1 Rob +1 Ross +.5

The proposal was accepted with the condition that no migration is required for existing AT content (i.e., an IUUID adapter for AT content is implemented to look up the UID()), and that the existing UID catalog is used.

The initial implementation deadline for your PLIP is October 1st, 2010. The Framework Team would certainly appreciate you finishing beforehand so that they may begin evaluating it as soon as possible. Announce its readiness here once your implementation is ready for review.

comment:15 Changed 6 years ago by optilude

As per r39259, plone.uuid now uses plone.app.testing for test setup. See also #10846.

comment:16 Changed 6 years ago by optilude

(In [39393]) Shift the focus of this package a little.

We now re-use the UID catalog metadata/column that is already used for Archetypes content. We still register an indexer for this, which will work with any IUUIDAware content, but we don't need to install a new catalog index anymore.

The rationale behind this is that Archetypes 1.7 (or at least the plip10778-plone.uuid branch, which will hopefully be part of 1.7) now has its own plone.uuid integration, using it to generate new UUIDs and return UUIDs from its UID() function (which basically just becomes a BBB shim around IUUID(object)). By using the existing catalog setup, we can avoid Archetypes migration and make compatibility with old code feasible.

Refs #10778.

comment:17 Changed 6 years ago by optilude

Note: To test this PLIP, use the PLIP #9938 buildout. This was done because #9938 also uses plone.uuid.

The buildout file is in plips/plip9938-output-filters.cfg

comment:18 Changed 6 years ago by optilude

(In [39400]) Add checkouts for PLIP refs #10778 to this buildout. Also refs #9938.

comment:19 Changed 6 years ago by optilude

(In [39430]) Use UUID4 instead of UUID1, since UUID1 can compromise privacy. Refs #10778

comment:20 Changed 6 years ago by optilude

If this is merged, I'd like to also update the following places to use the IUUID() pattern:

  • archetypes.referencebrowserwidgets's browser/view.py has a 'getUidFromReference()' method that uses obj.UID()
  • plone.app.contentrules's handlers.py does 'getattr(obj, 'UID', None)'
  • TinyMCE has a lot of code related to link-by-UID. However, I'm hoping that will be fixed by #9938.
  • TinyMCE's Upload.py does something like 'return self.okMessage("resolveuid/%s" % (obj.UID()))' if link-by-uid is enabled

I think that this, plus the changes already made on the AT branch, would allow AT content to establish references (at least simple references) to any content item providing the IUUID() protocol, whether or not it implements IReferenceable, so long as that content is also indexed in the uid_catalog and reference_catalog. It should be pretty straightforward to implement a package to add support for that to Dexterity content, for example.

Martin

comment:21 Changed 6 years ago by esteele

A note that ATCT's topic will need to be updated to not use the uid_catalog for reference criteria as well. Or, if #10902 is approved, it'll need to happen there.

comment:22 Changed 6 years ago by robgietema

Is this PLIP ready for review? If it is please add a buildout for it.

comment:23 Changed 6 years ago by optilude

Use the 9938 buildout. See above for details.

comment:24 Changed 6 years ago by robgietema

thanks Martin, sorry for not reading the above comments more careful.

Review added:  http://svn.plone.org/svn/plone/buildouts/plone-coredev/branches/4.1/plips/plip10778-review-robgietema.txt

comment:25 Changed 6 years ago by cah190

(In [40123]) Review for PLIP 10778. Refs #10778.

comment:26 Changed 6 years ago by cah190

  • Cc plip-advisories@… added

comment:27 Changed 6 years ago by alecm

(In [40362]) Added review for PLIP 10778. Refs #10778

comment:28 Changed 6 years ago by davisagli

The test failure in archetypes.referencebrowserwidget looks really trivial -- it simply uses a regexp that needs to be updated to match UIDs with dashes in them.

comment:29 Changed 6 years ago by alecm

(In [40410]) One more review note. Refs #10778

comment:30 Changed 6 years ago by optilude

Note: the catalog.xml in plone.app.uuid was deleted, but I forgot to 'svn rm' it. Sorry for the confusion!

Martin

comment:31 Changed 5 years ago by optilude

I just wanted to address a few other comments from the review:

"- The test code is in testing.py and tests.py, I prefer this being in a tests

folder to not mess up the package root." (Rob)

testing.py is a plone.testing (and ZTK) convention. Please see plone.testing for rationale.

"- The @@redirect-to-uuid view in plone.app.uuid throws an exception if

no UUID is provided, perhaps it should return something like a 400 Bad Request or a 404 Not Found instead of raising a KeyError?" (Craig)

I think it should return a 404 (doesn't it already?) if a UUID is given and not found. If it isn't, that's an error, so raising an exception seems reasonable to me. A 404 is misleading, since the view was found, it just wasn't used correctly. I'm not sure anyone would know what to do with a 400 request?

"- The @@redirect-to-uuid view causes a redirection loop to

@@redirect-to-uuid/None if an invalid/missing UUID is provided. I would expect a 404 Not Found if the UUID cannot be resolved." (Craig)

Indeed - I'll look into this.

"- There is a failure in archetypes.referencebrowserwidget which

appears to be caused by some change in the PLIP #9938 buildout." (Alec)

I'll look into this too upon merging.

"- It is not clear to me what the purpose of the IUUIDAware interface

is. Why aren't IUUID and IAttributeUUID be sufficient?" (Alec)

Some views are registered on IUUIDAware, so it's a nice generic implement. IAttributeUUID implies a specific implementation of UUID assignment and storage and enables an event handler for object creation. More specifically: IUUIDAware applies to AT objects, but IAttributeUUID does not.

"- The plone.app.uuid code is straightforward and well tested. I would

prefer if the utility functions defined in plone.app.uuid.utils followed PEP8 naming conventions." (Alec)

If you're referring to using names_with_underscores(), then I think that would be wrong. The Zope/Plone convention is mixedCase(). We went through a lot of pain in Dexterity starting with the former and having to migrate to the latter as the inconsistency just became too arbitrary and confusing.

"- There is a test in plone.app.uuid for a browser view defined in

plone.uuid, this is unconventional." (Alec)

I think this was due to difficulties in coming up with a good enough test fixture in plone.uuid without adding too many dependencies or assumptions. Or laziness. Or both. :)

"- It would be nice to see the manage_afterAdd and manage_afterCopy

methods in AT's Referenceable class go away in favor of an event based system for managing uuids. This would bring the AT IUUID implementation a little closer to the default one." (Alec)

Agree, but I think this is outside the scope of the PLIP.

comment:32 Changed 5 years ago by optilude

  • Description modified (diff)

comment:33 Changed 5 years ago by optilude

As a bit of a note to self, these are some other files that may need updating to use the IUID() pattern:

./archetypes/kss/fields.py:        uid = aq_inner(self.context).UID()
./archetypes/referencebrowserwidget/browser/view.py:        return getattr(aq_base(item), 'UID', None)
./archetypes/schemaextender/extender.py:                key = context.UID() or str(id(context))
./plone/app/contentrules/handlers.py:        uid_method = getattr(obj, 'UID', None)
./plone/app/iterate/copier.py:        new_baseline._setUID( baseline.UID() )
./plone/app/iterate/relation.py:            wc.addReference( ref.targetUID, ref.relationship, referenceClass=ref.__class__ )
./plone/app/linkintegrity/info.py:        deleted = set([obj.UID() for obj in self.getDeletedItems()])
./plone/app/linkintegrity/info.py:        deleted.update([obj.UID() for obj in breaches])
./plone/app/linkintegrity/info.py:                if source.UID() in deleted:
./Products/ATContentTypes/criteria/base.py:    def _uncatalogUID(self, *args, **kwargs): pass
./Products/Marshall/namespaces/atns.py:        value = getattr(instance, atcfg.UUID_ATTR, "")
./Products/Marshall/namespaces/atns.py:        existing = getattr(instance, atcfg.UUID_ATTR, _marker)
./Products/Marshall/namespaces/atns.py:            instance._setUID(at_uid)
./Products/TinyMCE/adapters/Upload.py:            return self.okMessage("resolveuid/%s" % (obj.UID()))
./Products/TinyMCE/browser/url.py:    def getPathByUID(self):

comment:34 Changed 5 years ago by toutpt

This one is done: ./archetypes/referencebrowserwidget/browser/view.py: return getattr(aq_base(item), 'UID', None)

comment:35 Changed 5 years ago by toutpt

I have branched and fixed most of the place left without IUUID patterns

changesets:

Plone 46374 46375 46376 46379 46381 46382 46387

Archetypes 13358 13359 13360 13361 13362

Now we have two places where the code use _setUID. Quite hard but plone.uuid doesn't support this at the moment. I have discussed about this with Martin and here is the conclusion: We need to add a IMutableUUID component with a set method and implements it for Archetypes and Dexterity

I'm working on this now

comment:36 Changed 5 years ago by toutpt

Well I have developped the MutableUUID adapter but ...

_setUID is doing many more things than just set the UID. And I don't understand it. It's about references and I really don't know how references will be managed. So I will not update the code which use _setUID at the moment.

There are nothing more to do to implement the plip plone.uuid, all plone and archetypes code is plone.uuid aware.

comment:37 Changed 5 years ago by esteele

  • Status changed from new to closed
  • Resolution set to fixed

Merged.

comment:38 Changed 4 years ago by davisagli

  • Component changed from Infrastructure to General
Note: See TracTickets for help on using tickets.