Ticket #13866 (new Bug)
WorkflowTool updateRoleMappings performance
Reported by: | thepjot | Owned by: | |
---|---|---|---|
Priority: | minor | Milestone: | 4.x |
Component: | Unknown | Version: | 4.3 |
Keywords: | workflow, migration, performance | Cc: |
Description
I've only tested this with Plone 4.0.9 so far, but I assume this will apply to later versions as well. Source of the problem is in Products.CMFCore (2.2.4) tho. But could be fixed with an override in CMFPlone if need be.
Background
When upgrading plone, one of the steps is to introduce the "Site Administrator" role with a given set of permissions. Following this addition, the workflowtool is triggered to 'updateRoleMappings'.
Problem
For our customer, I noticed this took over an hour (7370 portal catalog records, 9.4GB of blobstorage). I also noticed that the blobcache was being filled. (Which can be countered with 'zeo-client-blob-cache-size')
The culprint appears to be the 'reindexObject' method, or more exactly, the metadata. In the workflow tool we explicitly reindex the 'allowedRolesAndUsers' index. But without explicit mention, this also means the entire metadata sheet is re-made. Which causes the whole file to be loaded to fill in the required fields.
Fix
This can be averted by adding a 'update_metadata=0' keyword to the reindexObject in the '_recursiveUpdateRoleMappings' function. Which speeds up the performace significantly. Added a file with the function as I made it. Only added the update_metadata keyword.
Use case
1) Change workflow for Image/File to 'one_state_workflow' 2) Update security settings 3) Change workflow for Image/File to 'intranet_workflow' 4) Update security settings (I time this)
I did this twice, once with the default behaviour. Once with a minor change.
Test 1: TO 'intranet_workflow' WITH metadata (default)
1515 object(s) updated. updateRoleMappings function took 79624.944 ms
Test 2: TO 'intranet_workflow' WITHOUT metadata (change)
1515 object(s) updated. updateRoleMappings function took 17730.919 ms
Conclusion
This is quite a significant change. To my knowledge, there is nothing in the metadata set that is used for workflow, security or permission purpose.
Attachments
Change History
Changed 2 years ago by thepjot
-
attachment
recursiveUpdateRoleMappings.txt
added
comment:1 Changed 2 years ago by thepjot
Forgot to mention, this also occurs in Plone 4.3.2 since I upgraded our customer that version.
comment:3 follow-up: ↓ 4 Changed 2 years ago by maartenkling
i like, can you make a pull request for this?
comment:4 in reply to: ↑ 3 Changed 2 years ago by thepjot
Replying to maartenkling:
i like, can you make a pull request for this?
Override function in CMFPlone?
comment:5 Changed 2 years ago by maartenkling
Yes, if you make a pull request in Products.CMFPlone it can be reviewed and merged if okay
comment:6 Changed 2 years ago by thepjot
Hi Maarten,
I made a pull request; https://github.com/plone/Products.CMFPlone/pull/186 But think I picked the wrong branch, figure I need to go for 4.3.X ?
comment:8 Changed 2 years ago by thepjot
Update, made a new pull request for the right branch: https://github.com/plone/Products.CMFPlone/pull/187
comment:9 Changed 2 years ago by thepjot
Check :)
For master: https://github.com/plone/Products.CMFPlone/pull/186
For branch 4.3.x: https://github.com/plone/Products.CMFPlone/pull/187
comment:10 follow-up: ↓ 11 Changed 2 years ago by thepjot
Seems I was wrong. There is a 'allowedUsersAndGroups' in the metadata, which is used in various places. Making it required that that field is updated..
comment:11 in reply to: ↑ 10 Changed 2 years ago by stevenlooman
Replying to thepjot:
Seems I was wrong. There is a 'allowedUsersAndGroups' in the metadata, which is used in various places. Making it required that that field is updated..
Can you show me where allowedUsersAndGroups is referenced?
comment:12 Changed 2 years ago by thepjot
I just noticed today that that was in custom code I made for a client. So my fix is still valid, haven't had time to redo the pull.
comment:13 Changed 2 years ago by stevenlooman
Great!
comment:14 Changed 2 years ago by thepjot
Okay, re-did my pull requests.
For 4.x.3: https://github.com/plone/Products.CMFPlone/pull/191
For master: https://github.com/plone/Products.CMFPlone/pull/192
fixed function