Ticket #9352 (closed PLIP: fixed)
Improved search results
Reported by: | laurenskling | Owned by: | spliter |
---|---|---|---|
Priority: | major | Milestone: | 4.2 |
Component: | General | Version: | |
Keywords: | search | Cc: | plip-advisories@…, dthomas218, potzenheimer, eleddy, ggozad |
Description (last modified by elvix) (diff)
This PLIP is a combination of the two accepted PLIP's about the search results page. Tickets #9271 and #9282 by Laurens Kling and Geir Baekholt.
Motivation
“Search-dominant” navigating is one of the three popular ways to reach goals on a website. Search isn't just useful for finding content, it's how many people navigate. The search of Plone could do better to help these people. Here are some small adjustments that could help out big time. (this PLIP is only about normal search, not advanced or live search). Plone uses search extensively. The search forms and results have, however, not been improved in a long time. There are several small steps that can be done to make the presentation of search results much more comfortable and practical to use.
Proposal
- Location (path/URL) One of the main things the search results are missing is the location of the item. If you archive old items in folders called 'archive', try searching that one archive folder you are looking for. Adding the location (URL) gives users feed forward about the item. Google has a nice url display in green. This one can possibly (depending on performance impact) be improved even further by listing the titles of containing folders rather than their path.
- Relevance. I have never understood the relevance of showing the relevance. It just adds noise and doesn't help the user. Sorting by relevance helps, not printing the relevance.
- Search only in current section. Plone gives the opportunity to search within current section, which sadly is gone on the new plone.org. When checked, the resultpage doesn't give any feedback that the results are only from the certain section. It also doesn't give the opportunity to search again within this section. Adding a simple line states “searched in <section>” and a checkbox to search within it again could solve this.
- Sorting. The search results can't be sorted in any way at the moment. Sort by: relevance (i.e the default), date (newest first), alphabetically
- easy-to-read dates: "Last modified Nov 10, 2007 10:43 PM" is painful to read. "Last modified 2 weeks ago" makes sense when you read results and is more scannable.
- Left align the description with the title, so the results are less painful to read.
Deliverables
- A search results view that supports sorting
- Updated ZPT and CSS for presenting the search results
Risks
Potential of small performance loss by looking up real names and folder paths.
Search should stay as easy as possible. Adding checkboxes to (normal) search makes it bigger then “just type and press enter”. If the the checkbox is unchecked by default this is still a simple searchbar, so I do not see any problems. Sorting search results will make the page need adjustments, there has to come a good UI for it.
Progress
Implemented in plone.app.search
Attachments
Change History
Changed 7 years ago by alecm
-
attachment
Picture 4.png
added
Proposed improved search results presentation
comment:2 Changed 7 years ago by alecm
- Keywords plip-advisories@lists.plone.org added
Added Geir's attachments from #9271
comment:3 Changed 7 years ago by alecm
- Cc plip-advisories@… added
- Keywords plip-advisories@lists.plone.org removed
comment:5 Changed 7 years ago by jaroel
Maybe http://babel.edgewall.org/wiki/Documentation/dates.html#time-delta-formatting would be nice to format the time delta's?
comment:6 Changed 7 years ago by elvix
Removing the time-deltas from the scope of this PLIP for now. Too many considerations regarding i18n to introduce them on this level. They should be their own PLIP for wider usage in Plone.
comment:7 Changed 7 years ago by elvix
- Description modified (diff)
Ready for review (with some small bits still missing, noted in plip9352-search-results.txt)
comment:9 Changed 7 years ago by davisagli
I'm not signed up to do a full review of this PLIP, but I have been using it to benchmark plone.app.contentlisting by comparing the new search results to the old page template.
To be specific, I did the following:
- used collective.contentgenerator's Public Site profile to populate the site with 4000 items
- used ab -n 100 to benchmark the old search results (/search?SearchableText=Britain&b_size=2000) and the new ones (/@@search?SearchableText=Britain&b_size=2000) -- the large batch size here is because the new results don't support batching yet...in my case the search for 'Britain' gave about 500 results.
Results:
- old search.pt: 6.32 requests/sec.
- new search browser view: 1.09 requests/sec. Ouch.
Some other observations:
- the results are not batched
- the results omit the standard coloring of items based on workflow state
Given the rather large performance penalty of the new approach, you might want to consider branching this PLIP and doing an implementation that doesn't rely on IContentListing. (Caveat: I haven't done detailed profiling to confirm that the new content listing approach is the culprit.)
comment:10 Changed 7 years ago by elvix
The reason the template takes longer to render is not IContentListing itself, but the fact that all the results are rendered, in the absence of batching.
Batching support is on its way. We have just been having some discussions on how to make this work in the cleanest nicest possible way.
The options are:
1) Provide an adapter from PloneBatch to IContentListing — This will remove your error, and make everything work sort-of like before, where the template author is responsible for making and managing the batch from a template. Ugly, but not worse than what we have today.
2) Make ContentListing import PloneBatch and support batching directly inside the listing by calling a batch() method on it. This will make batching simpler for integrators. It will no longer be necessary to mess with batch creation in the template. It will also give us predictable ways of getting all the sequences of content we might need batched or not from one place. Doesn't add anything except practicality, simplicity and predictability for integrators.
I am leaning towards (2) but that will need more code done in little time.
comment:11 Changed 7 years ago by davisagli
The reason the template takes longer to render is not IContentListing itself, but the fact that all the results are rendered, in the absence of batching.
I don't believe that that is the reason for the difference. As I said, I set the batch size very large so both the old search.pt and the new search view were rendering all the results.
comment:12 Changed 7 years ago by elvix
ouch. The results are not supposed to be that much slower by far. Must be a sneaky bug somewhere. I'll kill it.
comment:13 Changed 7 years ago by elvix
For the record:
I have found the bug causing the performance problems.
I have added a couple of crude performance tests with testbrowser to plone.app.search to show that performance is rather similar to the old search results page. These tests should probably be disabled when/if the plip is included in Plone as they are not very fast to run.
comment:14 Changed 7 years ago by davisagli
I found an error in my initial ab benchmark. I thought that I had convinced the old search page to not batch, but it was still doing so. Here are my results after temporarily modifying the old search.pt to not batch:
- old: 1.88 req/sec.
- new: 1.17 req/sec. (e.g. 38% slower)
However, in realistic cases search results will be batched. Here are my results after reverting to the original search.pt, then modifying the IContentListing-based search view to slice its catalog results to 30 before adapting to IContentListing (to simulate batching, since it is not supported yet by the search view):
- old: 6.21 r/s
- new: 5.26 r/s (15% slower)
It makes no significant difference if I slice the new search results *after* the IContentListing adaptation (as I would expect, since the implementation in plone.app.contentlisting doesn't adapt individual items to IContentListingObject until they are retrieved):
- old: 6.29 r/s
- new: 5.27 r/s
I'm still a bit concerned about even a 15% slowdown, given that the #1 feature request we have on plone.uservoice.com right now is "Improve overall speed," and given that content listings are an important and not very trivially cachable part of most sites.
comment:15 Changed 7 years ago by cjohansen
Accessibility check with FAE ( http://fae.cita.uiuc.edu/) identical with baseline.
comment:16 Changed 7 years ago by rossp
FWT review finished in r29733. There seems to be a lot of work to do yet, but if it gets done then I'm +1.
comment:17 Changed 7 years ago by davisagli
From my own benchmarking with ab, I suspect that the performance degradation Ross discussed in his review is due to the fact that the new search results do not do batching, while the old search results do. Batching really is essential; can the PLIP authors please comment on how soon we can expect to see it implemented?
comment:18 Changed 7 years ago by davisagli
Never mind; I didn't see that batching has been added since the last time I looked at the code. So I'm not sure what the cause of the performance issue is.
comment:19 Changed 7 years ago by esteele
comment:20 Changed 7 years ago by esteele
Your PLIP has passed the Framework team's initial review. Feel free to discuss any suggested changes either here in the PLIP ticket or on the mailing lists. Final deadline for this PLIP is set for September 30.
comment:21 Changed 6 years ago by rossp
Addressed:
- queryparser
Unaddressed:
- Noisiness
- Search only in current section
- Left align the description with the title
- To do's in plip9352-search-results.txt
- relevance
- spurious comments
- configurable
What has been done is a minor incremental improvement and much less than proposed, but it still has some value that may be worth including. It might, however, be better to postpone to 4.1 and get more completion.
I'm +0 for merge.
comment:22 Changed 6 years ago by elvix
Sorry. have been moving house and largely offline. The remaining work shouldn't be more than a couple of hours. Do i stand a chance if i get it done before the end of the week-end?
comment:23 Changed 6 years ago by elvix
Did the alignment and noisiness on the location formatting. The only really critical remaining missing feature is to respect the configurability i think.
comment:24 Changed 6 years ago by elvix
I won't have time to do more on this until tuesday/wednesday. Most of the remaining issues are very simple cut/paste+tests. Nothing dramatic, but i understand if we'll have to postpone to 4.1
comment:27 Changed 6 years ago by spliter
- Status changed from assigned to new
- Owner changed from elvix to spliter
During the Cathedral sprint we have decided to pick this and keep working on improving the search results page. Some main points that have to be done:
- merge "search results" and "advanced search form" into one template in order to let people to tweak their search terms without going back and forth. Switching the context just to tweak a search term is bad idea from UI point of view.
- when embedded into the search results, fields of "advanced search form" (further "filter") should behave more like a search filter: ajax calls for immediate update of the results, show/hide the fields in order to provide the quick access to the fields
- hiding/showing of the filter's fields should happen with jquery, but the solution should be still accessible for those without JS enabled in the browsers
- one non-trivial addition could be the breadcrumbs-like bar that would show current search terms used like google.com shows search options.
- when search terms are visible in a bar there should be a quick access to this particular field in the filter without need of showing the whole form. Form should be shown only in case the new search term (not still in the terms bar) has to be added.
Screenshot of a prototype and updates are coming.
comment:28 follow-up: ↓ 29 Changed 6 years ago by yurj
why don't add http://pypi.python.org/pypi/collective.searchtool?
In this way, we could add "per type" result customization, for example showing thumbnail for IATImage and so on.
comment:29 in reply to: ↑ 28 Changed 6 years ago by ldr
Replying to yurj:
why don't add http://pypi.python.org/pypi/collective.searchtool?
In this way, we could add "per type" result customization, for example showing thumbnail for IATImage and so on.
Per type / by adaption lead in customisation makes a lot of sense, not just for search results but also for the summary view for folder listings / collections. There are performance considerations though, especially for search - search results are currently rendered from the catalog result brain (or the equivalent from solr), so do not need the full object loading.
comment:30 Changed 6 years ago by limi
- Priority changed from minor to n/a
- Milestone changed from 4.x to 4.1
Moving to 4.1 at Geir's request.
comment:31 Changed 6 years ago by thet
having the search form searching for searchterms as substrings instead of whole word matches would also be a nice feature (as like appending wildcards to the searchterms like: *searchterm*). this way the behavior is more aligned to google's search behavior. currently (4.0) livesearch takes the searchterms as substrings, the searchform itself doesn't.
comment:32 Changed 6 years ago by rossp
- Summary changed from combination of search results plips to Improved search results
comment:33 Changed 6 years ago by esteele
Your PLIP has been accepted for consideration for Plone 4.1.
Framework Team voting on this PLIP was: Alec +1 Craig +1 Elizabeth +1 Laurence +1 Martijn +1 Matthew +1 Rob +1 Ross +1
The initial implementation deadline for your PLIP is October 1st, 2010. The Framework Team would certainly appreciate you finishing beforehand so that they may begin evaluating it as soon as possible. Announce its readiness here once your implementation is ready for review.
comment:34 Changed 5 years ago by khink
#11201 (Site property for batch size (search results and folder contents)) is related.
comment:35 Changed 5 years ago by spliter
The PLIP is ready for review with some TODO items left (listed in TODO.txt file of plone.app.search package).
comment:36 follow-up: ↓ 37 Changed 5 years ago by esteele
Looking great!
One note: modifying filters on the left and then clicking the big search button in the middle of the page does the search, but with the options in their previous state. Should we support this?
comment:37 in reply to: ↑ 36 Changed 5 years ago by esteele
Replying to esteele:
Scratch that, the "new items since" option sets itself to "yesterday" after clicking either "search" button.
comment:38 follow-up: ↓ 43 Changed 5 years ago by spliter
- Status changed from new to assigned
One question about architecture of this PLIP. Right now the JS that manages ajax calls and other search-related tasks is placed in Products.CMFPlone. Would it make sense to host that JS in p.a.search itself?
comment:39 Changed 5 years ago by timo
There seems to be a problem with plone.app.layout in the current buildout:
./bin/buildout -c plips/plip9327-contentlisting.cfg
timo@rudolf:~/workspace/plone/plone-4.1-coredev$ ./bin/instance fg 2010-11-27 14:10:37 INFO ZServer HTTP server started at Sat Nov 27 14:10:37 2010 Hostname: 0.0.0.0 Port: 8080 /home/timo/.buildout/eggs/Products.CMFCore-2.2.2-py2.6.egg/Products/CMFCore/FSObject.py:21: DeprecationWarning: RoleManager is deprecated. RoleManager is no longer part of AccessControl, please depend on Zope2 and import from OFS.role or use the new minimal RoleManager class from AccessControl.rolemanager. from AccessControl.Role import RoleManager 2010-11-27 14:10:40 INFO ZODB.blob (5997) Blob directory `/home/timo/workspace/plone/plone-4.1-coredev/plips/../var/blobstorage` is unused and has no layout marker set. Selected `bushy` layout. 2010-11-27 14:10:40 INFO ZODB.blob (5997) Blob temporary directory '/home/timo/workspace/plone/plone-4.1-coredev/plips/../var/blobstorage/tmp' does not exist. Created new directory. 2010-11-27 14:10:40 WARNING ZODB.blob (5997) Blob dir /home/timo/workspace/plone/plone-4.1-coredev/var/blobstorage/ has insecure mode setting Traceback (most recent call last): File "/home/timo/.buildout/eggs/Zope2-2.13.0-py2.6.egg/Zope2/Startup/run.py", line 72, in <module> run() File "/home/timo/.buildout/eggs/Zope2-2.13.0-py2.6.egg/Zope2/Startup/run.py", line 21, in run starter.prepare() File "/home/timo/.buildout/eggs/Zope2-2.13.0-py2.6.egg/Zope2/Startup/__init__.py", line 86, in prepare self.startZope() File "/home/timo/.buildout/eggs/Zope2-2.13.0-py2.6.egg/Zope2/Startup/__init__.py", line 259, in startZope Zope2.startup() File "/home/timo/.buildout/eggs/Zope2-2.13.0-py2.6.egg/Zope2/__init__.py", line 47, in startup _startup() File "/home/timo/.buildout/eggs/Zope2-2.13.0-py2.6.egg/Zope2/App/startup.py", line 118, in startup load_zcml() File "/home/timo/.buildout/eggs/Zope2-2.13.0-py2.6.egg/Zope2/App/startup.py", line 52, in load_zcml load_site() File "/home/timo/.buildout/eggs/Zope2-2.13.0-py2.6.egg/Zope2/App/zcml.py", line 46, in load_site _context = xmlconfig.file(site_zcml) File "/home/timo/.buildout/eggs/zope.configuration-3.7.2-py2.6.egg/zope/configuration/xmlconfig.py", line 653, in file context.execute_actions() File "/home/timo/.buildout/eggs/zope.configuration-3.7.2-py2.6.egg/zope/configuration/config.py", line 606, in execute_actions callable(*args, **kw) File "/home/timo/.buildout/eggs/AccessControl-2.13.3-py2.6-linux-i686.egg/AccessControl/security.py", line 165, in protectClass permission = getUtility(IPermission, name=permission_id) File "/home/timo/.buildout/eggs/zope.component-3.9.5-py2.6.egg/zope/component/_api.py", line 169, in getUtility raise ComponentLookupError(interface, name) zope.configuration.config.ConfigurationExecutionError: <class 'zope.component.interfaces.ComponentLookupError'>: (<InterfaceClass zope.security.interfaces.IPermission>, 'cmf.ModifyPortalContent') in: File "/home/timo/workspace/plone/plone-4.1-coredev/src/plone.app.layout/plone/app/layout/viewlets/configure.zcml", line 319.4-325.10 <browser:viewlet name="plone.lockinfo" manager=".interfaces.IAboveContent" class="plone.locking.browser.info.LockInfoViewlet" permission="cmf.ModifyPortalContent" for="plone.locking.interfaces.ITTWLockable" />
comment:40 Changed 5 years ago by spliter
Timo, you try to build with plip9327-contentlisting.cfg. Does it mean you have reported to the wrong plip? This plip is about search results, not contentlisting ;)
comment:41 Changed 5 years ago by timo
Yes, of course. This was meant to be for #9327, sorry. Btw: the new search results page is awesome! Great work!
comment:42 Changed 5 years ago by davisagli
comment:43 in reply to: ↑ 38 Changed 5 years ago by esteele
Replying to spliter:
One question about architecture of this PLIP. Right now the JS that manages ajax calls and other search-related tasks is placed in Products.CMFPlone. Would it make sense to host that JS in p.a.search itself?
I think that's a reasonable move.
comment:44 Changed 5 years ago by davisagli
Regarding this feedback in my review—
Trying to modify the criteria or sort order seems to lose all the results. Investigating further, it looks like this is because the 'Creator' and 'created' criteria are getting passed to the catalog in the wrong format. The default setting for the Creator criteria is (empty string), which filters out everything rather than including everything as it is intended. The date criterion appears to be querying for one single date rather than a date range.
—this was a side effect of Hanno's remove of the ZCatalog's support for parsing _usage parameters in the request. I adjusted it to use Zope's records marshalling instead, so it works now.
comment:45 Changed 5 years ago by spliter
Seems like I have messed up the branches - forgot to merge back my changes from branch to trunk so anybody who reviewed this PLIP recently could notice failing test and no AJAX in the search results page. In the buildout I see only review from Ross being done at this time (David's review has been done before). I fixed the issue and informed Ross about this. So whoever started reviewing the PLIP, please consider updating to see if the issues you had are gone now. Sorry.
comment:46 Changed 5 years ago by rossp
comment:47 Changed 5 years ago by rossp
comment:48 Changed 5 years ago by spliter
comment:49 Changed 5 years ago by spliter
comment:50 Changed 5 years ago by spliter
comment:51 Changed 5 years ago by spliter
comment:52 Changed 5 years ago by rossp
(In [46246]) More accessible GS profile title and description. Refs #9352.
Sorry, I should have been clearer about this in my review. I think we're trying to make sure that GS profile and descriptions are more human accessible and require less technical/developer knowledge to understand what they are. Rather than bug Denys again, I've just committed it myself, feel free to change it if I've gotten something wrong.
Of course, this is largely irrelevant now that the profile is a dependency of Plone's and has been hidden from the Plone UI as well but my sense is that it's still best practice.
comment:53 Changed 5 years ago by rossp
comment:54 Changed 5 years ago by eleddy
comment:55 follow-up: ↓ 56 Changed 5 years ago by rossp
(In [46338]) Add a review note about reusing plone.app.querystring. Refs #9352.
It occurrs to me that this PLIP could have and probably should have re-used the plone.app.querystring package created for plone.app.collection which provides their AJAX search query building. The stuff used by plone.app.collection is pretty slick and we should avoid multiple ways of doing things. It seems like this ship has already sailed so it may be too late but I wanted to note it.
comment:56 in reply to: ↑ 55 ; follow-up: ↓ 116 Changed 5 years ago by davisagli
Replying to rossp:
(In [46338]) Add a review note about reusing plone.app.querystring. Refs #9352.
It occurrs to me that this PLIP could have and probably should have re-used the plone.app.querystring package created for plone.app.collection which provides their AJAX search query building.
IMHO a javascript-based query builder like that is appropriate for a site admin function like editing a collection, but not for an end-user function like buildout a search, which should still function with javascript disabled.
comment:57 Changed 5 years ago by jaroel
The querystring widget is designed to work with javascript disabled. Take a look if it is reusable. If not, just contact me and we'll figure that out somehow.
comment:58 Changed 5 years ago by spliter
comment:59 Changed 5 years ago by spliter
comment:60 Changed 5 years ago by spliter
Update status: Livesearch, search portlet and their 'Advanced Search' links have been updated to link to the new @@search view.
comment:61 Changed 5 years ago by dthomas218
- Cc dthomas218 added
- Keywords search added
Search in current section is a great addition to Plone 4, but it is only an option for the initial basic search, not for Advanced Search. The latter should allow the same level of granularity as the former. This is especially useful if one wants to remove "Search in current section" from basic search.
Apologies if I'm not posting this in the right place.
comment:62 Changed 5 years ago by rossp
The README.txt and setup.py description still need to be fixed. This is so easy and very much needed. Can you please just add this? That's the only thing I want to see before merging.
comment:63 follow-up: ↓ 64 Changed 5 years ago by elvix
Willdo
comment:64 in reply to: ↑ 63 Changed 5 years ago by elvix
comment:65 Changed 5 years ago by spliter
comment:66 Changed 5 years ago by potzenheimer
- Cc potzenheimer added
- Priority changed from n/a to major
(If this isn't the right place to report than please point me into the right direction...)
Out of curiority I just played around with `plone.app.search
bin/buildout -c plips/plip9352-search-results.cfg
I noticed that as soon as you apply any advanced search filters the listing bar points to @@updated_search which obviously will not work.
Setting the batch_base_url as a quick workaround fixes this issue.
<tal:batchmacros tal:define="batch_base_url string:${context/absolute_url}/@@search"> ... use-macro batch navigation </tal:batchmacros>
comment:67 Changed 5 years ago by esteele
- Milestone changed from 4.1 to 4.2
Moving to 4.2 milestone. I want this one. :)
comment:68 follow-up: ↓ 70 Changed 5 years ago by eleddy
spliter - are you still willing and able to finish this up for 4.2? We'd love to see it in the release.
comment:70 in reply to: ↑ 68 ; follow-up: ↓ 72 Changed 5 years ago by spliter
Replying to eleddy:
spliter - are you still willing and able to finish this up for 4.2? We'd love to see it in the release.
Hi, I am willing to keep working on this one. But can not keep it going for the next month more or less. But the PLIP should be ready by 4.2 I suppose.
comment:71 Changed 5 years ago by rossp
As I recall the biggest remaining issues were the lack of history for AJAX searches and the loss of search context on some operations. Can you add some detail of how you plan to address those issues?
comment:72 in reply to: ↑ 70 ; follow-up: ↓ 73 Changed 5 years ago by eleddy
Replying to spliter:
Hi, I am willing to keep working on this one. But can not keep it going for the next month more or less. But the PLIP should be ready by 4.2 I suppose.
Just checking in here - I heard that things have been pretty busy for you - would you like me to try and find you some help on this PLIP? We really want to see this make 4.2.
comment:73 in reply to: ↑ 72 Changed 5 years ago by spliter
Replying to eleddy:
Just checking in here - I heard that things have been pretty busy for you - would you like me to try and find you some help on this PLIP? We really want to see this make 4.2.
I have spent some time today checking what is done and what needs to be done more for the PLIP. Seems like there is not much to do — the most important is the AJAX go-back functionality that has to be solved with some external JS library. All the rest is UI imporovments mainly. The list in TODO.txt is valid, so if anybody is willing to help with the PLIP I would appreciate it of course since don't want to be a bottleneck for the PLIP to be merged.
I plan to spend some more time on this the comming weeks.
comment:74 Changed 5 years ago by spliter
(In [49416]) Categories in the search results listing are linked to updated search page with SearchText passed as a parameter.
comment:75 Changed 5 years ago by spliter
(In [49427]) Re-organization of the @@search markup to reveal RSS icon and to make search filter work better in different setups incl. fixed-width sites.
comment:76 Changed 5 years ago by spliter
comment:77 Changed 5 years ago by spliter
Hello,
I am planning to re-arrange the search filters we have on the search results page and make it becave as a normal dropdown we have in Plone (for personal tools, etc.) without fancy animation (putting the column on the left doesn't work in fixed-width sites). Problem is that dropdown.js responsible for providing the effect is restricted to authenticated users only. What would be the best way to do the same in the new search? Remove the restriction from dropdown.js or provide copy of the behavior in search.js (plone.app.search specific javascript)?
comment:78 Changed 5 years ago by mj
Don't duplicate JS code, simply make dropdown.js available for anonymous as well.
comment:79 follow-up: ↓ 90 Changed 5 years ago by eleddy
Hey guys - we are looking at a feature freeze on June 30th. Are you close to being ready for reviews yet?
comment:80 Changed 5 years ago by spliter
Working on it.
comment:81 Changed 5 years ago by spliter
comment:82 Changed 5 years ago by spliter
comment:83 Changed 5 years ago by spliter
comment:84 Changed 5 years ago by spliter
comment:85 Changed 5 years ago by spliter
comment:86 Changed 5 years ago by spliter
comment:87 Changed 5 years ago by rossp
Since we need to have a feature freeze by June 30th, we need implementations finished by next week's framework team meeting on Tuesday, June 14th. IOW, implementation will need to be finished on Monday, June 13th.
Will you be able to have the implementation done by then?
comment:88 Changed 5 years ago by spliter
Hello Ross,
No guaranties for the next Monday. Sorry. What I have heard was June 30th and was aiming for that. Didn't realize that FWT will need the PLIP ready 2 weeks before that.
comment:90 in reply to: ↑ 79 ; follow-up: ↓ 91 Changed 5 years ago by rossp
Replying to spliter:
No guaranties for the next Monday. Sorry. What I have heard was June 30th and was aiming for that. Didn't realize that FWT will need the PLIP ready 2 weeks before that.
Replying to eleddy:
we are looking at a feature freeze on June 30th
So that's a feature freeze, meaning we'd have to make a decision on the PLIP before that date in order to figure out what features makes it into the freeze, meaning we'd have to have finished the implementation detail before then.
comment:91 in reply to: ↑ 90 Changed 5 years ago by spliter
Replying to rossp:
Replying to eleddy:
we are looking at a feature freeze on June 30th
So that's a feature freeze, meaning we'd have to make a decision on the PLIP before that date in order to figure out what features makes it into the freeze, meaning we'd have to have finished the implementation detail before then.
Sure, Ross, thanks for the explanation — it makes total sense. I just mean I didn't understood that feature freeze is not the actual deadline for the implementation. And, of course, it's my problem — I will do my best to finish before Monday, but not sure whether it will work out or not.
comment:92 Changed 5 years ago by rossp
Request for delay to the list.
comment:93 Changed 5 years ago by ggozad
Some remarks about things that did not work for me. Maybe you can fix them in time for review?
- I would love to see JSON passed around instead of HTML in your AJAX requests. This would make it faster and reusable.
- INavigationRoot is not respected when you calculate sections.
- A little more in terms of documentation ;)
comment:94 Changed 5 years ago by rossp
FWT has approved a delay of one week for the implementation. So happy coding and have the implementation done next Monday!
comment:95 Changed 5 years ago by spliter
comment:96 Changed 5 years ago by spliter
comment:97 Changed 5 years ago by spliter
comment:98 Changed 5 years ago by spliter
comment:99 Changed 5 years ago by spliter
comment:100 Changed 5 years ago by spliter
(In [50548]) Back-button works with AJAX calls in modern browsers supporting HTML5 History API
comment:101 Changed 5 years ago by spliter
comment:102 Changed 5 years ago by spliter
comment:103 Changed 5 years ago by spliter
comment:104 Changed 5 years ago by spliter
comment:105 Changed 5 years ago by spliter
comment:106 Changed 5 years ago by spliter
comment:107 Changed 5 years ago by spliter
comment:108 Changed 5 years ago by spliter
comment:109 Changed 5 years ago by spliter
comment:110 Changed 5 years ago by spliter
The PLIP is ready for review. There are some notes though.
- JSON is not integrated. Have it half-done saved on the disk, but had no time to finish it. So HTML responses from AJAX for now.
- AJAX URL updates and back button work in browsers supporting HTML5 History API. No fallback (yet) for those browsers that don't support it.
- No upgrade steps. Will need help with this I assume.
- Didn't write more documentation yet. Will do after we merge this into trunk and I know what is there and how it works :)
- Didn't yet fix INavigationRoot handling for the sections, as Yiorgis asks in #comment:93.
All of these don't feel critical to me at the moment. Unless something major pops up these should not be the blockers for merging the PLIP I hope.
comment:111 Changed 5 years ago by spliter
comment:112 follow-up: ↓ 117 Changed 5 years ago by spliter
To make it clear, HTML to JSON stuff is on the branch of plone.app.search and should not affect the reviewing process.
comment:113 Changed 5 years ago by spliter
comment:114 Changed 5 years ago by spliter
comment:115 follow-ups: ↓ 118 ↓ 129 Changed 5 years ago by rossp
comment:116 in reply to: ↑ 56 Changed 5 years ago by rossp
Replying to davisagli:
Replying to rossp:
(In [46338]) Add a review note about reusing plone.app.querystring. Refs #9352.
It occurrs to me that this PLIP could have and probably should have re-used the plone.app.querystring package created for plone.app.collection which provides their AJAX search query building.
IMHO a javascript-based query builder like that is appropriate for a site admin function like editing a collection, but not for an end-user function like buildout a search, which should still function with javascript disabled.
I wasn't necessarily suggesting we expose the entire dynamic query building support of the package, but re-use the common bits. There end up being a lot of edge case bugs with dynamic stuff like this, and it's a shame to fix the bugs in two places. Especially given that p.a.search is still so buggy.
comment:117 in reply to: ↑ 112 Changed 5 years ago by rossp
Replying to spliter:
To make it clear, HTML to JSON stuff is on the branch of plone.app.search and should not affect the reviewing process.
Actually, that isn't so much clear as it confuses me. :-) If we're going to make this change, I think it would be best to do it before we include this in Plone core. Once we include it in Plone core, users/integrators/developers will start customizing it and it would be better not to pull the rug out from under them.
comment:118 in reply to: ↑ 115 ; follow-up: ↓ 119 Changed 5 years ago by spliter
Replying to rossp:
(In [50577]) Update review for advanced seaarch PLIP. Refs #9352.
Also, I would want to see the current status of the performance impacts before approving for merging. Can someone run these and update the PLIP ticket?
There are performace tests in the package. Please check tests/performance. I bet you could run those yourself, Ross.
comment:119 in reply to: ↑ 118 Changed 5 years ago by rossp
Replying to spliter:
Replying to rossp:
(In [50577]) Update review for advanced seaarch PLIP. Refs #9352.
Also, I would want to see the current status of the performance impacts before approving for merging. Can someone run these and update the PLIP ticket?
There are performace tests in the package. Please check tests/performance. I bet you could run those yourself, Ross.
I bet you or elvix could run them more readily yourselves, Denys. See comment 13. You're more familiar with them and have more of a sense of the context. Please run them and just report in a brief comment as to whether performance has improved or degraded since the last time they were discussed. It will help all the reviewers of this PLIP and the FWT discussion.
comment:120 Changed 5 years ago by spliter
Fair point. Then let me give the information about the performance of the new search indeed. This is a copy/paste from my terminal when running bin/test -s plone.app.search --tests-pattern=performance:
… Set up plone.app.search.tests.base.Search Performance 1000:Functional in 0.000 seconds. Running: 1/1 (100.0%)******************** old search page vs new search page 1.28667593002 vs 0.809792041779 ******************** … Set up plone.app.search.tests.base.Search Performance 100:Functional in 0.000 seconds. Running: 1/1 (100.0%)******************** old search page vs new search page 0.168622016907 vs 0.127992868423 ******************** …
I think this obviously shows that the new search is faster both for 100 items and for 1000 items.
comment:121 Changed 5 years ago by spliter
comment:122 Changed 5 years ago by spliter
comment:123 follow-up: ↓ 125 Changed 5 years ago by spliter
comment:124 Changed 5 years ago by spliter
comment:125 in reply to: ↑ 123 ; follow-up: ↓ 126 Changed 5 years ago by rossp
comment:126 in reply to: ↑ 125 ; follow-up: ↓ 127 Changed 5 years ago by spliter
Replying to rossp:
Replying to spliter:
(In [50617]) OMG! I actually forgot to register Modernizr.js in jsregistry.xml even though it has been in the plone_ecmascripts/ folder. References #9352
Might this comment be about #11300 instead?
No, Ross, that comment is correct. I asked FWT team whether it is ok to rely on Modernizr in this PLIP as well or I should wait for #11300 to be merged. And the answer was from Elizabeth "I don't think there has to be any problem with the library in PLIP 11300 and it's safe to rely on it in this PLIP as well, but would like to get a confirmation.". I didn't get any other responses, so accepted this as the final FWT's decision. The reason why I added Modernizr in Plone for this PLIP as well is simply because #11300 is not merged yet, thus I didn't really have Modernizr available for this PLIP before.
comment:127 in reply to: ↑ 126 Changed 5 years ago by rossp
Replying to spliter:
Replying to rossp:
Replying to spliter:
(In [50617]) OMG! I actually forgot to register Modernizr.js in jsregistry.xml even though it has been in the plone_ecmascripts/ folder. References #9352
Might this comment be about #11300 instead?
No, Ross, that comment is correct. I asked FWT team whether it is ok to rely on Modernizr in this PLIP as well or I should wait for #11300 to be merged...
Ah, that's right. I'd forgotten, sorry. Thanks for the refresher.
comment:128 Changed 5 years ago by spliter
(In [50621]) Switched 'Select All/None' switcher from using 'click' to using 'change' event to unify the behavior between browsers. The order of firing up events is broken in Chrome ("submit" is fired up before "click") that led to un-expected results when used in search form's filter. References #9352
comment:129 in reply to: ↑ 115 Changed 5 years ago by spliter
Replying to rossp:
(In [50577]) Update review for advanced seaarch PLIP. Refs #9352. … Browser history restores selected criteria but results don't reflect them.
This should work fine now. I was not able to reproduce the issue itself before some other commits. But I followed the steps in the review and the results I have got were correct (strictly following the steps I have got only one document listed after getting back to the results at the end). I assume the issue was a mix of bugy popstate event in Chrome and missing Modernizr library. Both of these had been fixed by now, so the is not valid anymore.
Tested in Chrome 13, Firefox 4.0.1, Safari 5.0.5, Opera 11.1 (even though the location bar is not updated in this browser, I have got one result whne getting back to the search)
comment:130 Changed 5 years ago by spliter
comment:131 Changed 5 years ago by spliter
comment:132 Changed 5 years ago by spliter
comment:133 Changed 5 years ago by spliter
comment:134 Changed 5 years ago by hannosch
I'm in the process of helping spliter to fix some batching issues and do a general code cleanup. Please hold of with any reviews until Sunday, as I'm probably leaving the code in a broken state for a while.
comment:135 Changed 5 years ago by hannosch
I've done my updates - in the end quite a lot of them. Denys will have another look to fix some small issues like highlighting and maybe some styling tweaks or update the batch navigation to work without full page reloads.
comment:136 Changed 5 years ago by spliter
comment:137 follow-up: ↓ 141 Changed 5 years ago by spliter
(In [50811]) AJAX click handler for the links in the batch navigation.
(In [50812]) Taking advantage of the built-in everything-is-object nature of javascript — we, now, have global Search object with the following methods:
- getResults() makes actual ajax call, gets results and updates the elements
- pushState() updates URL and pushes the state to browser's history
- popState() reverts search results to the previous history state (used by browser's Back button)
- updateResults() combination of getResults() and pushState()
comment:138 follow-up: ↓ 147 Changed 5 years ago by ggozad
comment:139 Changed 5 years ago by spliter
Updated information on the performance of the new search.
After Hanno helped me to fix the issue of the batches (the problem was that batches didn't work, so in tests for 100 and 1000 items I was comparing 1000 items of the new search with 30 items of the old search) the new search shows even better results than before. The tendency is the more items we get in search totally, the more significant the difference is. The new search is pretty consistent in the time spent for a search query. Here is the updated test output information:
… Set up plone.app.search.tests.base.Search Performance 1000:Functional in 0.000 seconds. Running: 1/1 (100.0%)******************** old search page vs new search page 1.31023287773 vs 0.0718610286713 ******************** Ran 1 tests with 0 failures and 0 errors in 3.547 seconds. Running plone.app.search.tests.base.Search Performance 100:Functional tests: Tear down plone.app.search.tests.base.Search Performance 1000:Functional in 0.000 seconds. Tear down plone.app.search.tests.base.SearchPerformance1000Layer in 4.011 seconds. Set up plone.app.search.tests.base.SearchPerformance100Layer Testing performance with 100 pages in 5.204 seconds. Set up plone.app.search.tests.base.Search Performance 100:Functional in 0.000 seconds. Running: 1/1 (100.0%)******************** old search page vs new search page 0.171858072281 vs 0.0708620548248 ******************** …
comment:140 Changed 5 years ago by ggozad
I can also verify spliter's comment on performance. During review I also made sure that the preheating did not have any effect on the measurements. With small variations the worse case scenario I got was a gain of a factor of 2.
comment:141 in reply to: ↑ 137 ; follow-up: ↓ 142 Changed 5 years ago by ldr
Replying to spliter:
(In [50811]) AJAX click handler for the links in the batch navigation.
(In [50812]) Taking advantage of the built-in everything-is-object nature of javascript — we, now, have global Search object with the following methods:
- getResults() makes actual ajax call, gets results and updates the elements
- pushState() updates URL and pushes the state to browser's history
- popState() reverts search results to the previous history state (used by browser's Back button)
- updateResults() combination of getResults() and pushState()
Is the Search object really in the global scope? The code executes in an anonymous block, so I can't see how it ends up in the global scope. The normal way to do this is to set it explicitly on the window object. If it is meant to be in the global scope then it should be called something like plone_search instead of Search to avoid clashes.
comment:142 in reply to: ↑ 141 ; follow-up: ↓ 144 Changed 5 years ago by spliter
Replying to ldr:
Is the Search object really in the global scope? The code executes in an anonymous block, so I can't see how it ends up in the global scope. The normal way to do this is to set it explicitly on the window object. If it is meant to be in the global scope then it should be called something like plone_search instead of Search to avoid clashes.
That's a good point, Laurence. The search object is really not in the global scope indeed. I am currently working on moving the object to the global scope and some other improvements to the javascript. Those are not really blockers for the merge I suppose, but will be good to have before final release. The plan is to move the object to the global scope and switch to JSON in the ajax call to make search results available everywhere with a simple javascript call. Next step will be to implement such a JS call in the livesearch in order to provide an example. Then the same approach might be used in the new collections or listings in order to fetch the results.
So… I am on it :)
comment:143 Changed 5 years ago by spliter
comment:144 in reply to: ↑ 142 Changed 5 years ago by spliter
Replying to spliter:
The plan is to move the object to the global scope and switch to JSON in the ajax call to make search results available everywhere with a simple javascript call.
Had the discussion with Hanno about this today. Seems like it doesn't make a lot of sense to provide JSON objects or add any global functions/methods in the plone.app.search package. The reason is — the search, as it is now, returns results in plone batches, is based on a certain template output structure, uses breadcrumbs for the entries etc. All of these are not very JSON-friendly. At least it doesn't really make a lot of sense to spend time on getting JSON results as the output of the search. So, for now, plone.app.search will be based on HTML structure being returned with the AJAX calls.
References #9327. Instead, maybe we could make plone.app.contentlisting providing JSON objects on demand in order for them to be re-usable in different places when using AJAX (for example, for dynamicly updated listing portlets like News/Event Items or something more dynamic). I suppose this is not the highest priority, but would be nice to have a unified story for listings, nomatter whether they are used in python or JS.
comment:145 Changed 5 years ago by spliter
comment:146 Changed 5 years ago by spliter
(In [51087]) Fixed breakage in @@updated_search for the cases when we don't have search results. References #9352 and the traceback at http://pastebin.com/BbTnHdU1
comment:147 in reply to: ↑ 138 Changed 5 years ago by spliter
Would like to comment on some issues, mentioned by Yiorgis in the review.
- RSS link:
Add a few items, mixing private and published content. Perform a search with no SearchableText and no additional filters. There are no results. Consecutively clicking on the RSS feed shows all items on the site. So it's inconsistent with the search results but also returns private content.
It is updated now and, when one searches for an empty string, search returns *all* results that makes sense. The same behavior is in the RSS link, so search results and their RSS listing are consistent now. Concerning the private items, I could not reproduce them being shown in the RSS results.
Nicely enough, if you DO filter by date (say display results since yesterday) only published items are shown.
Did this and got the same results in both — view and RSS listing. So, I would assume the other fixes to, probably, @@updated_search and search.js made RSS link work as expected and consistent with the main search results view.
- @@updated_search error when there are no results:
I've somehow managed to produce some errors on @@updated_search... traceback: http://pastebin.com/BbTnHdU1
Should not happen anymore.
- Clean URL for when only the SearchTerm is updated:
While on the search results page, updating the searchable text and clicking
on search again, includes all the filter parameters
Fixed (In [51086])
- Another issue with empty search results:
Searching with searchable text empty, show on the screen
"No results were found", under "0 items matching your search terms". Searching with searchable text that will not produce any results does not display "No results were found".
Missing text in this case was relevant to the aforementioned breakage in @@updated_search. As a repeating sidenote, "searching with searchable text empty" gives all results now, not 0.
- Search in the current section:
The "only in current section" setting is not preserved.
Right, it is not preserved when one performs a filtered search (using the options within the "Filter the results" list). It is preserved with the simple search, involving SearchableText updates and sorting options though. I am not sure whether we need to invest time in fixing this though — I would say, getting rid of that redundant "only in this section" checkbox should be another PLIP. Or, if we really need to preserve it, it should be concealed in the "Filter the results" list of the search results page. There is 1 out of 1000 designs that might need this option to be so prominent as it is now.
So, appart from the search within a section, I think issues in Yiorgis' review are handled and should nto be a blocker for the merge.
comment:148 Changed 5 years ago by rossp
FWT voted to merge this PLIP.
comment:149 Changed 5 years ago by esteele
comment:150 Changed 5 years ago by esteele
comment:151 Changed 5 years ago by esteele
comment:152 Changed 5 years ago by esteele
comment:153 Changed 5 years ago by esteele
comment:154 Changed 5 years ago by esteele
comment:155 Changed 5 years ago by esteele
- Status changed from assigned to closed
- Resolution set to fixed
Approved by FWT vote (see #9271 and #9282)