Ticket #9321 (closed PLIP: fixed)

Opened 7 years ago

Last modified 6 years ago

Reimplement the search form with an eye on usability

Reported by: csenger Owned by: csenger
Priority: minor Milestone: 4.0
Component: Unknown Version:
Keywords: Cc: plip-advisories@…, grahamperrin@…

Description

Motivation

The aim of this PLIP is to

  • integrate the "normal" and the advanced search form
  • provide a default advanced search form that is agressively optimized for ease of use and works in most cases with the possible drawback that it does not scale in all cases and removes "features" power users may like.

Rewriting the search form has two main motivations: Customizing the search form is a fairly extensive task and the current search form ui has some usability problems. It presents too many options whose use is not obvious to an average user. A summery of the problems using the (extreme) example of plone.org: There are 44 Item Types, among that 13 topic criteria (which is not possible with the plone control panel) and 3 different folder types. Many of them are "useless" as a selectable options like SoftwareCenter. The average Visitor will navigate to them. He won't use it as a criterion in the search. OTOH removing it from the search form implies that it will be not included in the full text search if the user searches for all content. There are 50 review states, among them 4 x *draft and 2 x 'Open for submission' It's not possible to see which states belongs which type of content. There are also review states listed, even if no object is in this state or is viewable by the user. These review states mean nothing to anonymous users (There are excepts like POI review states, but the global search form is not the appropriate tool to use then). Even active Plone.org members do probably not understand most of them. The most valuable criterion is probable "Viewable by anonymous visitors" or not.

This huge amount of information that stretches across 4 pages vertically is presented to all users (anonymous and authenticated). The add on [ http://pypi.python.org/pypi/collective.advancedsearchtame collective.advancedsearchtame] ( [ http://www.netsight.co.uk/blog/2009/3/27/taming-plone-s-advanced-search example] ) tries to reduce the overwhelming impression of the search form, but not the key problems.

Assumptions

Optimizations for usability has some costs:

  • Performance: Querying the catalog to refine the UI is more expensive then the current approach.
  • Administrative complexity: The control panel will contain more options that may be difficult to handle for some administrators. This is largely justifiable cause searching itself is a more important functionality.
  • Pure power users may prefer an unoptimized default. Sorry guys.

Not covered by this PLIP are changes to the search result listing as there are two PLIPs already for it.

The details for the UI, especially the admin UI, are not fixed. They depend on the general direction of the Plone UI and possible UI guidelines. Possible "power user" features (like switching from grouped review states to the full list of review states) have to be usability tested/reviewed.

Proposal

The general goals are

  1. provides fewer options to the user (e.g. collapse advanced options to only show the title
  2. groups options so their functionality is better understandable for users
  3. removes all options that have no effect for a search
  4. Is more responsive to the user.

There will be only one search form (@@search). It shows a simple full text search field by default (like the current search form) and has a link "Advanced Search" to reload the form with all search options.

The new advanced search options will be partly configurable. The portal administrator can disable all or single advanced search options for anonymous or authenticated separately if they are not useful, e.g. if categories are only used for internal purposes. The form does not present options to anonymous users if they don't provide value for them. Less frequently used options are collapsed by default.

The proposed changes in detail:

Provide only one input field for Title and Description

Both information are sometimes "more important" than the full text index, but providing separate input boxes clutters the interface. Querying them independently is a technical requirement of the portal catalog. Adding another joint index is no option. The limitation can be overcome with AdvancedQuery. The alternative plan is to drop the Description search.

Use a better widget for Categories

Will be presented with a better widget if javascript is activated. Will not be shown if there are no categories.

Modify 'New Item since'

  • Split into 'created' and 'modified' criteria
  • use a flexible widget instead of the fixed list, e.g. a slider with options with javascript and "between <datewidget> and <datewidget>" without.
  • Group it under "Created or Modified" and collapse the details by default

"Item Types" - allow grouping and other things

Having all searchable content types listed individually is often not necessary or confusing when there are more that Plone's out-of-the-box content types. Grouping them is an option to clean up the user interface and guide users.

Possible generic categories are e.g.: Documents, Events, News, Downloads, Multimedia, others.

There are some options grouping can be implemented.

  1. Predefine a set of groups
    • (+) these groups can be fully translated
    • (-) Only useful for a part of the use-cases
    • (+) can be extended if (policy) can register additionally groups

(like roles translation)

  1. Give add ons a way to group their types
    • (-) Grouping will be inconsistent (can be overwritten by a policy product)
    • (+) can be changed with policy products
  2. Let admins define the group TTW

I tend to option 3 as is the most flexible for simple deployments with the only drawback that is is not translatable TTW if needed, but file system products are required for customization in the other cases too.

  • Grouping of content types will be optional as it make no sense out-of-the-box.
  • There should be a way to switch to the full list of content types for authenticated users.
  • A content type is only listed if there is at least one findable object in the site
  • This option will be collapsed by default.
  • Details of the user interface still have to be explored. If the user interface get's to complex, grouping of content types might be dropped.

Authors

Authors will be displayed with their full name.

If there are many users, the current form uses a plain text line input. with javascript enabled, a more convenient widget should be used, e.g. an autocomplete widget.

Allow group of review states

The list of review states will not be displayed for anonymous users by default. Either they can view an object or not. A search option does not provide much value.

For authenticated users, review states will be grouped into

  • Viewable by all visitors
  • Viewable by (some) Authenticated users

This cleans up the UI and does not allow to select a review state where the user does not have view permissions. Authenticated users can switch to the full list of review states. This list should be limited to the review states that will give a result (present in the site and the object is accessible by the user).

The list of options should be disabled/enabled depending on the selection of content types.

This option will be collapsed by default.

Display options for the search results

The results can be sorted by relevance, creation and modification date and batched in a selectable size. These options are collapsed by default. They may be available in the simple search also.

Help

Providing a complex search form to anonymous visitors is only a half-assed solution. The form and the individual search options should have a help text that gives the user more information so the descriptions and the whole form can be kept compact.

Integration of "normal" and Advanced search

There will be only one search form (@@search). It shows a simple full text search field by default (like the current search form) and has a link "Advanced Search" to reload the form with all search options. If javascript is activated, it will either toggle the display of the advanced search options or reload this part with ajax. If a simple search was successful the form will show the full text search and the search results (current behavior). If an advanced search was successful, it will show a link "adjust search" that reloads the form (no javascript) or slides in the form (javascript). Both will be preserving the search options.

Compatibility

The search_rss script will be kept for backward compatibility.

Implementation

Depending on implementation and progress of the folder UI improvements GSoC project, parts of the code can be shared.

Form building

This search will be moved skin templates and scripts to a BrowserView. The Form will be a z3c.form. The individual search options will be subforms that can be customized independently. Major drawbacks of formlib/z3cforms are the excessive query strings and the effort to customize details of the template code.

Query Building

The construction of searches queries will be done with AdvancedQuery cause it allows OR queries across indexes and incremental sorting. Possible problems with LinguaPlone and catalogqueryplan have to be evaluated. AdvancedQuery is a dependency in Plone 3 even though this may disappear if wicked is dropped. The extend advantages has to be considered, but AdvancedQuery is no hard requirement for the PLIP.

Performance

I don't expect performance problems even so there is a number of catalog calls to construct and update the search form, but they should be fairly cheap even in big installation.

An autocomplete widget for the author search may be problematic if an external/big user database is used. If so, the full username support can be replaced by using the unique values from the catalog depending on the "Many Users?" setting in the control panel.

Javascript

The javascript support will be done with jquery naturally. Input to keep this accessible for people with disabilities is appreciated.

Deliverables

  • The new search form with the included end user documentation
  • The control panel form with admin documentation
  • Documentation how to customize (and maybe extend) the form
  • Changes to Documentation on plone.org (to be listed)
  • unit tests
  • selenium tests to test the javascript support
  • load tests to guarantee that the solution scales
  • localization of user interface and help

Risks

Bookmarked links to searches will break

Participants

Carsten Senger (csenger)

Change History

comment:1 Changed 7 years ago by csenger

  • Owner set to csenger
  • Status changed from new to assigned

comment:2 Changed 7 years ago by csenger

  • Status changed from assigned to new

ups, set back status to new

comment:3 Changed 7 years ago by erikrose

  • Owner csenger deleted

Clearing Owner field of 4.0 PLIPs so we can use it to mean "implementor". (Many of these owners were automatically assigned from choosing a Component that had a default owner.)

comment:4 Changed 7 years ago by smcmahon

  • Cc plip-advisories@… added

comment:5 Changed 7 years ago by csenger

  • Cc csenger added

add to cc to get updates again

comment:6 Changed 7 years ago by rossp

  • Cc csenger removed
  • Owner set to csenger

Owner here was intentional

comment:7 Changed 7 years ago by rossp

FWT vote: -1. While we very much need to improve the search form, I think the work slated for Plone 5 is very likely to end up replacing the search form and is likely to offer significant improvements. I don't want to create a disruptive experience for integrators and developers by replacing it in two successive major versions. Can we move this to 5.0 and coordinate with the work there?

comment:8 Changed 7 years ago by MatthewWilkes

Given that 4.x is mostly intended as a backporting release, and this sounds like a large thing to me, I don't think I'm comfortable with its inclusion. Tbh, I don't think you can get all this done by the implementation deadline on your own.

I'd like to bump this to Plone 5.x and do it properly for the long-term there.

FWT Vote: -1

comment:9 Changed 7 years ago by davisagli

FWT vote: -1. The current advanced search form is non-ideal, but it is also one of the most easily hackable parts of Plone. My fear is that the work proposed here would create something that is just as non-ideal for many use cases, but slower and harder to customize.

comment:10 Changed 7 years ago by csenger

I think it's the right decision to postpone big design decisions to Plone 5. As longer as I think about it the easy customizability is an important factor too. But instead of moving the whole PLIP to Plone 5 I like to implement some of the visual changes for Plone 4 that do not change a lot, are cheap to implement and partly even bugfixes:

  1. remove categories if there are no categories
  2. remove workflow states for anonymous visitors
  3. only list content types and workflow states for which at least one object exists (unique indexed values)
  4. add display options for sort order and batch size
  5. remove title and description input boxes, just leave Searchable Text.
  6. collapse "New Item since", "Item Type", "Author" and "Review State" by default if javascript is enabled, which leaves "Search Text", "Categories", "Sort on" and batch size.
  7. If an advanced search happened, the variables are encoded into the link to the advanced search and the user get's back the form with the options he choose. The form get's a clear button

The forms will remain as search.pt and search_form.pt. I implement configuration options for 2 and 6 in the search control panel. The Title and Description input boxes will be removed with a tal:replace="nothing" block.

New logic will partly move into a helper view to keep the forms clean and not add scripts to the skin layers.

These changes are fairly riskless.

  • Customized search forms will keep working.
  • If integrators (partly) prefer the old form, they can customize the templates and are done.
  • There are no new dependencies or technologies involved (except of the helper view that has a short life time, but I don't see a case where the view itself needs to be customized).
  • There are no catalog queries involved, only queries to uniqueValues().

comment:11 Changed 7 years ago by davisagli

+1 for the modified proposal csenger just posted.

comment:12 Changed 7 years ago by rossp

FWT vote revision: +1 for new proposal

comment:13 Changed 7 years ago by calvinhp

FWT Vote: +1 based on the new proposal (I had comments and different vote based on the PLIP as it was yesterday, one disadvantage of offline PLIP review)

comment:14 Changed 7 years ago by raphael

FWT Vote: +1 based on the new proposal

comment:15 Changed 7 years ago by esteele

Approved by FWT vote.

comment:16 Changed 7 years ago by csenger

  • Status changed from new to assigned

comment:17 Changed 7 years ago by grahamperrin

  • Cc grahamperrin@… added

comment:18 Changed 7 years ago by grahamperrin

accessible for people with disabilities

Subject to approval of PLIP #9250 for Plone 4.0, would you treat jQuery Tools as a sound foundation?

 http://www.diigo.com/bookmark/http%3A%2F%2Fflowplayer.org%2Fforum%2F20%2F20502?tab=comment&uname=grahamperrin for highlights from a relevant discussion.

comment:19 Changed 7 years ago by csenger

(In [28987]) Branch for PLIP #9321 refs #9321

comment:20 Changed 7 years ago by csenger

(In [28988]) Buildout for PLIP #9321 refs #9321

comment:21 Changed 7 years ago by csenger

(In [29038]) branch for Plip implementation (refs #9321)

comment:22 Changed 7 years ago by csenger

(In [29080]) add Dokumentation for PLIP, refs #9321, PLIP is ready for review

comment:23 Changed 7 years ago by csenger

(In [29102]) Add information about implementation Details to Plip Documentation refs #9321

comment:24 Changed 7 years ago by erikrose

(In [29634]) Completed review of at least the UI portions of the advanced search form PLIP. Refs #9321.

comment:25 Changed 7 years ago by cjohansen

Accessibility check with FAE ( http://fae.cita.uiuc.edu/) identical with baseline.

comment:26 Changed 7 years ago by esteele

(In [29748]) Review for #9321. Refs #9321.

comment:27 Changed 7 years ago by esteele

Your PLIP has passed the Framework team's initial review. Feel free to discuss any suggested changes either here in the PLIP ticket or on the mailing lists. Final deadline for this PLIP is set for September 30.

comment:28 Changed 6 years ago by csenger

(In [30166]) create branch for plip refs #9321

comment:29 Changed 6 years ago by csenger

(In [30169]) ready for merge

I added comments to the review in plip9321-searchform.txt

refs #9321

comment:30 Changed 6 years ago by esteele

(In [30207]) Updated review of #9321. Refs #9321.

comment:31 Changed 6 years ago by erikrose

(In [30250]) Updated review for merge vote day. +1 for me. Refs #9321.

comment:32 Changed 6 years ago by rossp

FWT vote: +1 for merge

comment:33 Changed 6 years ago by esteele

This PLIP has been accepted for merging into Plone 4.0

The final vote was: Alec Mitchell +1 David Glick +1 Erik Rose +1 Laurence Rowe +1 Matthew Wilkes - Ross Patterson +1

Please merge your branches into the Plone 4.0 head by end-of-day Friday Oct 16. If you need assistance with merging, please contact me.

We'll be assigning a documentation ticket to this PLIP shortly. Please assist the docs team in documenting the changes and new features that this PLIP introduces.

comment:34 follow-up: ↓ 36 Changed 6 years ago by esteele

  • Status changed from assigned to closed
  • Resolution set to wontfix

Closing this PLIP as it wasn't completed for 4.0. Please feel free to reopen and submit to a 4.x release.

comment:35 Changed 6 years ago by esteele

  • Status changed from closed to reopened
  • Resolution wontfix deleted

comment:36 in reply to: ↑ 34 Changed 6 years ago by esteele

Replying to esteele:

Closing this PLIP as it wasn't completed for 4.0. Please feel free to reopen and submit to a 4.x release.

Please disregard this. It was meant for another PLIP.

comment:37 Changed 6 years ago by esteele

Please assist the doc team in creating/updating documentation relating to this PLIP. See #9616.

comment:38 Changed 6 years ago by esteele

  • Status changed from reopened to closed
  • Resolution set to fixed

(In [30477]) Removing PLIP branches. Closes #9321.

Note: See TracTickets for help on using tickets.