Ticket #11708 (closed PLIP: fixed)

Opened 5 years ago

Last modified 2 years ago

Implement a Selenium 2 (Webdriver) testing framework for Plone core

Reported by: esteele Owned by:
Priority: n/a Milestone: 5.0
Component: Sysadmin and Development Infrastructure Version: 4.2
Keywords: Cc: emanlove

Description (last modified by rossp) (diff)

Proposer: Eric Steele
Seconder: Ross Patterson (rossp)

Motivation

Each new release of Plone introduces more JavaScript-based functionality. While some of those features see test coverage from (jUnit) unit tests, there is very little functional testing being done. This proposal aims to standardize and facilitate the means for creating such tests.

Proposal & Implementation

First some disambiguation... Selenium 1 used JavaScript to control browser interaction. This allowed it to work across browsers, but the JavaScript sandbox prevented it from interacting with things like file dialogs and drag-and-drop. Selenium 2 merged the Selenium project with Webdriver, providing a plugin-based interaction which removed these sandbox restrictions. More information is available at  http://seleniumhq.org/docs/01_introducing_selenium.html#brief-history-of-the-selenium-project For the purposes of this PLIP, when "Selenium" is used, it refers to "Selenium 2" or "Selenium Webdriver"

A plone.seleniumtest package will be created containing a base testcase relying on plone.app.testing's SELENIUM_FIXTURE. Existing Selenium tests in core will use this as their base.

An effort should be made to factor implementation details of interacting with specific Plone features to their particular packages. For instance, typing in a TinyMCE widget requires moving the focus to a specific frame, identifying an element within, and inserting the text before finally returning to the page itself to continue with the test. In this example, the TinyMCE package would provide a TinyMCEElement class which exposed to Selenium methods like 'type', 'insertImage', and other such common functions. Using this method, tests from across Plone core and in add-ons could interact with the widget even though the underlying Selenium interactions required to do so may have changed in a new release the TinyMCE package. Doing so, we hope to avoid the brittleness we've encountered in many of our current browsertests.

Selenium tests will intentionally not be run alongside Plone's standard tests from the bin/test command. A standardized location of "selenium/" for Selenium tests is proposed to separate them from the standard tests. A second zc.recipe.testrunner part with a --tests-pattern flag may be used to run them. The Selenium tests will be runnable locally without additional Selenium driver installations, but will also support a distributed testing environment in our current continuous integration infrastructure. Please note that this PLIP aims to provide the framework for creating these tests, and not the tests themselves.

Deliverables

  • plone.seleniumtest package containing base testcase and layers
  • Additions to Products.CMFPlone, plone.app.jquerytools, Products.TinyMCE, Products.Archetypes to provide the interaction layers.
  • Documentation: Creating tests, running tests

Risks

  • More tests to write/break/update
  • Those using plone.app.testing 1.0's selenium helper methods may need to rewrite their tests.

Participants

  • Eric Steele (esteele)

Progress

Evaluation of Webdriver and work on early examples was started during the 2010 Plone Conference sprints. A Selenium testing layer was subsequently added to plone.app.testing. Several packages currently make use of it to run Selenium tests.

Change History

comment:1 Changed 5 years ago by rossp

  • Description modified (diff)

I worked on this at the last conference bristol sprint and was very impressed. I'm very excited to see this and will both second this and be the FWT champion for this if allowed.

comment:2 Changed 5 years ago by esteele

comment:3 Changed 5 years ago by emanlove

  • Cc emanlove added

The selenium testing PLIP looks good. There are three issues, IMHO, that are important to get a good foundation for selenium testing; base and/or selenium layers, the PageObjects concept, and the testcase vs. layer/helper function debate.

Rereading the doc for plone.testing and plone.app.testing, I am having a hard time imagining anything beyond the base selenium_layer already found in plone.app.testing. As more tests are developed I'm sure that there will arise situations where common resources makes sense but I did want to raise this question to see if anyone else sees another "base" layer.

The PageObjects concept is probably the most important of the three; allowing for separation between "find this particular element" and "test this concept or procedure or steps" should allow for some future-proofing. For more information about PageObjects see [1] and [2] . I'll look at the current tests to see if I can refactor them to exhibit this.

Finally there is the (dreaded) testcase vs. layer/helper function debate. I'm sorry for making a big deal out of this but I want to set us off right; for which I don't know the best answer myself. There has been voiced, mostly offline, preference to use the testcase model but I wanted to move in the direction agreed upon by the PLIP process, including PLIP 10846. One change could be to move the helper functions into the layer class and make them a resource or methods of the seleniuim layer. This could be done within another layer using the current selenium layer as its base layer. Since I don't feel personally to be an expert in the new testing methodology I wanted to try to build consensus on this issue.

[1]  http://pragprog.com/magazines/2010-08/page-objects-in-python

[2]  https://github.com/adamgoucher/python-selenium-pageobjects

comment:4 Changed 5 years ago by eleddy

  • Milestone set to 4.2

comment:5 Changed 5 years ago by rossp

If accepted, we will require that PLIPs for future releases that involve significant JS behaviour must have selenium tests as a part of PLIP merging acceptance. Completion of this PLIP must also include adding the appropriate documentation to that effect.

Provided the above are addressed and it's finished by the deasdline the FWT approved this PLIP for inclusion in 4.2.

comment:6 Changed 5 years ago by esteele

I'm in the process of implementing Page Objects for anther project at the moment. I'm really liking that there are no actual Selenium calls in my test. Here's an example of testing Plone's login overlay/form.

    def test_incorrect_login(self):
        h = HomePage(self.portal)
        h.open_default_url()
        lo = h.login()
        lo.username = "foo"
        lo.password = "bar"
        lo.submit()
        self.assertEqual(l.status_message, "Login failed. Both login name and password are case sensitive, check that caps lock is not enabled.")

comment:7 Changed 5 years ago by esteele

(In [49094]) Branch CMFPlone. Refs #11708.

comment:8 Changed 5 years ago by rossp

Discussed briefly in todays meeting. Eric's gonna make a pass at this before next weeks meeting and will discuss then.

comment:9 Changed 5 years ago by rossp

  • Milestone changed from 4.2 to 4.3

Won't be ready for 4.2

comment:10 Changed 5 years ago by rossp

The FWT has wrapped up work on 4.2 and can start work on 4.3 whenever we have PLIPs to review. So can you as proposers or implementers please check in on your PLIPs and let us know what the status is and when we can expect issues to be addressed and implementations complete so we can review them for merge in 4.3.

comment:11 Changed 5 years ago by emanlove

I've converted over one of the previous selenium tests (testCollections) to now use page objects. To give this a test run try the following instructions

~$ svn co  https://svn.plone.org/svn/plone/buildouts/plone-coredev/branches/4.2/ ./p42sel
~$ cd p42sel/
~/p42sel$ python bootstrap.py 
~/p42sel$ ./bin/buildout -c plips/plip11708-selenium.cfg
~/p42sel$ ./bin/test -s Products.CMFPlone --tests-pattern=selenium -t pageobject

I left the odd test code so one can compare the difference in using pageobjects or not. To run both tests, with and without pageobjects, using the following command

~/p42sel$ ./bin/test -s Products.CMFPlone --tests-pattern=selenium -t collection

It is recommended to use virutalenv and setup a clean sandbox. Also the above buildout downloads selenium so there is no extra steps necessary to get, build, or run selenium. Buildout will do it all for you. If you have any issues contact me, emanlove on plone.org or t55e on #plone irc.

comment:12 follow-up: ↓ 13 Changed 4 years ago by gotcha

At Living Statutes sprint, Jan Murre and I started to work on Selenium tests of CMS UI and Deco.

 https://github.com/gotcha/buildout.deco/blob/deco-tests/acceptance-tests/edit.txt

We used robotframework ( http://code.google.com/p/robotframework/), acceptance tests tool.

We succeeded to make various tests including a drag and drop test in deco.

I plan to work on this during PloneConf sprint.

comment:13 in reply to: ↑ 12 Changed 4 years ago by rossp

Replying to gotcha:

I plan to work on this during PloneConf sprint.

That's what I'd likt to sprint on as well.

comment:14 Changed 4 years ago by esteele

  • Milestone changed from 4.3 to Future

comment:15 Changed 4 years ago by davisagli

  • Component changed from Infrastructure to General

comment:16 Changed 4 years ago by kleist

  • Status changed from new to confirmed
  • Component changed from General to Sysadmin and Development Infrastructure

comment:17 Changed 3 years ago by davisagli

  • Milestone changed from Future to 4.4

Can someone update this with the current status of what needs to be done? (Is it correct to think there's some consensus around plone.act as a way to move forward?) Is anyone currently able to put some time into finishing integrating this?

comment:18 Changed 3 years ago by esteele

  • Milestone changed from 4.4 to 5.0

The Framework Team has decided to move on to Plone 5. Updating milestones accordingly.

comment:19 Changed 2 years ago by davisagli

  • Status changed from confirmed to closed
  • Version set to 4.2
  • Resolution set to fixed

We have robotframework now.

Note: See TracTickets for help on using tickets.