Cross-browser testing strategy #42
Description
Overview
An issue was raised in a recent meeting when the torque edit feature didn't seem to work on an older version of Firefox. It was brought up that we may need a way to test our MediaWiki add-ons.
Drawbacks
The complexity of user interfaces means additional complexity when it comes to testing them. There's a variety of ways to do this:
- Unit testing javascript code
- Functional testing javascript code
- Visual diffing tests such as percy or storybook
- End 2 end testing using selenium
Beyond this, due to the browers ecosystem, it's difficult to control for things such as:
- browser version, js engine, etc.
- addons modifying the web page
Possible Solutions
Browser compatibility testing
Generally, testing browser compatibility uses selenium under the hood. In most cases, browser compat tests will test the latest version of different browsers. It is possible to test a variety of different browser versions but this requires multiple machines/VMs to be configured.
There's a number of pay-for services which do browser testing for you.
They're both quite good, I've used browser stack before. Pricing varies and due to the nature of Selenium being a fairly resource intensive application, I'd expect to pay a fair bit more than your usual CI bill.
If we're looking at doing some less intensive selenium testing (perhaps just testing the latest version of each browser), this is more feasible and MediaWiki does seem to have some native support of Selenium tests. This would require further discussion on what sort of instance this would run on, what it would test, etc.
Pre-deploy javascript & CSS transpilation
We can use an application such as babel to ensure the javascript being deployed on our mediawiki servers is compliant with older javascript engines and lessens the likelihood of being mangled by MediaWiki's minifier. This would likely require we commit compiled javascript code & have some sort of javascript build process.