How I do automated accessibility testing for my web site

This article covers including accessibility assessments to your website utilizing Pa11y (pa11y-ci with axe) and Cypress (with cypress-axe) in GitLab CI/CD. I exploit a Jekyll web site for example, however any web site expertise that runs in CI/CD can leverage this setup.

Prep your web site

In addition to getting your web site to run in CI/CD, I like to recommend enabling an XML sitemap function. A sitemap permits the accessibility assessments to parse all URLs to seek out accessibility points throughout the location. I like to recommend the jekyll-sitemap plugin for Jekyll websites.

Collecting a listing of all main URLs is an efficient alternate step if a sitemap just isn’t doable. The URLs ought to cowl all potential layouts of the web site, equivalent to pages with the best site visitors or probably the most landings. This strategy will not catch all accessibility points, particularly content material stage issues, however it would take a look at the format and predominant pages.

This state of affairs requires the npm or yarn package deal managers. I used npm for this text. If your challenge does not have npm initialized, run the npm init command to create the package deal.json file.

Begin with Pa11y

Pa11y is a free and open supply software program that assessments web sites for accessibility points. Pa11y-ci is the command line utility geared in the direction of steady integration (CI). Install pa11y-ci as a growth dependency with npm:

$ npm i --save-dev pa11y-ci

After you full the set up, edit the package deal.json and add the next instructions to the scripts part:

"start-detached": "bundle exec jekyll serve --detach",
"pa11y-ci:home": "pa11y-ci http://127.0.0.1:4000",
"pa11y-ci:sitemap": "pa11y-ci --sitemap http://127.0.0.1:4000/sitemap.xml --sitemap-find https://accessibility.civicactions.com --sitemap-replace http://127.0.0.1:4000 --sitemap-exclude "/*.pdf""

• start-detached: Starts the net server that can run Jekyll for testing.
• pa11y-ci:residence: Runs pa11y-ci assessments on the house web page. Useful for troubleshooting.
• pa11y-ci:sitemap: Runs pa11y-ci assessments utilizing the sitemap and excludes PDFs. The sitemap will check with the dwell website URLs, so change these with native URLs for testing within the CI pipeline.

Add a JSON file named .pa11yci that configures pa11y-ci with various options. Here is a pattern file:

{
  "defaults": {
    "concurrency": 1,
    "standard": "WCAG2AA",
    "runners": ["axe", "htmlcs"],
    "ignore": [
      "color-contrast",
      "frame-tested"
    ],
    "chromeLaunchConfig": {
      "args": ["--disable-dev-shm-usage", "--no-sandbox", "--disable-gpu"]
    },
    "reporters": [
      "cli",
      ["./pa11y-reporter-junit.js", { "fileName": "./pa11y-report-junit.xml" }]
    ]
  }
}

• concurrency: I diminished this set to 1 as a result of growing it induced errors (https://github.com/pa11y/pa11y-ci/issues/168 covers the bug, which is likely to be mounted).
• customary: I’ve caught with the default WCAG2AA because the objective for this website.
• runners: I ran axe (run assessments utilizing axe-core) and htmlcs (default, run assessments utilizing HTML CodeSniffer) to cowl all potential accessibility points.
• ignore: With newer variations of axe and a few of the modifications to the location, I bumped into colour distinction false positives. I even have an embedded iframe that requires separate testing that axe will report about. I’ve follow-up points to look at the axe outcomes, so I’m ignoring these standards for now.
• chromeLaunchConfig: pa11y-ci makes use of Chrome, and I discovered that the GitLab CI pipeline requires that the Chrome browser runs correctly within the pipeline.
• stories: I exploit the default command line reporter, however I additionally added a customized reporter that stories on the pa11y-ci leads to a junit format. This got here in helpful for reporting the leads to the GitLab CI pipeline.

That’s it. Run this setup domestically utilizing npm, and you will notice the next output (truncated for brevity):

dmundra in ~/workspace/accessibility/accessibility on department predominant > npm run start-detached

> start-detached
> bundle exec jekyll serve --detach

Configuration file: /Users/dmundra/workspace/accessibility/accessibility/_config.yml
            Source: /Users/dmundra/workspace/accessibility/accessibility
       Destination: /Users/dmundra/workspace/accessibility/accessibility/_site
 Incremental construct: disabled. Enable with --incremental
      Generating...
                    completed in 8.217 seconds.
 Auto-regeneration: disabled when operating server indifferent.
    Server handle: http://127.0.0.1:4000
Server indifferent with pid '14850'. Run `pkill -f jekyll' or `kill -9 14850' to cease the server.
dmundra in ~/workspace/accessibility/accessibility on department predominant > npm run pa11y-ci:sitemap

> pa11y-ci:sitemap
> pa11y-ci --sitemap http://localhost:4000/sitemap.xml --sitemap-exclude "/*.pdf"

Running Pa11y on 110 URLs:
 > http://localhost:4000/guide/glossary - 0 errors
 > http://localhost:4000/guide/introduction - 0 errors
 > http://localhost:4000/guide/history - 0 errors
 > http://localhost:4000/guide/design - 0 errors
...

✔ 110/110 URLs handed

The website passes the assessments. Here is an instance job operating in GitLab. The pa11y configuration continues to check all website pages for accessibility points and report on them.

What does an error appear to be? Here is an instance:

 > http://localhost:4000/guide/introduction - 1 errors

Errors in http://localhost:4000/guide/introduction:

 • <ul> and <ol> should solely instantly include <li>, <script> or <template>
   components (https://dequeuniversity.com/rules/axe/3.5/list?application=axeAPI)

   (#main-content > div:nth-child(2) > div > div > div > div:nth-child(1) > nav
   > ul)

   <ul class="usa-sidenav">

You get a rely of the variety of errors at a given URL after which particulars on the accessibility situation. It additionally shows a hyperlink to the factors being violated and the situation within the HTML of the problem.

Try Cypress

Cypress is a JavaScript testing framework and may be very useful in writing assessments that work together with the location and assert that options work as anticipated. The setup for Cypress is similar to pa11y-ci by way of set up with npm.

$ npm i --save-dev cypress cypress-axe cypress-real-events

After the set up is full, edit the package deal.json and add the next instructions to the scripts part:

"cypress-tests": "cypress run --browser chrome --headless"

• cypress-tests: Run the Cypress assessments with a headless Chrome browser.

When launching Cypress for the primary time, you get a wizard to create the configuration file. Here is a pattern file:

const { defineConfig } = require('cypress')

module.exports = defineConfig({
  video: true,
  videosFolder: 'cypress/outcomes',
  reporter: 'junit',
  reporterOptions: {
    mochaFile: 'cypress/outcomes/junit.[hash].xml',
    toConsole: false,
  },
  screenshotsFolder: 'cypress/outcomes/screenshots',
  e2e: {
    // We've imported your outdated cypress plugins right here.
    // You could wish to clear this up later by importing these.
    setupNodeOccasions(on, config) {
      return require('./cypress/plugins/index.js')(on, config)
    },
    baseUrl: 'http://localhost:4000',
  },
})

• video: Take movies of the assessments, that are useful for troubleshooting.
• videosFolder: Defines the video storage folder.
• reporter: Set to junit to make it simpler to report the leads to the GitLab CI pipeline.
• reporterOptions: Includes a path for the junit recordsdata and the key phrase [hash] to protect distinctive stories for every take a look at file (in any other case, the file is overwritten). Skip the console output for the reporter and use the default output.
• screenshotsFolder: Defines the screenshot storage folder (helpful for troubleshooting).
• e2e: References the native URL of the location and the plugins.

After establishing Cypress and writing some assessments (see beneath for examples), run the assessments domestically utilizing npm. You will see the next output (truncated for brevity):

dmundra in ~/workspace/accessibility/accessibility on department predominant > npm run cypress-tests

> cypress-tests
> cypress run --browser chrome --headless

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        11.2.0                                                                         │
  │ Browser:        Chrome 109 (headless)                                                          │
  │ Node Version:   v18.10.0 (/usr/native/Cellar/node/18.10.0/bin/node)                             │
  │ Specs:          5 discovered (accordion.cy.js, residence.cy.js, pictures.cy.js, menu.cy.js, search.cy.js)  │
  │ Searched:       cypress/e2e/**/*.cy.{js,jsx,ts,tsx}                                            │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  search.cy.js                                                                    (5 of 5)

  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        1                                                                                │
  │ Passing:      1                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     2 seconds                                                                        │
  │ Spec Ran:     search.cy.js                                                                     │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


  (Video)

  -  Started processing:  Compressing to 32 CRF                                                     
  -  Finished processing: /Users/dmundra/workspace/accessibility/accessibility/cypres    (0 seconds)
                          s/outcomes/search.cy.js.mp4     
 
...
  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  search.cy.js                             00:02        1        1        -        -        - │

...

While Pa11y-ci can take a look at interactivity, Cypress and its plugins can do way more. For a Jekyll website, I discovered that pa11y-ci didn’t catch any accessibility points in cell drop-down menu, dynamic search, or accordion options. I ran Cypress assessments to work together with the weather (like performing searches, clicking menus, or clicking the accordion) after which checked if the outcomes nonetheless handed accessibility assessments. Here is the search example:

describe('Search', () => {
  it('needs to be accessible', () => {
    cy.go to('/search')
    cy.get('#search-input').sort('accessibility')
    cy.checkA11yWithMultipleViewPorts()
  })
})

Here is a fast video of the operating take a look at.

The above take a look at visits the search web page, varieties the phrase “accessibility” within the search area, after which checks the outcomes for accessibility points. I exploit the cypress-axe plugin to verify accessibility points with axe core, identical to pa11y-ci. I’ve wrapped the cypress-axe capabilities in a perform to check a number of window sizes and report on the problems in a desk format.

I additionally use the plugin cypress-real-events to work together with the location with a keyboard to verify that the options are keyboard-accessible. Keyboard accessibility is a crucial consideration (Operable precept of WCAG), and having an automatic take a look at that may affirm the options are keyboard accessible implies that, possibly, there’s one much less take a look at to run manually. You can see an instance of the take a look at here.

Here is an instance of what an error seems like:

  Running:  a11y/anonymous_a11y.cy.js                                                      (1 of 36)
cy.log(): Accessibility scanning: Home (/)
cy.log(): 4 accessibility violations have been detected
┌─────────┬────────────────────────┬────────────┬────────────────────────────────────────────────────────────────────────────────────┬───────┐
│ (index) │           id           │   impression   │                                    description                                     │ nodes │
├─────────┼────────────────────────┼────────────┼────────────────────────────────────────────────────────────────────────────────────┼───────┤
│    0    │      'image-alt'       │ 'crucial' │   'Ensures <img> components have alternate textual content or a task of none or presentation'   │   4   │
│    1    │      'link-name'       │ 'critical'  │                       'Ensures hyperlinks have discernible textual content'                        │   3   │
│    2    │ 'page-has-heading-one' │ 'reasonable' │ 'Ensure that the web page, or a minimum of considered one of its frames comprises a level-one heading' │   1   │
│    3    │        'area'        │ 'reasonable' │                'Ensures all web page content material is contained by landmarks'                │   2   │
└─────────┴────────────────────────┴────────────┴────────────────────────────────────────────────────────────────────────────────────┴───────┘

Cypress logs present a rely of the variety of errors at a given URL after which particulars on what the accessibility situation is, the impression, and its location.

You can discover further particulars and examples within the Cypress folder.

Use the GitLab CI/CD

Now that you’ve got pa11y-ci and Cypress operating domestically, see the right way to run automated accessibility assessments in GitLab utilizing CI/CD features. The GitLab repository is accessible here. Here is the .gitlab-ci.yml file setup:

phases:
- take a look at

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - node_modules/
    - .npm/
    - vendor/ruby

default:
  picture: ruby:2
  before_script:
    - apt-get replace
    - apt-get set up -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libnss3 lsb-release xdg-utils wget libgbm1 xvfb
    - apt-get set up -y nodejs npm
    - bundle set up -j $(nproc) --path vendor/ruby
    - npm ci --cache .npm --prefer-offline
    - npm run start-detached

pa11y-tests:
  stage: take a look at
  script:
    - npm run pa11y-ci:sitemap
  artifacts:
    when: at all times
    stories:
      junit:
        - pa11y-report-junit.xml
    expire_in: 1 day

cypress-tests:
  stage: take a look at
  script:
    # Install chrome browser manually, taken from https://github.com/cypress-io/cypress-docker-images/blob/master/browsers/node16.14.2-slim-chrome100-ff99-edge/Dockerfile#L48
    - wget --no-verbose -O /usr/src/google-chrome-stable_current_amd64.deb "http://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_105.0.5195.125-1_amd64.deb"
    - dpkg -i /usr/src/google-chrome-stable_current_amd64.deb
    - rm -f /usr/src/google-chrome-stable_current_amd64.deb
    - npm run cypress-tests
  artifacts:
    when: at all times
    paths:
      - cypress/outcomes/
    stories:
      junit:
        - cypress/outcomes/*.xml
    expire_in: 1 day

The file at present defines just one stage take a look at and caches folders that retailer dependencies when put in. Then:

1. Steps utilized by all phases:
   1. Use the Ruby model 2 picture as a result of it’s suitable with the present Jekyll set up.
   2. I set up many dependencies primarily based on the documentation at running puppeteer on GitLab. Install node and npm to put in website dependencies.
   3. Install the Jekyll Ruby dependencies.
   4. Install the Cypress and pa11y-ci dependencies through npm.
   5. Start the net server.
2. Run the pa11y-ci to check the location and seize the output to a file.*
3. Install the Chrome browser dependencies in cypress-tests utilizing steps offered by Cypress of their Docker picture configurations. Run the Cypress assessments and seize the output to recordsdata.*

* Capture the output of Cypress and pa11y-ci assessments as junit XML recordsdata.

Here is an instance screenshot of the GitLab pipeline (taken from https://gitlab.com/civicactions/accessibility/-/pipelines/744894072):

Image by:

(Daniel Mundra, CC BY-SA 4.0)

Here is an instance of the take a look at leads to the identical pipeline:

Image by:

(Daniel Mundra, CC BY-SA 4.0)

GitLab CI/CD automatically take junit XML files and outputs them in a transparent format. Cypress assessments present the junit XML output as a part of their options (see above). I created a customized reporter for pa11y-ci to output the format in junit (credit score to macieklewkowicz/pa11y-reporter-junit).

Note: GitLab model 12.8+ helps Pa11y accessibility assessments (see https://docs.gitlab.com/ee/ci/testing/accessibility_testing.html for particulars). The above setup permits for personalisation of the pa11y-ci and likewise concentrating on of native URLs. I like to recommend utilizing their choices for dwell websites.

Wrap up

Using the above steps, you’ll be able to present accessibility testing in your website domestically and in CI. This course of helps you observe and repair accessibility points in your website and within the content material. An vital caveat about automated testing is that it solely catches 57% of issues, so that you undoubtedly wish to embody handbook testing along with your accessibility testing.

Further studying and examples

Thank you to Marissa Fox and Mike Gifford in your help, ideas, and suggestions.