Exploring Accessibility Testing - A Look at WCAG, Axe-Core Automation, and Issue Severity Levels

This guide provides an overview of accessibility testing, focusing on WCAG (Web Content Accessibility Guidelines) compliance, automated testing using the Axe-Core library, and severity levels of issues identified during testing.

Table of Contents

1. WCAG Levels
2. WCAG 2.0 Level A & AA Rules
3. Test Scope
4. Out of Scope
5. Automated Test
6. Axe-Core Report Severity Levels
7. Useful Links
8. Axe-Core Supported Browsers
9. Automated Accessibility Testing with Axe-Playwright

1. WCAG Levels

WCAG divides accessibility guidelines into three levels of conformance:

• WCAG Level A: The minimum level of conformance.
• WCAG Level AA: The target conformance level, often required by legal standards like ADA (Americans with Disabilities Act).
• WCAG Level AAA: The highest level of conformance, meeting all three levels' success criteria.

Compliance with Level AA implies compliance with Level A.

2. WCAG 2.0 Level A & AA Rules

For a comprehensive list of WCAG 2.0 Level A and AA rules, refer to this resource.

3. Test Scope

Your testing scope should align with WCAG Level AA compliance. According to WCAG, adherence to the following four main principles is crucial:

• Operable: Ensure users can navigate your website using a keyboard in addition to a mouse.
• Perceivable: Make information and content accessible through audio descriptions, transcriptions, and closed captions.
• Robust: Your site should be compatible with screen readers and adaptable to users' changing needs.
• Understandable: Ensure text is readable, expands abbreviations, and uses formats that don't require advanced reading abilities.

4. Out of Scope

Accessibility testing should not include testing with assistive technologies that real users use. For a more comprehensive understanding of users with disabilities, refer to WebAIM's survey results.

5. Automated Test

Automated accessibility testing is crucial for maintaining compliance. In the case of MAP, it will be automated using the Axe-Core library within the End-to-End (E2E) automation framework using Playwright. Configure Axe-Core to ensure MAP complies with WCAG AA rules for ADA compliance. Refer to Axe-Core on GitHub for detailed information.

6. Axe-Core Report Severity Levels

During automated testing with Axe-Core, issues are categorized into four severity levels:

• Minor: Considered nuisances or annoyances. Fix promptly if it takes minimal effort; otherwise, prioritize appropriately.
• Moderate: Result in some barriers but don't prevent access to fundamental features. Prioritize in the release if no higher-priority issues exist.
• Serious: Result in significant barriers and frustration for people with disabilities. Remediation should be a priority.
• Critical: Block content for people with disabilities, putting your organization at risk. Prioritize immediate remediation.

7. Useful Links

For further resources on accessibility testing and Axe-Core:

• Axe-Core on npm
• Axe-Playwright on GitHub
• Deque's Axe-Core Documentation

8. Axe-Core Supported Browsers

Axe-Core, used for automated accessibility testing, supports the following browsers:

• Microsoft Edge v40 and above
• Google Chrome v42 and above
• Mozilla Firefox v38 and above
• Apple Safari v7 and above

Ensuring accessibility in your digital content is essential to make it inclusive and ADA-compliant. By following these guidelines and leveraging Axe-Core for automated testing, you can proactively address accessibility issues and provide a more accessible online experience for all users.

9. Automated Accessibility Testing with Axe-Playwright

In this example, we will demonstrate how to perform automated accessibility testing using Axe-Playwright within the Playwright framework on the Wikipedia website. We will use TypeScript for scripting this test.

import { Page, BrowserContext, devices } from 'playwright';
import { setupPlaywright } from 'axe-playwright';
 
// Define the URL of the Wikipedia website you want to test.
const wikipediaURL = 'https://en.wikipedia.org';
 
// Define the Playwright device to emulate a desktop view.
const device = devices['Desktop 1920x1080'];
 
(async () => {
  // Launch a browser instance and create a browser context with the Playwright device configuration.
  const browser = await playwright.chromium.launch();
  const context: BrowserContext = await browser.newContext({
    ...device,
  });
 
  // Create a new page within the context and navigate to the Wikipedia website.
  const page: Page = await context.newPage();
  await page.goto(wikipediaURL);
 
  // Inject Axe-Core into the page for accessibility testing.
  const { configureAxe, toHaveNoViolations } = setupPlaywright();
  await configureAxe(page);
 
  // Run an accessibility scan on the page and check for violations.
  await page.evaluate(() => axe.run());
  const results = await page.waitForFunction(toHaveNoViolations, { timeout: 5000 });
 
  // Output the accessibility results to the console.
  console.log('Accessibility test results:', results);
 
  // Close the browser instance.
  await browser.close();
})();

import { Page, BrowserContext, devices } from 'playwright';
import { setupPlaywright } from 'axe-playwright';
 
// Define the URL of the Wikipedia website you want to test.
const wikipediaURL = 'https://en.wikipedia.org';
 
// Define the Playwright device to emulate a desktop view.
const device = devices['Desktop 1920x1080'];
 
(async () => {
  // Launch a browser instance and create a browser context with the Playwright device configuration.
  const browser = await playwright.chromium.launch();
  const context: BrowserContext = await browser.newContext({
    ...device,
  });
 
  // Create a new page within the context and navigate to the Wikipedia website.
  const page: Page = await context.newPage();
  await page.goto(wikipediaURL);
 
  // Inject Axe-Core into the page for accessibility testing.
  const { configureAxe, toHaveNoViolations } = setupPlaywright();
  await configureAxe(page);
 
  // Run an accessibility scan on the page and check for violations.
  await page.evaluate(() => axe.run());
  const results = await page.waitForFunction(toHaveNoViolations, { timeout: 5000 });
 
  // Output the accessibility results to the console.
  console.log('Accessibility test results:', results);
 
  // Close the browser instance.
  await browser.close();
})();

It launches a browser, navigates to the Wikipedia site, injects Axe-Core for accessibility testing, scans the page, and checks for violations. The results are then logged to the console. Ensure you have Playwright and Axe-Playwright installed in your project to run this code.