Automated Accessibility Testing Guide

Overview

This guide explains how to use the automated accessibility testing integrated into the Fawkes platform CI/CD pipeline. The system uses axe-core and Lighthouse CI to ensure WCAG 2.1 AA compliance for all components.

Table of Contents

• Quick Start
• Testing Tools
• Running Tests Locally
• CI/CD Integration
• Understanding Test Results
• Fixing Violations
• Dashboard and Metrics
• Troubleshooting

Quick Start

Run All Accessibility Tests

cd design-system
npm run accessibility:full

Run Only Axe-Core Tests

cd design-system
npm run test:a11y

Run Only Lighthouse CI

cd design-system
npm run build-storybook
npm run lighthouse:ci

Testing Tools

Axe-Core

axe-core is a fast, lightweight accessibility testing engine that runs automated WCAG 2.0, 2.1, and Section 508 accessibility tests.

What it tests:

• Color contrast
• ARIA attributes
• Semantic HTML
• Keyboard navigation
• Form labels
• Image alt text
• And 90+ other rules

Lighthouse CI

Lighthouse CI provides automated auditing, performance metrics, and best practices for web pages.

What it tests:

• Accessibility score (0-100)
• WCAG compliance
• Best practices
• SEO fundamentals

Running Tests Locally

Prerequisites

cd design-system
npm install

Run Axe-Core Tests

# Run all accessibility tests
npm run test:a11y

# Run in watch mode for development
npm run test:watch -- --testPathPattern=a11y

# Run with coverage report
npm run test:coverage

Run Lighthouse CI

# Build Storybook first
npm run build-storybook

# Run Lighthouse CI
npm run lighthouse:ci

# Or run individual commands
npm run lighthouse:collect
npm run lighthouse:assert

Writing Accessibility Tests

Create tests for your components using jest-axe:

import { render } from '@testing-library/react';
import { axe, toHaveNoViolations } from 'jest-axe';
import { MyComponent } from './MyComponent';

expect.extend(toHaveNoViolations);

describe('MyComponent Accessibility', () => {
  it('should have no accessibility violations', async () => {
    const { container } = render(<MyComponent />);
    const results = await axe(container);
    expect(results).toHaveNoViolations();
  });

  it('should be keyboard accessible', async () => {
    const { container } = render(<MyComponent />);
    const results = await axe(container, {
      rules: {
        'keyboard-access': { enabled: true }
      }
    });
    expect(results).toHaveNoViolations();
  });
});

CI/CD Integration

GitHub Actions

Accessibility tests run automatically on:

• Every pull request
• Push to main branch
• Daily at 9 AM UTC (scheduled)
• Manual workflow dispatch

Workflow: .github/workflows/accessibility-testing.yml

Jenkins Pipeline

For services using the Golden Path pipeline, add accessibility testing:

@Library('fawkes-pipeline-library') _

goldenPathPipeline {
    appName = 'my-service'
    language = 'node'
    // ... other config
}

// Add accessibility stage after unit tests
stage('Accessibility Tests') {
    steps {
        accessibilityTest {
            runAxeCore = true
            runLighthouse = true
            wcagLevel = 'AA'
            failOnViolations = true
            lighthouseScoreThreshold = 90
        }
    }
}

Quality Gates

The pipeline enforces these quality gates:

Gate	Threshold	Action
Axe-Core Tests	0 violations	❌ Fail build
Lighthouse Score	≥ 90/100	❌ Fail build
WCAG 2.1 AA	Must comply	❌ Fail build
Color Contrast	4.5:1 text, 3:1 large	❌ Fail build

Understanding Test Results

Axe-Core Results

Violations are categorized by severity:

• Critical: Serious accessibility issues that must be fixed immediately
• Serious: Major accessibility problems that significantly impact users
• Moderate: Noticeable accessibility issues
• Minor: Small accessibility improvements

Example output:

❌ 2 violations found:

1. color-contrast (serious)
   - Element: <button class="btn-primary">Submit</button>
   - Description: Element's background color could not be determined due to a background image
   - Fix: Ensure sufficient color contrast (4.5:1 for normal text)
   - More info: https://dequeuniversity.com/rules/axe/4.8/color-contrast

2. label (critical)
   - Element: <input type="text" id="email">
   - Description: Form element does not have an associated label
   - Fix: Add a <label> element or aria-label attribute
   - More info: https://dequeuniversity.com/rules/axe/4.8/label

Lighthouse Results

Lighthouse provides a score from 0-100:

• 90-100: ✅ Good (meets WCAG 2.1 AA)
• 70-89: ⚠️ Needs improvement
• 0-69: ❌ Poor (does not meet requirements)

Reports include:

• Overall accessibility score
• Detailed audit results
• Opportunities for improvement
• Passed audits

Fixing Violations

Common Violations and Fixes

1. Missing Alt Text

Problem:

<img src="logo.png" />

Fix:

<img src="logo.png" alt="Company logo" />

2. Poor Color Contrast

Problem:

.text {
  color: #999;
  background: #fff;
  /* Contrast ratio: 2.85:1 (fails) */
}

Fix:

.text {
  color: #666;
  background: #fff;
  /* Contrast ratio: 5.74:1 (passes) */
}

3. Missing Form Labels

Problem:

<input type="email" id="email" />

Fix:

<label htmlFor="email">Email Address</label> <input type="email" id="email" />

Or use aria-label:

<input type="email" aria-label="Email Address" />

4. Invalid ARIA Attributes

Problem:

<div role="button" aria-pressed="yes">Click me</div>

Fix:

<button aria-pressed="true">Click me</button>

5. Missing Keyboard Access

Problem:

<div onClick="{handleClick}">Clickable</div>

Fix:

<button onClick="{handleClick}">Clickable</button>

Or make div keyboard accessible:

<div role="button" tabindex="{0}" onClick="{handleClick}" onKeyPress="{(e)" ="">
  e.key === 'Enter' && handleClick()} > Clickable
</div>

Testing Your Fixes

1. Make the fix in your component
2. Run tests locally:

   npm run test:a11y
   

3. Verify the specific test passes
4. Commit and push
5. Check CI results

Dashboard and Metrics

Grafana Dashboard

Access the accessibility dashboard at:

https://grafana.fawkes.idp/d/accessibility

Dashboard features:

• Overall accessibility score trend
• Test pass/fail rate
• Violations by severity
• Violations by component
• WCAG 2.1 AA compliance status
• Historical metrics

Prometheus Metrics

Query these metrics in Prometheus:

# Accessibility score
lighthouse_accessibility_score

# Test pass rate
sum(accessibility_tests_passed) / sum(accessibility_tests_total)

# Critical violations
sum(axe_critical_violations)

# Violations by component
sum by (component) (axe_component_violations)

# WCAG compliance status
wcag_aa_compliance_status

Troubleshooting

Tests Pass Locally But Fail in CI

Cause: Environment differences, missing dependencies, or timing issues.

Solution:

1. Check CI logs for specific errors
2. Ensure all dependencies are in package.json
3. Run tests with CI flag locally:

   npm run test:a11y:ci
   

4. Check for race conditions in async tests

Lighthouse Score is Lower Than Expected

Cause: Storybook build issues, network problems, or threshold too high.

Solution:

1. Build Storybook locally and inspect:

   npm run build-storybook
   npx http-server storybook-static
   

2. Check Lighthouse report for specific issues
3. Review lighthouserc.json configuration
4. Temporarily lower threshold while fixing issues

False Positives

Cause: Axe-core might flag valid patterns as violations.

Solution:

1. Review the violation carefully
2. Check if it's a legitimate issue
3. If it's a false positive, add to .axerc.json:

   {
     "rules": {
       "rule-id": { "enabled": false }
     }
   }
   

4. Document why you disabled the rule

GitHub Actions Workflow Failing

Cause: Workflow configuration issues or missing secrets.

Solution:

1. Check workflow logs in GitHub Actions
2. Verify all required secrets are set
3. Check file paths are correct
4. Ensure Node version is compatible
5. Re-run workflow after fixes

Jenkins Pipeline Hanging

Cause: Lighthouse might be waiting for user input or timing out.

Solution:

1. Check Jenkins console output
2. Increase timeout in lighthouserc.json:

   {
     "ci": {
       "collect": {
         "numberOfRuns": 1
       }
     }
   }
   

3. Use headless mode in Jenkins agent

Resources

WCAG Guidelines

• WCAG 2.1 Quick Reference
• Understanding WCAG 2.1
• How to Meet WCAG

Tools Documentation

• axe-core Documentation
• jest-axe Documentation
• Lighthouse CI Documentation

Learning Resources

• WebAIM Articles
• A11y Project
• MDN Accessibility
• Inclusive Components

Browser Extensions

• axe DevTools
• WAVE
• Lighthouse

Getting Help

If you need help with accessibility testing:

1. Check this documentation
2. Review existing test examples in design-system/src/
3. Check the A11y Project
4. Ask in #accessibility Slack channel
5. Create an issue with the accessibility label

Contributing

To improve accessibility testing:

1. Add more comprehensive tests
2. Update test utilities
3. Improve documentation
4. Share learnings with the team
5. Propose new quality gates

---

Last Updated: 2025-12-24 Maintained By: Fawkes Platform Team