Automated Testing Tool Limitations
Most people today will agree that testing for accessibility, in particular Section 508 compliance, is a good idea. Unfortunately, it is unclear how best to test for accessibility in a fashion that provides a reasonable degree of testing coverage for compliance requirements.
In general most tests for accessibility in place today utilize an automated testing tool to find violations. These approaches concern themselves with testing a given IT system, usually a web site, against a set of automatic tests. This approach has some significant benefits: it is generally cheap, easy to use, and fast. While cheap and fast, however, it is effective only at validating a certain sub-set of accessible requirements and can not provide a comprehensive validation of accessibility. The scope of coverage varies amongst tools but is generally around 25% of the total accessibility requirements. As an example, nearly all tools can test for the presence of alternative text but no tools can test if alternative text is a meaningful replacement for an image. As a second example, tools can test for a form field label, but not whether an assistive technology user can fill out a form – the requirement of most accessibility standards.
Automated tools tend to be limited to a specific technology platform. Tools that test Web content are not going to test your PDF content. Additionally, tools do not exist for many major technology platforms in wide use today.
For the purposes of evaluating Section 508 compliance in particular it should be noted that the requirements of the Section 508 functional standards (§1194.31) by definition must be evaluated with Assistive Technology (AT) testing. As such, testing for Section 508 compliance requires testing both normative – or rules based – and AT based testing. Thus without the completion of AT testing you will not have information to create a responsible claim of compliance with the Section 508 requirements.
In sum, automatic testing is a cheap and easy way to validate a sub-set of 25% of compliance violation quickly. In general the tools are robust, and well developed tools but all suffer from the same limitations on what can be tested automatically. Thus, if automated testing is all your organization uses to determine compliance, you will be unable to make a responsible claim of accessibility or compliance.
There are many automated tools out on the market today. A group of them have jumped into the mainline Section 508 market, by adding some minimal functionality to an already existing testing tool platform, in hopes of grabbing additional customers. These tools in general do not measure up to the specific Section 508 testing tools on the market which SSB would recommend. Among the Section 508 specific tools, Deque, HiSoftware, and SSB BART Group all provide tools which widely overlap and all of them will provide you with an excellent automated testing tool.
It is important to try to educate fully any person that is part of the Section 508 testing process, so that we are not left with the danger of lack of understanding and the resulting frustration that results. Section 508 testing involves human interaction, but more specifically the human must have knowledge of accessibility issues and solutions. This includes the Section 508 law, the automated testing tool that they are using, the barriers that people with disabilities experience, and the assistive technologies that they use on a daily basis to gain access to IT. Lastly, it is worth noting that when we consider disabilities, we are not just considering blindness, but the entire spectrum of disabilities, including ones that we cannot identify visually. This total group of people that are impacted by inaccessible IT is 23% of our population.
For more information two articles by know accessibility experts in the field are recommended as areas of further research:
- Jim Thatcher.com, Web Accessibility Testing
- Myth: Evalutation Tools Can Determine Accessibility and conformance to Standards, Springer-Verlag, 2006