Use Case Testing: What is it, where does it fit, and is it right for your situation?
As a member of the Accessibility Services team at SSB BART Group (SSB), I work with many types of industry and government clients, some of which include use case testing in their engagement. This blog post will explain what use case testing is, what it can tell you, and when it is most effective to use.
What is use case testing?
At the end of the day, use case testing is real-world testing. That is, it seeks to simulate – as closely as possible – a “real” user’s experience with various facets of a client’s product, whether it be their website, mobile app or device. SSB uses our industry-leading experience to assist clients in triaging their various offerings with multiple testing strategies. For core scenarios integral to use of the products, use case testing is performed in addition to automatic and manual testing.
Use case testing is performed with users that have disabilities using assistive technology, accessibility features, or other strategies that a person with that disability would commonly use. The results of each use case is scored objectively. SSB uses a scoring system of one through five to rate individual use cases as well as an overall average score; five indicating no accessibility issues and one indicating severe problems that pose a barrier to access. While users can indicate efficiency and effectiveness in the use case notes, the intent of the score is to document the presence or absence of barriers and issues that would impact the user’s ability to access the service. The use case gets at what the impact to the user is, and how it might affect the user’s ability to carry out a given task. Code examples and solutions are not provided in the use case notes… a description of the challenges or lack thereof are documented by the user instead.
Use cases are selected from core tasks that are integral to use of a site or product. For example, a retail/ecommerce client may want to test the accessibility of various situations:
- creating a new account
- logging onto the site or mobile app
- searching for and investigating various core products within their catalog
- adding products to the shopping cart
- checking out and viewing order status
Each of these core scenarios can be broken down into more granular use cases as appropriate and to stay within budget. Error states and alternative paths should also be documented as part of the use case and tested during the use case.
A telecommunications company may wish to test accessibility with regard to the ability to order services, i.e., explore TV, Internet and phone options, customize those options, pay bills, schedule an appointment. Because many telecommunication providers offer viewing options on PCs, tablets, and mobile devices, being able to navigate electronic programming guides, schedule and view digital video recordings, and order pay-per-view is also core to the business, and each of these can make excellent use cases.
Travel clients could test the accessibility of booking travel, e.g., select destinations/accommodations, compare fares, purchase add-ons and pay for bookings. Additional use cases could also be performed testing loyalty program navigation, award redemption, etc.
Education clients could test the accessibility of navigating online textbooks, viewing grades, creating/completing assessments and interacting within discussion boards. In addition, applying for admission, registering for courses, navigating and signing up for events are great candidates for use case testing of university-specific scenarios.
State and city government clients may wish to test navigating government facilities, reporting issues to various departments, filling out forms, viewing government meetings/minutes, etc.
Finally, hospitality industry clients may wish to test the accessibility of their online ordering website and/or mobile app, accessibility of their in-house order terminals and delivery tracking systems.
Now that we have an idea of what different industries may wish to test using the use case methodology, the next question is:
Where Does It Fit?
SSB BART Group employs an end-to-end approach to accessibility testing, remediation and policy. It is beyond the scope of this blog post to outline all SSB’s offerings, but putting use case testing into context within the accessibility audit methodology can be helpful to understand where it fits.
Automated testing is just that: automated. A few examples of automated testing could include:
Spidering: Spidering technology can be used to “walk” through a site’s various web pages. As mistakes within the code related to accessibility are found, they are logged for further review.
In-page Testing: In this scenario, a script is inserted on the page and the page is automatically tested on load. Then, as the user interacts with the page, newly revealed content is tested automatically.
Behavior Driven Development (BDD) and Test Driven Development (TDD): This can be used to automatically test certain core tasks and pathways through a site, but does not address all areas of a user’s experience. For example, it can verify that certain accessibility information is present when a keystroke is pressed on a control or when a user action occurs, but it cannot describe the experience with a given assistive technology on the page holistically.
Providing specific URLs to test: This is more targeted than general “spidering” because only a specific list of web addresses are tested.
On the surface, the examples above may seem like the best possible ways to test for accessibility: automation keeps costs down, minimizes human error, can be performed extremely quickly, and is almost infinitely scalable with the right tools. So, are there any drawbacks? Indeed, there are!
It is quite possible to write code that, on the surface, meets accessibility standards but will be totally inaccessible or have very limited usability. Examples of this are alternative text and form labels. Random numbers and letters can be placed into alternative text tags, form fields and so on, and not be caught by today’s automated testing. While computer learning shows progress in this area, there remain some aspects of testing that require a human. To truly have a complete view of the accessibility of your site or application, manual and use case (human) testing must be an integral part of your audit plan.
The human side of SSB’s testing methodology comes in two categories: manual testing and – you guessed it – use case testing.
Manual testing is when a person reviews items that cannot be tested automatically or potential issues that were detected automatically. Sometimes automated tools can flag false positives and a manual review is needed to determine if the issue really is an accessibility issue. Manual testing may also involve using code inspection, using a tool, or testing using only the keyboard.
Another part of manual testing involves going through a site or application using one or more assistive technologies. These include, but aren’t limited to, voice dictation, switch controls, screen magnification/ high contrast and screen readers. So, an obvious question is, “What’s the difference between general manual testing and use case testing?”
Manual testing does not necessarily have to be performed by someone with a disability; manual testers will often not be native users of assistive technology. That is, someone manually testing using a screen reader may not have a visual disability, so they will be testing in a very “clinical” way, for lack of a better term. To clarify, this type of testing is not scenario-based. The manual tester will just “walk” through the page or screen (or “module” in SSB terminology), noting which best practices were violated, where the violation was located and why the violation occurred. In short, manual testing may catch individual component issues but doesn’t provide a full view of the task.
Historically, use case testing is conducted by a native user of an assistive technology, i.e., someone who is deeply familiar with that technology and its application/function. So, when this technique is used, the human element is put back into the testing/validation process. For instance, a manual tester would note a violation of the lack of alternative text on a photograph of the latest skinny jeans. A use case tester would note that they would not buy the skinny jeans because they did not know exactly what the color pallet was. If other images were present, showing an entire outfit for example, that also lacked alternative text, a non-visual user would not be able to receive this information, therefore decreasing the odds of buying additional accessories.
As you can see, use case testing can provide “color” to black-and-white manual testing, which brings us to our final question:
Is It Right for Your Situation?
Frankly, each client’s situation is and will be unique. As a matter of fact, clients who engage with SSB more than once will have different needs each time. However, there are some guidelines you can use to help you determine if you should include use case testing as part of your engagement:
- How far along the path is your product?
- Is this the initial release?
- Has it been tested at all for accessibility and compliance?
- Have remediations been done?
If you answered “no” to at least two of the above questions, then SSB would most likely recommend including use case testing in the mix. This would provide project managers and less technical folks with real-world scenarios they could present to upper management, which could potentially influence funding to fix any issues which were discovered. Simply relying on technical violations may not have the same impact on those outside of the technical team. On the other hand, if this is an update to an already-established, well-tested product, use case testing may not be necessary, or perhaps only targeted use case testing, e.g., testing only new or overhauled features rather than a more end-to-end approach, may suffice.
Primary use: Is your site or application for use by the public? If so, your exposure for accessibility risk could be high. If your product is specifically used by a small subset of the population, use case testing may not need to be done outside; you could do some in-house testing if you employ persons with disabilities, and/or your technical QA team could do its own manual testing using assistive technologies.
Are you a potential legal target? In recent years, litigation has been brought against companies in the retail and financial sectors, as well as state and local government. If you are a player in any of these industries, use case testing could provide additional support for your case, should you be the subject of legal action. Performing use case testing on the integral functions of your site can reduce the risk that users will run into issues on the core paths of your product. Performing use case testing does not reduce your need for the other types of testing discussed in this post however; SSB still recommends that automatic and manual testing be prioritized and performed, as the presence of automatically detectable violations can also pose risk to your organization.
Those are a few core examples to consider when determining if use case testing is appropriate to include in your SSB engagement. We are of course here to help you with those decisions and support your compliance needs, now and into the future. If you have additional questions about use case testing or SSB’s Accessibility Services team, please feel free to contact me at Loren.Mikola@ssbbartgroup.com.