Audit Getting Started Guide
SSB assessments are broken up into three phases: Groundwork, Testing, and Delivery. The Groundwork phase involves defining and securing the project resources and finalizing project scope. The Testing phase involves all the testing that is required to deliver an assessment to the client. The Delivery phase involves the analysis of the test findings and the delivery of those results.
The requirements of the Groundwork phase vary primarily based on whether the testing of the IT system is to be performed offsite at SSB or onsite at the client location. Generally, this is determined during the process of developing the Statement of Work (SOW). However, this issue can be subject to change. If you have questions about the location of testing or would like to explore other testing options, please contact your SSB Project Manager.
Offsite Testing Requirements
The most common testing type that SSB performs is offsite testing, from SSB offices. Offsite testing of IT Systems is performed over the Internet for web sites and web applications, via shipment of hardware, installation of software locally, use of a virtual server provided by the client, or via a Virtual Private Network.
Web Site and Web Application Offsite Testing Requirements
For publicly available web sites or web applications, SSB requires only a valid URL to begin testing. For private web sites or web applications, such as those only available on a private site or staging server, SSB requires the appropriate type of access. Generally, these sites or applications fall into two categories:
- Private sites or applications that can be accessed from a public URL. Such web systems are often protected by general authentication (plain text password). For such systems SSB requires a valid username and password to access the site.
- Private sites or applications that are located behind a firewall. For such sites, SSB generally requires VPN Access.
The preferred method for accessing a site or application is through a publicly available URL. While SSB has significant experience setting up VPN clients, this route is generally the most cumbersome, and requires significant coordination with the client IT staff. (For more information see VPN Access.)
For web applications that require a login to utilize the functionality, the client will be responsible for setting up the server, creating the appropriate user accounts, and maintaining the server’s availability. If the server or URL is unavailable, SSB cannot be held to the proposed schedules in the SOW or those dates that were outlined in the project. Another possibility is for the client to deliver to SSB a fully-installed and configured computer that SSB can use to perform its testing.
Whenever possible, it is useful for SSB to have access to the source files used to create the site or application. For example, if an application is produced using Java Server Pages (JSP) technology, SSB would require access to the individual JSP files. Access to source files is not a requirement for testing; however, access to the source code makes the testing results more actionable for a development team.
Software Offsite Testing Requirements
For desktop applications, the client is responsible for making installation materials available to SSB. The installation materials include: all files required for installation, product keys or authorization disks, and access to any manual or supporting documents that may be used during installation. For any piece of desktop software, two methods of installation are available. The most common and preferred method remains the usage of physical installation media, such as CDs or diskettes, to install software. This method is preferred by SSB because it generally is the most formal and well supported installation process, as well as the most likely to be experienced by the end user. For this method of installation, SSB requires three copies of the application of the physical medium. The second technique is to utilize a public download URL site for the software. While this technique is generally the most expedient, it often does not mimic the installation and setup process experienced by the end user. Therefore, SSB recommends this only as a secondary measure if physical installation media cannot be provided.
Virtual Server (VMWare) Testing Requirements
Clients can also deliver to SSB a complete turnkey system in the form of a VMWare image. This is a valuable option that allows the client to provide a system where all required applications and components are already installed, licensed and configured, where an appropriate body of sample data and accounts are already populated, and so on. This can dramatically reduce the amount of time required for system setup, reducing the costs to the client and allowing SSB to begin actual testing much sooner. It also avoids the provision and delivery of physical servers by the client to SSB, as the VMWare image can be delivered via a secure FTP site (the client’s or SSB’s).
SSB uses a variety of VMWare products to meet the varying needs of the applications being tested and the available hardware platforms. SSB has a VMWare ESXi server for Enterprise class applications that require significant hardware resources in order to run well. For the desktop, SSB uses the VMWare Workstation, Fusion, and Player products.
When preparing VMWare images for use by SSB, the following issues must be considered.
Compression and Delivery
- Image disks should be shrunk using the VMWare tools before delivery to reduce image size
- Unnecessary and temporary files should be deleted
- If the images will be uploaded to a secure FTP site, they should first be compressed, which can greatly reduce the download time for these very large files.
- The client may choose to provide credentials to its own secure FTP site, or SSB can provide a dedicated account on our secure FTP site.
- If the images will be delivered via physical media (DVDs), please contact your SSB Account Manager to confirm the address to which they should be sent
- Images should be built to be compatible with the most recent versions of VMWare ESXiand the desktop products (Workstation, Fusion, Player). For hardware compatibility, this would currently be version 7.0. Images made for earlier versions may still work, but there would be slight possibility that problems would occur (such as licensing or conversion needs)
- Images should be left in a raw form so they can be directly added into the VMWare inventory. Images delivered as appliances can take hours to import.
- Images should be delivered in a shut-down state with no enterprise VMWare features (such as VMotion) enabled, to ensure the image can run on a standalone VMWare product.
- The client shall provide a VMWare image that meets the system requirements of the application to ensure acceptable and realistic performance for a light user load.
- All applications and required components including sample data should be installed and ready to use on the image.
- License information for the OS and any Applications provided on the image should be provided and verified to be valid for 90 days from the start of testing. In the event licenses expire or are lost, the client must be able to provide new licenses in a timely fashion to avoid delays in the project.
- If possible, any tools and installers needed to repair the installed applications or data should be included. Image Snapshots may also be used for this purpose.
- Any documentation associated with the Image should either be included with the image delivery as a separate file or email or stored on the image in an obvious location (such as the desktop).
If testing is to be done directly on the VMWare Image rather than with client software or a browser on another machine, the following should be considered:
- Images built for the desktop should have a sound card (ESX compatible images will not have an option for sound)
- Images running on ESX will not have sound, and any testing that must be conducted using a screen reader on the image must be done either via Tandem (JAWS only) or Remote Desktop connection.
- Screen Magnification software does not currently work correctly with VMWare or is too slow to be usable, and any testing using Screen Magnification must be done on real hardware.
- The client should inform SSB of any known issues or differences in the product being tested when running in a VMWare image.
- Testing that needs the use of USB devices or sound cards (such as for Dragon Naturally Speaking, a speech input product) must be done using a VMWare desktop product (Workstation, Fusion or Player.) ESX cannot be used.
Hardware Offsite Testing Requirements
For IT systems that contain hardware, the client is responsible for shipping hardware to SSB. The hardware should be shipped to SSB and packaged in the fashion most likely to be experienced by the end user. This ensures that SSB can simulate the entire user experience.
For all applications that require complex setup, SSB recommends that this setup be performed by the client to avoid errors resulting from configuration mistakes. For applications setup by the client, this generally means access to a server behind the client’s firewall. Such access is granted through a Virtual Private Network, or VPN. This allows someone outside of the company network – such as SSB – to gain access to the authorized resources on the company network via a secure Internet connection. To setup VPN access, you will need to contact your IT staff and inform them of the project requirements. If your company has a VPN already in place, your IT staff should have a process in place for setting up new VPN accounts. If your IT staff is not familiar with VPN setup, it is strongly recommended that you setup the server on a publicly available URL.
A few specific items are required to ensure that VPN access occurs smoothly and without difficulty:
- VPN Clients – Often firewalls will require the usage of third party clients to access resources through a VPN. These VPN clients should be made available to SSB either through a public download URL or via e-mail.
- Documentation for VPN installation and configuration – Generally companies have internal documentation for VPN setup and configuration. Copies of this should be made available to SSB.
- Support for installation and configuration of the VPN – SSB will require access to a named IT resource familiar with issues pertaining to VPN access.
- SSB specific configuration information – SSB should be sent all server URLs, usernames, passwords, and related information required to access the application once the VPN is properly setup.
Setting up and configuring VPN access can often be the most cumbersome part of the assessment process. To ensure that the process goes as smoothly as possible, SSB strongly urges the client to secure the items outlined above as early as possible.
While most testing performed by SSB is performed offsite, sometimes it is convenient or even required to have testing performed onsite, at the client’s location. To support testing that is performed onsite, a few basic requirements must be met.
Setup of Testing Location
To perform testing on an IT system, SSB requires access to a computer with Internet connectivity and the permissions (or availability of IT staff) to install the Assistive Technologies to be utilized during testing. The Assistive Technologies required are outlined in the SOW, and should be setup by the client organization. Although trial versions of Assistive Technologies can be used, they are not preferred because the feature set of trial versions is often not congruent with those of production versions and/or the time-limited trial versions make for inefficient use of testing time. If the computer requires a username and password to login, these should be supplied to SSB .
Physical site access must be provided for the duration of the onsite testing. While escorted site access is acceptable, past experience has shown that it can often be impractical, and unescorted site access is preferred. For sites that require security badges for unescorted site access, contractors should be supplied with the appropriate materials before work begins on the project.
Travel costs to and from the site are invoiced independently of costs associated with the preparation of the assessment. While travel costs do not necessarily need to be pre-approved, generally it is helpful to specify cost budgets before the project begins. Pre-approved travel cost plans should be negotiated with SSB as part of the SOW.
Once SSB has gained access to the IT system, a test set for the system is developed. The test set includes two components: the module list and the use case list. The module list is a list of individual components within the application. The module list components differ for web, software, and hardware based systems. The use case list defines the core uses for an IT system. Use cases define specific sets of actions that users are likely to take within the system. The client should offer, or be asked to provide, guidance that would affect the selection of modules and use cases. Examples include whether to include or exclude the online help, features that will be discontinued or totally re-designed in the next major release, features that are not yet stable enough to assess, that are used only by certain user types (such as administrators), or features for which accessibility complaints have been received.
Module List Development
The module list contains that set of modules to be tested within the application. For web based and software based systems, the module list is the set of visual components to be tested within the application. For hardware systems, the set of components is the combination of individual controls, standard control groups, and displays on the device.
Often a list of modules is already available internally, and can be utilized by SSB as a basis for the final test set. This list is generally maintained by individuals working in a design or QA capacity within the organization. To determine if such a list exists for your organization, contact the individuals working with the design or QA group of your organization. If you are unsure if your organization has such a list, get in touch with your SSB Project Manager.
Working from the client supplied module list or creating a list if no such list exists, the final list of visual modules is developed by working collaboratively with SSB. This process requires the following steps:
- A walkthrough of the application (SSB + Client)
- Development of a draft list of the components within the application (SSB)
- Review of the draft list and identification of changes (SSB + Client)
- Final customer approval of changes (SSB + Client)
Web and Software Modules
Visual components refer to the various pieces of the application’s Graphical User Interface. In web sites and web applications, this refers to the various pieces of the standard look and feel, and most often maps to individual JSP, ASP, or other files. In software applications, this generally refers to the set of classes used to display the GUI components in the application. SSB performs assessments along the lines of these various components to ensure that testing results do not repeat across various pages. The list of these visual components is developed in conjunction with the client to ensure the entire GUI of the site or application is covered, or at least a representative cross-section or the highest-profile pages.
Hardware modules are based on the physical aspects of a hardware device, and include individual controls, standard control groups, and displays. Individual controls are primary use controls on a device, such as an on/off switch, that stand-alone and are not used in conjunction with other controls. Standard control groups, such as a dial pad on a telephone, are related sets of controls. These controls are used in conjunction with one another and form a logical group of controls. Finally, displays on the device are considered a final type of module. Displays include LCD multi-function displays, status lights, and tactile status indicators.
Use Case Development
Use case development is the process for developing a list of use cases for a IT system. The use cases are meant to define the core functionality that is in use within a system. For example, on a job search web site a likely set of use cases would be: search for a job, post a resume, etc. For a desktop editing application, such as Microsoft Word™, use cases would include creating a new file, saving the file, editing the file, etc. For any given application the goal of use case development is to ensure that the most frequent use cases are captured in the document.
Organizations that perform usability testing as part of their standard QA process will likely have a set of use cases already defined. These use cases can be used as a basis for developing the SSB use cases, which are then used throughout the testing phase.
If your organization does not perform usability testing as part of your QA process, a list of use cases can be created by working collaboratively with SSB. This process requires the following steps:
- A walkthrough of the application (SSB + Client)
- Development of a draft list of use cases (SSB)
- Review of the draft list and identification of changes (SSB + Client)
- Final customer approval of changes (SSB + Client)
One item to consider when developing use cases is whether an “accessibility mode” must be activated in the application. “Accessibility mode” generally refers to any user setting that must be active for the user to access accessibility enhancements within the application. Such modes are often defined in applications to ensure that only users with accessibility features active are exposed to accessibility enhancements. During the process of the application, client Project Managers should ensure that any user settings required for accessibility mode are active and clearly indicated to SSB. The use case for activating accessibility mode features should be the first use case tested.
Issues to Consider in Selecting Modules and Use Cases
A variety of issues should be considered when selecting the modules and use cases to be included in (or excluded from) the assessment. Some are technical, and should be readily identified by the test developers who generate the proposed module and use case lists. Other issues are driven mainly by sales or accessibility compliance and liability considerations, making it more important to get active guidance from the client’s sales and product management. These issues are discussed in more detail in the Module List Creation Guide, and can be discussed further with SSB staff.
- Occurrence Rate – Is there a core set of pages and features that are most commonly used, or others that are seldom used?
- User Types – Should administrator-only features (for example) be excluded?
- Complex Features – Are there any relatively complex features, such as dynamic navigation structures, wizards, e-commerce capabilities, or other such features that present unique accessibility challenges? Of particular importance are features that involve “simulated controls”, such as simulated pull-down menus.
- Deprecated Features – Are there features that will be removed from future versions and may not need to be tested?
- Components Still in Development – Are there screens or components whose design has not been finalized? If yes, the decision could be to exclude them from testing, or to test them so accessibility problems could be fixed prior to release.
- Documentation – Should it be part of the assessment? Commonly, we test the basic structure/framework of the online help and a few help topics, not the entire help system.
- Proprietary Content – Does the IT system include PDF documents, Flash content, Java applets, or other proprietary technologies?
- Complaints – Have there been reports about the inaccessibility of certain features? If yes, the relevant areas should definitely be included.
- Government-Facing Content – Are there features within the IT system that are targeted to the federal government or are specific to the public sector?
- High-Touch Points for Disabled Users – Are there contact pages, complaint or support forms, or accessibility sections, whose accessibility would all be particularly important?
- Accessibility Documents – Are there any documents—such as a VPAT—that specify the level of accessibility of the IT system, or that describe any known accessibility issues?
During the testing phase, the only resources required from the client are those relating to the support of the testing process. For testing that occurs offsite, this includes access to IT resources that maintain servers accessed by SSB personnel and the VPN connection utilized by SSB. For testing that is performed onsite, this includes ensuring access to site and IT support for issues relating to computer usage and interaction.
If requested by the client and included in the Statement of Work, SSB can also perform multiple rounds of testing. Typically, this involves the audit of an early version of the product, after which SSB delivers an Initial Audit Report. After the client’s development team addresses whatever accessibility issues it can, SSB then performs one or two Regression Audit rounds of testing. This testing focuses on verifying the fixes and changes that have been made since the initial testing, after which an updated report is created. Note: if the project includes a VPAT or other Compliance Statement, it should be based on the final version tested.
This final phase includes the delivery of the full, written assessment report, a slideshow-based presentation of the report’s highlights, and (if included in the Statement of Work) a Voluntary Product Accessibility Template (VPAT) or WCAG Compliance Statement.
After testing is completed, SSB will analyze the results of the global, manual, automated and use case testing. The Project Lead will enter the findings into the SSB database, for output in the report appendices and to generate violation and compliance metrics. The Project Lead will also write a detailed Audit Report. After a high-level Executive Overview, the report will illustrate and discuss the most significant problems that were found during testing, across all relevant technology platforms, media types and issue categories. The issues will appear in descending order of priority, with the most important issues first. For every type of violation that was found, the report will include a description of the issues, offer compliant and non-compliant code examples, recommendations for fixes, and the specific accessibility standard that the issue is mapped to. The report will also include several data appendices, including a listing of every problem found on every module tested, a table of violation patterns, the results of the use case (disabled user) testing, a statistically-generated matrix for prioritizing fixes, and a document with the name and screenshot of each module for cross-referencing purposes.
Report Findings Presentation
At the client’s direction, SSB will arrange either an online or onsite presentation of the report findings. The Audit findings will be provided in the form of a presentation that will serve to present and discuss the highlights of the report. The presentation will offers the opportunity for the client to ask questions and to discuss “next steps”. SSB participants will include the Project Lead, Senior Accessibility Engineer(s), and (optionally) management. Recommended participants from the client’s organization include Development Lead(s), the Product Manager, and Accessibility Program Manager.
VPAT and WCAG Conformance Statement
At the clients direction, SSB will create a third-party Voluntary Product Accessibility Template (VPAT) or Web Content Accessibility Guidelines (WCAG) conformance statement. A VPAT details an IT system’s level of compliance with each relevant section and paragraph of Section 508. In a similar fashion, a WCAG conformance statement summarizes the system’s compliance with the Web Content Accessibility Guidelines. Both statements will be printed on SSB letterhead and certified by the SSB accessibility lead for the project.