Why am I Getting Different Results when I use Different Accessibility Checkers?

We are often asked, "Why am I getting different results when I use CommonLook's validation checker as opposed to others?" 

While we can't speak to the functionality, or development decisions that have gone into non-CommonLook products, we can offer insight into our own testing process. CommonLook tests against provided structural and accessibility standards and gives a range of feedback that will guide a remediator through fixing their issues.

Verification Results & Examples of Differences

Failures

Failures are results that CommonLook has determined to be a direct violation of the standard. These are fairly straightforward and, in most cases, allow the remediator to use our Fix Wizard to solve problems. These are typically interpreted similarly across standards checkers, but our development team has also made some decisions to promote the use of accepted best practices where possible. For example, some tables only have row headers. This is not a common occurrence as the vast majority of tables include column headers or both column and row headers. This is a best practice for table layout and design, and as a result, our development team has chosen to mark any table with only row headers as a Failure. This is designed to encourage proper table formatting but is technically only a suggestion as having a data table with just row headers doesn't violate the standards. You might be shocked by this Failure, but can always manually change the result to "Passed" if appropriate. 

Warnings

Warnings are results that CommonLook wants you to be aware of, but are not a direct violation of the standard. Many of our clients get confused when CommonLook identifies and flags something as a Warning when it is not flagged by other checkers.  The short answer as to why is because we don't want anything to slip past during remediation. For example, table summaries are not required under WCAG (or any accessibility standard). If you do not provide one, you will not receive a Failure, but will be given a Warning. We view this as a gentle reminder that if you want to add one, you can, but it is not a requirement. This is one of many examples of Warnings that help our clients make informed, voluntary, decisions as to what to provide in their documents. Another example is around the use of header or footer Pagination artifacts. While they aren't always required, Warnings will pop up, serving as a reminder that, if you didn't use Pagination artifacts, and you should have, it would be good to go and fix that.

User Verifications

The User Verifications are for checkpoints that, by definition, require human verification, because no automated checker can test everything. Depending on the content in your documents, this can be a sizable portion of your standard check. Many clients are shocked by how much requires manual verification considering other checkers don't flag them at all, and quite frankly, it is something we are most proud of here at CommonLook.For a document to be 100% accessible and standards-compliant, these manual checkers are mandatory. For example, our checker will ask you to verify that a Figure tag's Alternative text is correct. While this is absolutely critical to ensuring content is being accurately conveyed, other checkers might simply "trust" that the remediator got it right the first time or already verified it. That is not a risk we are willing to take, and while there could be numerous verifications to work through, the process for manual verification with CommonLook is very efficient. 

Moving Forward

On a side note, it's possible to edit the checkpoints in CommonLook. So, if you don't agree with our development team and want to change the settings so a specific checkpoint doesn't get flagged, you can certainly do that. In general, we discourage clients from doing this, but the functionality is available. 

In general, if you are having trouble with a portion of your document validation or are confused by provided results, check here, in our Knowledge Base. If you still don't find the answer you're looking for, feel free to contact us. We'll do our best to provide thorough explanations as to what a status or error message means and how to address it.