How to Move Accessibility from Aspirational to Measurable

If you’ve been working in UX for any length of time, it’s almost certain that accessibility in ux has crossed your desk at least once or come up in meetings you’re part of.  Between the Americans with Disabilities Act, Barrier Free Canada Act, and the recent European Union accessibility deadline, the risks of not thinking about accessibility are rising ever higher. 

On top of that, people with disabilities are the largest untapped market segment in the world, and as the population ages, one that only continues to grow.  Making sure the experiences you create are accessible is becoming table stakes, for both legal and economic reasons.  

Many of us are aware, of course, that performing UX research involving people with disabilities is the solution to the endless loop of ineffective audits and incomplete remediation.  But the step that’s all too frequently missed is how to turn this research into something measurable, repeatable, and actionable. 

Based on my own lengthy experience in the accessibility field, both as participant and researcher, and Fable’s deep expertise in conducting UX Research involving people with disabilities, here are some metrics that can point the way forward.

Ready to build that trust and kickstart your research?

let’s make trust the foundation of every project you work on.

Try UXArmy today

Who to Test With

When testing with people with disabilities, you should try to get:

• Multiple assistive technology types (screen magnification, voice control, screen readers)
• Across multiple platforms (Windows, Mac, Android, iOS)

Unlike other types of UX research, what you’re checking is if your website is compatible, easy to use, and creating the expected experience across different types of assistive technology.  If the VoiceOver screen reader can’t activate a button in your website or app, this will remain true for young college students in LA, middle-aged office workers in London, and retirees in Toronto. This means that small sample sizes (in the 5-10 user range) are perfectly acceptable. It is more important to test and re-test frequently in order to collect up to date metrics, than it is to increase your sample size.

Critical Flows

Before collecting metrics, you first have to decide what to measure. In general, it’s better to look at accessibility through the lens of task flows, than it is single pages. If a particular page is accessible, what does it matter if the five other actions required to get there are not? On the other hand, if the social media links in your footer have no alt-text, this will show up as an error on every single page, even if it in no way prevents users from logging in and interacting with your product or website. 

When choosing your critical task flows, consider:

• Gateways: if this is a gateway flow, like registration or login, if it is inaccessible it will block access to all other flows
• Frequency: how frequently will this flow be accessed? If I must get help from a sighted friend to order cheques from the bank, this is annoying, but infrequent. If I can’t check my bank balance or pay bills independently, though, I’m going to switch banks!
• Privacy and dignity: Sadly, the online form I have to fill out to donate blood is inaccessible with my screen reader. My only other option is to go through my sexual history and other medical information with a stranger. Instead, I choose not to give blood at all. If getting help with a flow would require a person with a disability to surrender our privacy or dignity (think medical information, credit card numbers, etc.) it should be considered a critical flow.

Task Completion Rates and Blockers

Now that you’ve figured out what to test, and who to test it with, it’s time to get to measuring. 

Whenever “completion rates” are first mentioned as a metric, UX Researchers frequently respond with a look of shock.  However, it is an unfortunate reality that many people with disabilities are completely unable to complete critical task flows on your website:

• The CAPTCHA that means a low vision or older user can’t create an account
• The menu system that doesn’t work at all with a keyboard, leaving screen reader users unable to navigate the website
• The contact us page that only lists a phone number, locking deaf users out entirely

You may have the most accessible credit card entry form in the world. But it does you no good if a screen reader can’t find or activate the add to cart button at all: they won’t be making a purchase, today. This is why it is useful to measure both task completion rates and blockers.

• Task completion rates: the number of people with disabilities able to complete critical flows
• Blockers: the number of blockers (i.e. a particular step) that prevented people from continuing (the CAPTCHA they couldn’t solve, the login button they couldn’t click, and so on)

The Accessible Usability Scale

Once you have tasks that people with disabilities can complete, and thus a website that works for the most part, you can begin looking at the experience people are actually having. An experience that “mostly works” is far from what you want for your non-disabled users, and it’s not good enough for people with disabilities, either. 

The best tool to measure this is Fable’s Accessible Usability Scale, or AUS. The AUS is a free questionnaire you can administer to participants that was created based on SUS, the System Usability Scale, with modifications to make it more suitable for users of assistive technology.  Once users complete the questionnaire, an overall experience score is calculated. 

Fable has collected thousands of AUS scores over the years and has published data on the average AUS score for each assistive technology type. So, when you use AUS, you will be able to compare yourself to the average, giving you a much better understanding of how well you’re really doing. Not only that, but Fable has released the AUS as a completely free tool that anyone can use. 

You can adapt the questions and scoring to be part of your own research, or you can use Fable’s fully accessible AUS survey and have your participants fill it out directly.  The AUS is the tool of choice for many large enterprises, and AUS scores are even integrated into accessibility related OKRs, allowing organizations to set measurable, trackable, and achievable accessibility goals.  You can find out more about the Accessible Usability Scale, and start using it right away yourself, at Fable’s website.

Conclusion

In order to have an effective and measurable accessibility effort, you should have a set of critical flows that you regularly test and keep an eye on, with multiple assistive technologies. You will also want to consider how you can test new flows or changes to old ones before release, to save yourself the time and effort of remediation. If it’s all working as it should, you should see:

• Completion rates going up
• Number of blockers going down
• AUS scores going up

As your accessibility program expands and matures, you should:

• Continue to expand the number of flows you test
• Expand the types of assistive technologies you test with
• Test critical flows and changes as soon as possible in order to prevent rework
• Involve people with disabilities in design and ideation to catch problems before development begins

It doesn’t matter where the numbers start out. What matters is that you’re putting in the work and moving the numbers in the right direction.