I agreeto Idea Encourage usability testing with people with disabilities
Voting is Disabled

29 votes

I disagree

Rank2

Idea#43

This idea is active.
Increase Transparency »

Encourage usability testing with people with disabilities

It's hard to design, write, code, etc. for people you don't know. One of the value of usability testing with people with a variety of disabilities is that it puts a human face on the goals of Section 508.

I do NOT mean using people with disabilities as "accessibility QA testers." This is about observing - and understanding - why each requirement in Section 508 (or WCAG 2.0) exists: what human communication need it meets.

Submitted by Whitney Q. 2 years ago

Vote Activity Show

(latest 20 votes)

Events Show

Comments (12)

  1. I agee about putting a human face on accessibility and usability design. Giving developers design criteria may be less effective, but finding a pool of disabled testers that large is difficult. Can you suggest a way that large agencies can do this when they have thousands of developers?

    2 years ago
    0 Agreed
    0 Disagreed
  2. Whitney Q. Idea Submitter

    I agree that gathering a pool of participants is a first step. But, I'd say that it doesn't take many participants to make a point. They also don't have to be in Washington DC. Remote testing techniques have gotten very good - and allow the participants to use their own computer setup. I know of at least one non-profit project that is gathering a list of possible participants, but I'd also hope that this might encourage commercial recruiting services to make an effort to add people with disabilities to their own databases. Many things seem difficult when they are first attempted... but get easier with practice.

    2 years ago
    2 Agreed
    0 Disagreed
  3. Usability, Accessibility, and Assistive Technology-compatibility should be mandatory, not a 'should' or encouraged. We've passed ten years of this law - we know what's right and what's wrong, we know what works and what doesn't work, we know what's accessible and what's not accessible (even if it's not allowed to be documented so that plausible deniability can be claimed by 'management.')

    2 years ago
    1 Agreed
    0 Disagreed
    1. Moderator

      Is usability is clearly defined - what is usable to one person may not be usable to another?

      2 years ago
      0 Agreed
      0 Disagreed
  4. I strongly agree that we should use people with disabilities in 508. I work for Columbia Lighthouse for the Blind in Washington, DC and we currently have visually impaired and blind employees who work with us on 508.

    2 years ago
    1 Agreed
    0 Disagreed
  5. I agree that developers, vendors, ICT managers, etc. should all understand *why* 508 exists and *who* it benefits. It doesn't always take product testing to do this, though. There should be a way to make compliance more efficient -- bureaucratic, even -- without ignoring the goals of full accessibility and inclusion, or weakening the human element as part of the path towards those goals. Constant progress towards clear and objective criteria may actually improve accessibility while reducing the need for explicit user testing. I know that's dangerous territory, but relying on large scale user testing is also dangerous, because it's expensive, time consuming, and subjective. I think both approaches should be developed thoroughly, and used in hybrid processes.

    2 years ago
    0 Agreed
    0 Disagreed
  6. Whitney Q. Idea Submitter

    To the question of whether usability is clearly defined. Usability is defined by whether the people for whom the product is intended can use it appropriately with efficiency, effectiveness and satisfaction.

    But I'm not suggesting large scale testing, or usability testing as a metric.

    I am suggesting that when we (everyone on the design and development team) observe people using our products, it is easier to understand the problems they encounter... and thus to fix them.

    I have seen a lot of developers trying to meet 508 or WCAG who seem to have absolutely no idea why the requirements exist. This not only leads to poor compliance, but makes it hard to innovate and problem-solve.

    For an example of the type of usability testing I'm suggesting, you might take a look at the First Fridays program at the GSA: http://www.howto.gov/web-content/usability/first-fridays

    2 years ago
    0 Agreed
    0 Disagreed
  7. Whitney Q. Idea Submitter

    Jim, I did not say "large scale user testing" (and I mean testing the products for usability, not testing the users).

    I certainly did not say to substitute usability testing for other ways of testing for compliance with 508.

    The testing tools in the kit are, broadly:

    1. Automated testing for the presence or absence of technical attributes

    2. Manual inspection, walking through a site and inspecting the underlying code

    3. Expert review, using assistive technology or simulators

    4. Testing through actual use, usability testing with people with disabilities

    I suggest three things:

    First, that all of these techniques are needed

    Second, that they should be done in roughly this order. Each step builds on the last and also takes more resources. So I would not recommend usability testing until you are sure that the site meets technical requirements, for example.

    Third, that a site may meet the letter of the regulation, as tested by #1 and #2, but still not meet the spirit of 508, which is to enable people with disabilities to use ICT.

    2 years ago
    0 Agreed
    0 Disagreed
  8. I agree with Whitney Q about a site meeting the letter of the law, but not necessarily the spirit of 508. Take this platform for instance. It was used for a number of other initiatives, so it must have met compliance regulations. Usability testing with people using a variety of assistive technologies could point out usability issues.

    2 years ago
    0 Agreed
    0 Disagreed
  9. There's another flavor of testing that's important. I don't know what it's called -- maybe something like 'environmentalized testing' or 'real usage testing'. Too often usability testing and accessibility testing are done out of context, as if the product itself or just in concert with an assistive technology is all that's important. If you think about actual usage, it's always in a complicated context of technological and non-technological factors. Here's an example. I create a document in an application, and then want to attach it to an email. But where is the document? The application that created it usually has a default storage location that I may not know about, especially if I'm not paying attention. So I'm stymied not by the document application or the email application, but the crack that has opened up between them.

    This kind of testing, because it calls for thinking about deep contexts and may involve hundreds of permutations of product-to-product interactions, may often not be considered feasible. I think many of these situations affect accessibility (the one I raised is especially toxic to people with any cognitive disabilities). So how do we institutionalize such an approach?

    2 years ago
    0 Agreed
    0 Disagreed
  10. Whitney Q. Idea Submitter

    Jim, great point about the need to test not just the discrete interactions, but the users's goals. These often cross products, media or time. There are ways to gather this data: diary studies are one.

    When I think about research methods, I try to pick the most economical - that is, the one that requires the fewest resources, the least time, and the least effort from everyone involved.

    There is no point in usability testing a product before we have solved the absolute barriers to use, so you want to start by getting the basic technology issues right. Similarly, if I can't figure out how to use the product at all, the question of context is less relevant. It's a hierarchy of experience, to mangle a phrase.

    The real problem comes when you finally release a product (presumably having tested along the way) only to find that it doesn't fit into the real user environment. A very trivial example (from Ginny Redish) is of a hand-held device meant to be used in a very cold space, where the keys are too small to be used with gloved hands).

    That's why user experience research - that initial work to understand users and their context of use is so critical.

    And, why a well-constructed usability test attempts to create as natural an interaction as possible. For example, I often start off by giving participants no task at all, just using interview techniques to set them off on a self-defined activity. After that, I can ask the participants to try specific kinds of activities to focus on the area of the product I'm interested in. It's not as "efficient" a test, but it gives me a much better idea how people solve problems or meet goals on their own, even in a "lab."

    2 years ago
    0 Agreed
    0 Disagreed
  11. Whitney Q. Idea Submitter

    For testing with people using assistive technology, it's really helpful to let them work at their own pace, with their own setup, in their own environment. I find that remote usability methods work well for this. They can be structured tasks, or even open tasks in a sort of mini-diary study. Knowbility has a program like this (just getting off the ground) that matches a panel of participants with various disabilities to an accessible remote testing platform. http://www.knowbility.org/v/service-detail/AccessWorks-Testing-Portal/3k/

    2 years ago
    0 Agreed
    0 Disagreed