We recommend where possible, you involve your organisations digital accessibility officers in the review of any supplier responses received during a procurement exercise. It is important to get the feedback of accessibility professionals to be able to correctly judge the quality of documentation provided by suppliers.

For example, if a supplier provides an accessibility roadmap it should be reviewed by an appropriate accessibility professional outside the project. They weigh up the potential user impact, proposed mitigations, and consider the availability of alternative products. They may approve or not; approval may be subject to specific mitigations being implemented as a prerequisite to purchase.

If you do not have accessibility specialists to review your supplier responses, this guide should help you understand the types of responses you might receive from suppliers and what you should be looking for in different response and documentation types.

Once you have read this guide, you may find more itemised support useful in scoring supplier responses. The MTA Procurement scoring and maturity guide and template can help you to begin turning the learning from this guide into practical steps during your procurement process.

Assuming you have used the MTA procurement requirements template to get information from suppliers on how their systems will meet your requirements, you should have received some information which you need to assess to ensure that the supplier’s approach to accessibility fits with your organisational requirements.

The below sections provide further information on what these responses might look link and how best to respond to or interpret different responses and evidence.

Our product fully complies

Many suppliers will claim that their product fully complies with all WCAG A and AA success criteria. In our experience this is most often incorrect, but not due to a lack of good intentions.

If you receive documentation which shows a clean slate for accessibility, be somewhat sceptical and ask to be able to do your own testing. As we often say, most of the time any suitably large system will not be perfect, and sometimes that must be accepted. You are looking for an accurate state of the system not clean documentation just to get past the check.

Most suppliers that have a fully compliant VPAT or other “clean” evidence have likely put in work to test their platforms. Ask the supplier about their testing processes and if/how they have included accessibility checking into the development cycle.

Suppliers may also answer follow up questions explaining that there are a limited number of areas where the product has current issues against WCAG. This might be in a written response to one of the tender questions or can be identified through documentation such as a VPAT or audit report as mentioned below.

If you receive a response from a supplier which says, “fully compliant” and then receive documentation which does not say the platform is 100% perfect, you should raise this inconsistency and double check the causes and what suppliers are doing about issues.

Consider:

• Who wrote the report?
• Was it in-house or a reputable 3rd party auditor?
• To what extent the supplier understands and can meet your requirements if they are making contradictory statements.

If you find small discrepancies between their documentation and your own testing, this may be due to the changing nature of platforms, not a lack of effort. If the supplier has other supporting evidence, this may be a good response that can give you confidence in a supplier.

VPAT

Voluntary Product Accessibility Templates (VPATs) are a type of document which provides a very basic level of information on the accessibility of a digital product. This will normally be in the form of a table listing each of the WCAG success criteria, identification of their compliance, and some notes. VPAT's are part of the American Section 508 requirements but have become internationally used pieces of accessibility documentation.

The problem with VPATs is that they are normally not detailed enough to give useful information on the true level of impact for a service.

If you receive a VPAT that includes vague statements such as the above, which could mean a potential risk, or you are not sure, ask the supplier for more detailed information on the exact areas of the product that the issue covers. A VPAT can only have been produced correctly following in-depth testing. VPATs are summary documents and so the detailed test results must exist.

You should also be asking for a detailed remediation roadmap. If the supplier is already aware of several issues as detailed in their VPAT, ask them for evidence of how they are planning to fix those identified issues, or what actions they have already taken if the VPAT is more than a few months old.

If the supplier cannot give you more detail than the VPAT alone, be sceptical of the documentation. You know there are some issues at this point, but the supplier cannot give you any evidence as to the impact on their product or what they are doing about it. This can present a high risk and may suggest that getting accessibility fixes made may be difficult. Avoid the supplier in this case.

If the supplier can follow up the VPAT with more detailed documentation and a roadmap, or if the VPAT itself already contains more detailed specifics about the impacts of individual success criteria, and the supplier has a roadmap, then this may be a good response that can give you confidence in a supplier.

Detailed testing documentation

If the supplier responds with detailed testing documentation for the product, you should look through the documents carefully. But this is generally an initial good sign. You will want to look at:

• How the supplier tested the product
• What tools the supplier has used
• Is there a mix between manual, automated and assistive technology testing
• How large was the scope of testing
• How serious the issues are or if the descriptions are vague
• What priority ratings the supplier has given for issues

If there are any issues present in the detailed documentation you should ask for a detailed remediation roadmap. If the supplier is already aware of issues, ask them for evidence of how they are planning to fix those identified issues.

If the supplier has no answer, cannot show actions of accessibility fixes, or does not want to commit to a roadmap, this can present a high risk and may suggest that getting accessibility fixes made may be difficult. Avoid the supplier in this case.

If the supplier has detailed documentation, and can show how they are testing, what they are testing, and what they are doing about results through a roadmap, this may come close to a model answer that can give you confidence in the supplier.

Accessibility Statement

The accessibility statement is the end of the documentation journey. If the supplier can provide you with an accessibility statement, this is an insight into their testing process. If the statement is good and has useful information on issues and even suggested workarounds already this could be a positive sign.

If the statement is poor and does not include specific information about technical issues and their effects on users, but instead includes vapid statements about "commitments to support all users" this could be a sign that the supplier does not have meaningful practical documentation to back this up and show what actions they are taking to deliver on those commitments.

In either case, if an accessibility statement is provided as the main piece of evidence, you should respond asking for other detailed testing documentation which must have been used to inform a good accessibility statement.

Remediation roadmaps

In the event that you receive detailed documentation from a supplier that includes identifying issues as well as remediation roadmaps which show when each issue is planned to be fixed, this is an excellent response and should show that the supplier is aware of issues with their product, are working to fix them, and can provide you as the customer with significant evidence to support due diligence.

At this stage, you can talk with the supplier about prioritisation and timelines. What issues has the supplier prioritised first? The biggest, most impactful issues, or the “quick wins”. Speak with them to better understand their current position and what mitigations and guidance material you may have to put in place in the interim period to support users while fixes are being made.

If any significant milestones within the roadmap have been passed, ask if items on the roadmap have now been completed, this is a good chance to see if the roadmap is a “tomorrow never comes” situation, or if actions are being taken and individual issues are being resolved.

Plans for testing

Sometimes, particularly with new products that are either being designed bespoke for you or are a new launch from the supplier, thorough testing and road mapping may not yet have taken place. In this case you would expect to see information from the supplier about how they plan to test the product and what they will do about issues found during that testing.

Ideally you will be looking for suppliers to commit to testing against the latest WCAG A & AA success criteria as a minimum. This can be done either in house if the supplier has the skills, or by a subcontractor such as professional accessibility auditing companies. In any case the testing should include:

• A significant scope which covers the majority of components across a platform through a representative sample.

• A mix of manual, automated and assistive technology testing.

  • Manual testing is required to confirm several WCAG criteria that can only be done with human judgement such as whether things are correct in context.
  • Automated testing tools can be used to deal with repeat issues that are easier to check with a computer such as comparing colour values for contrast, rather than trying to guess by eye.
  • Assistive technology testing is vital to check that the platform works with common assistive technology and operating system pairings. This would include checking with common screen readers, dictation software, magnification techniques and non-pointer input devices such as keyboard controls.

• A prioritisation of issues based on impact to users.

• Recommendations on how to fix issues. This may be more present in common easy to fix issues, rather than more complex issues for significant components which may require replacement.

If suppliers are providing you with plans for future testing, a good response must cover the points above. If the supplier suggests testing with only automated tools for example, this is not sufficient. As this is most likely occurring as part of a more bespoke offering you have the opportunity to demand that the above points for testing be mandatory as part of any delivery. At that point it is on the supplier to provide you with a compelling explanation that they understand and have the tools / skills to conduct that level of testing in-house or tell you who they may subcontract out to as an accessibility testing partner to deliver this.

No information or other negative response

Unfortunately, not all suppliers have model answers for accessibility documentation. Many may not even be aware of these requirements or may be new to this area of compliance. This means that for many responses you may receive sub-standard or no information.

If a tender response cannot provide any information to show the supplier is aware of the legal accessibility requirements for their customer base and know how to deal with requests and have appropriate evidence, then they are significantly higher risk.

The important thing is how the supplier responded to these requirements. If the supplier approaches it proactively and try to make fixes and are operating in good faith then that is one thing, but conversely, they may dismiss the need because it has been "unimportant" up till now. The supplier's direction from this point is important.

The range of poor responses you might receive from suppliers can be quite broad. You may receive answers such as the following, which are all real responses we have seen before. For each we have provided some notes on why these can be a problem for prospective customers:

• No answers at all – The issue with this should be self-evident. If the supplier has nothing to say about accessibility, then they are an unknown risk. You wouldn’t expect a supplier to leave other important compliance sections like cyber security blank.
• Accessibility is not on our roadmap – This shows a lack of importance to the supplier and means their product is likely not very accessible and will present a higher risk, and that getting them to take action will be difficult.
• We have not done any testing to check compliance – This may be down to ignorance of a lack of importance to the supplier. If they can commit to getting testing done and taking action on the results, you may be able to justify mitigating the risk. But if you feel it is more to a lack of care, and there is no commitment to get tested before contract signing, then avoid the supplier.
• No one has ever asked about this before (including “you are the first Council/Uni/College/etc. to ask for this”) – Accessibility requirements have been in UK law now since 2010 if you are being generous. If the supplier has never considered this up until now, they are either lying or get away with being incredibly lax about understanding laws that affect their products, services, and customer base.
• We are on X framework (such as Crown Commercial Services frameworks) so we must be fine, no further evidence supplied – Being on a framework does not in itself count as completed due diligence. Yes, some frameworks ask if you meet accessibility standards, but the framework owners cannot check thousands of applicants for WCAG compliance. Each supplier must maintain their own documentation.
• We have done testing with disabled users back in X year. No, we cannot find or provide you with any evidence even anecdotally about what the results of that testing were – A supplier can say all sorts of things, but these must be backed up with documentation. Any claims the supplier makes, you must follow up on by asking what the results of an exercise, or audit were.
• We have Y overlay product, so this solves all our accessibility – Obvious high risk given all MTA template questions specifically ask about overlays and highlight the problems with them in the Overlay Fact Sheet and our own guide on Overlays and other fix-it products. We have seen suppliers respond to every question with a one line link to an overlay product before. Think about how different each question is and consider how lazy a response a one line overlay link is.
• Our product is customisable to you, we can’t give you documentation because it depends on your customisation, or what components you choose out of our library – There is no guarantee that any component in their library is accessible. The supplier should be able to evidence that their pick and choose components are individually built to accessibility standards, or that their platform enables customers to build compliant platforms. Customisability is not an excuse for lack of due diligence.

Once you have read this guide, you may find more itemised support useful in scoring supplier responses. The MTA Procurement scoring and maturity guide and template can help you to begin turning the learning from this guide into practical steps during your procurement process.

Once you have completed your tendering process and have decided to enter into an agreement with a supplier, it is important to keep accessibility compliance in the requirements.

The best way to do this is to include accessibility into contracts with suppliers. For more information, read our guide on accessibility in supplier contracts which includes example contract clauses.