Accessibility Acceptance Criteria and Test Protocols for Transit Voice AI
A practical, procurement-ready framework for specifying accessibility acceptance criteria and test protocols for Voice AI in public transit. For procurement, operations, accessibility and IT leaders who must evaluate vendors, scope implementation, phase rollout, test readiness, and assign accountability.
1. Decision context: why explicit accessibility acceptance matters
Transit Voice AI projects frequently fail at operational handoff—not because the model is wrong, but because procurement lacked measurable acceptance gates for accessibility, integration, and scalar resilience. This section aligns leaders on the decisions they must make before technical evaluation.
Operational risks and scope boundaries
Before specification, decide which user intents the Voice AI will own and which remain human-handled. Common split: timetable and fare information, account or concession queries, and basic route directions. Reserve safety-critical and emergency interactions for trained staff. Document boundaries explicitly in the Statement of Work and acceptance tests.
- Owned intents (examples): scheduled arrival time lookups, fare category explanation, lost-and-found intake.
- Escalation-only intents: medical emergencies, safety incidents, criminal reporting.
- Failure boundary: when confidence < vendor-agreed threshold, call must route to live staff with a clear transcript and context token.
Schedule vs alerts: separate canonical sources
Require vendors to implement a controlled knowledge store for scheduled information (timetables, stop names, accessibility features), and separate, authenticated API adapters for service alerts and detours. Acceptance tests must verify the Voice AI responds to scheduled queries only from the controlled store and uses the service-alert API for dynamic disruptions.
- Scheduled knowledge: authoritative export of agency GTFS/static or equivalent—read-only for the voice pipeline.
- Service alerts: authenticated, real-time API (approved feed) that overrides or annotates schedule answers.
- Test requirement: cross-source divergence test demonstrating consistent precedence rules.
2. Accessibility acceptance criteria: explicit, measurable, and audit-ready
Translate accessibility policy into acceptance criteria that are verifiable. Avoid vague wording like 'accessible'—specify tasks, measurable outcomes, and required assistive-technology compatibility.
Task-based success metrics
Define a set of core tasks and measurable pass/fail criteria: e.g., a visually impaired rider must successfully obtain step-free route options and boarding instructions using the Voice AI within two interaction turns and without human assistance in 95% of scripted test cases during pilot.
- Examples of core tasks: request stop accessibility features; confirm step-free path along a route; get fare concession info.
- Measure: completion rate, number of clarification prompts, average interaction time, and need for human transfer.
- Acceptance gate: defined minimums for each KPI during controlled pilot.
Assistive-technology compatibility
Require compatibility testing with common assistive technologies used in your jurisdiction (screen readers, TTS engines, switch devices, relay services). Include human testers who use these tools as part of acceptance testing. Automated checks are necessary but not sufficient.
- Test with representative devices: screen readers, in-call TTY/relay, external amplification, and alternative input devices.
- Require vendor to provide an accessibility conformance report and remediation roadmap for any failures.
- Document consent and recording rules for users relying on relay or third-party devices.
Language, dialects and minority-language service
Specify languages and dialects required at launch and in later phases. For jurisdictions with concentrated non-dominant language communities, require supplier testing with native speakers and human-in-the-loop fallback when ASR confidence is low.
- Define supported languages at contract award and acceptable quality thresholds (per-language completion rates).
- Plan phased inclusion for lower-priority languages with explicit pilot acceptance gates.
- Consider linking to local-language accessibility guidance; test with native speakers rather than synthetic benchmarks.
3. Test protocols: from unit tests to live pilot validation
Acceptance requires a layered test strategy: component unit tests, integration and API contracts, accessibility conformance, load and resilience testing, and live pilot assessments with real riders and staff.
Component and integration tests
Require vendor-supplied test harnesses demonstrating deterministic outputs for canonical inputs. Tests should include ASR reproducers, intent resolution reproducibility, and API contract tests for schedule and service-alert adapters.
- ASR test corpus: representative utterances, noise profiles, accents; pass/fail thresholds per utterance class.
- Intent determinism: identical canonical input should map to the same intent and slot values in repeatable runs.
- API contract tests: authenticated feeds must produce expected override behavior for alerts/closures.
Accessibility conformance and human-centred testing
Accessibility testing must combine automated conformance (WCAG where applicable to web portals or transcripts) with human testers who rely on assistive technologies. Include predefined scenarios for low-vision, deaf, and neurodivergent users.
- Include scripted walkthroughs and unscripted exploratory sessions with trained accessibility testers.
- Log acceptance metrics: completion rate, retries, transfers to human agents, and user-reported usability scores.
- Require vendor remediation schedules for any failures before production rollout.
Pilot, load, and resilience testing
Define realistic concurrency and fault-injection tests. Pilots should run under realistic traffic, covering peak commute times and lower-resource scenarios (poor cellular coverage, high background noise). Load tests must verify degradation modes—e.g., graceful fallback to human agents.
- Simulate peak inbound call patterns and concurrent transcription loads connected to real-time alert feeds.
- Inject service-alert API failures and verify the Voice AI's safe fallback behavior and user messaging.
- Define acceptable recovery time objectives (RTO) for critical integrations and human-handoff SLAs.
4. Procurement and vendor-evidence checklist
Procurement must require demonstrable, reproducible evidence before accepting systems. Ask for records, sandbox access, and binding operational commitments.
Documentation and demonstrable evidence
Require reproducible test bundles: ASR corpora, accessibility test reports, API sandbox with representative schedule and alerts, and sample transcripts with redacted data. Insist on runbooks for common failures and a vendor-signed remediation SLA.
- Provide a vendor test sandbox that mirrors production API authentication and latency characteristics.
- Demand retention and deletion policies for recordings and transcripts, and subprocessors list with locations.
- Request a reproducible script to run acceptance tests locally or in your CI environment.
Data geography, subprocessors and privacy
Specify hosting region, backup region, allowed subprocessors, remote-support access, and transfer mechanisms in the contract. Request clear retention and recording-consent procedures for voice data and ensure these are testable during acceptance.
- Contract items to require: hosting region(s), backup region(s), subprocessors list, and notification obligations for changes.
- Define consent banners/prompts required in voice interactions and test that they are presented in applicable jurisdictions.
- Specify breach notification timelines and evidence of vendor incident response capability.
Service levels and observability
Acceptance should include observable metrics and dashboards integrated with your monitoring. Require event-level logging for transcripts, intent decisions, handoffs, and service-alert ingestion, retained for audit windows required by your jurisdiction.
- Minimum observability: per-call latency, ASR confidence distribution, intent confidence distributions, handoff rates, and per-language KPIs.
- Require integration with your incident management and logging; test alerting during pilot.
- Define acceptable error budgets and remediation workflows when SLAs are breached.
5. Rollout phasing and operational readiness gates
Use a conservative, gate-driven rollout. Each phase must close acceptance criteria before advancing and define rollback triggers.
Phase definitions and gates
Example phases: internal acceptance (vendor and agency staff), controlled pilot (limited routes or trip types), expanded pilot (more routes and languages), and full production. Each phase has explicit acceptance gates tied to the metrics defined earlier.
- Internal acceptance: pass unit/integration tests and accessibility conformance with vendor-provided testers.
- Controlled pilot: 4–8 weeks of live traffic on low-risk routes with human agents on immediate handoff.
- Gate criteria: per-task completion rates, handoff latency, incident counts below defined thresholds.
Staffing, training and human handoff
Plan human-agent staffing and scripts for transfers. Acceptance tests must validate that transcripts, context tokens, and suggested resolution forms are delivered to agents within SLA.
- Define handoff payloads: user transcript, intent, slots, confidence scores, and suggested dynamic form pre-fill.
- Test end-to-end: successful transfer, agent receipt of context, and agent ability to close the case.
- Ensure agents are trained for low-confidence and assistive-technology scenarios.
Rollback and fail-safe triggers
Define clear rollback triggers (e.g., sustained drop in accessibility KPIs, critical incident related to safety communications, or data-residency violation). Acceptance testing must include simulated rollback drills.
- Triggers should be numeric and observable (e.g., accessibility completion rate < X% for Y days).
- Practice rollback during pilot and verify the system returns to the prior state without data loss.
- Require vendor support windows and escalation contacts in the contract.
6. Governance, roles and accountable controls
Establish a RACI that covers pre-production acceptance, live operations, accessibility remediation, and incident handling. Align governance with AI risk-management practices and transport-sector resilience.
RACI for acceptance and operations
Assign precise ownership for each artifact: test script ownership, accessibility test panel coordination, data-retention signoff, and SLA enforcement. Ensure the agency retains ultimate sign-off for public-facing behavior and incident declarations.
- Typical roles: Product owner (agency), Vendor technical lead, Accessibility officer, IT/security lead, Operations lead (customer-service).
- Agency retains authority to pause public traffic for any unresolved accessibility or safety failure.
- Require monthly review cadence for logs, KPIs, and change requests during the first 12 months.
Incident handling and audit trails
Acceptance must ensure auditable trails: call metadata, ASR transcripts, intent classification history, developer model versions, and change logs for knowledge sources. Define retention windows and access controls.
- Audit data must be tamper-evident and accessible to authorized agency staff for investigations.
- Define who may access raw recordings and under what process—test access during acceptance.
- If cross-border processing is involved, require evidence of lawful transfer mechanisms.
Related Peak Demand resources
Industry and AI sources reviewed
- OECD AI PrinciplesOrganisation for Economic Co-operation and Development
- Transportation Systems SectorCybersecurity and Infrastructure Security Agency (CISA)
- Artificial Intelligence Risk Management Framework (AI RMF 1.0)National Institute of Standards and Technology (NIST)
Privacy, telecommunications, recording-consent, cybersecurity, consumer-protection, employment, and records obligations vary by jurisdiction and use case. This article is operational guidance, not legal advice; organizations should confirm applicable requirements with qualified professionals.
Frequently asked questions
Good starting points include lost property, complaints and feedback, stop or shelter issues, fare-machine faults, non-emergency accessibility service requests, schedule information from approved sources, and structured routing to customer service or field teams.
Official reference: Artificial Intelligence Risk Management Framework (AI RMF 1.0)
Use GTFS Realtime only when the agency exposes suitable feeds and the workflow genuinely needs service alerts, trip updates, or vehicle positions. The integration should validate freshness and availability, and the agent should avoid presenting stale feed data as a guaranteed arrival prediction.
Emergency, security, injury, crime, and safety-critical reports should follow approved transfer or emergency-routing procedures. Voice AI may detect and route the call, but it should not make operational safety decisions or replace trained personnel.
Official reference: Transportation Systems Sector
Request realistic call testing, feed and system failure handling, service-request integration, transfer context, audit logs, accessibility channels, monitoring, change control, and evidence that the agent distinguishes scheduled information from dynamic service alerts.
Official reference: Artificial Intelligence Risk Management Framework (AI RMF 1.0)
Turn Voice AI infrastructure into a managed enterprise operation
Peak Demand designs, integrates, deploys, monitors, and improves Voice AI systems across customer service, enterprise systems, governance, escalation, and reporting.
Schedule a discovery call
