Running controlled experiments on how education and literacy impact voice interaction outcomes

The literacy and technology gap

Brazil faces significant functional illiteracy challenges, yet voice messaging via WhatsApp is ubiquitous across all education levels. This paradox raised a research question: if people comfortably use voice messaging, why don't they adopt voice assistants like Google Assistant equally? Our HCI research team (3 researchers at Universidade de Fortaleza) investigated how education levels affect voice interaction technology adoption and effectiveness.

Experiment design to reduce bias

Designed controlled experiment with 15 participants stratified across three education levels: incomplete elementary, complete high school, and university degree. Carefully avoided bias by using comic strip prompts (visual scenarios with minimal text) that presented identical situations to all participants regardless of reading ability. This ensured natural interactions without literacy barriers influencing results.

Five standardized voice tasks

Participants performed identical tasks with Google Assistant:

1. Weather query — "What's today's weather?"

2. Time query — "What time is it?"

3. Search query — "Find nearby coffee shops"

4. Navigation query — "How do I get to [address]?"

5. Multi-step query — Complex request requiring context

Each task observed how participants formulated requests: phrasing choices, formality level, request reformulation after failures, and interaction patterns. Recorded audio and screen capture documented complete interactions for detailed analysis.

Analysis and insights

Measured three key metrics: number of attempts until task completion, total task completion time, and qualitative patterns in request formulation. Results revealed significant performance gaps correlated with education level.

Less educated participants:

• Required more attempts per task (average 3.2 vs. 1.4 for university educated)

• Longer total completion times (average 4.5 minutes vs. 2.1 minutes)

• Used informal phrasing, complete sentences, and hesitation patterns

• Struggled reformulating failed requests

• Showed lower confidence in voice technology

Higher educated participants:

• Concise, keyword-based phrasing

• Immediate reformulation strategies when misunderstood

• Greater persistence through multiple attempts

• Treated assistant as search engine (keyword queries) rather than conversation partner

Implications for inclusive voice interface design

The research revealed that voice assistants, despite being voice-based, still require certain linguistic and technical literacy. Less educated users approached voice interaction conversationally (as they do with WhatsApp voice messages), while systems were optimized for keyword-based queries.

Reproducible protocols and publication

Authored detailed experimental protocols ensuring reproducibility: participant recruitment criteria, comic strip design methodology, task instructions, data collection procedures, and analysis frameworks. Consolidated quantitative metrics (attempt counts, timing) with qualitative analysis (phrasing patterns, error types, user confidence).

Contribution to accessible voice interaction

Provided evidence-based insights for designing more inclusive voice interfaces for diverse populations. The work demonstrated that voice technology accessibility requires more than just speech input/output—it requires interface design that accommodates different interaction models and educational backgrounds. The findings informed recommendations for: conversational error recovery, adaptive phrasing guidance, confidence-building feedback, and educational scaffolding for voice assistant adoption.