AI can describe a picture, but it still struggles to understand blind users

AI can describe a picture, identify objects, and answer questions about what a camera sees. But when a blind user needs a quick answer, getting an entire spoken essay about the scene isn’t exactly helpful. Researchers at the University of Maryland (UMD) have found that AI-powered visual assistance can leave blind users doing too much work to get the information they actually need.

The study, titled Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Users, examined how people interact with Be My AI, an AI-powered feature of the Be My Eyes app. The researchers found that users often received lengthy descriptions that didn’t adequately address their immediate goals, forcing them to ask follow-up questions or find ways to steer the AI towards a more useful answer.

AI gives blind users more information than they need

The researchers recruited 11 blind participants, who first tested Be My AI in a laboratory before using it in their daily lives for 10 days. The team analyzed 418 interactions, examining how participants asked questions, responded to the AI’s answers, and adapted their approach when the results missed the mark.

The results revealed a mismatch between what users asked and what the AI delivered. Participants typically asked questions of around 10 to 20 words, while the AI’s responses were more than 10 times longer. Conversations averaged three turns, but some stretched to 21 exchanges. Since the answers are read aloud, finding one useful detail can mean listening through a lot of unnecessary information first.

One example involved a blind participant checking whether sausages were cooked. The AI repeatedly reported that some parts still looked pink, but the user wanted practical guidance on what to do next, rather than another description of the food. In another case, a participant looking for her keys received a lengthy description of the image instead of a response focused on the missing item. More than half the participants considered the app’s habit of starting with a full image description a significant problem.

Blind users are finding workarounds for AI’s limitations

Participants tried several ways to get more useful answers. Some explained why they were asking a question, giving the AI context about their goals. Others broke complicated tasks into smaller questions. One participant checking bread dough asked separately about its size, surface, air bubbles, and flour, saying this was the only way to be sure the answers were correct.

While users found ways to steer the AI, the study exposed other limitations, including difficulty estimating distances and changes over time, inaccessible image framing, and limited guidance on camera positioning. The researchers argue that blind users shouldn’t have to master complex prompts to get simple answers. For AI visual assistance to be truly accessible, it must focus on what users need to know and deliver clear, useful answers without making them work around the technology’s shortcomings.

Because when someone asks where their keys are, they probably don’t need a guided tour of the entire room.

RELATED COVERAGE: