Welcome to the Spring 2024 Issue of AccessWorld! This issue marks a year since we refocused the AccessWorld publication. Over the last year, we have been thrilled to bring our readers more knowledge content useful to developers of websites, apps, and other software. Not only that, but it has also been exciting to expand AccessWorld into new mediums.

Starting in October of 2023, we launched our AccessWorld podcast, where we cover topics related to blindness/low vision and accessibility, with a specific focus on digital inclusion. We frequently have guests from the accessibility industry and related fields present on the podcast as well. If you are interested in learning more about our podcast, you can find past episodes and other information on this page.

We have also expanded AccessWorld content to AFB's main blog. If you follow AFB's blog, you will have noticed that much of the content that was a staple of AccessWorld in years past, such as conference coverage and operating system update guides and reviews, is now present on the blog. This allows us to continue providing this popular content alongside the more focused digital inclusion-based content of AccessWorld while also reaching new readers. If you would be interested in more AccessWorld content outside of the publication proper, be sure to check out our blog here and sign up for alerts. Notably, we publish content on the blog in between our monthly releases.

For this anniversary of our refocused AccessWorld, the issue includes the following articles:

First, Jamie Pauls brings us a review of the latest releases of Zoom's essential series of standalone audio recorders. We were particularly interested in reviewing these recorders as Zoom has now included accessibility features in the form of speech feedback and other auditory cues. This is noteworthy because, due to the nature of these recorders as standalone devices, accessibility had to be implemented from scratch, and developers could not rely on existing screen readers to provide accessibility.

Next, writer and web developer Hannah Thompson offers her debut article. In her piece, Hannah details her experience learning and implementing accessibility fixes into her web application without having prior knowledge of accessibility beforehand. We are thrilled to offer this article, as many developers still are not aware of accessibility and its nuances. We wanted to illustrate the process of what it might be like for someone new to the topic and seeking information and resources on how to implement access changes.

Hannah's piece inspired me to write an article for web developers who would like to use a screen reader to test the accessibility and usability of their websites on their own, either to determine if accessibility fixes need to be made or when testing fixes that have been applied to a page or web app. In the article, I detail how a developer can go about using the NonVisual Desktop Access (NVDA) screen reader, with the aim of providing a clear and concise guide to get someone up and running with NVDA on the web.

My second article in this issue details an accessibility feature released by Apple that I believe could be a major game-changer in accessibility. Apple's Screen Recognition feature has been available for several years and uses machine learning to recognize user interface (UI) elements in an app, even if the app itself is completely inaccessible to VoiceOver. I use Screen Recognition frequently to access apps that have no accessibility whatsoever, where VoiceOver cannot detect anything, and I find it to be an amazing tool. With the recent developments in AI vision technology and as more action-based AI models are on the horizon, it felt like a great time to spotlight this amazing technology.

Finally, Michele McDonnall of the The National Research and Training Center on Blindness and Low Vision (NRTC) brings us a report of findings from a research project conducted by the NRTC and us here at AFB. The project aims to determine the accessibility of job application websites by surveying in detail 30 Fortune 500 companies' job application websites and the applications themselves. In this article, it was discovered that though applications have improved in accessibility since research was done 10 to 12 years earlier, there are still significant barriers to applying for jobs as someone who is blind or low vision and uses a screen reader on the web. Considering the importance of employment for people who are blind and low vision and the struggles that still persist in this area, we felt it vitally important to bring knowledge of this topic to our readership.

I hope that you have enjoyed the refocused AccessWorld over the past 12 months as well as the current issue. If you have any comments, questions, or other thoughts, feel free to reach out to me by email at apreece@afb.org.

Sincerely,

Aaron Preece

AccessWorld Editor-in-Chief

American Foundation for the Blind

Jamie Pauls

My wife is seriously into photography. I am told that she takes wonderful pictures. As someone who has been totally blind since birth, I rely on her descriptions of wildlife, sunsets, and other things that catch her eye and cause her to pull our car off to the side of the road for a quick pic. I, like many other people who are blind or have low vision, am into audio. I enjoy making good-quality recordings of the sounds I hear around me. I am also a musician and need to capture a practice session from time to time. The problem with using a high-quality recorder is making sure that everything is set the way I want it before I press the record button. And once I do press Record, how can I be certain that I am actually capturing the sounds I want?

Zoom makes professional-quality recorders that don’t break the bank. Some of them start at just $99. For years, people who are blind or have low vision have been memorizing menus, counting button presses, and sharing this knowledge with others so that Zoom’s recorders as well as those made by other companies can be used by those of us who are blind or low vision.

In the first part of this year, Zoom released three recorders that have proven to be a game-changer for totally blind audio enthusiasts. Zoom’s H1Essential, H4Essential, and H6Essential recorders all have accessibility built in—something that has not been present with any previous Zoom product. In this article, I would like to highlight some important ways in which Zoom has gotten accessibility right and a few areas still in need of improvement.

First, all three of the recorders mentioned above start up with speech enabled. It is easy enough to turn it off, but whether one is sighted or blind, they will hear the spoken prompts that allow a user to decide whether or not they wish to use the built-in accessibility these recorders provide.

Second, almost all menus speak. The only exception to this is the file list that allows a user to move through files currently on the SD card that must be in the recorder in order to capture audio. Zoom is aware of this issue and hopefully will have a fix for this problem in a future firmware release.

Beeps are available as well as spoken prompts for certain actions. Two of the three recorders allow multiple inputs to be used simultaneously, creating a multitrack recording that can be manipulated in an audio editor or, to some extent, on the recorder itself. It is necessary to arm or disarm these inputs. Upon first release, it was not possible just by listening to the beeps to determine whether or not an input such as a microphone was armed for recording. This has recently been rectified so that a blind user can determine the state of the input source in question.

The volume level of the spoken prompts can be adjusted, but not the speed. As a user who is blind becomes more familiar with the recorder, it would increase productivity to be able to speed up prompts in order to more quickly move through menus and receive important alerts.

Buttons are very tactile in nature and have some distinct differences from one another. Although this is almost certainly not a design that was intended specifically for people who are blind or have low vision, the fact that, for example, the record button is large and easily identifiable adds to the ease of use of these recorders by people who are blind or have low vision. Another design element that aids in accessibility is the fact that these recorders use 32-bit float technology. In the simplest of terms, this means that it is possible to record very loud sounds without clipping or distorting the audio being recorded. It is also possible to make really good recordings of quiet sounds as well. A blind person does not need to constantly keep track of audio levels during recording, but can mix the recording as desired later in an audio editor of their choosing.

One area where accessibility is lacking is in the creation of the manuals for these recorders. Graphics are used to indicate buttons that should be pressed, so a person who is blind attempting to use the manual to learn how to use their recorder will encounter the frustration of not being able to determine what button they should press in order to complete a task. Zoom has made some of their most recent documentation accessible, and hopefully will create all of their manuals in an accessible manner moving forward.

When new firmware is released, the process of updating one’s recorder does not include spoken prompts for the duration of the process. Fortunately, Zoom has created some documentation and YouTube videos giving people who are blind or have low vision added instruction in this process.

If accessible manuals aren’t currently available for these recorders, fortunately audio tutorials produced by people who are blind or have low vision are readily available. I and many others have received much help from the reviews of all three recorders produced by Jonathan Mosen. These tutorials are available in at least two places, one of which is the Blind Podmaker podcast.

There are a number of places where you can purchase the Zoom H1Essential, H4Essential, and H6Essential recorders. One of them is AT Guys.

If you are not part of an audio email list or following someone on social media, it may be difficult to find the latest firmware and accessible guides from Zoom. Perhaps a well-structured site relating to accessibility would be of help, but I have not yet found it if it exists. One good option for learning about the latest and greatest in all things blindness related, Zoom Recorders included, is Jonathan Mosen’s Living Blindfully podcast which you can access online or using any podcast client of your choosing.

The Zoom H1Essential, H4Essential, and H6Essential recorders are mainstream products of high quality with accessibility built in for low vision users. Zoom is to be commended for their work in this area as well as their willingness to engage in regular dialog with the blind community in order to constantly improve their products.

Hannah Thompson

As a junior software developer, I was asked the question: "What accessibility features does your web application incorporate?" Admittedly, I had not yet integrated any such features. This interaction prompted me to begin the journey of understanding the implementation of accessibility features to better enhance the experience for all users.

This article will include details about the findings of my research, documenting my encounters with:

• NVDA (Non-Visual Desktop Access)
• WCAG (Web Content Accessibility Guidelines)
• Implementation of accessibility features in React code

NVDA: Non-Visual Desktop Access

NVDA, short for Non-Visual Desktop Access, is a software tool designed to assist blind and low vision individuals in interacting with the Windows operating system. I began my dive into accessibility with the installation of this tool. The process was pretty straightforward — I visited the NVDA website, followed the download instructions provided, and proceeded to install the software on my local computer.

Upon completing the download and installation, I was surprised to find my laptop audibly engaging with me. Every action I performed, from moving the mouse to navigating through applications, was followed by detailed verbal descriptions. Initially overwhelmed, I gradually acclimated to NVDA's precise audio feedback as I explored various elements on my screen.

While NVDA doesn't directly assist in coding implementation, it has significantly enhanced my understanding of accessibility. By providing specific descriptions of my actions on the computer screen and detailing what I was hovering over, NVDA highlighted the importance of features like alt labels and alt text in improving web application accessibility. I highly recommend downloading this software, as it is invaluable for testing accessibility features. Additionally, NVDA supports various languages, integrates with popular web browsers, and is regularly updated to ensure compatibility with new technology and standards.

WCAG: Web Content Accessibility Guidelines

My next stop was exploring the Web Content Accessibility Guidelines (WCAG). In my opinion, understanding the WCAG is crucial for anyone involved in creating or managing web applications. WCAG provides a set of standards aimed at improving website accessibility for people with disabilities. It addresses various aspects such as content visibility, ease of use, and compatibility across different devices. By adhering to WCAG guidelines, developers can ensure that their content is accessible to all users, regardless of their abilities, thus providing a better user experience.

By exploring the WCAG, I gained a clearer understanding of the specific features required to enhance the already existing functionality of my web application. Put simply, I learned what I needed to fix to improve the accessibility of my website based on what features I offered users.

Implementation of Accessibility Features: React

I designed a website using React, a language that is filled with pre-made components for building websites or applications. My web application, Bookshelf, functions as a social media platform enabling users to manage their personal libraries. During user testing, feedback highlighted the need for better integration of alt text and labels within the platform. For example, my images didn’t have labels, my edit and delete buttons weren’t labeled, and my edit profile form inputs were labeled incorrectly. Therefore, the user who was testing my application could not use it properly due to my lack of accessibility implementation.

Initially, I decided to read more into the accessibility features provided by React. After a quick Google search, I found that React offers an article that revolves around implementation practices. In this documentation, I discovered that React follows the WCAG guidelines and even includes links for developers to reference. Additionally, React uses components such as <label> tags, <Fragment> tags, and `aria-* attributes to provide developers with essential tools for enhancing the accessibility of their own applications.

I feel that it’s important to note that incorporating aria-labels incorporates accessibility by providing an accessible name for an element that does not have visible text or a visible label, while the alt attribute creates alternative text for images, both aiding users with visual impairments in understanding their purpose and function. While they are similar in function, they are used on different components. Also, using these attributes is important because implementing alt text ensures that screen-readers can effectively interpret the content rendered on the user's screen.

Before resorting to aria-labels, developers should ensure accessibility by adhering to several key practices. Begin with semantic HTML and ensure all interactive elements have visible labels. Use native HTML attributes like alt for images and labels for form inputs. These steps lay a solid foundation for accessibility, allowing aria-labels and other ARIA attributes to serve as supplementary enhancements when needed. However, since I used React to create my web application, I incorporated aria-labels as they are a key part of React's template structure.

After finishing the article, I began the process of incorporating accessibility features directly into my own code. My primary focus was on the addition of alt labels for all buttons, and form fields and alt text for all images. I noticed that certain components I had used from React already included the alt = "" or aria-label ="" attributes, making for an easier refactoring process. However, in other instances, I had to incorporate accessibility code from scratch, in these instances I used my resources such as React’s documentation for guidance. In addition, I also had to refactor incorrectly labeled form fields which was a simple fix once I had examples of properly functioning code. In the end, I tested the functionality of my website by navigating through it using NVDA software. This ensured that all alt-text was properly incorporated.

Code Examples

For those of you who are interested to see examples of my code implementation, I've included a few below:

Note: The aria-label attribute is typically used to provide alternative text for non-text elements, such as icons or images when the alt attribute is not applicable or sufficient.

Code Examples

Aria-Label on A Form Input: In this example, I am using an aria-label to label my input form for “User Name.” When the screen-reader hovers over the button, it will say “User Name.”

<p> User Name <Input name="userName" type="text" value={editUser.userName} onChange={handleControlledInputChange} aria-label="User Name" /> </p>

Aria-Label on a Button: In this example, I am using an aria-label to label my pencil icon as “Edit.” When the screen-reader hovers over the button, it will say “Edit.”

<Button color="primary" onClick={() => setShowForm(bookClub.id)}> <Pencil aria-label="Edit" /> </Button>

Note: Notice the aria-label was placed within the icon, not the actual tag.

Alt-Text on an Image: In this example, I'm using the alt attribute with the value "User Profile Picture" to provide alternative text. When the screen-reader hovers over the image, it will say “User Profile Picture”

<img style={{ width: "170px", float: "left", marginRight: "20px", borderRadius: "50%", border: "2px solid #000060" }} src={user.imageUrl} alt="User Profile Picture" />

Final Thoughts

As a junior developer new to accessibility, I found the process of learning and implementing these features fulfilling. Again, I highly recommend downloading NVDA on your local computer, as well as referencing the WCAG guidelines when you begin to add accessibility functionality into your own code. From understanding screen-reading software to coding implementation, it was an enlightening and necessary journey. It not only improved my skills as a developer but it also taught me the importance of inclusivity in technology. I hope this article prompts you to realize the importance of accessibility and begin to consider adding accessibility implementation into your own code.

Resources

Listed below are the online resources I used throughout the process:

NonVisual Desktop Access (NVDA): NVDA is free screen reader software for Windows, aiding visually impaired users to access digital content. Web Content Accessibility Guidelines (WCAG): WCAG provides the standards for making web content more accessible to people with disabilities. Accessibility Documentation (React): This is the documentation page on React's website discussing accessibility features; it covers guidelines, best practices, and tools for enhancing the accessibility of React-based web applications.

Aaron Preece

When you're making your website accessible, there are many guidelines and suggestions to help you do so. Front and center is the Web Content Accessibility Guidelines (WCAG), which are incredibly detailed and comprehensive in detailing what is needed to bring accessibility to your website or web app. If you would like to take things even further, you can use the same software that a blind or low vision person would use to navigate the web to ensure your website is fully accessible and usable.

In this article, we will detail everything you need to know to start testing your website for accessibility using the Non-Visual Desktop Access (NVDA) screen reader on Windows. We choose NVDA as the screen reader of choice for web developers for several reasons. First, it is one of the two most popular screen readers on Windows, the other being JAWS by Vispero. Fortunately, NVDA and JAWS have many commonalities in how they interact with the web and present it to users. Secondly, NVDA is free software that can be downloaded and used by anyone free of charge. Finally, much like JAWS, the free software is relatively complex and full-featured.

Screen Reader Basics

Before we dive into NVDA itself, let's take a moment to understand what a screen reader is and how it functions for a blind or low vision user. A screen reader, in simple terms, takes visual screen content and outputs it as audible speech using a speech synthesizer. It's important to note that in addition to outputting content as speech, a screen reader provides alternative mechanisms for navigating user interfaces to compensate for controls that would normally be accessed by mouse. On desktop computers, this primarily takes the form of keyboard commands. On mobile, screen readers use specific multi-finger gestures to accomplish the same. For this piece, we specifically focus on desktop screen readers as websites on a desktop tend to be displayed in the most complex form.

Getting Started with NVDA

To use NVDA, you will first need to download the software from this page and install it. When you open the installer on your Windows computer, you will hear a musical intro play before the installer loads. When the installer loads, you will be presented with speech feedback as a temporary copy of NVDA is loaded onto your system so that you may complete the installation successfully if you are someone who is blind or has low vision. Be aware of the speech, but everything you need to install the program will be presented to you visually, and you can use your mouse as normal to complete the installation on your computer.

When NVDA first launches, you will be presented with a dialog box welcoming you to NVDA and allowing you to set your keyboard layout as well as choose if you wish the Caps Lock key to be the NVDA modifier key. The NVDA modifier key is incredibly important and used for a massive amount of the commands present in NVDA. I highly recommend that you check the checkbox to make Caps Lock the NVDA modifier key as it significantly improves your ability to input NVDA-specific commands efficiently. When Caps Lock is a modifier key, you can still use the Caps Lock function by pressing it twice quickly.

With the decisions in the welcome dialog made and the dialog closed, you will most likely wish to change some default settings in NVDA such as the speech synthesizer or the rate of speech. NVDA is primarily a keyboard-command-driven application with little visible UI. That being said, there are UI elements for settings and tools that you may wish to access. To gain access to NVDA settings and preferences, open the main NVDA menu by pressing and holding the NVDA key and then pressing N. So going forward, we would say something like, "To open the NVDA menu, press NVDA+N."

Now you should have a visual menu that you can use your mouse or arrow keys to interact with. I would first recommend going into Preferences and then into Settings to adjust the speech settings. Make changes to the rate and other options as you wish and then apply the changes and close this window. NVDA will automatically save your changes by default, but in case you would like to save them manually, hold NVDA and Ctrl and press C. You should hear "configuration saved," alerting you that you have saved your current settings manually.

Before we open a web browser and begin exploring the web with NVDA, we need to make one more change. Go back into Preferences and Settings, and navigate to Browse Mode. Here, you want to uncheck the box that says "Use screen layout when supported." The most drastic change to turning this off will be that now, all links will show up as the first item on their line when navigating the web using arrow keys. When this option is left on, inline links are left inside of the sentences where they appear. This can make navigation much more cumbersome when using the web, in my personal experience.

Before we begin using NVDA on the web, there is an NVDA add-on that is incredibly helpful to anyone using NVDA visually online. NVDA can be augmented through add-ons—small application packages that can be installed for greater functionality. For some examples of those I use personally to augment the accessibility of my computer usage, see this blog post. The add-on we wish to find is called Focus Highlight. The link provided in the previous sentence will work properly with the latest version of NVDA at the time of this article's publication. Focus Highlight is incredibly useful as it will show a visual marker on the screen illustrating where NVDA is currently focused. In addition, the different colors of the marker around the focus element will show if the element is one that can be navigated to using the keyboard normally or is an element only navigable because of NVDA's processing of the web page.

When you use a screen reader on the web, essentially the screen reader takes the HTML code of the site that has been displayed and structures it into one large column. Think about the situation where you have a navigation bar going left to right as well as a sidebar going vertically that is next to the main content of your page. For NVDA, your navigation bar will probably appear first if you have constructed it so in your HTML, and each item in the navigation bar, even though it is visually arranged horizontally, will be navigated vertically using the up and down arrows. Subsequently, you will find your sidebar and then below that your main content of your page or potentially vice versa depending on your layout in the actual HTML code. A good way to think of this is that every single web page is displayed as if it were a Word document. So up and down arrows move line by line, left and right move character by character, and other keyboard commands you can use in a Word document function similarly, such as Ctrl+left and right for navigating word by word or Home and End for navigating to the beginning or end of the line. This is vitally important to keep in mind because you want to be sure that the focus order in your actual HTML code is displayed in such a way that you want the screen reader user to see your page when they visit it.

Using the up and down arrow to navigate through most pages does become quite cumbersome due to the sheer size and complexity of most websites. Fortunately, NVDA includes a set of single-key keyboard shortcuts that will allow you to navigate by different elements on the page. This is important to keep note of as most blind or low vision people have been trained to use various shortcuts frequently to efficiently navigate web pages. For example, headings have for decades been the most commonly suggested method for navigating to key content on a website. The assumption is that if, say, you're reading an article on a news website, the main content or the article will be preceded by a heading level one. By jumping directly to this heading, the blind or low vision user can efficiently navigate past all of the header information to get to the content they're actually interested in. If you do not have a heading, it makes it much more difficult for someone using a screen reader to access this content. However, if there is a reason why you would not want to implement a proper HTML heading, you can use something like an ARIA landmark, which also has a keyboard shortcut to allow users to jump by different specifically landmarked portions of your page.

Here are some common keyboard shortcuts for navigating to different elements: - H: Moves forward down the page by all heading levels. - 1-6: Navigates forward by specific heading levels. This is extremely valuable when navigating a lengthy web page, such as a Wikipedia article, that has many sections and subsections. - F: Navigates by form elements. This can be anything including edit fields, buttons, radio buttons, checkboxes, and combo boxes. All of these elements also have shortcut keys, such as E for edit field, R for radio button, B for button, C for combo box, and X for checkbox. - K: Navigates by links. - N: Navigates to the next plain text after a block of links. This does not always function properly, but it is intended to allow a user to skip past things like a navigation bar where you have a long list of links one after another. The assumption being the next bit of plain text will probably be the main content that someone is searching for or potentially the title of the next list of content.

There are several other less commonly used keyboard shortcuts; you can see these in the NVDA Commands Quick Reference by going into the NVDA menu with NVDA+N, selecting Help, and selecting Commands Quick Reference. All of these keys navigate the user forward on the page. If the user wishes to move backwards through the page, they can use the same exact keys by simply holding Shift while doing so. So, holding Shift and pressing H will navigate backwards by headings.

When you're navigating a webpage in this way using the arrow keys as well as the single-key shortcut keys to navigate by elements, you're in what is considered NVDA Browse Mode. If you intend to type anything into an edit field or, in many cases, use shortcut keys provided by a web app, the single-letter shortcut keys for elements are going to interfere. This is where Focus Mode is helpful. Focus Mode can either be entered automatically upon NVDA's focus landing on an element in which Focus Mode would be preferred, such as an edit field, or can be manually triggered by pressing NVDA+Spacebar. When you do so, you will hear a mechanical click or button-press type sound to indicate that you're in Focus Mode. Pressing it again will return you to Browse Mode, whereupon you will hear a beep sound.

As mentioned, entering Focus Mode is useful when interacting with certain page elements such as edit fields and combo boxes. But it can also be incredibly useful when using modern web apps. For example, Google Docs employs a significant number of keyboard shortcuts that can only be used when in Focus Mode. Likewise, the way accessibility has been implemented for a screen reader to accurately present the Google Docs app to the user requires Browse Mode to be turned off.

The Bottom Line

It's heartening to see the number of resources available to web developers who seek to make their websites and web apps accessible. Being able to use the same tools as someone who is blind or low vision when using your website can provide you with instant feedback on your accessibility implementations. It can also allow you to find accessibility or usability gaps that you might not otherwise detect when simply using instructional resources such as the Web Content Accessibility Guidelines.

Aaron Preece

In the last year, significant improvements in AI vision technology have emerged, most notably OpenAI's GPT-4 and its integration with the Be My Eyes app. Previously a novelty in most cases, image recognition has now advanced to a point where it can be incredibly valuable in multiple real-world situations. With technologies such as image AI, it seems inevitable that they will be adapted into screen reading and other accessibility technologies to aid people who are blind or have low vision.

For this piece, I wanted to highlight a groundbreaking feature that takes a significant step to use machine learning to increase accessibility: Apple's Screen Recognition feature integrated with VoiceOver. This feature scans the currently visible UI displayed on your phone or tablet, automatically recognizes UI elements, and makes them available to VoiceOver, even if they were completely invisible to the screen reader before. Notably, Screen Recognition takes place on the user's device and does not require cloud access.

Getting Started with Screen Recognition

To use Screen Recognition, you will need an iOS device no older than an iPhone Xs. A full list of supported devices can be found on Apple's official website. Using Screen Recognition is fairly simple. First, go to Settings > Accessibility > VoiceOver > VoiceOver Recognition, and turn Screen Recognition on. Most likely, your phone will need to download the machine learning model used for recognizing content.

Once Screen Recognition is active, there is one other aspect to consider to ensure you can use it to its full extent. Oftentimes, apps that are entirely inaccessible register to VoiceOver as direct touch apps. Direct touch is a feature that allows a user to interact directly with an app, bypassing VoiceOver's usual commands. This theoretically allows an app to be navigated with its own built-in gestures and inputs while still being able to output information to VoiceOver. However, whenever an app automatically turns on direct touch, it becomes inaccessible with VoiceOver gestures, making it impossible to use with Screen Recognition.

You can manually turn off direct touch in specific apps by going to Settings > Accessibility > VoiceOver > Rotor and then selecting Direct Touch Apps. Additionally, you can disable direct touch on the fly by doing a two-finger quadruple tap to bring up VoiceOver Quick Settings. Here, you will find an option to turn off direct touch altogether. Note that it may turn on again later and might need to be deactivated again. Also, you will need to turn off direct touch before entering an app that registers as a direct touch app, as once you're in the app, you will not be able to perform this command.

Using Screen Recognition Effectively

Once you have installed Screen Recognition, activated it, and ensured that the app you wish to use will not automatically enable direct touch, you are ready to begin. I personally recommend adding Screen Recognition to your Rotor settings so it will be easy to turn on and off for specific apps on the fly. Go to Settings > Accessibility > VoiceOver > Rotor > Rotor Items and select Screen Recognition to ensure you always have access.

Screen Recognition is most useful in apps that do not provide any accessibility, but it can be used in any app, even in stock apps such as Safari. This is where having Screen Recognition in your Rotor will come in handy, as it doesn't simply recognize inaccessible items but every item in your currently visible UI. This means that if you open an app that is partially accessible already, Screen Recognition will essentially re-identify all the elements on the screen, which might introduce accessibility errors elsewhere.

Remember, Screen Recognition can recognize UI elements such as buttons and sliders, as well as all the text displayed on the screen. However, it may not necessarily know what the image on a button means. For example, if Screen Recognition sees a button with a gear icon for settings in an app, it recognizes that it is a button but might not recognize that it is a settings button, unless the word "settings" is nearby to alert Screen Recognition that this label is associated with the button. More commonly, you will need to use "explore by touch" to find text labels and their associated buttons. Not ideal, but still quite usable.

When using Screen Recognition, be aware that elements that are distinct might be grouped into one large element or otherwise recognized incorrectly. I found it incredibly useful to rapidly turn VoiceOver off and back on to force Screen Recognition to re-identify the screen, which often corrects any strange recognition errors. Additionally, if a screen is scrollable, just scrolling the screen slightly might cause Screen Recognition to identify elements differently.

The Bottom Line

As smartphones and tablets continuously increase in power, it's exciting to consider where technologies like Screen Recognition will be in the future. Currently, Screen Recognition is functional for apps that use a specific layout or feature text labels heavily but might still be impractical for many others. However, as device processing power and storage capacity increase, the assumption is that the recognition model will become more powerful as well.

At the time of publication, OpenAI had just released GPT-4O, their latest large language model with image processing capabilities. GPT-4O has the capability to live recognize streamed video content, a major leap compared to the static image recognition available just a year earlier. This industry moves incredibly fast, and I have high hopes for the future of AI-based accessibility tools.