Accessibility service: get views you can interact with instantly (same as Voice Access)?

Question

The goal is to obtain views that can be interacted with instantly (that can be clicked right now and something would happen). If the view is visible and clickable in general but hovered by another view/menu/side panel, it should be omitted.

Voice Access do that. And it seems to use Accessibility API.

The perfect example is the bottom menu in Google Maps. When it expands, "Search along the route" button underneath is still visible but it's not highlighted by the app.

---

So what do we have?

• There is a stream of AccessibilityEvent . The most useful is AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED , so we can be notified when something is happening.
• With getSource() we can get an instance of AccessibilityNodeInfo that triggered the event. Or we can get a root of a window with AccessibilityService.getRootInActiveWindow() . And having that we are able to traverse the whole hierarchy within an app.
• AccessibilityNodeInfo doesn't provide any information about z-order of views, so it's not possible to understand what is above and what is beneath.
• The bottom menu is in the same window (it's not modal).
• If you try to click "Search along the route" button while the bottom menu is expanded, the bottom menu collapses. So you can't actually click it, it's beneath the menu.

• I've looked through all parameters of the AccessibilityNodeInfo , like isVisibleToUser() , isClickable() , isContextClickable() , isSelected() , isFocusable() , isFocused() , isAccessibilityFocused() and the button has the same parameters when the bottom menu is collapsed/expanded. It's visible to the user, focusable and clickable.

• I've looked into hidden APIs and don't see anything that can be useful.

What I'm missing?

---

Answer 1

The key point is that in an AccessibilityService.onAccessibilityEvent() the tree hierarchy is not final. To get views that are interactable at the moment, AccessibilityService.getRootInActiveWindow() should be called with a delay.

Answer 2

AccessibilityNodeInfo#getDrawingOrder() will probably help you. Note that you need to do tree traversal to determine what is on top of what.

There are still corner cases with transparent views that will give you trouble, but that should get you 95% of the way there. We're working on a better answer for that case.