SwitchAccess compatible virtual View nodes' deep tree hierarchy

Question

I am trying to create an accessible custom View in Android. The View consists of a virtual nodes' tree hierarchy which can be multi-level deep in my case. The tree is not necessarily binary: each node on each level may have as many children as the user wants. Specifically, I am trying to make it compatible with TalkBack's SwitchAccess accessibility service. The end goal is to make it compatible with all the TalkBack's accessibility services, but it must be compatible with SwitchAccess at least. In this post I am trying the single switch access. The structure of the tree is not known beforehand, neither is its scanning sequence. They are all defined at runtime. I am using the class ExploreByTouchHelper which is a convenience wrapper around AccessibilityDelegate .

The following resembles an image of the sequence of events along with the virtual tree:

An example scanning sequence is denoted by the numbers inside the nodes. From the user's prespective, they see the leaves as rectangles of some random color. This is what is only drawn to the custom View . The scanning sequence depicted is as follows:

1. First the user sees all the nodes being scanned together (there is the root being scanned). They press the switch and the root is selected, moving the scanning procedure into the second level of the tree.
2. Then the first inner node is scanned, which the user ignores, so the scanning moves on to the second root's child, on which the user clicks the switch, moving the scanning procedure into the third level of the tree, ie the clicked node's children.
3. And so on, until the user clicks/selects a leaf node (at step named 7. Click! ) in which case a custom action takes place.

Nomenclature:

1. What is SwitchAccess? It is a service which lets users type keys/letters by connecting a single (or a few) hardware switch(es) to their phones. You can imagine it like a physical keyboard for people with disabilities which has a single (or a few) switch(es) instead of a switch per letter. The users type a letter assisted by a process called scanning in which each letter of the alphabet (or any type of key) is scanned one by one and when the users click their single switch then the corresponding letter is typed. It's like clicking a switch and the frequency of clicks being translated to letters by this service.
2. What is TalkBack? A set of accessibility services (including SwitchAccess) available to Android users. It's a normal Android application (the Android Accessibility Suite ) which, if not included in your system already, can be downloaded from here .

What I've tried (and failed):

1. Experimenting with focus state, checked state and accessibility focus state of each AccessibilityNodeInfo .
2. Reporting only the subtree part I am interested in, for every click. That means reporting different tree on every click (via getVisibleVirtualViews ).
3. Experimenting with AccessibilityNodeInfo.CollectionInfo and AccessibilityNodeInfo.CollectionItemInfo , inspired by GridView 's implementation.
4. Sending AccessibilityEvent s only when the user clicks a leaf (and not for inner nodes), so as to let the scanning continue on clicking inner nodes.
5. Reporting only groups of leaves instead of the inner nodes.
6. Reinstalling/Changing the entire AccessibilityNodeProvider and/or AccessibilityDelegate on each user's click.

Follows my best effort so far (which is a combination of some of my efforts above), on which we can also have the discussion on:

import android.content.Context;
import android.graphics.Canvas;
import android.graphics.Color;
import android.graphics.Paint;
import android.graphics.Point;
import android.graphics.Rect;
import android.graphics.RectF;
import android.os.Bundle;
import android.view.KeyEvent;
import android.view.MotionEvent;
import android.view.View;
import android.view.ViewGroup;
import android.view.accessibility.AccessibilityEvent;

import androidx.annotation.NonNull;
import androidx.annotation.Nullable;
import androidx.appcompat.app.AppCompatActivity;
import androidx.core.view.ViewCompat;
import androidx.core.view.accessibility.AccessibilityNodeInfoCompat;
import androidx.customview.widget.ExploreByTouchHelper;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Locale;
import java.util.Random;

public class MainActivity extends AppCompatActivity {

    public static int randomColor(final int alpha, final Random rand) {
        final byte[] components = new byte[3];
        rand.nextBytes(components);
        return Color.argb(alpha, components[0] & 0xFF, components[1] & 0xFF, components[2] & 0xFF);
    }

    public static class Node {
        public Node parent = null;
        public final ArrayList<Node> children = new ArrayList<>();
        public final Point index = new Point(), //Location of this Node in its parent. Root Node does not use this.
                            size = new Point(); //Number of children this Node has per dimension (x == columns, y == rows).
        public String text = null; //The text associated with each Node.
        public int id = -1, //It is initialized at Tree construction.
                color = 0; //It is only used for leaves, but for simplicity included in every Node.

        /**
         * Used as a default way to create the children of this Node.
         * @param title Some value used to construct the text of each children.
         * @param sizeX Number of columns each children will be initialized to have.
         * @param sizeY Number of rows each children will be initialized to have.
         * @param rand Used for producing each child's color.
         */
        public void addChildren(final String title, final int sizeX, final int sizeY, final Random rand) {
            for (int row = 0; row < size.y; ++row) {
                for (int col = 0; col < size.x; ++col) {
                    final Node child = new Node();
                    child.parent = this;
                    children.add(child);
                    child.index.set(col, row);
                    child.size.set(sizeX, sizeY);
                    child.text = String.format(Locale.ENGLISH /*Just use ENGLISH for only the demonstration purposes.*/, "%s|%s:%d,%d", text, title, row, col);
                    child.color = randomColor(255, rand);
                }
            }
        }

        /** @param bounds Serves as input (initialized with the root Node's bounds) and as output (giving the bounds relative to root for the calling Node). */
        public void updateBounds(final RectF bounds) {
            if (parent != null) {
                parent.updateBounds(bounds);
                //Adjust parent bounds to locate the current node:
                final float cellWidth = bounds.width() / parent.size.x, cellHeight = bounds.height() / parent.size.y;
                bounds.left += (cellWidth * index.x);
                bounds.top += (cellHeight * index.y);
                bounds.right -= (cellWidth * (parent.size.x - index.x - 1));
                bounds.bottom -= (cellHeight * (parent.size.y - index.y - 1));
            }
        }
    }

    /**
     * Gets a subtree (starting from the given Node) of nodes into the given lists.
     * @param node the root of the subtree we are interested in.
     * @param allNodes all nodes of the subtree will go in here.
     * @param leavesOnly only the leaves of the subtree will go in here.
     */
    public static void getNodes(final Node node, final ArrayList<Node> allNodes, final ArrayList<Node> leavesOnly) {
        allNodes.add(node);
        if (node.children.isEmpty())
            leavesOnly.add(node);
        else
            for (final Node child: node.children)
                getNodes(child, allNodes, leavesOnly);
    }

    /** Sacrificing memory for speed: this is essentially a huge cache. */
    public static class Tree {
        public final Node root;
        public final List<Node> nodes, //All nodes of the tree.
                                leaves; //Only leaves of the tree (which will exist in both 'nodes' property and in 'leaves' property).

        public Tree(final Node root) {
            this.root = root;
            final ArrayList<Node> nodesList = new ArrayList<>();
            final ArrayList<Node> leavesList = new ArrayList<>();
            getNodes(root, nodesList, leavesList);
            nodes = Collections.unmodifiableList(nodesList);
            leaves = Collections.unmodifiableList(leavesList);
            final int sz = nodes.size();
            for (int i = 0; i < sz; ++i)
                nodes.get(i).id = i; //As you can see the id corresponds exactly to the index of the Node in the list (so as to have easier+faster retrieval of Node by its id).
        }
    }

    /** @return a Tree for testing. */
    public static Tree buildTestTree() {
        final Random rand = new Random();
        final Node root = new Node();
        root.size.set(2, 1); //2 columns, 1 row.
        root.text = "Root";
        root.addChildren("Inner", 2, 1, rand); //2 columns, 1 row.
        for (final Node rootChild: root.children) {
            rootChild.addChildren("Inner", 2, 1, rand); //2 columns, 1 row.
            for (final Node rootInnerChild: rootChild.children)
                rootInnerChild.addChildren("Leaf", 1, 1, rand); //1 column, 1 row. Basically a leaf.
        }
        return new Tree(root);
    }

    /** @return a value conforming to the measureSpec, while being as close as possible to the preferredSizeInPixels. */
    public static int getViewSize(final int preferredSizeInPixels,
                                  final int measureSpec) {
        int result = preferredSizeInPixels;
        final int specMode = View.MeasureSpec.getMode(measureSpec);
        final int specSize = View.MeasureSpec.getSize(measureSpec);
        switch (specMode) {
            case View.MeasureSpec.UNSPECIFIED: result = preferredSizeInPixels; break;
            case View.MeasureSpec.AT_MOST: result = Math.min(preferredSizeInPixels, specSize); break;
            case View.MeasureSpec.EXACTLY: result = specSize; break;
        }
        return result;
    }

    /** The custom View which maintains the tree hierarchy of virtual views. */
    public static class HierarchyView extends View {
        private final MyAccessibilityDelegate delegate; //The ExploreByTouchHelper implementation.
        public final Tree tree; //The tree of virtual views.
        public Node selected; //The last 'clicked' node from all the nodes in the tree.
        private final int preferredWidth, preferredHeight; //The preferred size of this View.
        private final Paint tmpPaint; //Used for drawing.
        private final RectF tmpBounds; //Used for drawing.

        public HierarchyView(final Context context) {
            super(context);
            tmpPaint = new Paint();
            tmpBounds = new RectF();
            selected = null;

            //Hardcoded magic numbers for the dimensions of this View, only in order to keep things simple in this demonstration:
            preferredWidth = 600;
            preferredHeight = 300;

            tree = buildTestTree();
            super.setContentDescription("Hierarchy");
            ViewCompat.setImportantForAccessibility(this, ViewCompat.IMPORTANT_FOR_ACCESSIBILITY_YES);
            ViewCompat.setAccessibilityDelegate(this, delegate = new MyAccessibilityDelegate(this));
        }

        @Override
        protected void onMeasure(final int widthMeasureSpec, final int heightMeasureSpec) {
            setMeasuredDimension(getViewSize(preferredWidth, widthMeasureSpec), getViewSize(preferredHeight, heightMeasureSpec));
        }

        /**
         * Use this method instead of {@link Node#updateBounds(RectF)}, which (this method) will properly initialize the root Node's bounds.
         * @param bounds The output bounds for the given Node.
         * @param node The input Node to get the bounds for.
         */
        public void updateBounds(final RectF bounds, final Node node) {
            bounds.left = bounds.top = 0;
            bounds.right = getWidth();
            bounds.bottom = getHeight();
            node.updateBounds(bounds);
        }

        @Override
        protected void onDraw(final Canvas canvas) {
            for (final Node leaf: tree.leaves) {
                tmpPaint.setColor(leaf.color);
                tmpPaint.setAlpha(selected == leaf? 255: 64);
                updateBounds(tmpBounds, leaf); //Not the most efficient (needs logN), but remember this is just a demo.
                canvas.drawRect(tmpBounds, tmpPaint);
            }
        }

        @Override
        public boolean dispatchHoverEvent(final MotionEvent event) {
            //This is required by ExploreByTouchHelper's docs:
            return delegate.dispatchHoverEvent(event) || super.dispatchHoverEvent(event);
        }

        @Override
        public boolean dispatchKeyEvent(final KeyEvent event) {
            //This is required by ExploreByTouchHelper's docs:
            return delegate.dispatchKeyEvent(event) || super.dispatchKeyEvent(event);
        }

        @Override
        protected void onFocusChanged(final boolean gainFocus, final int direction, final @Nullable Rect previouslyFocusedRect) {
            super.onFocusChanged(gainFocus, direction, previouslyFocusedRect);
            //This is required by ExploreByTouchHelper's docs:
            delegate.onFocusChanged(gainFocus, direction, previouslyFocusedRect);
        }

        /*
        @Override
        public boolean onTouchEvent(final MotionEvent event) {
            final int virtualViewId = delegate.getVirtualViewAt(event.getX(), event.getY());
            if (virtualViewId != ExploreByTouchHelper.INVALID_ID)
                selected = tree.nodes.get(virtualViewId);
            invalidate();
            return super.onTouchEvent(event);
        }
        */
    }

    public static class MyAccessibilityDelegate extends ExploreByTouchHelper {
        private final HierarchyView host;

        /**
         * This is used as the <b>parent</b> of each node that should be interactive. If null, then the
         * root should be interactive, otherwise if not null, then the its children should be interactive.
         */
        private Node last;

        public MyAccessibilityDelegate(final @NonNull HierarchyView host) {
            super(host);
            this.host = host;
            last = null; //Start with root.
        }

        /** Helper method to retrieve a read-only Iterable of the nodes that should be interactive. */
        private Iterable<Node> readVisibleNodes() {
            return last == null? Collections.singletonList(host.tree.root) : Collections.unmodifiableList(last.children);
        }

        @Override
        protected int getVirtualViewAt(final float x, final float y) {
            final RectF bounds = new RectF();
            for (final Node node: readVisibleNodes()) {
                host.updateBounds(bounds, node);
                if (bounds.contains(x, y))
                    return node.id;
            }
            return INVALID_ID;
        }

        @Override
        protected void getVisibleVirtualViews(final List<Integer> virtualViewIds) {
            for (final Node node: readVisibleNodes())
                virtualViewIds.add(node.id);
        }

        @Override
        protected void onPopulateNodeForVirtualView(final int virtualViewId, final @NonNull AccessibilityNodeInfoCompat info) {
            final Node node = host.tree.nodes.get(virtualViewId);

            //Just set all text to node#text for simplicity:
            info.setText(node.text);
            info.setHintText(node.text);
            info.setContentDescription(node.text);

            //Get the node's bounds:
            final RectF bounds = new RectF();
            host.updateBounds(bounds, node);

            /*Although deprecated, setBoundsInParent is actually what ExploreByTouchHelper requires, and itself
            then computes the bounds in screen. So lets just setBoundsInParent, instead of setBoundsInScreen...*/
            if (node.parent == null) { //If node is the root:
                info.setParent(host); //The View itself is the parent of it (or maybe not, I am not sure).
                info.setBoundsInParent(new Rect(Math.round(bounds.left), Math.round(bounds.top), Math.round(bounds.right), Math.round(bounds.bottom)));
            }
            else {
                /*To get the bounds of any node which is not the root, I simply subtract the parent's bounds
                with the current node's bounds. I know... not the most efficient, but it's just a demo now.*/
                info.setParent(host, node.parent.id);
                final RectF parentBounds = new RectF();
                host.updateBounds(parentBounds, node.parent);
                info.setBoundsInParent(new Rect(Math.round(bounds.left - parentBounds.left), Math.round(bounds.top - parentBounds.top), Math.round(bounds.right - parentBounds.left), Math.round(bounds.bottom - parentBounds.top)));
            }

            //As I have found out, those calls are absolutely necessary for the virtual views:
            info.setEnabled(true);
            info.setFocusable(true);

            //These calls seem to not be absolutely necessary, but I am not sure:
            info.setVisibleToUser(true);
            info.setImportantForAccessibility(true);

//            info.setContentInvalid(false);
//            info.setAccessibilityFocused(last == node);
//            info.setFocused(last == node);
//            info.setChecked(last == node);
//            info.setSelected(last == node);

            if (node.parent == last) { //This is the way I am testing if the current node should be interactive.
                info.setClickable(true);
                info.setCheckable(true);
                //info.setCanOpenPopup(true);
                //info.setContextClickable(true);
                //info.addAction(AccessibilityNodeInfoCompat.ACTION_SELECT);
                info.addAction(AccessibilityNodeInfoCompat.ACTION_CLICK);
            }
            if (!node.children.isEmpty()) {
                info.setCollectionInfo(AccessibilityNodeInfoCompat.CollectionInfoCompat.obtain(node.size.y, node.size.x, true, AccessibilityNodeInfoCompat.CollectionInfoCompat.SELECTION_MODE_SINGLE));
                for (final Node child: node.children)
                    info.addChild(host, child.id);
            }
            if (node.parent != null)
                info.setCollectionItemInfo(AccessibilityNodeInfoCompat.CollectionItemInfoCompat.obtain(node.index.y, 1, node.index.x, 1, false, false));
        }

        @Override
        protected boolean onPerformActionForVirtualView(final int virtualViewId, final int action, final @Nullable Bundle arguments) {
            if (action == AccessibilityNodeInfoCompat.ACTION_CLICK) {
                host.selected = host.tree.nodes.get(virtualViewId);
                last = host.selected.children.isEmpty()? null: host.selected;
//                if (host.selected.children.isEmpty()) {
                invalidateVirtualView(virtualViewId); //, AccessibilityEventCompat.CONTENT_CHANGE_TYPE_SUBTREE);
                sendEventForVirtualView(virtualViewId, AccessibilityEvent.TYPE_VIEW_CLICKED);
                host.invalidate(); //To redraw the UI.
//                }
//                else
//                    invalidateRoot();
//                    invalidateVirtualView(virtualViewId);
//                    host.sendAccessibilityEvent(AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED);
//                    host.sendAccessibilityEvent(AccessibilityEvent.TYPE_VIEW_CLICKED);
//                    host.sendAccessibilityEvent(AccessibilityEventCompat.TYPE_VIEW_CONTEXT_CLICKED);
//                    sendEventForVirtualView(virtualViewId, AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED);
//                    invalidateVirtualView(virtualViewId, AccessibilityEventCompat.CONTENT_CHANGE_TYPE_SUBTREE);
                return true;
            }
            return false;
        }
    }

    @Override
    protected void onCreate(final Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(new HierarchyView(this), new ViewGroup.LayoutParams(ViewGroup.LayoutParams.WRAP_CONTENT, ViewGroup.LayoutParams.WRAP_CONTENT));
    }
}

which results in the scanning freezing on click (but at least it goes one level deeper in the tree), while the actual needed behaviour is to let the scanning continue until a leaf is clicked where the scanning should end. Almost like software keyboards are scanned (ie first a group of keys, then, on click, the scanning continues with each key, and so on).

Note that the tree constructed by the given code is not the exact same as the image's one. The image serves as a visual example, while the code serves as a base to have the discussion on.

I am basically trying to find what rules in SwitchAccess will make my application compatible with it. Ideally I would like the user to see groups of leaves being scanned, rather than single nodes each time as in the provided code, but that is a different story I guess.

I think what I am asking is possible, because otherwise methods like setParent wouldn't be included in the AccessibilityNodeInfo class.

I was also thinking about expreimenting with drawing order but I don't know if it is even related at all.

I am working with minimum SDK version 14, if at all related.

There exist some examples on the Internet about single level deep virtual trees, but I just couldn't figure out how to make them multi-level deep.

Note that several steps were taken to shorten the code, so it is not going to follow best practices about Object Oriented Programming, neither time complexity, neither memory usage, etc, because it just serves as a demonstration.

Some resources:

1. The corresponding Google I/O 2013 video (starting at the presentation of an ExploreByTouchHelper use case).
2. TalkBack's source code . I had many looks on it, but I still couldn't figure out how to solve my issue.
3. How to use SwitchAccess on your phone (user's perspective).

Answer 1

I don't believe what you want is possible. Or at least, I don't believe what you want has "Explicit APIs" that support it. Let's talk about what Switch Access will focus. Basically, it will focus anything that

• Isn't explicitly marked as Not Important for Accessibility
• That has some type of action associated with it
  • Tap
  • Tap and Hold
  • Custom Actions
  • etc.

The ways in which Switch Access groups things don't respond to any particular API, but rather are calculated from existing information that is related to the Standard User Experience and based on the Switch Access configuration. You might have this be based on different thigns for Row-Column scanning vs Group Selection. In general the things that matter are:

• Order in the View Hierarchy
• Position on Screen

Manipulating these for "precise groupings" is going to be very difficult. Each version of Switch Access can do what they want. There is no documented API that says "Here are the groupings". Switch Access is just doing its best to make sense of standard Android APIs.

Represent the information you have accurately from a User's Perspective.

• Put groups of things into virtual layouts.
• Make sure everything that can be interacted with is marked as such.
• Put your views in a sensible order in your virtual hierarchy

That's all you can really do. In fact, trying to manipulate it any more than this would potentially make things confusing for users.