How to get url from browser using accessibility service

Question

I have enabled accessibility-service permission and now i want to get url from address bar.

I have tried below thing :

accessibility_service_config.xml

<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
    android:accessibilityEventTypes="typeAllMask"
    android:accessibilityFeedbackType="feedbackAllMask"
    android:accessibilityFlags="flagDefault"
    android:canRetrieveWindowContent="true"
    android:description="@string/accessibility_service_description"
    android:notificationTimeout="0"
    android:canRequestFilterKeyEvents="true"
    android:settingsActivity="com.example.android.accessibility.ServiceSettingsActivity" />

AccessService.java

public class AccessService extends AccessibilityService {

    @Override
    public void onAccessibilityEvent(AccessibilityEvent event) {
        AccessibilityNodeInfo source = event.getSource();
        if (source == null)
            return;
        final String packageName = String.valueOf(source.getPackageName());
        String BROWSER_LIST = "com.android.chrome";
        List<String> browserList
                = Arrays.asList(BROWSER_LIST.split(",\\s*"));
        if (event.getEventType()
                == AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED) {
            if (!browserList.contains(packageName)) {
                return;
            }
        }

        if (browserList.contains(packageName)) {
            try {
                if (AccessibilityEvent
                        .eventTypeToString(event.getEventType())
                        .contains("WINDOW")) {
                    AccessibilityNodeInfo nodeInfo = event.getSource();
                    getUrlsFromViews(nodeInfo);
                }
            } catch (StackOverflowError ex) {
                ex.printStackTrace();
            } catch (Exception ex) {
                ex.printStackTrace();
            }
        }
    }

    public void getUrlsFromViews(AccessibilityNodeInfo info) {

        try {
            if (info == null)
                return;
            if (info.getText() != null && info.getText().length() > 0) {
                String capturedText = info.getText().toString();
                Bundle arguments = new Bundle();
                if (capturedText.contains("https://")
                        || capturedText.contains("http://")) {

                   if (capturedText.contains("facebook.com")) {
                     // open new tab
                  }
                }
            }
            for (int i = 0; i < info.getChildCount(); i++) {
                AccessibilityNodeInfo child = info.getChild(i);
                getUrlsFromViews(child);
                if (child != null) {
                    child.recycle();
                }
            }
        } catch (StackOverflowError ex) {
            ex.printStackTrace();
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }

    @Override
    public void onInterrupt() {

    }

    @Override
    protected void onServiceConnected() {
        super.onServiceConnected();
    }
}

The problem here i am facing here is when i type facebook.com in address bar and hit the url then i am getting only facebook.com or m.facebook.com and for this reason i am not able to take any action.

I want to get URL only after it is hit in address bar. Also i want to open new tab and close existing tab when it will facebook.com in address bar.

Is there any proper way to do this ?

Answer 1

I have something to add here and how we can improve your solution. It could be better if you accept the following limitations:

• The application has an embedded list of supported browsers. Any other browser won't be supported.
• The accessibility service will look for an id of an address bar text field and try to intercept the URL from there. It is not possible to catch directly the URL which is going to be loaded. To find this id we should perform a bit of reverse engineering of the target browser: collect all fields by accessibility service and compare their ids and values with user input.
• From the previous point, the next limitation is that we are going to support only the current version of the browser. The id could be changed in the future by 3rd-party browser developers and we will have to update our interceptor to continue support. This could be done either by updating an app or by providing the browser package to id mapping by the remote server
• We detect either manual user input or redirect on link pressing (because in this case, the new URL will be also visible in an address bar). BTW it is not clear what you have mean when said

I want to get the URL only after it is hit in the address bar.

• The last limitation. Despite we are able to intercept the URL and redirect user to another page we can't prevent the site from being loaded or being started to load due to delays of asynchronous parsing a browser app screen. Eg it is not really safe to protect a user from any access to a fraud site

The implementation:

    public class UrlInterceptorService extends AccessibilityService {
        private HashMap<String, Long> previousUrlDetections = new HashMap<>();

        @Override
        protected void onServiceConnected() {
            AccessibilityServiceInfo info = getServiceInfo();
            info.eventTypes = AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED;
            info.packageNames = packageNames();
            info.feedbackType = AccessibilityServiceInfo.FEEDBACK_VISUAL;
            //throttling of accessibility event notification
            info.notificationTimeout = 300;
            //support ids interception
            info.flags = AccessibilityServiceInfo.FLAG_REPORT_VIEW_IDS |
                    AccessibilityServiceInfo.FLAG_RETRIEVE_INTERACTIVE_WINDOWS;

            this.setServiceInfo(info);
        }

        private String captureUrl(AccessibilityNodeInfo info, SupportedBrowserConfig config) {
            List<AccessibilityNodeInfo> nodes = info.findAccessibilityNodeInfosByViewId(config.addressBarId);
            if (nodes == null || nodes.size() <= 0) {
                return null;
            }

            AccessibilityNodeInfo addressBarNodeInfo = nodes.get(0);
            String url = null;
            if (addressBarNodeInfo.getText() != null) {
                url = addressBarNodeInfo.getText().toString();
            }
            addressBarNodeInfo.recycle();
            return url;
        }

        @Override
        public void onAccessibilityEvent(@NonNull AccessibilityEvent event) {
            AccessibilityNodeInfo parentNodeInfo = event.getSource();
            if (parentNodeInfo == null) {
                return;
            }

            String packageName = event.getPackageName().toString();
            SupportedBrowserConfig browserConfig = null;
            for (SupportedBrowserConfig supportedConfig: getSupportedBrowsers()) {
                if (supportedConfig.packageName.equals(packageName)) {
                    browserConfig = supportedConfig;
                }
            }
            //this is not supported browser, so exit
            if (browserConfig == null) {
                return;
            }

            String capturedUrl = captureUrl(parentNodeInfo, browserConfig);
            parentNodeInfo.recycle();

            //we can't find a url. Browser either was updated or opened page without url text field
            if (capturedUrl == null) {
                return;
            }

            long eventTime = event.getEventTime();
            String detectionId = packageName + ", and url " + capturedUrl;
            //noinspection ConstantConditions
            long lastRecordedTime = previousUrlDetections.containsKey(detectionId) ? previousUrlDetections.get(detectionId) : 0;
            //some kind of redirect throttling
            if (eventTime - lastRecordedTime > 2000) {
                previousUrlDetections.put(detectionId, eventTime);
                analyzeCapturedUrl(capturedUrl, browserConfig.packageName);
            }
        }

        private void analyzeCapturedUrl(@NonNull String capturedUrl, @NonNull String browserPackage) {
            String redirectUrl = "your redirect url is here";
            if (capturedUrl.contains("facebook.com")) {
                performRedirect(redirectUrl, browserPackage);
            }
        }

        /** we just reopen the browser app with our redirect url using service context
         * We may use more complicated solution with invisible activity to send a simple intent to open the url */
        private void performRedirect(@NonNull String redirectUrl, @NonNull String browserPackage) {
            try {
                Intent intent = new Intent(Intent.ACTION_VIEW, Uri.parse(redirectUrl));
                intent.setPackage(browserPackage);
                intent.putExtra(Browser.EXTRA_APPLICATION_ID, browserPackage);
                intent.addFlags(Intent.FLAG_ACTIVITY_CLEAR_TOP | Intent.FLAG_ACTIVITY_CLEAR_TASK | Intent.FLAG_ACTIVITY_NEW_TASK | Intent.FLAG_ACTIVITY_SINGLE_TOP);
                startActivity(intent);
            }
            catch(ActivityNotFoundException e) {
                // the expected browser is not installed
                Intent i = new Intent(Intent.ACTION_VIEW, Uri.parse(redirectUrl));
                startActivity(i);
            }
        }

        @Override
        public void onInterrupt() { }

        @NonNull
        private static String[] packageNames() {
            List<String> packageNames = new ArrayList<>();
            for (SupportedBrowserConfig config: getSupportedBrowsers()) {
                packageNames.add(config.packageName);
            }
            return packageNames.toArray(new String[0]);
        }

        private static class SupportedBrowserConfig {
            public String packageName, addressBarId;
            public SupportedBrowserConfig(String packageName, String addressBarId) {
                this.packageName = packageName;
                this.addressBarId = addressBarId;
            }
        }

        /** @return a list of supported browser configs
         * This list could be instead obtained from remote server to support future browser updates without updating an app */
        @NonNull
        private static List<SupportedBrowserConfig> getSupportedBrowsers() {
            List<SupportedBrowserConfig> browsers = new ArrayList<>();
            browsers.add( new SupportedBrowserConfig("com.android.chrome", "com.android.chrome:id/url_bar"));
            browsers.add( new SupportedBrowserConfig("org.mozilla.firefox", "org.mozilla.firefox:id/url_bar_title"));
            return browsers;
        }
    }

and accessibility service config:

<accessibility-service
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:description="@string/accessibility_service_description"
    android:canRetrieveWindowContent="true"
    android:settingsActivity=".ServiceSettingsActivity" />

Feel free to ask any questions and I'll try to help

Answer 2

first, add flag and package in accessibility_service XML

<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
    android:accessibilityEventTypes="typeAllMask"
    android:accessibilityFeedbackType="feedbackAllMask"
    android:accessibilityFlags="flagDefault|flagIncludeNotImportantViews|flagRequestTouchExplorationMode|flagRequestEnhancedWebAccessibility|flagReportViewIds|flagRetrieveInteractiveWindows"
    android:canRetrieveWindowContent="true"
    android:description="@string/accessibility_service_description"
    android:notificationTimeout="0"
    android:canRequestFilterKeyEvents="true"
    android:packageNames="com.android.chrome"
    android:settingsActivity="com.example.android.accessibility.ServiceSettingsActivity" />

in AndroidManifest:

<service android:name=".MyAccessibilityService"
        android:label="@string/accessibility_title"
        android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
        <intent-filter>
            <action android:name="android.accessibilityservice.AccessibilityService"/>
        </intent-filter>
        <meta-data
            android:name="android.accessibilityservice"
            android:resource="@xml/accessibility_service" />
    </service>

class java:

public class ASUrl extends AccessibilityService {

      @Override
      public void onAccessibilityEvent(AccessibilityEvent event) {
  if(AccessibilityEvent.eventTypeToString(event.getEventType()).contains("WINDOW")){
         AccessibilityNodeInfo nodeInfo = event.getSource();
         dfs(nodeInfo);
    }
      }

      /**
      * Method to loop through all the views and try to find a URL.
      * @param info
      */
      public void getUrlsFromViews(AccessibilityNodeInfo info) {  
        if(info == null)
          return;

        if(info.getText() != null && info.getText().length() > 0)
          System.out.println(info.getText() + " class: "+info.getClassName());

        for(int i=0;i<info.getChildCount();i++){
         AccessibilityNodeInfo child = info.getChild(i);
         getUrlsFromViews(child);
          if(child != null){
            child.recycle();
          }
        }
      }
 
  }