How do I recurse in RxJava while only using a single thread?

Question

I had some working recursion code in my RxJava-based network stack for flatMapping the HTTP body string file listing into an Observable of custom POJOs representing the files.

However, I was using .scheduleOn(Schedulers.newThread()) , which uses multiple new threads to do the work. I just found out one of the devices I need to connect to only allows a single HTTP connection per sixty seconds for locking purposes. This means I need to use the same thread and connection for all connections to this device.

The following code deadlocks when I call it with a .scheduleOn(Schedulers.from(Executors.newSingleThreadExecutor())) . How could I rewrite this RxJava recursion to make it function better when using only a single worker thread? I've seen examples in Rx.NET using nested Actions that lead me to believe that this is possible.

public Observable<? extends MyFile> getAllFiles(final String path) {

    Observable<String> response = NetworkUtils.getRequestAsString(GET_FILE_LIST + path);

    return response.flatMap(new Func1<String, Observable<MyFile>>() {
        @Override
        public Observable<MyFile> call(String s) {
            List<MyFile> files = new ArrayList<MyFile>();
            Observable<MyFile> fileObservable = Observable.empty();

            try {
                ObjectMapper mapper = new ObjectMapper();
                JSONObject jo = new JSONObject(s);
                JSONArray a = jo.names();
                for (int i = 0; i < a.length(); i++) {
                    JSONObject o = jo.getJSONObject(a.getString(i));
                    MyFile file = mapper.readValue(o.toString(), MyFile.class);
                    file.setDirectory(path);
                    if (file.getType().equals("dir")) {
                        fileObservable = fileObservable.mergeWith(getAllFiles(file.getFullPath())); // add files from subdirectories
                    } else if (file.getType().equals("file")) {
                        files.add(file);
                    }
                }
                return fileObservable.mergeWith(Observable.from(files)); // add files in the current directory
            } catch (Exception e) {
                return Observable.error(e);
            }
        }
    });
}

I'm calling the function like this where mExecutor is initialized as Executors.newSingleThreadExecutor() :

mDevice.getAllFiles("/")
 .toList()
 .subscribeOn(Schedulers.from(mExecutor))
 .observeOn(Schedulers.trampoline())
 .subscribe(
   //...
 );

getRequestAsString looks like this:

public static Observable<String> getRequestAsString(final String urlString) {

    return Observable.create(new Observable.OnSubscribe<String>() {
        @Override
        public void call(Subscriber<? super String> subscriber) {
            HttpURLConnection urlConnection = null;
            try {
                // Open connection
                URL url = new URL(urlString);
                urlConnection = (HttpURLConnection) url.openConnection();

                // Set parameters
                urlConnection.setDoOutput(false);
                urlConnection.setRequestMethod(REQUEST_METHOD_GET);
                urlConnection.setRequestProperty(ACCEPT_PROPERTY, ACCEPT_VALUE);
                urlConnection.setRequestProperty(CONTENT_TYPE_PROPERTY, CONTENT_TYPE_VALUE);
                urlConnection.setUseCaches(false);
                urlConnection.connect();

                // Get response code
                int code = urlConnection.getResponseCode();
                if (code == RESULT_OK) {

                    // Get and return string
                    BufferedInputStream stream = null;
                    stream = new BufferedInputStream(urlConnection.getInputStream());
                    String s = stringFromStream(stream);
                    subscriber.onNext(s); // Emit string network response to Observable
                    subscriber.onCompleted(); // Complete Observable
                    urlConnection.disconnect();
                    stream.close();
                }
            } catch (Exception e) {
                subscriber.onError(e);
            } finally {
                if (urlConnection != null)
                    urlConnection.disconnect();
            }
        }
    });
}

Answer 1

I've tried to make an analogue to your code by recursively listing a local directory and I don't experience any hangs.

My guess is that your single connection enforced by the server, the recursive call tries to establish a new connection before the previous is terminated. I'd move urlConnection.disconnect(); before the onNext so a new connection may be established immediately.

Answer 2

I think you code can be easier to understand/use, and then it will be easier to found your issue if your replate your flatMap part.

You try to merge maybe a lot of observable and using it against, etc

Why you don't flatMap and return a new Observable that will emit values (instead of trying to do the same by merging observable)

yourObs.flatMap(v -> Observable.create(new FromJson(v)).subscribe();

And in your FromJson class, emit each files using :

subscriber.onNext(yourFile);