在 Kotlin 中让 Coroutine 等待 Jsoup 响应

Question

I'm kind of new developing with Android, and have problems understanding how things works. I made an app to scrape content from a page, and download elements, first with AsyncTask and worked, but since AsyncTask doesn't let me communicate with the UI Activity on progress, i decided to change to coroutines, checked an example, and the same code i used doesn't seems to work.

I used a few logs to try to determine the problem, and seems like it doesn't wait for the Jsoup request. The coroutine first calls a method scrapePage() to download the HTML and scrape the links, and then calls downloadImages() to add the links to Android's DownloadManager. In the logs Log.d("action", "Start Scraping") is printed, but Log.d("action", "Page downloaded") doesn't, still we get Log.d("action", "End") from the coroutine, which makes me think that instead of waiting for the Jsoup request to answer, it goes with an empty response, causing the rest of the code to not work correctly.

DownloadService.kt

object DownloadService {

    private val parentJob = Job()
    
    ...

    private val coroutineScope = CoroutineScope(Dispatchers.Main + parentJob +
            coroutineExceptionHandler)

    fun StartService(URL: String, location:String, contx:Context) {

        coroutineScope.launch(Dispatchers.Main) {
            Log.d("action", "Start")
            val links = scrapePage(URL)
            val download = downloadImages(links, location, contx)
            Log.d("action", "End")
        }
    }

    private suspend fun scrapePage(url: String): MainActivity.Scraped =
        withContext(Dispatchers.IO) {
            var URL = url
            var scrape = MainActivity.Scraped()
            try {
                Log.d("action", "Start Scraping")
                var response = Jsoup.connect(URL).get()
                Log.d("action", "Page downloaded")
                response.getElementsByClass("link").forEach {
                    /*Scrape URLs*/
                    Log.d("action", "Add "+link)
                }
            } catch (e: Exception) {
                when(e) {
                    is HttpStatusException -> {
                        System.out.println(e.getStatusCode())
                        scrape.error = true
                        error = true
                    }
                }
            }
            return@withContext scrape
        }

    ...
}

MainActivity.kt

class MainActivity : AppCompatActivity() {
    ...
    fun makeRequest(URL : String) {
        WorkingURL = URL
        var uri = ""
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) {
            uri = MediaStore.Downloads.EXTERNAL_CONTENT_URI.toString()
            log.text = uri
        } else {
            uri = getStoragePath()
            log.text = uri
        }
        DownloadService.StartService(URL, uri, this)
        Log.d("links", DownloadService.getError().toString())
    }
}

I am not sure where the problem is, nor where to start searching. I know the code for the Scraping works, because i used it before with AsyncTask, so the problem seems to be passing it to coroutines.

Answer 1

Here Jsoup.connect(URL).get() is throwing an error. so, Log.d("action", "Page downloaded") is not called.

But since you are handling the exception, the code runs the catch part and completes the suspend function and moves on to downloadImages() .

---

Solution

First, add a log in the catch part of scrapePage() function and find out what is causing the exception. Everything else in your code is good.