How to wait until URL is ready in Java?

Question

I created method that reads bytes from an URL with image and writes it to the S3 bukcet.

public void saveFileToStorage(String url, Long timestamp, Integer vehicleId) {
    try {
        URL link = new URL(url);
        Thread.sleep(1500);//wait until URL is ready for download
        byte[] contentBytes = IOUtils.toByteArray(link);
        Long contentLength = (long) contentBytes.length;
        repository.uploadFile(timestamp + ".jpg", link.openStream(), vehicleId.toString() + "/", contentLength);
    } catch (IOException | InterruptedException e) {
        log.error(e.getMessage() + " - check thread sleep time!");
        throw new RuntimeException(e);
    }

}

Repository:

public void uploadFile(String keyName, InputStream file, String folder, Long contentLength) {
    ObjectMetadata folderMetadata = new ObjectMetadata();
    folderMetadata.setContentLength(0);
    ObjectMetadata fileMetadata = new ObjectMetadata();
    fileMetadata.setContentLength(contentLength);
    s3client.putObject(bucketName, folder, new ByteArrayInputStream(new byte[0]), folderMetadata);
    s3client.putObject(new PutObjectRequest(bucketName, folder + keyName, file, fileMetadata));
}

My main problem with this method was the time of URL readiness, when I run the app it always threw a RuntimeException because the URL wasn't ready to read. After some tests where I was looking at the URL response time, I added Thread.sleep and after that the apps work fine, but the response time can vary and I still get these errors from time to time. What is the best way to check if a URL is ready? I'm trying to use the code below to check for readiness, but I can't figure out how to "wait" until the URL is ready to use

 public static boolean check(String URLName){
    try {
        HttpURLConnection.setFollowRedirects(false);
        HttpURLConnection con = (HttpURLConnection) new URL(URLName).openConnection();
        con.setRequestMethod("HEAD");
        return (con.getResponseCode() == HttpURLConnection.HTTP_OK);
    }
    catch (Exception e) {
       return false;
    }

Answer 1

I would write the method something like this:

create and configure HttpURLConnection for GET requests
for retries in 0 ... N:
    try:
        connect
        if status code == 200:
            open and read connection input stream
            return
        else:
            if status code not retriable:
                fail
    catch FileNotFound:
        // noop
    catch IOException:
        fail
    sleep S seconds
fail
     

There some key differences between this and your approach.

• This doesn't do a HEAD. That requires an extra round trip to the server. Just do a GET each time.
• This doesn't retry for all status codes. For example, if you got a 403 you shouldn't retry.
• This retries for a FileNotFound exception which you will get for a 404 response; see the javadoc. It shoulkdn't retry for other IO exceptions... unless they are likely to succeed on retry.
• The max number of retries and delay between retries should be tuned.