当RestTemplate调用外部API（Cloudflare服务器）时禁止

Question

I'm using one REST API inside my REST service. Everything works when I call the API from Chrome or Postman, but a Forbbiden response is returned when I call from my application.

PS: I'm using a Java Spring Boot project.

Test Method:

public static void main(String[] args) {
    final String uri = "https://swapi.co/api/planets?search=Alderaan";
    System.out.println(new RestTemplate().getForObject(uri, String.class));
}

Produces:

20:58:01.436 [main] DEBUG org.springframework.web.client.RestTemplate - HTTP GET swapi.co/api/planets?search=Alderaan 
20:58:01.461 [main] DEBUG org.springframework.web.client.RestTemplate - Accept=[text/plain, application/json, application/*+json, */*]
20:58:02.577 [main] DEBUG org.springframework.web.client.RestTemplate - Response 403 FORBIDDEN
Exception in thread "main" org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden

The external API: https://swapi.co/documentation

Answer 1

Your code actually works for me:

public static void main(String[] args) {
    final String uri = "https://swapi.co/api/planets?search=Alderaan";
    System.out.println(new RestTemplate().getForObject(uri, String.class));
}

The output is:

01:20:49.564 [main] DEBUG org.springframework.web.client.RestTemplate - HTTP GET https://swapi.co/api/planets?search=Alderaan
01:20:49.573 [main] DEBUG org.springframework.web.client.RestTemplate - Accept=[text/plain, application/json, application/*+json, */*]
01:20:51.177 [main] DEBUG org.springframework.web.client.RestTemplate - Response 200 OK
01:20:51.179 [main] DEBUG org.springframework.web.client.RestTemplate - Reading to [java.lang.String] as "application/json"
{"count":1,"next":null,"previous":null,"results":[{"name":"Alderaan","rotation_period":"24","orbital_period":"364","diameter":"12500","climate":"temperate","gravity":"1 standard","terrain":"grasslands, mountains","surface_water":"40","population":"2000000000","residents":["https://swapi.co/api/people/5/","https://swapi.co/api/people/68/","https://swapi.co/api/people/81/"],"films":["https://swapi.co/api/films/6/","https://swapi.co/api/films/1/"],"created":"2014-12-10T11:35:48.479000Z","edited":"2014-12-20T20:58:18.420000Z","url":"https://swapi.co/api/planets/2/"}]}

Maybe you have sent too many requests to a service within a minute or an hour and they have started to block your IP / User Agent or what so ever.

Answer 2

Everything works when I specify some "user-agent" in the Headers.

It seams to be a restriction on CloudFlare: https://support.cloudflare.com/hc/en-us/articles/200170086-What-does-the-Browser-Integrity-Check-do-

The user-agent of my application is my Java version (Java/1.8.0_151). If you try this user-agent you will receive a restrict access message from CloudFlare.

curl:

curl -H "User-Agent: Java/1.8.0_151" https://swapi.co/api/planets/?search=Alderaan

response: Access denied | swapi.co used Cloudflare to restrict access

This code solve the problem:

 HttpHeaders headers = new HttpHeaders();
 headers.add("user-agent", "Application");
 HttpEntity<String> entity = new HttpEntity<>(headers);

 String planetFound = restTemplate.exchange(findPlanetUri, HttpMethod.GET, entity, String.class).getBody();

Another solution:

 String planetFound = restTemplate.getForObject(findPlanetUri, String.class);

and

   @Bean
    public RestTemplate restTemplate(RestTemplateBuilder restTemplateBuilder) {
        ClientHttpRequestInterceptor interceptor = (request, body, execution) -> {
            request.getHeaders().add("user-agent", "Application");
            return execution.execute(request, body);
        };
        return restTemplateBuilder.additionalInterceptors(interceptor).build();
    }