从 nifi 连接到代理后面的外部 HTTP api

Question

I have a apache/nifi:latest instance spun inside an Amazon Linux 2 EC2. For reference, see this guide: here

I have a QuerySalesforceObject ver. 1.18.0 that makes use of StandardOauth2AccessTokenProvider.

The oauth2 provider url is configured at https://test.salesforce.com/services/oauth2/token

I can curl this url from the box and from inside the docker container just fine (I don't get a timeout).

[root@ip-10-229-18-107 \~\]# docker exec -it nifi_container_persistent /bin/sh

printenv | grep -i proxy

HTTPS_PROXY=http://proxy.MY_DOMAIN.com:3128

no_proxy=localhost,127.0.0.1,MY_DOMAIN.com,.amazonaws.com

NO_PROXY=localhost,127.0.0.1, MY_DOMAIN.com,.amazonaws.com

https_proxy=http://proxy.MY_DOMAIN.com:3128

http_proxy=http://proxy.MY_DOMAIN.com:3128

HTTP_PROXY=http://proxy.MY_DOMAIN.com:3128

curl https://test.salesforce.com/services/oauth2/token

{"error":"unsupported_grant_type","error_description":"grant type not supported"}#

But when I run the task, oauth2 fails with an error

java.io.UncheckedIOException: OAuth2 access token request failed

Caused by: java.net.SocketTimeoutException: connect timed out

This leads me to believe the proxy settings are not being honored by the class. How can I fix this?

Here's more info on this class: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-oauth2-provider-nar/1.17.0/org.apache.nifi.oauth2.StandardOauth2AccessTokenProvider/index.html

Answer 1

The standard way to interface with HTTP resources with a proxy in Nifi is via StandardProxyConfigurationService : https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-proxy-configuration-nar/1.19.1/org.apache.nifi.proxy.StandardProxyConfigurationService/index.html

If a component does not have this property, then it means it does not support it.

You can try bootstrapping proxy settings into nifi with /opt/nifi/nifi-current/conf/bootstrap.conf . But there is no standard and support of proxy is not guaranteed. Implementation (bugs and all) depends on the library. aws-java-sdk ver. 1x, for example, has a bug where nonProxyHosts is not honoured. https://github.com/aws/aws-sdk-java/issues/2797

java.arg.18=-Dhttp.nonProxyHosts="foo|localhost|*.bar.org"
java.arg.19=-Dhttp.proxyHost=proxy.foo.com
java.arg.20=-Dhttp.proxyPort=123
java.arg.21=-Dhttp.proxyUser=foo
java.arg.22=-Dhttp.proxyPassword=bar

java.arg.23=-Dhttps.nonProxyHosts="foo|localhost|*.bar.org"
java.arg.24=-Dhttps.proxyHost=proxy.foo.com
java.arg.25=-Dhttps.proxyPort=123
java.arg.26=-Dhttps.proxyUser=foo
java.arg.27=-Dhttps.proxyPassword=bar