WSO2 4.0.0 control-plane as key manager : Error while parsing JWT - Connection refused to localhost:443
Question
I use two WSO2 Api Manager v4.0.0 nodes inside a private.network:
- reverse is the Nginx node that proxy requests to destination ports as a gateway/entrypoint
- apim-bo is the WSO2 control-plane. It hosts devportal/publisher and is the key manager. IP 192.168.0.6, WSO2 listen on port 9443, domain apimbo.mysite.com
- apim-gw is the WSO2 gateway worker used to handle requests. IP 192.168.0.5, WSO2 listen on port 8243, domain apim.mysite.com
- I use a wildcard certificate shared between nginx and the two WSO2 nodes, same certificates and keystore files on all nodes
It works pretty much fine but I can see a lot of errors in apim-gw logs:
[2023-01-16 16:19:34,010] ERROR {org.wso2.carbon.apimgt.impl.jwt.JWTValidatorImpl} - Error while parsing JWT org.apache.http.conn.HttpHostConnectException: Connect to localhost:443 [localhost/127.0.0.1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(Retry Exec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at org.wso2.carbon.apimgt.impl.utils.JWTUtil.retrieveJWKSConfiguration_aroundBody0(JWTUtil.java:58) at org.wso2.carbon.apimgt.impl.utils.JWTUtil.retrieveJWKSConfiguration(JWTUtil.java:52) at org.wso2.carbon.apimgt.impl.jwt.JWTValidatorImpl.retrieveJWKSet_aroundBody22(JWTValidatorImpl.java:231) at org.wso2.carbon.apimgt.impl.jwt.JWTValidatorImpl.retrieveJWKSet(JWTValidatorImpl.java:228) at org.wso2.carbon. apimgt.impl.jwt.JWTValidatorImpl.validateSignature_aroundBody8(JWTValidatorImpl.java:145) at org.wso2.carbon.apimgt.impl.jwt.JWTValidatorImpl.validateSignature(JWTValidatorImpl.java:135) at org.wso2.carbon.apimgt.impl.jwt.JWTValidatorImpl.validateToken_aroundBody0(JWTValidatorImpl.java:62) at org.wso2.carbon.apimgt.impl.jwt.JWTValidatorImpl.validateToken(JWTValidatorImpl.java:57) at org.wso2.carbon.apimgt.impl.jwt.JWTValidationServiceImpl.validateJWTToken_aroundBody0(JWTValidationServiceImpl.java:44) at org.wso2.carbon.apimgt.impl.jwt.JWTValidationServiceImpl.validateJWTToken(JWTValidationServiceImpl.java:36) at org.wso2.carbon.apimgt.gateway.handlers.security.jwt.JWTValidator.getJwtValidationInfo_aroundBody18(JWTValidator.java:519) at org.wso2.carbon.apimgt.gateway.handle rs.security.jwt.JWTValidator.getJwtValidationInfo(JWTValidator.java:487) at org.wso2.carbon.apimgt.gateway.handlers.security.jwt.JWTValidator.authenticate_aroundBody0(JWTValidator.java:170) at org.wso2.carbon.apimgt.gateway.handlers.security.jwt.JWTValidator.authenticate(JWTValidator.java:138) at org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator.authenticate_aroundBody4(OAuthAuthenticator.java:323) at org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator.authenticate(OAuthAuthenticator.java:115) at org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.isAuthenticate_aroundBody42(APIAuthenticationHandler.java:428) at org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.isAuthenticate(APIAuthenticationHandler.java:422) at org.8828 8002658488.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.handleRequest_aroundBody36(APIAuthenticationHandler.java:353) at org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.handleRequest(APIAuthenticationHandler.java:317) at org.apache.synapse.api.API.process(API.java:389) at org.apache.synapse.api.AbstractApiHandler.apiProcessNonDefaultStrategy(AbstractApiHandler.java:107) at org.apache.synapse.api.AbstractApiHandler.identifyAPI(AbstractApiHandler.java:127) at org.apache.synapse.api.AbstractApiHandler.dispatchToAPI(AbstractApiHandler.java:59) at org.apache.synapse.api.rest.RestRequestHandler.dispatchToAPI(RestRequestHandler.java:84) at org.apache.synapse.api.rest.RestRequestHandler.process(RestRequestHandler.java :70) at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:54) at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:344) at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:101) at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:375) at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:189) at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java :628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) at java.base/java.net.Socket.connect(Socket.java:591) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOper ator.java:142)
Do you have an idea of what I am missing please?
reverse apim-bo vhost:
upstream apimbo {
server 192.168.0.6:9443;
}
server {
# Listen
server_name apimbo.mysite.com;
listen 443 ssl http2;
listen [::]:443 ssl http2;
# Certs
ssl_certificate /home/user/certs/mysite.com.fullchain.pem;
ssl_certificate_key /home/user/certs/mysite.com.privkey.key;
# Proxy conf
location / {
proxy_pass https://apimbo;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_connect_timeout 90s;
proxy_send_timeout 90s;
proxy_read_timeout 90s;
}
}
reverse apim-gw vhost:
upstream apim {
server 192.168.0.5:8243;
keepalive 32;
}
server {
# Listen
server_name apim.mysite.com;
listen 443 ssl http2;
listen [::]:443 ssl http2;
# Certs
ssl_certificate /home/user/certs/mysite.com.fullchain.pem;
ssl_certificate_key /home/user/certs/mysite.com.privkey.key;
# Proxy conf
location / {
proxy_pass https://apim;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header Connection "";
proxy_connect_timeout 90s;
proxy_send_timeout 90s;
proxy_read_timeout 90s;
}
}
apim-bo /etc/hosts:
127.0.1.1 apimbo.mysite.com
127.0.0.1 localhost
192.168.0.5 apim.mysite.com
apim-bo deployment.toml:
[server]
hostname = "apimbo.mysite.com"
base_path = "${carbon.protocol}://${carbon.host}:${carbon.management.port}"
server_role = "control-plane"
[super_admin]
username = "admin"
password = "adminpass"
create_admin_account = true
[user_store]
type = "database_unique_id"
[database.apim_db]
type = "mysql"
hostname = "192.168.0.4"
name = "apimdb"
port = "3306"
username = "apimdbuser"
password = "apimdbpass"
driver = "com.mysql.cj.jdbc.Driver"
[database.shared_db]
type = "mysql"
hostname = "192.168.0.4"
name = "apimshareddb"
port = "3306"
username = "apimshareddbuser"
password = "apimshareddbpass"
driver = "com.mysql.cj.jdbc.Driver"
[keystore.tls]
file_name = "wso2carbon.jks"
type = "PKCS12"
password = "wso2"
alias = "apim"
key_password = "wso2"
[keystore.primary]
file_name = "wso2carbon.jks"
type = "{{ wso2_keystore_type }}"
password = "{{ wso2_keystore_new_password }}"
alias = "{{ wso2_keystore_new_alias }}"
key_password = "{{ wso2_keystore_new_password }}"
[keystore.internal]
file_name = "wso2carbon.jks"
type = "PKCS12"
password = "{{ wso2_keystore_new_password }}"
alias = "{{ wso2_keystore_new_alias }}"
key_password = "{{ wso2_keystore_new_password }}"
[truststore]
file_name = "client-truststore.jks"
type = "PKCS12"
password = "wso2"
alias = "symmetric.key.value"
algorithm = "AES"
# Custom : apim gateway env
[[apim.gateway.environment]]
name = "Default"
type = "hybrid"
display_in_api_console = true
description = "This is a hybrid gateway that handles both production and sandbox token traffic."
show_as_token_endpoint_url = true
service_url = "https://apim.mysite.com/services/"
username= "${admin.username}"
password= "${admin.password}"
ws_endpoint = "ws://apim.mysite.com:9099"
wss_endpoint = "wss://apim.mysite.com:8099"
http_endpoint = "http://apim.mysite.com"
https_endpoint = "https://apim.mysite.com"
[apim.cache.tags]
expiry_time = "2m"
[apim.cache.jwt_claim]
enable = true
[apim.cache.gateway_token]
enable = true
[apim.cache.resource]
enable = true
[apim.cache.km_token]
enable = true
[apim.cache.scopes]
enable = true
[apim.oauth_config]
revoke_endpoint = "https://apimbo.mysite.com/oauth2/revoke"
[apim.devportal]
url = "https://apimbo.mysite.com/devportal"
api_key_alias = "apim"
display_multiple_versions = true
[apim.cors]
allow_origins = "*"
allow_methods = ["GET","PUT","POST","DELETE","PATCH","OPTIONS"]
allow_headers = ["authorization","Access-Control-Allow-Origin","Content-Type","SOAPAction", "apikey","Internal-Key"]
allow_credentials = false
[apim.workflow]
token_endpoint = "https://apimbo.mysite.com/oauth2/token"
[apim.notification]
hostname = "apimbo.mysite.com"
[[event_handler]]
name="userPostSelfRegistration"
subscriptions=["POST_ADD_USER"]
[service_provider]
sp_name_regex = "^[\\sa-zA-Z0-9._-]*$"
[database.local]
url = "jdbc:h2:./repository/database/WSO2CARBON_DB;DB_CLOSE_ON_EXIT=FALSE"
[[event_listener]]
id = "token_revocation"
type = "org.wso2.carbon.identity.core.handler.AbstractIdentityHandler"
name = "org.wso2.is.notification.ApimOauthEventInterceptor"
order = 1
[event_listener.properties]
notification_endpoint = "https://localhost:${mgt.transport.https.port}/internal/data/v1/notify"
username = "${admin.username}"
password = "${admin.password}"
'header.X-WSO2-KEY-MANAGER' = "default"
[transport.https.properties]
proxyPort = 443
apim-gw /etc/hosts:
127.0.1.1 apim.mysite.com
127.0.0.1 localhost
192.168.0.6 apimbo.mysite.com
apim-gw deployment.toml:
[server]
hostname = "apim.mysite.com"
server_role = "gateway-worker"
[super_admin]
username = "admin"
password = "adminpass"
create_admin_account = true
[user_store]
type = "database_unique_id"
[database.shared_db]
type = "mysql"
hostname = "192.168.0.4"
name = "apimshareddb"
port = "3306"
username = "apimshareddbname"
password = "apimshareddbpass"
driver = "com.mysql.cj.jdbc.Driver"
[keystore.tls]
file_name = "wso2carbon.jks"
type = "PKCS12"
password = "wso2"
alias = "apim"
key_password = "wso2"
[keystore.primary]
file_name = "wso2carbon.jks"
type = "PKCS12"
password = "wso2"
alias = "apim"
key_password = "wso2"
[keystore.internal]
file_name = "wso2carbon.jks"
type = "PKCS12"
password = "wso2"
alias = "apim"
key_password = "wso2"
[truststore]
file_name = "client-truststore.jks"
type = "PKCS12"
password = "wso2"
alias = "symmetric.key.value"
algorithm = "AES"
[apim.key_manager]
service_url = "https://apimbo.mysite.com:9443${carbon.context}/services/"
username= "$ref{super_admin.username}"
password= "$ref{super_admin.password}"
[apim.jwt]
enable = true
encoding = "base64"
claim_dialect = "http://wso2.org/claims"
header = "X-JWT-Assertion"
signing_algorithm = "SHA256withRSA"
[apim.sync_runtime_artifacts.gateway]
gateway_labels = ["Default"]
[apim.throttling]
service_url = "https://apimbo.mysite.com:9443/services/"
username = "$ref{super_admin.username}"
password = "$ref{super_admin.password}"
enable_unlimited_tier = true
enable_header_based_throttling = false
enable_jwt_claim_based_throttling = false
enable_query_param_based_throttling = false
throttle_decision_endpoints = ["tcp://apimbo.mysite.com:5672","tcp://apimbo.mysite.com:5672"]
[[apim.throttling.url_group]]
traffic_manager_urls=["tcp://apimbo.mysite.com:9611"]
traffic_manager_auth_urls=["ssl://apimbo.mysite.com:9711"]
[apim.analytics]
enable = false
config_endpoint = "https://analytics-event-auth.choreo.dev/auth/v1"
auth_token = "ffffffff-ffff-ffff-ffff-ffffffffffff"
[apim.cache.jwt_claim]
enable = true
expiry_time = 900
[apim.cache.gateway_token]
enable = true
expiry_time = 15
[apim.cache.resource]
enable = true
[apim.cache.km_token]
enable = true
[apim.oauth_config]
remove_outbound_auth_header = true
auth_header = "Authorization"
[apim.devportal]
api_key_alias = "apim"
[apim.cors]
allow_origins = "*"
allow_methods = ["GET","PUT","POST","DELETE","PATCH","OPTIONS"]
allow_headers = ["authorization","Access-Control-Allow-Origin","Content-Type","SOAPAction", "apikey"]
allow_credentials = false
[transport.http]
properties.port = 9763
properties.proxyPort = 80
[transport.https]
properties.port = 9443
properties.proxyPort = 443
[passthru_http]
worker_thread_keepalive_sec = 60
worker_pool_queue_length = -1
core_worker_pool_size = 400
max_worker_pool_size = 400
io_buffer_size = 16384
http.socket.timeout = 180000
[message_builder]
json = "org.apache.synapse.commons.json.JsonStreamBuilder"
2 answers
solution1
0 2023-01-16 16:45:05
Add the following configs to your control plane deployment.toml
If you have a single server.
[apim.key_manager]
service_url = "https://localhost:9443/services/"
username = "admin"
password = "admin"
If you have a HA setup
[apim.key_manager]
service_url = "https://CP_LB_HOST/services/"
username = "admin"
password = "admin"
Update
Add the following to change the JWKS endpoint.
[[apim.jwt.issuer]]
name = "https://localhost:9443/oauth2/token"
jwks.url = "https://CP_HOST:9443/oauth2/jwks"
solution2
0 2023-01-17 11:19:43
Ok I managed to configure the control-plane as the key manager, so my errors are finally gone.
Basically the gateway tried to claim /oauth2/jwks on the gateway side instead of calling it through the control-plane. Special thanks to you @ycr you helped me a lot.
Nothing worked on deployment.toml (it looks like a bug to me), but I succeeded using the admin panel:
- Go to https://apimbo.mysite.com/admin/dashboard
- Browse the Key Managers menu
- Click on "Resident Key Manager" to edit it
- Inside "Certificates", change https://localhost:9443/oauth2/jwks to https://apimbo.mysite.com:9443/oauth2/jwks
I restarted both gw and cp to make sure that the value was not overwritten after a restart, and it's not.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.