Unable to create oauth keys using wso2 apim 3.1.0 and IS 5.10.0

Question

I am using wso2 APIM 3.1.0 with Wso2 IS 5.10.0 as key manager. This error happens when I create the API in the API publisher portal and publish it. Using API developer portal when I try to subscribe to the API created I am getting the below exception at the generate keys step when I click on the Next button as shown in the below image. Also on the Identity server side I see the logs. I believe these logs confirm that everything is fine on the IS side

[2020-05-31 09:29:30,166] [d407196c-7d23-45b4-b5f7-fc6136106c4e]  INFO 
{org.wso2.carbon.identity.oauth.config.OAuthServerConfiguration} - An 
instance of org.wso2.carbon.apimgt.keymgt.issuers.APIMTokenIssuer is created 
for Identity OAuth token generation.
[2020-05-31 09:29:30,221] [d407196c-7d23-45b4-b5f7-fc6136106c4e]  INFO 
{org.opensaml.core.config.InitializationService} - Initializing OpenSAML 
using the Java Services API

I followed this link for configuring wso2 Is as key manager in API manager https://apim.docs.wso2.com/en/latest/install-and-setup/setup/distributed-deployment/configuring-wso2-identity-server-as-a-key-manager/#step-4-configure-the-wso2-is-as-key-manager-with-wso2-api-manager Is this bug in API manager?

ERROR {org.wso2.carbon.apimgt.rest.api.store.v1.impl.ApplicationsApiServiceImpl} - Error while generating keys for application dfe28a41-4ff5-48eb-bf91-8d2125a344b9 org.wso2.carbon.apimgt.api.APIManagementException: org.wso2.carbon.apimgt.impl.workflow.WorkflowException: Error occurred while executing SubscriberKeyMgtClient.
    at org.wso2.carbon.apimgt.impl.APIConsumerImpl.requestApprovalForApplicationRegistration_aroundBody144(APIConsumerImpl.java:3907)
    at org.wso2.carbon.apimgt.impl.APIConsumerImpl.requestApprovalForApplicationRegistration(APIConsumerImpl.java:3745)
    at org.wso2.carbon.apimgt.rest.api.store.v1.impl.ApplicationsApiServiceImpl.applicationsApplicationIdGenerateKeysPost(ApplicationsApiServiceImpl.java:505)
    at org.wso2.carbon.apimgt.rest.api.store.v1.ApplicationsApi.applicationsApplicationIdGenerateKeysPost(ApplicationsApi.java:125)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)
    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
    at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:193)
    at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:103)
    at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
    at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
    at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
    at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
    at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
    at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:216)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:220)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:276)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:86)
    at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:110)
    at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:75)
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
    at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:49)
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
    at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:145)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:688)
    at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
    at org.wso2.carbon.tomcat.ext.valves.RequestCorrelationIdValve.invoke(RequestCorrelationIdValve.java:119)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1639)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.wso2.carbon.apimgt.impl.workflow.WorkflowException: Error occurred while executing SubscriberKeyMgtClient.
    at org.wso2.carbon.apimgt.impl.workflow.ApplicationRegistrationSimpleWorkflowExecutor.complete_aroundBody2(ApplicationRegistrationSimpleWorkflowExecutor.java:82)
    at org.wso2.carbon.apimgt.impl.workflow.ApplicationRegistrationSimpleWorkflowExecutor.complete(ApplicationRegistrationSimpleWorkflowExecutor.java:66)
    at org.wso2.carbon.apimgt.impl.workflow.ApplicationRegistrationSimpleWorkflowExecutor.execute_aroundBody0(ApplicationRegistrationSimpleWorkflowExecutor.java:54)
    at org.wso2.carbon.apimgt.impl.workflow.ApplicationRegistrationSimpleWorkflowExecutor.execute(ApplicationRegistrationSimpleWorkflowExecutor.java:47)
    at org.wso2.carbon.apimgt.impl.APIConsumerImpl.requestApprovalForApplicationRegistration_aroundBody144(APIConsumerImpl.java:3876)
    ... 55 more
Caused by: org.wso2.carbon.apimgt.api.APIManagementException: Error occurred while executing SubscriberKeyMgtClient.
    at org.wso2.carbon.apimgt.impl.utils.APIUtil.handleException_aroundBody78(APIUtil.java:1932)
    at org.wso2.carbon.apimgt.impl.utils.APIUtil.handleException(APIUtil.java:1930)
    at org.wso2.carbon.apimgt.impl.workflow.AbstractApplicationRegistrationWorkflowExecutor.dogenerateKeysForApplication_aroundBody8(AbstractApplicationRegistrationWorkflowExecutor.java:170)
    at org.wso2.carbon.apimgt.impl.workflow.AbstractApplicationRegistrationWorkflowExecutor.dogenerateKeysForApplication(AbstractApplicationRegistrationWorkflowExecutor.java:123)
    at org.wso2.carbon.apimgt.impl.workflow.AbstractApplicationRegistrationWorkflowExecutor.generateKeysForApplication_aroundBody6(AbstractApplicationRegistrationWorkflowExecutor.java:119)
    at org.wso2.carbon.apimgt.impl.workflow.AbstractApplicationRegistrationWorkflowExecutor.generateKeysForApplication(AbstractApplicationRegistrationWorkflowExecutor.java:116)
    at org.wso2.carbon.apimgt.impl.workflow.ApplicationRegistrationSimpleWorkflowExecutor.complete_aroundBody2(ApplicationRegistrationSimpleWorkflowExecutor.java:78)
    ... 59 more
Caused by: org.wso2.carbon.apimgt.api.APIManagementException: Error occurred while calling token endpoint: HTTP error code : 400
    at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.getNewApplicationAccessToken_aroundBody10(AMDefaultKeyManagerImpl.java:389)
    at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.getNewApplicationAccessToken(AMDefaultKeyManagerImpl.java:304)
    at org.wso2.carbon.apimgt.impl.workflow.AbstractApplicationRegistrationWorkflowExecutor.dogenerateKeysForApplication_aroundBody8(AbstractApplicationRegistrationWorkflowExecutor.java:159)
    ... 63 more

Answer 1

I assume you have 1 APIM node and 1 keymanager node.

When you click generate keys 2 things happen. 1. Creating an OAuth app. 2. Generate an access token. In this case, what has failed is the step 2. The token gen call has received 400 (Bad request) response. There must be another error log related to this, in one of the nodes (most probably in the KM node).

Do you have the following configured by any chance?

[apim.oauth_config]
revoke_endpoint = ??