WSO2 APIM doesnt update a published API in a cluster setup

Question

Setup: Two WSO2 APIM pointing to same MYSQL. These two WSO2 instances are behind a LB and publisher session is sticky. The API session is not sticky. I have published API "/public/1.0.0/abc" which previously pointed to "/api/app/v1/xyz". I updated the published API to now point to "/api/app/v1/abc".

Issue: Now after the update when the published API is accessed through curl, at times returns back expected result but other times it throws 403 error. It seems update in one instance did not propagate to another? Below steps to make it work means downtime and we are trying to avoid the downtime.

The only way to make this work is by doing the following:

1. Shutdown the wso2am app on the working instance (Instance1)
2. Update the API again on WSO2 publisher so Instance2 picks up the change
3. Start back the Instance1 wso2am app

WSO2 Instance1 logs where the call works:

==> /usr/lib64/wso2/wso2am/2.6.0/repository/logs/http_access_.log <==
- <private-subnet-ip> - - [04/Jun/2019:20:02:00 +0000] "GET /services/Version HTTP/1.1" - - "-" "ELB-HealthChecker/2.0"
<my-ip> <private-subnet-ip> - - [04/Jun/2019:20:02:22 +0000] "GET /public/1.0.0/abc HTTP/1.1" - - "-" "curl/7.54.0"
<my-ip> <private-subnet-ip> - - [04/Jun/2019:20:02:22 +0000] "GET /api/app/v1/abc HTTP/1.1" - - "-" "Synapse-PT-HttpComponents-NIO"
- <private-subnet-ip>  - [04/Jun/2019:20:02:00 +0000] "- - " 200 - "-" "-"
- <private-subnet-ip>  - [04/Jun/2019:20:02:22 +0000] "- - " 200 - "-" "-"
- <private-subnet-ip>  - [04/Jun/2019:20:02:22 +0000] "- - " 200 - "-" "-"

WSO2 Instance2 logs where the call doesnt works:

==> /usr/lib64/wso2/wso2am/2.6.0/repository/logs/wso2carbon.log <==
TID: [-1234] [] [2019-06-04 20:01:24,511]  WARN {org.apache.synapse.rest.API} -  Trying to access API : admin--PublicAPI on restricted transport chanel [https] {org.apache.synapse.rest.API}

==> /usr/lib64/wso2/wso2am/2.6.0/repository/logs/wso2-apigw-errors.log <==
2019-06-04 20:01:24,511 [-] [PassThroughMessageProcessor-123]  WARN API Trying to access API : admin--PublicAPI on restricted transport chanel [https]

==> /usr/lib64/wso2/wso2am/2.6.0/repository/logs/wso2carbon.log <==
TID: [-1234] [] [2019-06-04 20:01:24,511]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = /public/1.0.0/abc {org.apache.synapse.mediators.builtin.LogMediator}

==> /usr/lib64/wso2/wso2am/2.6.0/repository/logs/wso2-apigw-service.log <==
2019-06-04 20:01:24,511 [-] [PassThroughMessageProcessor-123]  INFO __SynapseService STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = /public/1.0.0/abc

Answer 1

This happens if you haven't shared the API artifacts between the two servers. Only one server has the updated artifact. When you publish an API, it creates a Synapse(API) artifact which stored in the file system of the server. The location is SERVER_HOME/repository/deployment/server/synapse/default/api location. If you are running more than one instance, you should have a mechanism to sync those artifacts between the servers. There are several options.

1. Use a shared file system for the nodes
2. Rsync option - you can keep always point to node 1 and publish the API. From node 1 you can rsync data to the node 2. In the api-manager.xml, APIGateway serverURL should be changed to node 1 in both nodes.