Cloud PubSub to App Engine memory Issue (and Should I move to DataFlow?)

Question

I currently run a "small" flask application on (Google Cloud's App Engine) that is used to integrate applications (it listens to webhooks and calls other APIs). The issue is that I consistently exceed the soft memory limit after 35 - 45 requests.

Memory footprint of the combined instances:

Since I intend to increase the load on this system by orders of magnitude this worries me.

There seems to be three possible solutions to me but I don't know where to start:

1. Switch to DataFlow: I already use Pub/Sub between two App Engine instances to add higher consistency, but maybe App Engine is the wrong platform or this kind of platform.
2. Fix memory leak: The issue here could be a possible memory leak, but I can't find the right tools to analyse the memory usage on the App Engine platform (on my local machine usage of the Python process hovers around 51MB)
3. Divide the system into multiple microservices to decrease the footprint per instance. (Maintaining the code base will probably be harder though).

Any advice or experience is very welcome.

Answer 1

If your case is indeed a memory leak, you need to verify your code as this will consistently lead to your application crashing. There are other posts, like this one that discusses tools and strategies to address memory issues in python code.

You could potentially use Dataflow or Cloud Functions in your project. If you provide more details about the nature of your use case in a separate question, one could evaluate if these options could be a better alternative to your current App Engine approach.

Finally, dividing your application into multiple services is likely the best long term solution to your issue, as it will make it easier to find any memory leak, to control costs and to generally maintain your application.

There are few pages in App Engine's documentation that discuss best practices in microservice design using App Engine [1] [2] [3]. Proper microservice-based applications permit clear logging and monitoring as well as an increase in application reliability and scalability , among other benefits [1]. These benefits that I mentioned here are important to you. Following the layout as discussed in [4], you can scale your services individually and independently of each other. If you believe that one of your services is more resource-demanding, you can adjust the scaling parameters in order to provide optimal performance for that service. For example, you can manage the numbers of instances that are fired during operations.

You can use the app elements max_concurrent_requests and target_throughput_utilization which you can define in your App Engine's configuration file, the app.yaml file. See [5]. To clarify, you want to reduce your max_concurrent_requests in your case.

Please note that, as discussed in previous comments, this road could lead to higher costs. If you are using the free tier then you will need to check [4] for available resources to you in this tier.

Regarding the issue of your instances running out of memory, if you find out that it is not due to a memory leak, then another solution would be to use a different instance_class which means that you can instantiate App Engines with higher compute resources (also higher costs). Please see [5] and [6].

[1] https://cloud.google.com/appengine/docs/standard/python/microservices-on-app-engine
[2] https://cloud.google.com/appengine/docs/standard/python/designing-microservice-api
[3] https://cloud.google.com/appengine/docs/standard/python/microservice-performance
[4] https://cloud.google.com/appengine/docs/standard/python/an-overview-of-app-engine
[5] https://cloud.google.com/appengine/docs/standard/python/config/appref
[6] https://cloud.google.com/appengine/docs/standard/#instance_classes