管理高流量连接应用程序的正确方法

Question

Introduction

I am working on a GPS Listener, this is a service build on twisted python, this app receive at least 100 connections from gps devices, and it is working without issues, each GPS send data each 5 seconds, containing positions. ( the next week must be at least 200 gps devices connected )

Database

I am using a unique postgresql connection, this connection is shared between all gps devices connected for save and store information, postgresql is using pgbouncer as pooler

Server

I am using a small pc as server, and I need to find a way to have a high availability application with out loosing data

Problem

According with my high traffic on my app, I am having issues with memory data after 30 minutes start to appear as no saved, however queries are being executed on postgres ( I have checked that on last activity )

Fake Solution

I have amke a script that restart my app, postgres ang pgbouncer, however this is a wrong solution, because each time that I restart my app, gps get disconnected, and must to reconnected again

Posible Solution

I am thinking on a high availability solution based on a Data Layer, where each time when database have to be restarted or something happened, a txt file store data from gps devices.

For get it, I am thing on a no unique connection, I am thinking on a simple connection each time one data must be saved, and then test database, like a pooler, and then if database connection is wrong, the txt file store it, until database is ok again, and the other process read txt file and send info to database

Question

Since I am thinking on a app data pooler and a single connection each time when this data must be saved for try to no lost data, I want to know

Is ok making single connection each time that data is saved for this kind of app, knowing that connections will be done more than 100 times each 5 seconds?

As I said, my question is too simple, which one is the right way on working with db connections on a high traffic app? single connections per query or shared unique connection for all app.

The reason on looking this single question, is looking for the right way on working with db connections considering memory resources.

I am not looking for solve postgresql issues or performance , just to know the right way on working with this kind of applications . And that is the reason on give as much of possible about my application

Note One more thing,I have seen one vote to close this question, that is related to no clear question, when the question is titled with the word "question" and was marked on italic, now I have marked as gray for notice people that dont read the word "question"

Thanks a lot

Answer 1

Databases do not just lose data willy-nilly. Not losing data is pretty much number one in their job description. If it seems to be losing data, you must be misusing transactions in your application. Figure out what you are doing wrong and fix it.

Making and breaking a connection between your app and pgbouncer for each transaction is not good for performance, but is not terrible either; and if that is what helps you fix your transaction boundaries then do that.