Python postgreSQL 很慢

Question

I'm doing a for Loop of x that is a list of 30-50 URLS through 1 table to see if any of the URL matches --- ( it took more than 5 Seconds )

About this system :

• this is a free database from Heroku.com i'm using it as a test
• as it says on the code i'm using " postgreSQL
• I'm using flask_sqlalchemy
• i'm running this Local hosted so the front end and the backend is my laptop which is i5 1.9ghz and 8g rams.
• time stamp difference is around 5 Sec just to check those 50 entries with 1 user online locally

python the code.

from flask_sqlalchemy import SQLAlchemy 
app.config['SQLALCHEMY_DATABASE_URI'] = "postgres://***" 
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = "True" 
db = SQLAlchemy(app,session_options={"autoflush": False}) 

class test1(db.Model): 
    id = db.Column(db.Integer, primary_key = True) 
    url = db.Column(db.String(255)) 
    code = db.Column(db.String(255)) 


    def __init__ (self,url,code): 
        self.url = url 
        self.code = code 

x = [url1,url2,url3,url4,....url50] 

t2 = datetime.time(datetime.now()) 
print("before : ",t2) 

for key in x: 

    y = test1.query.filter(test1.url == key).all() 
    if y: 
        print('----i got it') 
    else: 
        print('no') 

t3 = datetime.time(datetime.now()) 
print("after forloop : ",t3) 

I have asked many people in the chatrooms, many of them have their own different suspicions but no one can confirm why there is such a massive delay in a simple process.

Answer 1

You are creating separate queries for each test, which require a new SQL statement sent to the database server.

That's not that big an issue for a local database, living on the same server. But on Heroku, the database does not live on the same server. The database lives somewhere else in the cloud, and it depends on your specific configuration if those two locations are near to one another, or far apart.

Connecting to any Heroku-hosted Postgres database from Africa is going to be slowed down by latency. That latency is compounded by the number of queries you make.

You can test how much of a latency, with a simple command:

time psql postgresql://*** -c select 1;

The difference between the real and user components is roughly the round-trip time it takes for one very simple query to reach the database server and the response to come back to you.

If you are experiencing huge delays in queries when deployed on Heroku , then your database instance and web Dyno are probably located in two different regions. Relocate one or the other to live in the same region, at the very least. If you are connecting to the database from your own machine, then you'll also see large latencies.

On the command line, run the heroku info command to check what region your app is located in (look for the Region line). Or check the Settings tab on the Heroku dashboard for your app. See the Heroku Regions documentation for more information, and read up on migrating apps if you want to move it to another region.

If you want to move the database to the same region as your app, you'll have to re-create it, I believe. Create and download a full backup, drop the database add-on, re-create it, and it should be created in the same region as the app. Then restore the database from your backup.

Even if your app dyno and database are in the same region, you'll still see larger latencies than you'd see with Flask and Postgres installed on your local machine. How much latency you'll see also depends on the specific Heroku Postgres plan you pick . The Hobby tier (free) only allows for 20 connections and has no in-memory cache, so you'll need to re-use connections more often and you can't count on the server having cached table information that'd be useful for a series of related queries.

Answer 2

Instead of querying in the loop you can query one time using in_ and search in memory.

test1_objs = test1.query.filter(test1.url.in_(x)).all()
test1_urls = set()
for test1_obj in test1_objs:
    test1_urls.add(test1_obj.url)
for key in x: 
    if key in test1_urls: 
        print('----i got it') 
    else: 
        print('no') 

This will reduce the time but I think it is happening because of the network latency. If you use server on the cloud and try again the latency will be less.