Flask / Heroku / Redis / RQ内部服务器错误

Question

I have a recommendation site. Everything was working dandy, until at points when the site was under a decent amount of traffic, the recommendations would take longer than 30 seconds (Heroku's limit) and time-out, throwing a 500 error. I realize this is a very long time for a http request.

So, I read up online and implemented RQ with Redis. I got that to work, but after some testing, it will still throw the Internal Server Error, even though the requests are going through a queue.

I'm really just lacking knowledge here and I have no idea what to do. I think I'm missing the whole idea of rq and redis I guess? Here's some of my code if it helps, but I'm hoping for more of just guidance of where to go from here to fix this error.

worker.py

import os
import redis
from rq import Worker, Queue, Connection

listen = ['high', 'default', 'low']

redis_url = os.getenv('REDISTOGO_URL',
                      'redis://redistogo:sampleurl:portNo/')
if not redis_url:
    raise RuntimeError('Set up Redis to go first.')

conn = redis.from_url(redis_url)

if __name__=='__main__':
    with Connection(conn):
        worker = Worker(map(Queue, listen))
        worker.work()

part of my views.py

q = Queue(connection=conn)

@app.route('/')
def home():
    form = ArtistsForm()
    return render_template('home.html', form=form)


@app.route('/results', methods=['POST'])
def results():
    form = ArtistsForm()
    error = None
    try:
        if request.method == 'POST' and form.validate():
            table = 'Artists'
            artists = []
            for value in form.data.items():
                if (value[1] is not ''):
                    artists.append(value[1])

            results = q.enqueue_call(func=getArtists, args=(table, *artists))
            while results.result is None:
                time.sleep(1)
            results = results.result.values.tolist()

            return render_template('results.html', results=results)
        else:
            error = "Please be sure to enter 5 artists with correct spelling" \
                    " and punctuation"

    except pylast.WSError:
        return render_template('error.html')
    return render_template('home.html', form=form, error=error)

Any guidance is appreciated

Answer 1

You can always try dividing the work between a web dyno to simply acknowledge the request, then have worker dynos doing the heavy lifting.

Kafka or something similar could be used to accomplish this.