Flask多线程

Question

I'd like to call generate_async_audio_service from a view and have it asynchronously generate audio files for the list of words using a threading pool and then commit them to a database.

I keep running into an error that I'm working out of the application context even though I'm creating a new polly and s3 instance each time.

How can I generate/upload multiple audio files at once?

from flask import current_app, 
from multiprocessing.pool import ThreadPool
from Server.database import db
import boto3
import io
import uuid


def upload_audio_file_to_s3(file):
   app = current_app._get_current_object()
   with app.app_context():
      s3 = boto3.client(service_name='s3',
               aws_access_key_id=app.config.get('BOTO3_ACCESS_KEY'),
               aws_secret_access_key=app.config.get('BOTO3_SECRET_KEY'))
      extension = file.filename.rsplit('.', 1)[1].lower()
      file.filename = f"{uuid.uuid4().hex}.{extension}"
      s3.upload_fileobj(file,
         app.config.get('S3_BUCKET'),
         f"{app.config.get('UPLOADED_AUDIO_FOLDER')}/{file.filename}",
         ExtraArgs={"ACL": 'public-read', "ContentType": file.content_type})
      return file.filename

def generate_polly(voice_id, text):
   app = current_app._get_current_object()
   with app.app_context():
      polly_client = boto3.Session(
         aws_access_key_id=app.config.get('BOTO3_ACCESS_KEY'),                   
         aws_secret_access_key=app.config.get('BOTO3_SECRET_KEY'),
         region_name=app.config.get('AWS_REGION')).client('polly')
      response = polly_client.synthesize_speech(VoiceId=voice_id,
                     OutputFormat='mp3', Text=text)
      return response['AudioStream'].read()


def generate_polly_from_term(vocab_term, gender='m'):
   app = current_app._get_current_object()
   with app.app_context():
      audio = generate_polly('Celine', vocab_term.term)
      file = io.BytesIO(audio)
      file.filename = 'temp.mp3'
      file.content_type = 'mp3'
      return vocab_term.id, upload_audio_file_to_s3(file)

def generate_async_audio_service(terms):
   pool = ThreadPool(processes=12)
   results = pool.map(generate_polly_from_term, terms)
   # do something w/ results

Answer 1

This is not necessarily a fleshed-out answer, but rather than putting things into comments I'll put it here.

Celery is a task manager for python. The reason you would want to use this is if you have tasks pinging Flask, but they take longer to finish than the interval of tasks coming in, then certain tasks will be blocked and you won't get all of your results. To fix this, you hand it to another process. This goes like so:

1) Client sends a request to Flask to process audio files

2) The files land in Flask to be processed, Flask will send an asyncronous task to Celery.

3) Celery is notified of the task and stores its state in some sort of messaging system (RabbitMQ and Redis are the canonical examples)

4) Flask is now unburdened from that task and can receive more

5) Celery finishes the task, including the upload to your database

Celery and Flask are then two separate python processes communicating with one another. That should satisfy your multithreaded approach. You can also retrieve the state from a task through Flask if you want the client to verify that the task was/was not completed. The route in your Flask app.py would look like:

@app.route('/my-route', methods=['POST'])
def process_audio():
    # Get your files and save to common temp storage
    save_my_files(target_dir, files)

    response = celery_app.send_tast('celery_worker.files', args=[target_dir])
    return jsonify({'task_id': response.task_id})

Where celery_app comes from another module worker.py :

import os
from celery import Celery

env = os.environ

# This is for a rabbitMQ backend
CELERY_BROKER_URL = env.get('CELERY_BROKER_URL', 'amqp://0.0.0.0:5672/0')
CELERY_RESULT_BACKEND = env.get('CELERY_RESULT_BACKEND', 'rpc://')

celery_app = Celery('tasks', broker=CELERY_BROKER_URL, backend=CELERY_RESULT_BACKEND)

Then, your celery process would have a worker configured something like:

from celery import Celery
from celery.signals import after_task_publish

env = os.environ
CELERY_BROKER_URL = env.get('CELERY_BROKER_URL')
CELERY_RESULT_BACKEND = env.get('CELERY_RESULT_BACKEND', 'rpc://')

# Set celery_app with name 'tasks' using the above broker and backend
celery_app = Celery('tasks', broker=CELERY_BROKER_URL, backend=CELERY_RESULT_BACKEND)

@celery_app.task(name='celery_worker.files')
def async_files(path):
    # Get file from path
    # Process
    # Upload to database
    # This is just if you want to return an actual result, you can fill this in with whatever
    return {'task_state': "FINISHED"}

This is relatively basic, but could serve as a starting point. I will say that some of Celery's behavior and setup is not always the most intuitive, but this will leave your flask app available to whoever wants to send files to it without blocking anything else.

Hopefully that's somewhat helpful