The execution time of parallel queries in Redshift increases drastically with the no of queries

Question

I am new to Amazon Redshift. I have preloaded some data to a table and I am testing the query latency in Redshift. The fact that I have observed is that with parallel queries the execution time is going up highly as the no of queries fired parallely increases. The Redshift documentation points to the use of sort key and I have tried all that and the situation still remains the same. Kindly help me out in improving the parallel query execution time in Redshift.

Answer 1

It sounds like you are just issuing more queries and not changing the queue definitions, right?

Redshift uses a queuing system to manage the influx of queries called WLM (WorkLoad Manager). The WLM is configurable and limits the number of queries that are actually executed in parallel vs. held waiting in the queues. So the first question I'd want to look at is how much of the observed latency is due to "queue wait" time.

Now just increasing how many queries can execute in parallel is not going to fix the latency in many cases. Redshift can typically execute between 1 and 2 dozen queries at a time depending on the complexity of your queries. Allowing too many queries to execute at the same time can drastically reduce the total performance of the cluster. Most people have a mix of large and small queries and properly configuring the WLM can improve query latency by allowing simple and high priority queries to be executed quickly while big batch queries queue up.