Aws cloudwatch logs getQueryResults returns empty when tried with boto3

Question

Using boto3 of aws, I am trying to run start query and get the results using query id. but it didnt work as expected in python script. It returns the expected json output for start_query and able to fetch the queryID. But if i try to fetch the query results using queryID, it returns empty json.

<code>
import boto3
client = boto3.client('logs')
executeQuery = client.start_query(
       logGroupName='LOGGROUPNAME',
       startTime=STARTDATE,
       endTime=ENDDATE,
       queryString='fields status',
       limit=10000
   )
getQueryId=executeQuery.get('queryId')
getQueryResults = client.get_query_results(
    queryId=getQueryId
  )
</code>

it returns the reponse of get_query_results as {'results': [], 'statistics': {'recordsMatched': 0.0, 'recordsScanned': 0.0, 'bytesScanned': 0.0}, 'status': 'Running',

But if i try using aws cli with the queryID generated from script, it returns json output as expected.

Anyone could able to tell why it didnt work from boto3 python script and worked in cli?

Thank you.

Answer 1

The other issue you might be encountering is that the syntax for the queryString in the API is different from a query you would type into the CloudWatch console.

Console query syntax example:

{ $.foo = "bar" && $.baz > 0 }

API syntax for same query:

filter foo = "bar" and baz > 0

Source: careful reading and extrapolation from the official documentation plus some trial-and-error.

My logs are in JSON format. YMMV.

Answer 2

The query status is Running in your example. Its not in Complete status yet.

Running queries is not instantaneous . Have to wait a bit for query to complete, before you can get results.

You can use describe_queries to check if your query has completed or not. You can also check if logs service has dedicated waiters in boto3 for the results. They would save you from pulling describe_queries API in a loop waiting till your queries finish.

When you do this in CLI, probably there is more time before you start the query, and query results using CLI.

Answer 3

Not sure if this problem is resolved. I was facing the same issue with AWS Java SDK. But when i terminate the thread performing the executeQuery query and perform the get_query_results using a new thread and the old queryId. It seems to be working fine.

Answer 4

Adding sleep will work here. If the query is exceeding the Sleep time then again it will show as Running status. You can write a Loop where you can check the status Completed, if the status is Running you can run Sleep again for some second and retry. You can give some retry count here.

Sample Pseudocode:

function for sleep; (let's say SleepFunc())

Loop till retry count
check if the status is Completed;
If yes
break;
else call
SleepFunc();