Node.js - 如何继续描述 Cloudwatch 日志查询以获取特定查询的已完成状态

Question

So for anyone familiar with CW Logs..this is not the typical DB query call that returns with results. You send an API call to start a query, that returns a Query ID. You then send a different API call to get the query results, with the "hope" that the query completed, and if it hasn't, you're out of luck. That's where I am right now.

I have a query that takes a bit of time and I'm "guessing" the way to handle this is to keep looping the DescribeQueries call until I find a match among that returned array of Completed queries, then go on with the rest of the code. I am unable to pull this off! Grrrr!!

I have tried While and Do..While loops that completely do NOT work. I tried setting the escape condition value when a match is found.. but it never gets set and the Lambda function times out.

function checkQueryStatus (logGroup, qID, callback) {

      var params = {
      logGroupName: logGroup, 
      maxResults: '3',
      status: 'Complete'
    };

let found = 0;

      do {
      cwlogs.describeQueries(params, function(err, data) {
        if (err) console.log(err, err.stack); // an error occurred
        else {
          // console.log(data.queries);           // successful response
            const qList = data.queries;
            if (qList.some(query => query.queryId === qID)) {  
              console.log('Query Done');
              callback(1);
              found = 1;
          } else {
            console.log('Query not done');
          }
        }
      });
      } while (found == 0); 
}

checkQueryStatus(logGroupName, queryID, function(qStatus) {
  console.log('Query Status: ', qStatus);

  if (qStatus == 1) { 
  console.log('Query Done');

  <do other code...>

How can I do it? I'm now looking into Promises to see what thats all about.. If the DescribeQueries find a match, I want to trigger the GetQueryResults API call.

Answer 1

I've run into a similar problem with AWS Athena. When I start a query, I get back the response that is has started but get no notification when it finishes. The best solution I came up with was to use setTimeout to check its status every 100ms or so and continue when the query is completed. Hope that helps.

Answer 2

Here is a generic function which you can use to await until the query finish and returns the query result.

async function queryCloudWatch(queryRequest: StartQueryRequest): Promise<GetQueryResultsResponse> {
    const queryResponse: StartQueryResponse = await cloudwatchLogs.startQuery(queryRequest).promise()

    if (!queryResponse.queryId) return {}

    let response: GetQueryResultsResponse | undefined = undefined

    while (!response || response.status === 'Running') {
        response = await cloudwatchLogs.getQueryResults({
            "queryId": queryResponse.queryId
        }).promise()
    }

    return response;
}

Answer 3

You have two options: query log Insights or query log groups
If you would like to query log Insights:

const AWS = require('aws-sdk');
AWS.config.setPromisesDependency(require('bluebird'));
AWS.config.update({region: 'us-west-2'});
const cloudWatchLogs = new AWS.CloudWatchLogs({apiVersion: '2014-03-28'});
exports.handler = async (event) => {

  // Cloudwatch Log Group name
  const logGroupName = '/aws/lambda/<Name of your Log Group>';
  const timestamp = new Date();

  const params = {
    endTime: timestamp.getTime(),
    queryString: `fields @message, @timestamp
    | sort @timestamp desc
    | limit 10
    | filter @message like /(?i)("Error")/
    | stats count() by bin(1d)`, // Group by Day
    startTime: timestamp.setDate( timestamp.getDate() - 3 ), // Last 3 days
    logGroupName: logGroupName
  };
  
  // 1. Start the query. When we start a query, this returns a queryId for us to use on our next step.
  const data = await cloudwatchlogs.startQuery(params).promise();
  const { queryId } = data;
  console.debug('query id', queryId);

  while (true) {
    
    // 2. Send Insight query to CloudwatchLogs
    const insightData = await cloudwatchlogs.getQueryResults({ queryId })
        .promise();
    
    // 3. Check if it is available    
    if (Array.isArray(insightData.results) && insightData.status === 'Complete') {
      const insightResult = insightData.results;
      
      // Change this line to publish to SNS or send to Slack
      console.log(JSON.stringify(insightResult, null, 4))
      break;
    }
    
    // 4. Otherwise, Wait for 100 ms for insight api result
    await new Promise((resolve, reject) => setTimeout(resolve, 100));
  } 

  return 'ok';
}

To query log groups use filterLogEvents API:

const AWS = require('aws-sdk');
AWS.config.setPromisesDependency(require('bluebird'));
AWS.config.update({region: 'us-west-2'});
const cloudWatchLogs = new AWS.CloudWatchLogs({apiVersion: '2014-03-28'});

const timestamp = new Date();
const endtTime = timestamp.getTime();
const params = {
    endTime: endtTime,
    filterPattern: `"${stringToSearch}"`,
    startTime: new Date (endtTime - 5 * 60 * 60* 24 * 1000).getTime(), // Last 5 days
    logGroupName: 'myLogGroup',
    limit : 10
};

const events = await cloudWatchLogs.filterLogEvents(params).promise();
console.log(`successfully queryCloudWatchLogs ${stringToSearch} results: ${JSON.stringify(events)}`);
const results = events.events.map(e => e.message)
console.log(`successfully queryCloudWatchLogs ${stringToSearch} results (${results.length}): ${JSON.stringify(results)}`);