Rails formatting logs to use with aws-logs and CloudWatch

Question

AWS has this very cool log collection tool using aws-logs

However, I do not understand how I can format my log / configure the tool to be smarter and regroup the same error message. Right now AWS shows one message per line (because every line is timestamped)

My current log configuration indeed captures one new log entry per message. How can I go around it

[rails/production.log]
file = /var/www/xxx/shared/log/production.log
log_group_name = /rails/production.log
log_stream_name = {instance_id}
time_zone = LOCAL
datetime_format = %Y-%m-%dT%H:%M:%S

Answer 1

I actually partly solved the problem using lograge and JSON output which is parsed correctly by Amazon and lets your regroup most requests correctly.

However I still have some problems with errors, which are not outputted the same way, and still generate one line per call stack trace on awslogs

EDIT : We are now using a Rails API and regular exceptions thrown during JSON requests are rescued with a json:api error handler renderer. Furthermore, we are using Rollbar to log actual errors, so it becomes irrelevant to have the full error log

In our API::ApplicationController

# We don't want error reports for those errors
RESCUABLE_ERRORS = [
  ActionController::ParameterMissing,
  ActiveModel::ForbiddenAttributesError,
  StrongerParameters::InvalidParameter,
  Mongoid::Errors::Validations,
  Mongoid::Errors::DocumentNotFound
]

# Note that in tests, we want to actually do not want to rescue non-Runtime exceptions straight away because most likely this indicates a real bug that you should fix, but in production we want to rescue any error so the frontend does not get the default HTML response but a JSON:api error
rescue_from(Rails.env.test? ? RuntimeError : Exception) do |e|
  handle_exception(e)
  notify_exception(e, 'Rescued from API controller - Rendering JSONAPI Error')
end

rescue_from(*RESCUABLE_ERRORS) do |e|
  handle_exception(e)
end

In our controllers that inherit API::ApplicationController, we add as many lines of rescue_from depending whether we want to report the exception as an error ( notify_exception ) or just convert to a JSON payload ( handle_exception )

rescue_from(SPECIFIC_ERROR_CLASS) do |exception|
  handle_exception(exception) # will render a json:api error payload
  # notify_exception(exception) # Optional : ExceptionNotifier to broadcast the error to email/Rollbar, etc. if this error should not happen.
end