[英]AWS Elastic Beanstalk logging with python (django)
How do you manage your application logs in AWS elastic beanstalk?您如何在 AWS elastic beanstalk 中管理您的应用程序日志? Which file you write you application logs to?
您将应用程序日志写入哪个文件?
I'm using the following Logging configuration in my development environment but this doesn't work when I deploy in AWS.我在我的开发环境中使用以下日志记录配置,但是当我在 AWS 中部署时这不起作用。
DEBUG_LOG_DIR = BASE_DIR + "/django_debug.log"
LOGGING = {
'version': 1,
'disable_existing_loggers': True,
# How to format the output
'formatters': {
'standard': {
'format' : "[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s",
'datefmt' : "%d/%b/%Y %H:%M:%S"
},
},
# Log handlers (where to go)
'handlers': {
'null': {
'level':'DEBUG',
'class':'django.utils.log.NullHandler',
},
'log_file': {
'level':'DEBUG',
'class':'logging.handlers.RotatingFileHandler',
'filename': DEBUG_LOG_DIR,
'maxBytes': 50000,
'backupCount': 2,
'formatter': 'standard',
},
'console':{
'level':'INFO',
'class':'logging.StreamHandler',
'formatter': 'standard'
},
'mail_admins': {
'level': 'ERROR',
'class': 'django.utils.log.AdminEmailHandler',
},
},
# Loggers (where does the log come from)
'loggers': {
'repackager': {
'handlers': ['console', 'log_file'],
'level': 'DEBUG',
'propagate': True,
},
'django': {
'handlers':['console'],
'propagate': True,
'level':'WARN',
},
'django.db.backends': {
'handlers': ['console', 'log_file'],
'level': 'WARN',
'propagate': False,
},
'': {
'handlers': ['console', 'log_file'],
'level': 'DEBUG',
},
}
}
I had a similar issue but on Elastic Beanstalk, so I created a config file (eg applogs.config) in .ebextensions
folder of the app.我有一个类似的问题,但在 Elastic Beanstalk 上,所以我在应用程序的
.ebextensions
文件夹中创建了一个配置文件(例如 applogs.config)。 This creates the app-logs folder if it is not there already and sets the file permissions and owner so that the app can write its logs there.这将创建 app-logs 文件夹(如果尚不存在)并设置文件权限和所有者,以便应用程序可以在那里写入其日志。
commands:
00_create_dir:
command: mkdir -p /var/log/app-logs
01_change_permissions:
command: chmod g+s /var/log/app-logs
02_change_owner:
command: chown wsgi:wsgi /var/log/app-logs
Finally, in your Django settings:最后,在您的 Django 设置中:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'file': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': '/var/log/app-logs/django.log',
},
},
'loggers': {
'django': {
'handlers': ['file'],
'level': 'DEBUG',
'propagate': True,
},
},
}
Aditionally, if you want your log to be accessible from beanstalk logs using the web, add this to your file in .ebextensions此外,如果您希望使用网络从 beanstalk 日志访问您的日志,请将其添加到 .ebextensions 中的文件中
files:
"/opt/elasticbeanstalk/tasks/taillogs.d/django.conf":
mode: "000755"
owner: root
group: root
content: |
/var/log/app-logs/django.log
Ok, I figured out a way to do it.好的,我想出了一个方法来做到这一点。
First I connected via ssh to ec2 machine, then I create a folder in /var/log called app_logs with root user:首先我通过 ssh 连接到 ec2 机器,然后我在 /var/log 中创建一个名为 app_logs 的文件夹,root 用户:
mkdir /var/log/app_logs
After that I did the follow:之后我做了以下事情:
cd /var/log/
chmod g+s app_logs/
setfacl -d -m g::rw app_logs/
chown wsgi:wsgi app_logs/
That ensures that all the files created in this folder will have wsgi as owner and will be writable for the group that the file belongs.这确保在此文件夹中创建的所有文件都将 wsgi 作为所有者,并且对于该文件所属的组来说是可写的。 I had to do that because I noticed that the log file created by django app had root as owner and owner group but the application runs through wsgi user.
我不得不这样做,因为我注意到 django 应用程序创建的日志文件以 root 为所有者和所有者组,但应用程序通过 wsgi 用户运行。
Finally I changed DEBUG_LOG_DIR to /var/log/app_logs/django_debug.log最后我将 DEBUG_LOG_DIR 更改为 /var/log/app_logs/django_debug.log
There's a simple way that doesn't require any beanstalk configuration.有一种不需要任何 beanstalk 配置的简单方法。
In your django settings under LOGGING set up a handler directed to the file '/opt/python/log/{log_file_name}' .在LOGGING下的 Django 设置中,设置一个指向文件'/opt/python/log/{log_file_name}' 的处理程序。 The logs can then be accessed via the beanstalk environment menu under "Logs".
然后可以通过“日志”下的 beanstalk 环境菜单访问日志。
LOGGING = {
...,
'handlers': {
'logfile': {
'level': 'DEBUG',
'class': 'logging.handlers.RotatingFileHandler',
'filename': '/opt/python/log/{log_file_name}',
},
},
'loggers': {
'debugger': {
'level': 'DEBUG',
'handlers': ['logfile'],
'propagate': False,
},
}
This location is stated in the documentation here:此位置在此处的文档中说明:
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html#health-logs-instancelocation https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html#health-logs-instancelocation
-- edit -- - 编辑 -
This answer was originally written for Amazon Linux AMI , which has now reached end-of-life .这个答案最初被写了亚马逊的Linux AMI,目前已达到了结束生命。
To keeps things clear and separate, I wrote a new answer for Amazon Linux 2 .为了让事情清楚和分开,我为Amazon Linux 2写了一个新答案。
-- original answer -- ——原答案——
The simplest solution, in my opinion, is to log to the /opt/python/log
folder, as suggested by bewestphal and by @thierry-j (under steve-dunlop's answer ).在我看来,最简单的解决方案是登录到
/opt/python/log
文件夹,正如bewestphal和 @ thierry -j 所建议的(在steve-dunlop 的回答下)。
That is also what the official AWS EB Python Sample Application does: see python-v1.zip
source这也是官方 AWS EB Python示例应用程序所做的:参见
python-v1.zip
源
The log file will then be included automatically when you request logs from EB.当您从 EB 请求日志时,日志文件将自动包含在内。
This solution works out-of-the-box, without any modification of .ebextensions
, as long as you do not call django-admin.py
(or other django code) in your .ebextensions
.该解决方案的工作外的开箱,没有任何修饰
.ebextensions
,只要你不叫django-admin.py
在您(或其他Django的代码) .ebextensions
。
However, most apps do need to call django-admin.py
in .ebextensions
, eg in order to migrate
.但是,大多数应用程序确实需要在
django-admin.py
中调用.ebextensions
,例如为了migrate
。 That will cause the log file to be created prematurely , with root
owner and root
group.这将导致日志文件过早创建,具有
root
所有者和root
组。 This leads to permission errors, because the app runs as wsgi:wsgi
.这会导致权限错误,因为应用程序作为
wsgi:wsgi
运行。
This can be fixed by adding a new command, at the end of your container_commands
, to remove the "premature" log file, eg:这可以通过在您的
container_commands
末尾添加一个新命令来修复,以删除“过早”日志文件,例如:
container_commands:
...
9999_remove_root_log_file:
command: rm /opt/python/log/django.log
ignoreErrors: true
Details below.详情如下。
On a standard pre-configured Amazon Linux/Python platform, which uses Apache with mod_wsgi (see AWS platform docs ), the WSGIDaemonProcess
for the Django app runs as user wsgi
and group wsgi
(see /etc/httpd/conf.d/wsgi.conf
on your EC2 instance).在标准的预配置 Amazon Linux/Python 平台上,该平台使用 Apache 和 mod_wsgi(请参阅AWS 平台文档),Django 应用程序的
WSGIDaemonProcess
以用户wsgi
和组wsgi
(请参阅/etc/httpd/conf.d/wsgi.conf
在您的 EC2 实例上)。
In addition, the default folder permissions for the /opt/python/log
folder (on my standard EC2 instance) are: drwxrwxr-x 3 root wsgi 4096 Mar 5 14:08 .
此外,
/opt/python/log
文件夹(在我的标准 EC2 实例上)的默认文件夹权限是: drwxrwxr-x 3 root wsgi 4096 Mar 5 14:08 .
That is, the wsgi
group has all permissions ( rwx
), so the Django app (group wsgi
) can create log files there.也就是说,
wsgi
组拥有所有权限( rwx
),因此 Django 应用程序(组wsgi
)可以在那里创建日志文件。
This works, out-of-the-box, as demonstrated by the official AWS EB Python Sample Application ( python-v1.zip ).如官方 AWS EB Python 示例应用程序 ( python-v1.zip ) 所示,这是开箱即用的。
However, if you do anything in your .ebextensions
that causes the logging
file-handler to be initialized (like calling django-admin.py
), it will break.但是,如果您在
.ebextensions
中执行任何导致logging
文件处理程序被初始化的操作(例如调用django-admin.py
),它就会中断。
Here's how using django-admin.py
in .ebextensions
breaks your log file permissions:以下是在
.ebextensions
使用django-admin.py
破坏日志文件权限的方法:
Elastic Beanstalk container_commands
, in .ebextensions
, are executed as the root
user (see aws docs ). .ebextensions
Elastic Beanstalk container_commands
以root
用户身份执行(请参阅aws 文档)。
If you call django-admin.py
in any of the container_commands
, eg with collectstatic
or migrate
, that will cause your logging file handler(s) to be initialized.如果调用
django-admin.py
在任何的container_commands
,例如用collectstatic
或migrate
,这将导致你的日志文件处理程序(一个或多个)进行初始化。 If the specified log file does not exist yet, at that time, it will be created, with root
owner and root
group.如果指定的日志文件尚不存在,则届时将创建该文件,并具有
root
所有者和root
组。
That means the Django app, running as part of the wsgi
group, will not have permission to write to the log file (which belongs to the root
group).这意味着作为
wsgi
组的一部分运行的 Django 应用程序将无权写入日志文件(属于root
组)。
This leads to permission errors, eg: PermissionError: [Errno 13] Permission denied: '/opt/python/log/django.log'
这会导致权限错误,例如:
PermissionError: [Errno 13] Permission denied: '/opt/python/log/django.log'
The following snippet illustrates the permissions issue and shows how to fix it.以下代码段说明了权限问题并展示了如何修复它。
To reproduce the issue, add these container_commands
to a clean project (eg following the AWS EB Django tutorial ), configure Django settings.py
to log to /opt/python/log/django.log
, deploy to AWS EB, then check the eb-activity.log
to see the output of the container commands.要重现该问题,请将这些
container_commands
添加到一个干净的项目中(例如遵循AWS EB Django 教程),将 Django settings.py
配置为登录到/opt/python/log/django.log
,部署到 AWS EB,然后检查eb-activity.log
以查看容器命令的输出。
...
container_commands:
0100_show_current_user:
# show that we are running as root user
command: whoami
0200_try_to_remove_log_file:
# we need a clean slate for this example (make sure no log file owned by wsgi is present)
command: rm /opt/python/log/django.log
ignoreErrors: true
0300_break_log_file_permissions:
# this causes a new log file to be created, owned by root:root (instead of wsgi:wsgi)
command: django-admin.py
0400_show_log_file_permissions:
# prove that a log file was created by root, and show folder permissions
command: ls -la /opt/python/log
0500_fix_by_removing_log_file_after_all_django_admin_calls:
# remove the log file created by django-admin.py, to ensure that a new log file will
# be created when the server starts, owned by wsgi:wsgi
command: rm /opt/python/log/django.log
ignoreErrors: true
So, there is no need to mess with file/folder permissions explicitly.因此,没有必要明确地弄乱文件/文件夹权限。
If you don't call django code in .ebextensions
, logging to /opt/python/log
works, out-of-the-box.如果您不在
.ebextensions
调用 django 代码,则登录到/opt/python/log
即可开箱即用。
If you do call django code in .ebextensions
, eg django-admin.py collectstatic
, simply remove the log file at the end of your container_commands
section.如果您确实在
.ebextensions
调用 django 代码,例如django-admin.py collectstatic
,只需删除container_commands
部分末尾的日志文件。
NOTE: If you want to log files to persist between deployments, only remove them if they are owned by root
.注意:如果要在部署之间保留日志文件,请仅在它们由
root
拥有时删除它们。
Here's a DRY example:这是一个 DRY 示例:
In .ebextensions
config:在
.ebextensions
配置中:
option_settings:
# create EB environment property for the log file path
aws:elasticbeanstalk:application:environment:
LOG_FILE_PATH: /opt/python/log/django.log
...
container_commands:
...
# django code called here, e.g. "django-admin.py collectstatic"
...
9999_remove_any_existing_django_log_files:
command: rm $LOG_FILE_PATH
ignoreErrors: true
and in settings.py
:并在
settings.py
:
...
# get log path from environment variable, with fallback for local development
log_file_path = os.getenv('LOG_FILE_PATH', 'local.log')
# use this as 'filename' for the file handler, as described in the other answers
...
As a beginner in terms of linux permissions, it took me some time to get it to work.作为 linux 权限方面的初学者,我花了一些时间才让它工作。 Summarising the above given answers the following finally worked for me:
总结以上给出的答案,以下最终对我有用:
logging.config日志配置文件
commands:
00_create_dir:
command: mkdir -p /var/log/app-logs
01_change_permissions:
command: chmod g+s /var/log/app-logs
02_change_default_owner:
command: setfacl -d -m g::rw /var/log/app-logs
03_change_owner:
command: chown wsgi:wsgi /var/log/app-logs
settings.py设置.py
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'file': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': '/var/log/app-logs/django.log',
},
},
'loggers': {
'django': {
'handlers': ['file'],
'level': 'DEBUG',
'propagate': True,
},
},
}
With this I can see the logs as a separate section using 'eb logs' or within the Beanstalk environment, section "logs".有了这个,我可以使用“eb logs”或在 Beanstalk 环境中的“日志”部分将日志视为一个单独的部分。
This answer is for Amazon Linux 2 only.此答案仅适用于Amazon Linux 2 。 For those who have not migrated yet , please see my old answer for Amazon Linux AMI .
对于那些谁没有迁移还没有,请参阅我的亚马逊的Linux AMI老答案。
The official AWS Python sample-application for Amazon linux 2 uses the /tmp
folder for logging. 适用于Amazon linux 2的官方 AWS Python 示例应用程序使用
/tmp
文件夹进行日志记录。
However, custom log files added to /tmp
are not included automatically when requesting logs from Elastic Beanstalk.但是,从 Elastic Beanstalk 请求日志时,不会自动包含添加到
/tmp
自定义日志文件。 To include custom log files, we need to create logging tasks in subfolders of /opt/elasticbeanstalk/tasks
on the EC2 instance.要包含自定义日志文件,我们需要在 EC2 实例上的
/opt/elasticbeanstalk/tasks
子文件夹中创建日志记录任务。 See instructions in the documentation.见指示的文件中。
The sample app ( source ) accomplishes this using .ebextensions
.示例应用程序( 源代码)使用
.ebextensions
完成此操作。 However, the AWS Linux 2 migration docs suggest we should use .platform
hooks instead:但是, AWS Linux 2 迁移文档建议我们应该使用
.platform
hooks:
We recommend using platform hooks to run custom code on your environment instances.
我们建议使用平台挂钩在您的环境实例上运行自定义代码。 You can still use commands and container commands in .ebextensions configuration files, but they aren't as easy to work with.
您仍然可以在 .ebextensions 配置文件中使用命令和容器命令,但它们并不那么容易使用。 For example, writing command scripts inside a YAML file can be cumbersome and difficult to test.
例如,在 YAML 文件中编写命令脚本可能很麻烦且难以测试。
This has the additional advantage that output from platform hooks is collected in a separate log file, viz.这具有额外的优势,即平台挂钩的输出收集在单独的日志文件中,即。
/var/log/eb-hooks.log
, which makes debugging a bit easier. /var/log/eb-hooks.log
,这使得调试更容易一些。
Log level and log path are defined in one place, as Elastic Beanstalk environment properties, eg in .ebextensions/options.config
:日志级别和日志路径在一处定义,作为 Elastic Beanstalk 环境属性,例如在
.ebextensions/options.config
:
option_settings:
aws:elasticbeanstalk:application:environment:
LOG_LEVEL: INFO
LOG_FILE_PATH: /tmp/django-app.log
...
The LOG_FILE_PATH
environment property can now be used in a platform hook to create logging tasks:现在可以在平台挂钩中使用
LOG_FILE_PATH
环境属性来创建日志记录任务:
.platform/hooks/postdeploy/00_create_logging_tasks.sh
#!/bin/bash
TASKS_DIR=/opt/elasticbeanstalk/tasks
# include all app log files in bundle logs (replaces ".log" by "*")
echo "${LOG_FILE_PATH//.log/*}" > "$TASKS_DIR/bundlelogs.d/01-app-log.conf"
# include current app log file in tail logs
echo $LOG_FILE_PATH > "$TASKS_DIR/taillogs.d/01-app-log.conf"
Note that the platform hooks require execution permission, eg chmod +x 00_create_logging_tasks.sh
.请注意,平台挂钩需要执行权限,例如
chmod +x 00_create_logging_tasks.sh
。 On windows you can use git
as described here .在 Windows 上,您可以按照此处所述使用
git
。
We also use the LOG_LEVEL
and LOG_FILE_PATH
environment properties in our Django settings.py
:我们还在 Django
settings.py
使用了LOG_LEVEL
和LOG_FILE_PATH
环境属性:
...
# basic logging with file rotation ()
log_level = os.getenv('LOG_LEVEL', 'INFO')
handlers = dict(file={'class': 'logging.handlers.TimedRotatingFileHandler',
'filename': os.getenv('LOG_FILE_PATH'),
'when': 'midnight',
'interval': 1,
'backupCount': 1,
'encoding': 'utf-8'})
loggers = dict(django=dict(level=log_level, handlers=['file']),
myapp=dict(level=log_level, handlers=['file']))
LOGGING = dict(version=1,
disable_existing_loggers=False,
handlers=handlers,
loggers=loggers)
...
Some notes:一些注意事项:
We normally specify custom formatters
as well, but I left those out for clarity.我们通常也指定自定义
formatters
,但为了清楚起见,我将它们省略了。
The application itself can now be found on the EC2 instance in /var/app/current
.现在可以在 EC2 实例的
/var/app/current
找到应用程序本身。 Also see extending EB Linux platforms for more details.另请参阅扩展 EB Linux 平台以获取更多详细信息。
The application now runs as webapp
with group webapp
.现在,应用程序运行的
webapp
与组webapp
。
We also use platform hooks to run Django's migrate
and collectstatic
commands, as described here .我们还利用平台钩运行Django的
migrate
和collectstatic
命令,描述在这里。 We have not seen any issues with log file permissions yet.我们还没有看到任何日志文件权限问题。
EDIT:编辑:
As pointed out by @hax0 in the comments, file permission issues may arise if you try to run manage.py
commands on the EC2 instance, using SSH, after deployment.正如@hax0 在评论中指出的那样,如果您在部署后尝试使用 SSH 在 EC2 实例上运行
manage.py
命令,则可能会出现文件权限问题。
For example, when using eb ssh
, you are logged in as ec2-user
, but the log file is owned by the webapp
user, and, by default, only the owner has write permission ( 644
).例如,在使用
eb ssh
,您以ec2-user
身份登录,但日志文件归webapp
用户所有,并且默认情况下,只有所有者拥有写入权限( 644
)。 Thus, when running python manage.py
as ec2-user
, you will get an error saying it cannot configure the log file handler because permission is denied.因此,当以
ec2-user
身份运行python manage.py
时,您将收到一条错误消息,指出它无法配置日志文件处理程序,因为权限被拒绝。
A quick & dirty workaround is to change file permissions temporarily , eg using一个快速而肮脏的解决方法是临时更改文件权限,例如使用
sudo chmod 646 /tmp/django-app.log
Another workaround would be to run manage.py
as the webapp
user, for example like so:另一种解决方法是以
webapp
用户身份运行manage.py
,例如:
sudo su - webapp <<'EOF'
source $(find /var/app/venv/*/bin/activate)
export $(/opt/elasticbeanstalk/bin/get-config --output YAML environment |
sed -r 's/: /=/' | xargs)
python3 /var/app/current/manage.py showmigrations
EOF
None of the mentioned solutions worked for me as I am running my server on Amazon Linux 2
.由于我在
Amazon Linux 2
上运行我的服务器,因此上述解决方案均不适合我。
The problem wasn't in the logging configuration itself but in the .log
file permissions.问题不在于日志配置本身,而在于
.log
文件权限。
This is my related logger.config
file in .ebextensions:这是我在 .ebextensions 中的相关
logger.config
文件:
commands:
01_create_log_file:
command: 'touch /var/log/django.log'
02_change_log_file_permissions:
command: 'sudo chmod ugo+rwx /var/log/django.log'
As you can see I had to handle the permission problem by simply giving all permissions to all users.如您所见,我不得不通过简单地将所有权限授予所有用户来处理权限问题。
This is my Django logger configuration:这是我的 Django 记录器配置:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'file': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': '/var/log/django.log',
},
},
'loggers': {
'django': {
'handlers': ['file'],
'level': 'DEBUG',
'propagate': True,
},
},
}
默认情况下,在 elasticbeanstalk 中,您可以在此处查看 django 错误日志。
/var/log/httpd/error_log
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.