[英]“could not receive data from server: Connection timed out” or “connection not open” errors when connecting to PostGres DB on a separate AWS instance
I am using Ruby 1.9.3 in my app server that is running on an AWS EC2 instance. 我在我的应用服务器上使用Ruby 1.9.3,该服务器在AWS EC2实例上运行。 I have my Postgres DB running on a separate EC2 instance but both instances are in the same security group. 我在单独的EC2实例上运行Postgres DB,但两个实例都在同一个安全组中。 When m Ruby code connects to the DB, it uses the Sequel ORM gem ( http://sequel.rubyforge.org/ ) . 当Ruby代码连接到DB时,它使用Sequel ORM gem( http://sequel.rubyforge.org/ )。
Now, I have configured that Postgres 9.1.4 DB to be able to accept connections properly from the app server instance. 现在,我已将Postgres 9.1.4 DB配置为能够从应用服务器实例正确接受连接。
However, every now and then, I noticed on the app server's logs that it will have trouble connecting to the Postgres DB instance and I would see error messages like these : 但是,我时不时地在应用服务器的日志中注意到它将无法连接到Postgres数据库实例,我会看到如下错误消息:
PG::Error: could not receive data from server: Connection timed out
or 要么
PG::Error: connection not open
So I went to the Postgres DB instance and looked in /var/log/postgresql/postgresql-9.1-main.log and I see a bunch of messages like these: 所以我去了Postgres数据库实例并查看了/var/log/postgresql/postgresql-9.1-main.log,我看到了一堆这样的消息:
2012-11-07 08:15:17 UTC LOG: could not receive data from client: Connection timed out
2012-11-07 08:15:17 UTC LOG: unexpected EOF on client connection
I did searching around the web including stack overflow, and have made sure that my PostgreSQL does not have SSL enabled ( I have "ssl=off" inside my postgresql.conf file ) 我在网上搜索包括堆栈溢出,并确保我的PostgreSQL没有启用SSL(我的postgresql.conf文件中有“ssl = off”)
At this point, I am not exactly sure what exactly is the issue in the Postgres configuration. 在这一点上,我不确定Postgres配置中究竟是什么问题。 I do not what to mess with the maximum # of connections or maximum timeout values on my production server without a good proven reason. 如果没有充分证明的原因,我不会弄乱生产服务器上的最大连接数或最大超时值。
The app server can connect to the DB most of the time, and this problem only shows up intermittently. 应用服务器大多数时间都可以连接到数据库,此问题只会间歇性地出现。
On the Ruby-side, this is the error trace for "connection not open" when making the Postgres call : 在Ruby方面,这是在进行Postgres调用时“连接未打开”的错误跟踪:
PG::Error: connection not open
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:145:in `async_exec'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:145:in `block in execute_query'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/database/logging.rb:33:in `log_yield'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:145:in `execute_query'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:132:in `block in execute'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:111:in `check_disconnect_errors'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:132:in `execute'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:372:in `_execute'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:234:in `block (2 levels) in execute'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:379:in `check_database_errors'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:234:in `block in execute'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/database/connecting.rb:229:in `block in synchronize'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/connection_pool/threaded.rb:105:in `hold'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/database/connecting.rb:229:in `synchronize'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:234:in `execute'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/dataset/actions.rb:744:in `execute'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:483:in `fetch_rows'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/model/base.rb:785:in `primary_key_lookup'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/model/base.rb:124:in `[]'
Similarly, this is the trace for "could not receive data from server" : 同样,这是“无法从服务器接收数据”的跟踪:
PG::Error: could not receive data from server: Connection timed out
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:124:in `block'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:124:in `ensure in check_disconnect_errors'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:124:in `check_disconnect_errors'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:132:in `execute'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:372:in `_execute'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:234:in `block (2 levels) in execute'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:379:in `check_database_errors'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:234:in `block in execute'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/database/connecting.rb:229:in `block in synchronize'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/connection_pool/threaded.rb:105:in `hold'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/database/connecting.rb:229:in `synchronize'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:234:in `execute'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/dataset/actions.rb:744:in `execute'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/adapters/postgres.rb:483:in `fetch_rows'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/model/base.rb:785:in `primary_key_lookup'
/var/lib/gems/1.9.1/gems/sequel-3.38.0/lib/sequel/model/base.rb:124:in `[]'
I observed that if I have both the App server and the Postgres DB running on the same instance, then there are no connectivity issues, at least not yet. 我观察到如果我同时在同一个实例上运行App服务器和Postgres DB,那么就没有连接问题,至少现在还没有。 Maybe the Postgres is less lenient towards non-local DB connections? 也许Postgres对非本地数据库连接不太宽容?
Please let me know what I might have missed, I appreciate it! 请让我知道我可能错过了什么,我很感激!
IS IS
The usual explanation for this would be a connectivity problem. 通常的解释是连接问题。
Alternately, if it's not connectivity, it could be a protocol synchronisation issue. 或者,如果它不是连接,则可能是协议同步问题。 It looks like the two ends could be trying to read from the socket, with neither trying to write. 看起来两端可能试图从套接字读取,既没有尝试写入。 So maybe the client is expecting the server to send a response, while the server is expecting the client to send data. 因此,客户端可能希望服务器发送响应,而服务器期望客户端发送数据。
This will be very hard to debug if it's intermittent and occasional, as you can't really just tcpdump it and analyse it. 如果它是间歇性的和偶尔的,这将很难调试,因为你不能真正只是tcpdump并分析它。
I'd add more logging on the server side - log_statement = 'all'
, and a log_line_prefix
that shows the client IP, backend start time & backend pid. 我将在服务器端添加更多日志记录log_statement = 'all'
,以及一个log_line_prefix
,它显示客户端IP,后端开始时间和后端pid。 That way you can start trying to match these failures back to the session activity that happened before the failure, work out if it's particular clients, particular jobs, or really just random. 通过这种方式,您可以开始尝试将这些故障与故障前发生的会话活动相匹配,确定是特定客户端,特定工作,还是真正随机。
Is this Sequel ORM gem using libpq
at the bottom layer, or its own PostgreSQL protocol implementation? 这个Sequel ORM gem是在底层使用libpq
,还是它自己的PostgreSQL协议实现? If the latter, that's probably going to turn out to be the culprit. 如果是后者,那可能会成为罪魁祸首。
Update: It looks like it can use the pg
gem ( libpq
-based), the postgres
gem, or possibly postgres-pr
(whatever that is). 更新:看起来它可以使用pg
gem(基于libpq
), postgres
gem,或者可能使用postgres-pr
(无论是什么)。 It'll prefer pg
if it's installed. 如果已经安装,它会更喜欢pg
。
Since it seems you're already using the pg
gem, you're probably going to need to do some diagnostic work to track down where the problem arises - particular queries, particular clients, etc - and try to find a way to reproduce the issue. 由于您似乎已经在使用pg
gem,您可能需要做一些诊断工作来追踪问题出现的位置 - 特定查询,特定客户等 - 并尝试找到重现问题的方法。 PostgreSQL's csvlog
might be useful so you can load and analyse the logs more easily. PostgreSQL的csvlog
可能很有用,因此您可以更轻松地加载和分析日志。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.