简体繁体 English

如何将Sensu数据包追溯到其原始IP地址？

[英]How do I trace a Sensu packet back to its originating IP address?

原文 2014-01-15 21:15:06 7 1 amazon-web-services/ rabbitmq/ tcpdump/ sensu

I am trying to locate a machine in one of our environments that is causing a Sensu notification. 我正在尝试在引起Sensu通知的我们的环境之一中查找机器。 The hostname and IP address listed in the notification are all messed up, because at the time the machine was being created, it had different data. 通知中列出的主机名和IP地址都被弄乱了，因为在创建计算机时，它具有不同的数据。 So the wrong data stuck, and the machine is still alive and kicking... I mean, sending wrong data to the Sensu server from somewhere. 因此，错误的数据被卡住了，并且机器仍然处于运行状态并且正在踢...我的意思是，从某个地方将错误的数据发送到Sensu服务器。

I have attempted to track down the address of the machine. 我试图追踪机器的地址。 With help of tcpdump, I found the same kind of packet that I am looking for occurring in two places: 在tcpdump的帮助下，我在两个地方找到了与我正在寻找的相同类型的数据包：

1) At every machine that is running a Sensu client, I see packets with the right payload leaving for the Sensu server machine. 1）在运行Sensu客户端的每台计算机上，我看到具有正确有效负载的数据包留给Sensu服务器计算机。 Sensu config files tell me that Sensu is using RabbitMQ on the same machine as the Sensu server, and the packets are heading straight for that. Sensu配置文件告诉我，Sensu与Sensu服务器在同一台计算机上使用RabbitMQ，因此数据包正朝着该方向前进。

2) At the Sensu server, I see all of those packets incoming from a local 10. . 2）在Sensu服务器上，我看到了所有从本地10传入的数据包。 .* IP address, from all kinds of different ports. 。* IP地址，来自各种不同的端口。 When I probed that IP address with a wget, it game me the index.html of the Sensu dashboard, so that local address seems to be the same machine - probably RabbitMQ or something, since Sensu uses that. 当我用wget探测该IP地址时，它会用Sensu仪表板的index.html进行游戏，因此本地地址似乎是同一台机器-可能是RabbitMQ或其他东西，因为Sensu使用了它。

There's probably up to a hundred machines running a Sensu client in our environments, but there are nowhere nearly as many connections or source IP addresses in the incoming traffic. 在我们的环境中，可能有多达一百台计算机在运行Sensu客户端，但是传入流量中几乎没有连接或源IP地址。 So, I can't figure out how to find the right source machine other than brute force shutting down every machine one by one and seeing when a different notification pops up. 因此，除了蛮力逐个关闭每台计算机并查看何时弹出另一条通知之外，我无法弄清楚如何找到合适的源计算机。

Extra information: our machines are all in AWS, and are provisioned by Puppet after creation. 额外信息：我们的机器全部在AWS中，并在创建后由Puppet进行配置。 Sensu is baked into the base AMI so that we can get alerted if Puppet fails right away. Sensu被烘焙到基本的AMI中，因此如果Puppet立即失败，我们可以得到警报。 Except that Puppet didn't even know who he was at the time it failed. 除了Puppet失败时甚至都不知道他是谁。

EDIT: also, now that I think about it, it may be important that the Sensu server is sitting behind an Elastic Load Balancer, which is behind a Route 53 entry, which is where all the Sensu clients are sending stuff. 编辑：此外，现在考虑到这一点，将Sensu服务器置于Elastic Load Balancer（位于Route 53条目的后面）是很重要的，该条目是所有Sensu客户端正在发送东西的位置。

1 个解决方案

ELB turned out to be the trouble. ELB原来是麻烦。 As soon as I rerouted Route 53 directly to the Sensu server and (because of caching issues) took the Sensu server out of the ELB, all the incoming connections assumed correct IP addresses. 一旦我将Route 53直接重新路由到Sensu服务器，并且（由于缓存问题）将Sensu服务器从ELB中取出，所有传入的连接都假定使用正确的IP地址。 Wasn't a Sensu problem after all. 毕竟不是Sensu问题。