Ejabberd Server Application CPU Overload

Question

We have build Ejabberd in AWS EC2 instance and have enabled the clustering in the 6 Ejabberd servers in Tokyo, Frankfurt, and Singapore regions. The OS, middleware, applications and settings for each EC2 instance are exactly the same.

But currently, the Ejabberd CPUs in the Frankfurt and Singapore regions are overloaded. The CPU of Ejabberd in the Japan region is normal.

Could you please tell me the suspicious part?

Answer 1

You can take a look at the ejabberd log files of the problematic (and the good) nodes, maybe you find some clue.

You can use the undocumented "ejabberdctl etop" shell command in the problematic nodes. It's similar to "top", but runs inside the erlang virtual machine that runs ejabberd

ejabberdctl etop

========================================================================================
 ejabberd@localhost                                                        16:00:12
 Load:  cpu         0               Memory:  total       44174    binary       1320
        procs     277                        processes    5667    code        20489
        runq        1                        atom          984    ets          5467

Pid            Name or Initial Func    Time    Reds  Memory    MsgQ Current Function
----------------------------------------------------------------------------------------
<9135.1252.0>  caps_requests_cache     2393       1    2816       0 gen_server:loop/7   
<9135.932.0>   mnesia_recover           480      39    2816       0 gen_server:loop/7   
<9135.1118.0>  dets:init/2               71       2    5944       0 dets:open_file_loop2
<9135.6.0>     prim_file:start/0         63       1    2608       0 prim_file:helper_loo
<9135.1164.0>  dets:init/2               56       2    4072       0 dets:open_file_loop2
<9135.818.0>   disk_log:init/2           49       2    5984       0 disk_log:loop/1     
<9135.1038.0>  ejabberd_listener:in      31       2    2840       0 prim_inet:accept0/3 
<9135.1213.0>  dets:init/2               31       2    5944       0 dets:open_file_loop2
<9135.1255.0>  dets:init/2               30       2    5944       0 dets:open_file_loop2
<9135.0.0>     init                      28       1    3912       0 init:loop/1         
========================================================================================

Answer 2

I appreciate your quick response.

I did "ejabberdctl etop" command, but I got bellow response.

=======================

[root@trick01 ~]# ejabberdctl etop Erlang/OTP 22 [erts-10.4] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1]

Error Couldn't connect to node ejabberd

Usage of the Erlang top program

Options are set as command line parameters as in -node my@host or as parameters to etop:start([{node, my@host}, {...}]).

Options are: node atom Required The erlang node to measure port integer The used port, NOTE: due to a bug this program will hang if the port is not avaiable accumulate boolean If true execution time is accumulated lines integer Number of displayed processes interval integer Display update interval in secs sort runtime | reductions | memory | msg_q What information to sort by Default: runtime (reductions if tracing=off) tracing on | off etop uses the erlang trace facility, and thus no other tracing is possible on the node while etop is running, unless this option is set to 'off'. Also helpful if the etop tracing causes too high load on the measured node. With tracing off, runtime is not measured. setcookie string Only applicable on operating system command line, Set cookie for the etop node. must be same as the cookie for the measured node, This is not an etop parameter {"init terminating in do_boot","connection error"} init terminating in do_boot (connection error)

Crash dump is being written to: //var/log/ejabberd/erl_crash_20220204-050847.dump...done [root@trick01 ~]#