CHAINLINK NODE - Your node is overloaded and may start missing jobs ERROR

Question

Running a test node in GCP, using Docker 9.9.4, Ubuntu, Postgres db, Infura. I had issues with public/private IP, but once I cleared that up my node is up and running. I am now throwing the error below repeatedly, potentially due to the blockchain connection. How do I fix this?

[ERROR] HeadTracker: dropping head 26085153 with hash 0xf50e19099b7e343829935d70dd7d86c5bc0398286b7a4e4f32ac033ac60c3733 because queue is full. WARNING: Your node is overloaded and may start missing jobs. logger/default.go:155 stacktrace=github.com/smartcontractkit/chainlink/core/logger.Errorf

Answer 1

This log output is related to an overload of your blockchain connection. This notification is usually related to the usage of public websocket connections and/or free third party NaaS Provider. To fix this connection issue you can either run an own full node or change the tier or the third party NaaS provider. Also it is recommended to use Chainlink version 0.10.8 or higher, as the HeadTracker has been revised here and performs more efficient.

In regard to the question let me try to give you a small technical overview, which may clarify the payload of a Chainlink node to it's remote full node:

Your Chainlink node establishes a connection to a full node. There the Chainlink node initiates various subscriptions, which are a special feature of the websocket protocol to enable bidirectional communication. More precisely, this means that the Chainlink node is informed if a certain "state" of the subscription changes. Basically, the node interacts with using JSON-RPC methods and uses the following methods to initiate and process various functions internally: eth_getBlockByNumber , eth_getBalance , eth_getTransactionReceipt , eth_getTransactionCount , eth_getLogs , eth_subscribe , eth_unsubscribe , eth_sendRawTransaction and eth_Call https://ethereum.org/uk/developers/docs/apis/json-rpc/

The high amount of interactions of the Chainlink node are especially executed during the syncing process via the internal HeadTracker service. This service initiates a "head" subscription in order to interact with every single incoming new blockheader. During this syncing process it uses the JSON-RPC methods eth_GetBlockByNumber and eth_getBalance to get all the necessary information from the block. So these two methods are used/ executed every block. The number of requests now depends on the average blocktime of the network the Chainlink node is connected to

An example would be the Kovan Testnet:

The avg.blocktime here is 6.7sec, which means you get a daily request number of approx.21.000 During fulfilling job requests, those request also includes following methods: eth_getTransactionReceipt , eth_sendRawTransaction , eth_getLogs , eth_subscribe , eth_unsubscribe , eth_getTransactionCount and eth_call , which increases the total number significantly depending on the number of job requests. It should also be noted that especially with faster blockchains (eg polygon) there is a very high payload of the WebSocket and you have to deal with a good full node connection in detail, as many full nodes do not receive such a high number of requests permanently.