Near Lake NFT indexer access is slow

Question

We are using near Lake NFT indexer to listen to mainnet block information. We find that it is very slow to obtain data, sometimes in streamer_messages_queue.get() method waits for hundreds of seconds to get the data, and sometimes stops at the streamer_messages_queue.get() method, so we want to know if there is any configuration error or any restriction that causes us to get data very slowly?

stream_handle, streamer_messages_queue = streamer(config)
while True:
    start_time = int(time.time())
    logger.info("Start listening time:{}", start_time)
    streamer_message = await streamer_messages_queue.get()
    end_time = int(time.time())
    logger.info("streamer_messages_queue.get() consuming time:{}", start_time - end_time)
    logger.info(f"Block #{streamer_message.block.header.height} Shards: {len(streamer_message.shards)}")
    start_time = int(time.time())
    await handle_streamer_message(streamer_message)
    end_time = int(time.time())
    logger.info("handle_streamer_message consuming time:{}", start_time - end_time)

Output log:
Start listening time:1660711985
streamer_messages_queue.get() consuming time:-282
Block #71516637 Shards: 4
handle_streamer_message consuming time:0

When using the method provided by (near_lake_framework) to obtain data, it will wait for a long time at the above code (streamer_messages_queue.get())

Answer 1

I have heard such reports from people in some regions where AWS S3 is rate-limited. Try using VPN or running it on some server. You could also try Rust or JS versions of near-lake-framework (pick any of the tutorials ).