celery task scheduled but not executed at ETA

Question

I have my celery configured as following:

# create context tasks in celery
celery = Celery(
    __name__,
    # redis
    backend=app.config['CELERY_RESULT_BACKEND'],
    broker=app.config['CELERY_BROKER_URL'],
    include=['app.celery_tasks.tasks']
)
celery.conf.timezone = 'US/Pacific'
celery.conf.broker_transport_options = {'visibility_timeout': 3600*24}
celery.conf.task_routes = {
    'tasks.periodic': {
        'queue': 'periodic',
        'routing_key': 'tasks.periodic'
    },
    'tasks.generate_report': {
        'queue': 'report',
        'routing_key': 'tasks.generate_report'
    }
}

Then I'm using this helper method to get eta for all my report tasks

def get_eta_time(time_d=8):
    tz = timezone('US/Pacific')
    ct = datetime.now(tz=tz)
    eta = ct + timedelta(hours=time_d)
    return eta

The thing I'm encountering is I can see the tasks are scheduled using the celery control but they are not executed when the eta arrives. However, when I tried to restart my celery workers, these tasks got picked up immediately. Is there anything I missed in my celery config?

My tasks are triggered as following:

eta = get_eta_time()
generate_report.apply_async(args=(log_location, repetition_count+1), queue='report', eta=eta)

My periodical queue works as expected, but my report queue is not making any sense to me.

Answer 1

Turned out to be some hardware issue. I switch to a different machine and it is working now.