Django 1.6.5 with Celery 3.1 for Periodic Tasks

Question

I'm new with Django 1.6.5, and I have just completed the Poll tutorial:

https://docs.djangoproject.com/en/dev/intro/tutorial01/

My objective is to:

1. Setup periodic tasks so it fetches data from a Oracle Server, and copy their data to my SQLite database. (Per hour basis)
2. After copying the data, I should organize the data into my own format in the SQLite database.

Currently my folder contents are:

/Dashboard_Web
    /Dashboard_Web
        /settings.py
        /url.py
        /wsgi.py
        /__init___.py
    /WebApp
        /admin.py
        /models.py
        /tasks.py
        /tests.py
        /view.py
        /__init__.py
    /db.sqlite3
    /manage.py

There's currently not much except the models.py: Data from the oracle server will be copied to mapt, datt, and sett. I will then fit the data into the new table, called data_parsed, which is the combination of mapt, datt, and sett.

from django.db import models

# Create your models here.

class mapt(models.Model):
    s = models.IntegerField(primary_key=True)
    name = models.CharField(max_length=200)

class datt(models.Model):
    s = models.IntegerField(primary_key=True)
    setid = models.IntegerField()
    var = models.IntegerField()
    val = models.IntegerField()

class sett(models.Model):
    setid = models.IntegerField()
    block = models.IntegerField()
    username = models.IntegerField()
    ts = models.IntegerField()

class data_parsed(models.Model):
    setid = models.CharField(max_length=2000)
    block = models.CharField(max_length=200)
    username = models.CharField(max_length=200)
    data = models.CharField(max_length=2000)
    time = models.IntegerField()

I've read this question about django + periodic tasks

My question is: celery -A proj worker -B -l info command, where my question is, if I put my django project to a production server (Gunicorn, apache), does the celery worker get executed automatically?

==================================EDIT==================================

Sorry if I didn't say this earlier, I'm using Windows , and I cannot use a Ubuntu (I'd love to), since the corporate IT doesn't support it (I can install Ubuntu on a VMware, but it wouldn't have internet access.

Currently I followed celery's tutorial , and I did these steps:

1. Installed and downloaded Erlang and RabbitMQ on my Windows Machine
2. Created 3 files: celery.py, tasks.py, ____init____.py

Hence the current folder format look like:

/Dashboard_Web
    /Dashboard_Web
        /settings.py
        /url.py
        /wsgi.py
        /__init___.py <== Updated
        /celery.py <=== Added
    /WebApp
        /admin.py
        /models.py
        /tasks.py
        /tests.py
        /view.py
        /__init__.py
        /tasks.py <=== Added
    /db.sqlite3
    /manage.py

Contents of Celery.py:

from __future__ import absolute_import

import os

from celery import Celery

from django.conf import settings

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'Dashboard_Web.settings')

app = Celery('Dashboard_Web')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

Contents of ____init____.py:

from __future__ import absolute_import

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app

Contents of tasks.py:

from __future__ import absolute_import

from celery import shared_task


@shared_task
def add(x, y):
    return x + y


@shared_task
def mul(x, y):
    return x * y


@shared_task
def xsum(numbers):
    return sum(numbers)

@periodic_task(run_every=(crontab(hour="*", minute="*", day_of_week="*")))
def scraper_example():
    print("Hello World")
    logger.info("Start task")
    logger.info("Task finished: result")

I am able to execute:

python manage.py runserver
celery -A Dashboard_Web worker -l INFO 

without any problems, however, there is nothing shown in the console:

[2014-07-02 11:46:12,835: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2014-07-02 11:46:12,880: INFO/MainProcess] mingle: searching for neighbors
[2014-07-02 11:46:13,957: INFO/MainProcess] mingle: all alone
C:\Python27\lib\site-packages\celery\fixups\django.py:236: UserWarning: Using settings.DEBUG leads to a memory leak, nev
er use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '

[2014-07-02 11:46:13,994: WARNING/MainProcess] C:\Python27\lib\site-packages\celery\fixups\django.py:236: UserWarning: U
sing settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '

[2014-07-02 11:46:14,012: WARNING/MainProcess] celery@LEIMAX1 ready.

Answer 1

Here's the steps for supervisor

>> apt-get install supervisor
>> service supervisor restart

Then create a conf file in /etc/supervisor/conf.d/celery.conf that contains something similar to:

[program:celery-worker]
command=/root/.virtualenvs/test/bin/celery -A Dashboard_Web worker -l INFO
autostart=true
autorestart=true
stderr_logfile=/var/log/long.err.log
stdout_logfile=/var/log/long.out.log

[program:celery-beat]
command=/root/.virtualenvs/test/bin/celery -A Dashboard_Web beat -l INFO
autostart=true
autorestart=true
stderr_logfile=/var/log/long.err.log
stdout_logfile=/var/log/long.out.log

You should also set up celery.py using this tutorial .

Then run:

>> supervisorctl reread
>> supervisorctl update

See the status by running

>> supervisorctl status