使用celery 进行任务调度，做的主要任务就是使用phantomjs进行抓取网页内容，然后进行保存。
celery 配置代码:
BROKER_URL = 'redis:/*'
CELERY_RESULT_BACKEND = 'redis://:*'

CELERY_TASK_RESULT_EXPIRES = 300

CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_RESULT_SERIALIZER = 'json'

CELERY_DEFAULT_EXCHANGE = 'celery'
CELERY_DEFAULT_EXCHANGE_TYPE = 'direct'
CELERY_DEFAULT_QUEUE = 'celery'
CELERY_DEFAULT_ROUTING_KEY = 'celery'

CELERY_TIMEZONE = 'Asia/Shanghai'

CELERY_ENABLE_UTC = True

禁用所有速度限制

CELERY_DISABLE_RATE_LIMITS = True
CELERY_ACKS_LATE = True
CELERY_IGNORE_RESULT = True

调用phantomjs代理进行抓取网站内容
downloadByPhantomjsProxy(url, fileParh)

phantomjs 代理启动：
phantomjs phantomjs/phantomjs_fetcher.js 1234

当程序抓取一段时间后celery flower监控出现如下情况:

Worker Name Status Active Processed Failed Succeeded Retried Load Average

celery@localhost Offline 3 4303 0 4299 0