Crontab Schedule on Mac/Linux for Scrapy

1. Linux Crontab Conmand

Linux crontab is a command used to execute programs on a regular basis.

When the operating system is installed, this task scheduling command will be started by default.

The crond command will periodically check whether there is a job to be executed every minute, and if there is a job to be executed, the job will be executed automatically.

The time format is as follows:

f1 f2 f3 f4 f5 program

* * * * * 
- - - - - 
| | | | | 
| | | | + ----- week week (0--6) (Sunday is 0) 
| | | + ------- --- Month (1-12) 
| | +--------------- Day of the month (1-31) 
| +--------- ----------- hours (0-23) 
+ ------------------------- minutes (0-59)

2. Parameter description

• -e: Run a text editor to set the schedule. The default text editor is VI. If you want to use another text editor, please set the VISUAL environment variable to specify which text editor to use (for example setenv VISUAL Joe)
• -r: delete the current schedule
• -l: List the current schedule

3. Crontab Schedule on Mac/Linux

3.1 Linux

crontab -e

# python run print_time.py each minutes
# write to time1600.txt continuous 

*/1 * * * * python3 -u rsdata/print_time.py >> rsdata/timelog.txt 2>&1

:qw # save and quit

crontab -l  # check crontab schedule

cat timelog.txt  # check insert result from print_time.py

3.2 Mac

crontab - e

# env on anaconda3/scrapy_sina
# scrapy sina everyday 23:50
# write log to sina 2mysql.txt
# saved to mysql

50 23 */1 * * /opt/anaconda3/envs/scrapy_sina/bin/python -u /Users/user/PycharmProjects/scrapycourse/sina/sina/main.py >> /Users/user/PycharmProjects/scrapycourse/sina/sina/sina2mysql.txt 2>&1

:qw

crontab -l

cat sina2mysql.txt

4. Tricks

If we use crontab to execute the script regularly, it cannot be executed, but if it can be executed directly through the command (such as: ./test.sh), this is mainly because the environment variable cannot be read.

Solution:

• All commands need to be written in absolute path form, such as: /usr/local/bin/docker.
• Use the code at the beginning of the shell script.
• Add environment variables in /etc/crontab , and add commands before executable commands . /etc/profile;/bin/sh to make environment variables take effect.

reference link: Linux crontab.