XLP Literate Program for all cron jobs
A.J.Hurst
Version 0.6.7
Table of Contents
1. Introduction
This program generates, installs and initiates all cron jobs
associated with Hurst computers.
A number of systems have been decommissioned. Legacy software
relating to those decommissioned systems has been moved to the
subdirectory
$HOME/Computers.technical/cron/decommissioned.
<cron header 1.1> =##
## **********************************************************
## * DO NOT EDIT THIS FILE! *
## * Use $HOME/Computers/technical/cron.xlp instead *
## **********************************************************
##
<cron title 1.2> =##################################
## CRON FILE for machine ##
##################################
<cron trailer 1.3> =##
## The End
##
2. The Newport Cron
The newport cron is used to ensure that my current workspace is
backed up each night.
"cron.newport" 2.1 =
<cron title 1.2>(machine='NEWPORT ')
<cron header 1.1>
HOME=/home/ajh
LOGS=/home/ajh/logs/newport
DATE=`/home/ajh/bin/date`
INCRLOG=/home/ajh/logs/newport/backup/$DATE
PYTHONLIB=/home/ajh/lib/python
PYTHONPATH=/home/ajh/lib/python:/home/ajh/Computers/python:/usr/local/lib/python
PYTHON=/home/ajh/Computers/python
SHELL=/bin/bash
PATH=.:/home/ajh/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
XML_CATALOG_FILES=/home/ajh/etc/xml/catalog
#Mins Hour Day Month Weekday
# backup to backup disk
30 0 * * * $HOME/bin/backup >>$HOME/logs/backup.log 2>&1
<cron trailer 1.3>
3. The Burnley Cron
The burnley cron is used to ensure that my current workspace is
backed up to spencer each night.
"cron.burnley" 3.1 =
<cron title 1.2>(machine='BURNLEY ')
<cron header 1.1>
HOME=/home/ajh
LOGS=/home/ajh/logs/burnley
DATE=`/home/ajh/bin/date`
INCRLOG=/home/ajh/logs/burnley/backup/$DATE
PYTHONLIB=/home/ajh/lib/python
PYTHONPATH=/home/ajh/lib/python:/home/ajh/Computers/python:/usr/local/lib/python
PYTHON=/home/ajh/Computers/python
SHELL=/bin/bash
PATH=.:/home/ajh/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
XML_CATALOG_FILES=/home/ajh/etc/xml/catalog
#Mins Hour Day Month Weekday
# backup to spencer
30 0 * * * $HOME/bin/sync to spencer >>$HOME/logs/sync.log
<cron trailer 1.3>
4. The Spencer Cron
spencer is the disk farm, and these few cron jobs are related to
performing backups.
"cron.spencer" 4.1 =
<cron title 1.2>(machine='SPENCER ')
<cron header 1.1>
HOME=/home/ajh
LOGS=$HOME/logs
DATE=`$HOME/bin/date`
INCRLOG=$HOME/logs/spencer/backup/$DATE
PYTHONPATH=./$HOME/lib/python:/home/ajh/Computers/python:/usr/local/lib/python
SHELL=/bin/bash
PATH=.:/sw/sbin:/sw/bin:$HOME/bin/macosx:$HOME/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin
XML_CATALOG_FILES=$HOME/etc/xml/catalog
MAILTO=ajh@internode.on.net
#Mins
# Hour
# Day
# Month
# Weekday
20 1 * * * rsync -auv $HOME/* /lizard/ajh/
0 0 * * * $HOME/bin/backup >>$HOME/logs/backup.log
0 * * * * sleep 17 ; wget -O - http://freedns.afraid.org/dynamic/update.php?ZkhMd3JlaWZhS2FCSldOVmxReXQ6MTcyMzkwNzI= >>/tmp/freedns_@_ajh_id_au.log 2>&1 &
# wiki backup: start this a couple of minutes after the cleanWiki script should have run
31 3 * * * /usr/bin/rsync -auv --exclude=surgeprotect --exclude=__session__ gwuc-wiki.gwuc.org.au:/var/wiki/share/moin/server/data/* /lizard/backup/wiki/
0 7 * 1 * mv /lizard/backup/wiki /lizard/backup/wiki-`/bin/date "+%Y%m%d"` ; mkdir /lizard/backup/wiki
#0 1 * * * /usr/bin/rsync -auv root@gorg.njhurst.com:/lizard/njh/* /lizard/njh/ >>$LOGS/njhXfer.log
#0 1 * * * /usr/bin/rsync -auv gorg.njhurst.com:/lizard/bunny/* /lizard/bunny/ >>$LOGS/bunnyXfer.log
#0 2 * * * /usr/bin/rsync -auv /home/ajh/Pictures/Albums/* gorg.njhurst.com:/lizard/ajh/Pictures/Albums/ >>$LOGS/picsXfer.log
<cron trailer 1.3>
The backup cron job now (20200104:134652) seems to be
working OK, although it struggles when a full backup is
required, such as when creating new monthly backups (last full
cycle took 2.5 days!) Need to add a locking mechanism to avoid
overrunning on to next day's cycle.
20200104:134826 the gorg rsync jobs have been commented out
while Nathan's system is off-line.
5. The Albens Cron
Under development
"cron.albens" 5.1 =
<cron title 1.2>(machine='ALBENS')
<cron header 1.1>
HOME=/home/ajh
LOCAL=/home/ajh/local/albens
DATE=`/home/ajh/bin/date`
SHELL=/bin/bash
PATH=.:/sw/sbin:/sw/bin:/home/ajh/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin
#Mins Hour Day Month Weekday
0 1 * * * find /home/ajh/local/albens/htmls -mtime +14 -type f -exec rm {} \;
0 2 * * * find /home/ajh/local/albens/counters -mtime +14 -name personal-albums-\* -exec rm {} \;
<cron trailer 1.3>
The two cron entries do clean-up on the web page logging. The
first is to remove archived html files that have been cached
from the corresponding xml file rendering, and are older than 14
days.
The second entry removes counters from the photo albums. These
tend to be generated by web crawlers, and are of no intrinsic
interest. As they tend to clog up the counter directory, they
are removed, also after 14 days.
6. The Everton Cron
The everton cron is used to ensure that my current workspace on
my laptop is backed up each night.
"cron.everton" 6.1 =
<cron title 1.2>(machine='EVERTON ')
<cron header 1.1>
HOME=/home/ajh
LOGS=/home/ajh/logs/everton
DATE=`/home/ajh/bin/date`
INCRLOG=/home/ajh/logs/everton/backup/$DATE
PYTHONLIB=/home/ajh/lib/python
PYTHONPATH=/home/ajh/lib/python:/home/ajh/Computers/python:/usr/local/lib/python
PYTHON=/home/ajh/Computers/python
SHELL=/bin/bash
PATH=.:/home/ajh/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
XML_CATALOG_FILES=/home/ajh/etc/xml/catalog
#Mins Hour Day Month Weekday
# backup to backup disk
30 0 * * * $HOME/bin/backup >>$HOME/logs/backup.log 2>&1
<cron trailer 1.3>
7. The Dimboola Cron
Not currently in use.
"cron.dimboola" 7.1 =
<cron title 1.2>(machine='DIMBOOLA')
<cron header 1.1>
HOME=/Users/ajh
LOGS=/Users/ajh/logs/dimboola
DATE=`/Users/ajh/bin/date`
INCRLOG=/Users/ajh/logs/dimboola/backup/$DATE
PYTHONLIB=/Users/ajh/lib/python:/Users/ajh/Church/Computers/rosters
PYTHONPATH=/Users/ajh/lib/python:/home/ajh/Computers/python:/usr/local/lib/python
PYTHON=/Users/ajh/Computers/python
ROSTER=/Users/ajh/Church/Computers/rosters
HOUSE=/Users/ajh/Computers/House
SHELL=/bin/bash
PATH=.:/sw/sbin:/sw/bin:/Users/ajh/bin/macosx:/Users/ajh/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin
XML_CATALOG_FILES=/Users/ajh/etc/xml/catalog
PYTHONPATH=/home/ajh/lib/python:/usr/local/lib/python
#Mins Hour Day Month Weekday
# Pictures/albums/2016/07 synching
#0 3 * * 1 rsync -auv albens:/home/ajh/Pictures/Albums/2016/07/* $HOME/Pictures/Albums/2016/07/
# backup to spencer
30 0 * * * $HOME/bin/sync to spencer
<cron trailer 1.3>
8. The Makefile
"Makefile" 8.1 =SOURCEDIR=$(HOME)/Computers/technical/cron
default=program
GenFiles = .program install-program
include ${HOME}/etc/MakeXLP
install: install-echuca install-spencer install-wolseley
start: start-echuca start-spencer start-wolseley
all: install start
clean: litclean
-rm $(GenFiles)
start-%: install-%
if [ "$(HOST)" = "$*" ] ; then \
crontab $(SOURCEDIR)/cron.$* ;\
else \
ssh $* crontab $(SOURCEDIR)/cron.$* ;\
fi
touch start-$*
install-%: make-%
if [ "$(HOST)" != "$*" ] ; then \
rsync -auv cron.$* $*:$(SOURCEDIR)/ ;\
fi
touch install-$*
make-%: cron.tangle
touch make-$*
#cron.tangle: cron.xlp # from MakeXLP
9. Migration
A lot of the cron jobs defined above use legacy or out-of-date
software. In order to manage the migration to new systems, the
following dependencies are identified (by cron reference):
| Cron |
Main |
Location |
Dependencies |
| Albens |
allRosterMail |
/home/ajh/Church/Computers/rosters |
copy, getopt, sys, subprocess (all standard libraries) |
| ERegnans |
genRoster |
/home/ajh/Church/Computers/rosters |
calendarNames (lib/python), SaltDuties (.), rosterAccess3 (.),
datetime, getopt, math, re, sys (all standard libraries) |
| ERegnans |
rosterAccess3 |
/home/ajh/Church/Computers/rosters |
calendarNames (lib/python), wikiAccess (.), editmoin (lib/python),
datetime, math, os, os.path, re, string, sys, urllib2 (all standard libraries) |
10. Document History
| 20130820:111537 |
ajh |
0.0 |
first draft, built from individual cron files |
| 20131125:222602 |
ajh |
0.1 |
omit central crons |
| 20131227:165448 |
ajh |
0.2 |
added garedelyon cron table, and migration section |
| 20140127:141504 |
ajh |
0.3 |
replace central with lilydale |
| 20140425:114511 |
ajh |
0.4 |
replace lilydale with ringwood |
| 20150420:174011 |
ajh |
0.5 |
add orsay, start to remove obsolete code (ringwood) |
| 20150606:120513 |
ajh |
0.5.1 |
add bastille, take over from orsay |
| 20150717:094715 |
ajh |
0.5.2 |
add lilydale, take over from bastille |
| 20170223:153456 |
ajh |
0.6.0 |
General clean, remove a lot of obsolete entries
|
| 20170510:120728 |
ajh |
0.6.1 |
further tidy ups (remove Lilydale), and revisions of content
|
| 20170906:131409 |
ajh |
0.6.2 |
revised spencer cron, and fix HOME address for spencer
|
| 20171022:124649 |
ajh |
0.6.3 |
change spencer backup time |
| 20180706:182105 |
ajh |
0.6.4 |
add burnley, remove dimboola |
| 20181114:171620 |
ajh |
0.6.5 |
add newport |
| 20181115:104546 |
ajh |
0.6.6 |
remove echuca and wolseley crons |
| 20200309:220458 |
ajh |
0.6.7 |
add everton |
11. Indices
11.1 Files
| File Name |
Defined in |
| Makefile |
8.1 |
| cron.albens |
5.1 |
| cron.burnley |
3.1 |
| cron.dimboola |
7.1 |
| cron.everton |
6.1 |
| cron.newport |
2.1 |
| cron.spencer |
4.1 |
11.2 Chunks
| Chunk Name |
Defined in |
Used in |
| cron title |
1.2 |
2.1, 3.1, 4.1, 5.1, 6.1, 7.1
|
| cron trailer |
1.3 |
2.1, 3.1, 4.1, 5.1, 6.1, 7.1
|
11.3 Identifiers
| Identifier |
Defined in |
Used in |