XLP Literate Program for all cron jobs

A.J.Hurst

Version 0.6.7

Table of Contents

1 Introduction
2 The Newport Cron
3 The Burnley Cron
4 The Spencer Cron
5 The Albens Cron
6 The Everton Cron
7 The Dimboola Cron
8 The Makefile
9 Migration
10 Document History
11 Indices
11.1 Files
11.2 Chunks
11.3 Identifiers


1. Introduction

This program generates, installs and initiates all cron jobs associated with Hurst computers.

A number of systems have been decommissioned. Legacy software relating to those decommissioned systems has been moved to the subdirectory $HOME/Computers.technical/cron/decommissioned.

<cron header 1.1> =
## ## ********************************************************** ## * DO NOT EDIT THIS FILE! * ## * Use $HOME/Computers/technical/cron.xlp instead * ## ********************************************************** ##
Chunk referenced in 2.1 3.1 4.1 5.1 6.1 7.1
<cron title 1.2> =
################################## ## CRON FILE for machine ## ##################################
Chunk referenced in 2.1 3.1 4.1 5.1 6.1 7.1
<cron trailer 1.3> =
## ## The End ##
Chunk referenced in 2.1 3.1 4.1 5.1 6.1 7.1

2. The Newport Cron

The newport cron is used to ensure that my current workspace is backed up each night.

"cron.newport" 2.1 =
<cron title 1.2>(machine='NEWPORT ') <cron header 1.1> HOME=/home/ajh LOGS=/home/ajh/logs/newport DATE=`/home/ajh/bin/date` INCRLOG=/home/ajh/logs/newport/backup/$DATE PYTHONLIB=/home/ajh/lib/python PYTHONPATH=/home/ajh/lib/python:/home/ajh/Computers/python:/usr/local/lib/python PYTHON=/home/ajh/Computers/python SHELL=/bin/bash PATH=.:/home/ajh/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin XML_CATALOG_FILES=/home/ajh/etc/xml/catalog #Mins Hour Day Month Weekday # backup to backup disk 30 0 * * * $HOME/bin/backup >>$HOME/logs/backup.log 2>&1 <cron trailer 1.3>

3. The Burnley Cron

The burnley cron is used to ensure that my current workspace is backed up to spencer each night.

"cron.burnley" 3.1 =
<cron title 1.2>(machine='BURNLEY ') <cron header 1.1> HOME=/home/ajh LOGS=/home/ajh/logs/burnley DATE=`/home/ajh/bin/date` INCRLOG=/home/ajh/logs/burnley/backup/$DATE PYTHONLIB=/home/ajh/lib/python PYTHONPATH=/home/ajh/lib/python:/home/ajh/Computers/python:/usr/local/lib/python PYTHON=/home/ajh/Computers/python SHELL=/bin/bash PATH=.:/home/ajh/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin XML_CATALOG_FILES=/home/ajh/etc/xml/catalog #Mins Hour Day Month Weekday # backup to spencer 30 0 * * * $HOME/bin/sync to spencer >>$HOME/logs/sync.log <cron trailer 1.3>

4. The Spencer Cron

spencer is the disk farm, and these few cron jobs are related to performing backups.

"cron.spencer" 4.1 =
<cron title 1.2>(machine='SPENCER ') <cron header 1.1> HOME=/home/ajh LOGS=$HOME/logs DATE=`$HOME/bin/date` INCRLOG=$HOME/logs/spencer/backup/$DATE PYTHONPATH=./$HOME/lib/python:/home/ajh/Computers/python:/usr/local/lib/python SHELL=/bin/bash PATH=.:/sw/sbin:/sw/bin:$HOME/bin/macosx:$HOME/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin XML_CATALOG_FILES=$HOME/etc/xml/catalog MAILTO=ajh@internode.on.net #Mins # Hour # Day # Month # Weekday 20 1 * * * rsync -auv $HOME/* /lizard/ajh/ 0 0 * * * $HOME/bin/backup >>$HOME/logs/backup.log 0 * * * * sleep 17 ; wget -O - http://freedns.afraid.org/dynamic/update.php?ZkhMd3JlaWZhS2FCSldOVmxReXQ6MTcyMzkwNzI= >>/tmp/freedns_@_ajh_id_au.log 2>&1 & # wiki backup: start this a couple of minutes after the cleanWiki script should have run 31 3 * * * /usr/bin/rsync -auv --exclude=surgeprotect --exclude=__session__ gwuc-wiki.gwuc.org.au:/var/wiki/share/moin/server/data/* /lizard/backup/wiki/ 0 7 * 1 * mv /lizard/backup/wiki /lizard/backup/wiki-`/bin/date "+%Y%m%d"` ; mkdir /lizard/backup/wiki #0 1 * * * /usr/bin/rsync -auv root@gorg.njhurst.com:/lizard/njh/* /lizard/njh/ >>$LOGS/njhXfer.log #0 1 * * * /usr/bin/rsync -auv gorg.njhurst.com:/lizard/bunny/* /lizard/bunny/ >>$LOGS/bunnyXfer.log #0 2 * * * /usr/bin/rsync -auv /home/ajh/Pictures/Albums/* gorg.njhurst.com:/lizard/ajh/Pictures/Albums/ >>$LOGS/picsXfer.log <cron trailer 1.3>

The backup cron job now (20200104:134652) seems to be working OK, although it struggles when a full backup is required, such as when creating new monthly backups (last full cycle took 2.5 days!) Need to add a locking mechanism to avoid overrunning on to next day's cycle.

20200104:134826 the gorg rsync jobs have been commented out while Nathan's system is off-line.

5. The Albens Cron

Under development

"cron.albens" 5.1 =
<cron title 1.2>(machine='ALBENS') <cron header 1.1> HOME=/home/ajh LOCAL=/home/ajh/local/albens DATE=`/home/ajh/bin/date` SHELL=/bin/bash PATH=.:/sw/sbin:/sw/bin:/home/ajh/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin #Mins Hour Day Month Weekday 0 1 * * * find /home/ajh/local/albens/htmls -mtime +14 -type f -exec rm {} \; 0 2 * * * find /home/ajh/local/albens/counters -mtime +14 -name personal-albums-\* -exec rm {} \; <cron trailer 1.3>

The two cron entries do clean-up on the web page logging. The first is to remove archived html files that have been cached from the corresponding xml file rendering, and are older than 14 days.

The second entry removes counters from the photo albums. These tend to be generated by web crawlers, and are of no intrinsic interest. As they tend to clog up the counter directory, they are removed, also after 14 days.

6. The Everton Cron

The everton cron is used to ensure that my current workspace on my laptop is backed up each night.

"cron.everton" 6.1 =
<cron title 1.2>(machine='EVERTON ') <cron header 1.1> HOME=/home/ajh LOGS=/home/ajh/logs/everton DATE=`/home/ajh/bin/date` INCRLOG=/home/ajh/logs/everton/backup/$DATE PYTHONLIB=/home/ajh/lib/python PYTHONPATH=/home/ajh/lib/python:/home/ajh/Computers/python:/usr/local/lib/python PYTHON=/home/ajh/Computers/python SHELL=/bin/bash PATH=.:/home/ajh/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin XML_CATALOG_FILES=/home/ajh/etc/xml/catalog #Mins Hour Day Month Weekday # backup to backup disk 30 0 * * * $HOME/bin/backup >>$HOME/logs/backup.log 2>&1 <cron trailer 1.3>

7. The Dimboola Cron

Not currently in use.

"cron.dimboola" 7.1 =
<cron title 1.2>(machine='DIMBOOLA') <cron header 1.1> HOME=/Users/ajh LOGS=/Users/ajh/logs/dimboola DATE=`/Users/ajh/bin/date` INCRLOG=/Users/ajh/logs/dimboola/backup/$DATE PYTHONLIB=/Users/ajh/lib/python:/Users/ajh/Church/Computers/rosters PYTHONPATH=/Users/ajh/lib/python:/home/ajh/Computers/python:/usr/local/lib/python PYTHON=/Users/ajh/Computers/python ROSTER=/Users/ajh/Church/Computers/rosters HOUSE=/Users/ajh/Computers/House SHELL=/bin/bash PATH=.:/sw/sbin:/sw/bin:/Users/ajh/bin/macosx:/Users/ajh/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin XML_CATALOG_FILES=/Users/ajh/etc/xml/catalog PYTHONPATH=/home/ajh/lib/python:/usr/local/lib/python #Mins Hour Day Month Weekday # Pictures/albums/2016/07 synching #0 3 * * 1 rsync -auv albens:/home/ajh/Pictures/Albums/2016/07/* $HOME/Pictures/Albums/2016/07/ # backup to spencer 30 0 * * * $HOME/bin/sync to spencer <cron trailer 1.3>

8. The Makefile

"Makefile" 8.1 =
SOURCEDIR=$(HOME)/Computers/technical/cron default=program GenFiles = .program install-program include ${HOME}/etc/MakeXLP install: install-echuca install-spencer install-wolseley start: start-echuca start-spencer start-wolseley all: install start clean: litclean -rm $(GenFiles) start-%: install-% if [ "$(HOST)" = "$*" ] ; then \ crontab $(SOURCEDIR)/cron.$* ;\ else \ ssh $* crontab $(SOURCEDIR)/cron.$* ;\ fi touch start-$* install-%: make-% if [ "$(HOST)" != "$*" ] ; then \ rsync -auv cron.$* $*:$(SOURCEDIR)/ ;\ fi touch install-$* make-%: cron.tangle touch make-$* #cron.tangle: cron.xlp # from MakeXLP

9. Migration

A lot of the cron jobs defined above use legacy or out-of-date software. In order to manage the migration to new systems, the following dependencies are identified (by cron reference):

Cron Main Location Dependencies
Albens allRosterMail /home/ajh/Church/Computers/rosters copy, getopt, sys, subprocess (all standard libraries)
ERegnans genRoster /home/ajh/Church/Computers/rosters calendarNames (lib/python), SaltDuties (.), rosterAccess3 (.), datetime, getopt, math, re, sys (all standard libraries)
ERegnans rosterAccess3 /home/ajh/Church/Computers/rosters calendarNames (lib/python), wikiAccess (.), editmoin (lib/python), datetime, math, os, os.path, re, string, sys, urllib2 (all standard libraries)

10. Document History

20130820:111537 ajh 0.0 first draft, built from individual cron files
20131125:222602 ajh 0.1 omit central crons
20131227:165448 ajh 0.2 added garedelyon cron table, and migration section
20140127:141504 ajh 0.3 replace central with lilydale
20140425:114511 ajh 0.4 replace lilydale with ringwood
20150420:174011 ajh 0.5 add orsay, start to remove obsolete code (ringwood)
20150606:120513 ajh 0.5.1 add bastille, take over from orsay
20150717:094715 ajh 0.5.2 add lilydale, take over from bastille
20170223:153456 ajh 0.6.0 General clean, remove a lot of obsolete entries
20170510:120728 ajh 0.6.1 further tidy ups (remove Lilydale), and revisions of content
20170906:131409 ajh 0.6.2 revised spencer cron, and fix HOME address for spencer
20171022:124649 ajh 0.6.3 change spencer backup time
20180706:182105 ajh 0.6.4 add burnley, remove dimboola
20181114:171620 ajh 0.6.5 add newport
20181115:104546 ajh 0.6.6 remove echuca and wolseley crons
20200309:220458 ajh 0.6.7 add everton

11. Indices

11.1 Files

File Name Defined in
Makefile 8.1
cron.albens 5.1
cron.burnley 3.1
cron.dimboola 7.1
cron.everton 6.1
cron.newport 2.1
cron.spencer 4.1

11.2 Chunks

Chunk Name Defined in Used in
cron title 1.2 2.1, 3.1, 4.1, 5.1, 6.1, 7.1
cron trailer 1.3 2.1, 3.1, 4.1, 5.1, 6.1, 7.1

11.3 Identifiers

Identifier Defined in Used in

35 accesses since 16 Jan 2019, HTML cache rendered at 20200309:2214