Global Synchronization of Files

Version 2.1.7

John Hurst

1	Abstract
2	Introduction
3	More Detailed Description
4	Description of The Environment
5	Program Specification
6	Design Document
7	User Manual
8	Literate Data
9	Literate Code
	9.1	The blat module
	9.2	blat: define the main routine
	9.3	blat: main cli interpretation
	9.4	blat: define the blat class
		9.4.1	blat: class: define the commandMachine method
		9.4.2	blat: class: define the loadState method
		9.4.3	blat: class: define the saveState method
		9.4.4	blat: class: define the checkMachines method
		9.4.5	blat: class: define the blatFile method
		9.4.6	blat: class: define the blatDirectory method
		9.4.7	blat: class: define the updateToDos method
	9.5	The scanlogs.py program
		9.5.1	scanlogs: build data structure of files from blatmissing
		9.5.2	scanlogs: restructure data to identify most recent files
10	Makefile
11	Literate Tests
12	TODOs
13	Bibliography
14	Glossary
15	Indices
	15.1	Chunk Names
	15.2	Identifiers
16	Maintenance History
	16.1	Previous status of this document
	16.2	Current status of this document:

1. Abstract

This document describes a python suite to synchronize files across multiple machines, known by their domain names.

2. Introduction

The blat.py program is written as a module, containing a single class, blat. This class provides methods to perform all the necessary operations in copying files around the network (called "blatting" the file). When called as a standalone program, blat provides a simple interface for copying a list of files or directories.

3. More Detailed Description

Working with multiple machines raises the question of how one maintains consistency of working files across the multiple platforms. I have previously used various scripts with rsync as the key ingredient, but these have all been limited to just single peer-to-peer transactions. What I was looking for is a script that can operate on any machine in the network, and which will bring all copies anywhere on the network up-to-date.

There are some limitations to this. Obviously, the machines need to be available. If it is off-line, shut down, or otherwise unavailable, it cannot participate in the sychronize operation. So one of the first tasks of the script must be to ascertain what machines are available. My first pass at this required a separate program (online.py) to check all machines, so that this did not need to be done everytime a file was synchronized across the network.

A second phase, synchronize.py, was used to actually perform the synchronization. An optional parameter to this program could be used to force a rescan of the network, but usually, once this had been done, subsequent invocations did not need it, as machines did not change their status that quickly.

But then I realized that it did not need to be so decoupled. As attempts were made to get file status, and to copy files, information on the current machine states could be updated. This would only need to be done as required, rather than attempt to update all machines at once. Accordingly, the two programs synchronize.py and online.py were merged into a single piece of code, blat.py, and the whole suite was rewritten in Python 3 (currently 3.3 compatible, although developed under 3.5)

4. Description of The Environment

The program is intended to run under Unix, in all its various flavours, including Darwin (the MacOSX version of Unix). (It has not been tested it on all variants of Unix, however.)

5. Program Specification

Not yet written.

6. Design Document

Not yet written.

7. User Manual

The blat module can be imported, or used as a standalone program.

As a standalone program, it provides a simple interface to the facilities of the module. Here is the usage outline:

        blat [-V|--version] [-r|--reload] {file/directory}*

-V|--version: This option prints the current version then exits.
-r|--reload: This option forces a reload of the state table before anything else is done.
file|directory: This option is a list of files and/or directories for copying across the network, where the current state of the network is as defined by the state table.

As an imported module, it provides a blat class. Instances of the blat class have the following methods:

probeMachine(m): Check if machine m is alive or not. Return True or False as appropriate.
commandMachine(m,c)
loadState(file='~/etc/machine.txt'): Load the current state tables from a file
saveState(file='~/etc/machine.txt'): Save the current state tables to a file.
checkMachines(): Probe all known machines to confirm their current state.
blatFile(f): Write the file f to all currently active machines.
blatDirectory(d,r): Write the contents of directory d to all currently active machines. If r is True, perform this action recursively.
catchupBlatup(m): Not yet implemented.

The program writes two logfiles:

blat.log: A list of files copied by the program, along with their date/time of modification, and destination machine. Any extraordinary events (such as unable to contact a machine that should be on-line) are also logged.
blatmissed.log: A list of files not able to be copied for whatever reason, along with their date/time of modification, and destination machine. It is expected that this program will be modified in future so that this list will be crossed examined to check whether missing machines are now back on-line, and if so, the relevant files copied. This file will be updated in this event to reflect the new status of missing files.

There is one persistent data structure, the state table. This is maintained in a text file called machines.txt, described in the next section.

8. Literate Data

This message gets used in a number of places to ensure that all modifications are focussed through this document.

<edit warning 8.1> =

# DO NOT EDIT this file! see $HOME/Computers/Sources/blat/blat.xlp instead

Chunk referenced in 9.1

The file generated by the blat.py module contains lines of data indicating the machine, its IP address, the operating system used, its current state, together with a possible empty list of aliases used by the machine.

The possible states of a machine are

online: The machine is on-line and available.
outofuse: The machine may or may not be on-line, but we are not interested in its current state. This may happen if we know the machine is out of action, and there is no point in testing it, or where the particular updates being performed are not relevant to this particular machine.
offline: The machine is not responding to ssh requests.
unknown: The state of the machine is unknown. This is the default state for all machines until established otherwise.

Note that when the machine.txt file is first loaded, the machine state is taken as presumptive, meaning that if it is online, it is assumed to still be so, unless an attempt to talk to it establishes otherwise.

"machine.txt" 8.2 =

# machine.txt file format
#machine,IPaddress,OS,state,[(alias,aliasIP),...]
albens.ajhurst.org,45.55.18.15,Linux,online
bittern,10.0.0.176,Darwin,outofuse
burnley,10.0.0.8,Linux,online
everton,10.0.0.5,Linux,offline
gorg.njhurst.com,209.181.89.22,Linux,outofuse
kerang,10.0.0.20,Linux,online
newport,10.0.0.14,Linux,online
spencer,10.0.0.120,Linux,online,[(ajh.co,59.167.194.123),(ajh.id.au,59.167.194.123)]
#ajh.id.au,59.167.194.123,Linux,outofuse # alias for spencer

This code is included as an example only. The real file is updated on each use to reflect the state of each machine, toggling "online" and "offline" as appropriate.

Machines marked as "outofuse" are not probed for activity. Comments are elided in the updated file. The IP address is currently ignored.

9. Literate Code

9.1 The blat module

"blat.py" 9.1 =

#!/home/ajh/binln/python3
#
# A python 3 program to blat copies of files and directories across the network
#
#module to provide blat facilities

<edit warning 8.1>

import argparse
import datetime
import homenorm 
import os,os.path
import re
import socket
import subprocess
import sys

thismachine=socket.gethostname() 

Chunk defined in 9.1,9.2,9.3

We need to check upon which machine this code is running. Hence the call to socket.gethostname.

"blat.py" 9.2 =

verbose=False
debug=False
version="<current version 16.3>"

norm=homenorm.normalize() # create a normalization instance
BlatLogFiles=norm.path('/home/ajh/logs/blat.log')
MissedFilesLog=norm.path('/home/ajh/logs/blatmissed.log')

def toString(fObj):
  rtn=''
  try:
    for line in fObj:
      rtn+=line
  except:
    rtn="No such file"
  return rtn

Chunk defined in 9.1,9.2,9.3

The homenorm module provides the normalize class, instances of which provide the path method, which converts paths containing the /home directory (or equivalent) between the systems used on the various machines. For example, Unix machines use /home, while MacOS (Darwin) machines use /Users. The norm object is used frequently in the code chunks defined below, viz. 9.3.

The function toString has been introduced, because in python3, returned stdout/err files are utf-8 rather than ascii. This will cause a unicode error unless converted to ascii, so the loop picks out lines in the file object, converting them to ascii and returning the result. If things do go wrong, catch the exception, and just return a 'No such file' message that will weave its way through the resultant chaos. There must be a better way ...

"blat.py" 9.3 =

# main components
<blat: define the blat class 9.6>
<blat: define the main routine 9.4>
<blat: main cli interpretation 9.5>

Chunk defined in 9.1,9.2,9.3

9.2 blat: define the main routine

<blat: define the main routine 9.4> =

def main(blatter,path):
  if verbose: print(path)
  for f in path:
    f=os.path.abspath(f)
    if os.path.isdir(f):
      blatter.blatDirectory(f,True)
    else:
      blatter.blatFile(f)

Chunk referenced in 9.3

9.3 blat: main cli interpretation

<blat: main cli interpretation 9.5> =

if __name__ == '__main__':
  skipMachines=[]
  parser=argparse.ArgumentParser(description='Blat files and directories around network')
  parser.add_argument('-V','--version',action='version',version=version)
  parser.add_argument('-r','--reload',action='store_true')
  parser.add_argument('-v','--verbose',action='store_true')
  parser.add_argument('path',nargs='*')
  args=parser.parse_args()
  if args.verbose:
    verbose=True
  else:
    verbose=False
  if verbose: print(args)
  blatter=blat()
  curState=blatter.loadState()
  if args.reload:
    machines=blatter.checkMachines()
  main(blatter,args.path)
  if verbose:
    print("\n\nList of files copied:")
    for (t,m,f) in blatter.copied:
      print(t,m,f)
  blatter.saveState()
  #print("End of blat")

Chunk referenced in 9.3

This is my first attempt at using argparse, and I have to say, it was not intuitively obvious how to use it. But I have sort-of cracked it. We first build a parser for the cli arguments, and describe it appropriately. This description is used in the help menu (blat -h) to remind the user what the program is for.

Then we add the various cli arguments that can be used:

-V --version: Print the current version number and exit.
-r --reload: Probe every machine defined in the state table to determine its current status, before performing any other action.
path[s]: Remaining parameters are assumed to be a list of files or directories for 'blatting' (copying) across the network, according to what machines are accessible.

Then we can parse the arguments/parameters, creating a namespace of parameter values inside the object args. If the -v arg is given, just print the version number and exit.

Make a blatter object and load its state. If the reload flag is given, run through every machine in the state table and check its availability. Note that machines whose state is given as outofuse are ignored.

Then we are ready to run the main routine, and following the successful completion of this, we save the current state table, and exit the program.

9.4 blat: define the blat class

<blat: define the blat class 9.6> =

class blat():
  def __init__(self):
    self.machines={} # indexed by machine
    self.copied=[] # list of tuples describing files copied
    self.comments=[] # list of comments in machine state file

  <blat: class: define the commandMachine method 9.7>
  def probeMachine(self,m):
    '''
    Run a dummy ssh call to see if machine is alive
    '''
    (r,s)=self.commandMachine(m,['echo','"OK"'])
    return r==0

  def __str__(self):
    retstr=''
    for k in self.machines:
      (machine,ipadr,system,state,aliases)=self.machines[k]
      retstr+="machine {} is at {}\n".format(machine,ipadr)
      retstr+="  running {} and state {}\n".format(system,state)
      if aliases:
        retstr+="  aliases are {}\n".format(aliases)
    return retstr

  <blat: class: define the loadState method 9.9>
  <blat: class: define the saveState method 9.10>
  <blat: class: define the checkMachines method 9.11>
  <blat: class: define the blatFile method 9.12>
  <blat: class: define the blatDirectory method 9.18>
  <blat: class: define the updateToDos method 9.19>
  def catchupBlatup(self,m):
    '''
    m is a machine to check if now alive.
    Update list of machines depending on outome.
    If alive, check log, and blat all files mentioned there.
    Delete log entry if successful blatting.
    '''
    return OK

Chunk referenced in 9.3

This class provides all the methods needed to blat files and directories across the network. A key data structure is the state table, blat.machines (the file format for which is defined in section Literate Data).

9.4.1 blat: class: define the commandMachine method

<blat: class: define the commandMachine method 9.7> =

def commandMachine(self,m,cmd):
  '''
  Run a real ssh call to perform something.
  Update machine state if different from current.
  cmd is a list of tokens to form command
  '''
  # don't need ssh if local machine
  if m==thismachine:
    perform=cmd
  else:
    perform=['ssh','-o','ConnectTimeOut=10',m]+cmd
  p=subprocess.Popen(perform,\
                     universal_newlines=True,\
                     stdout=subprocess.PIPE,\
                     stderr=subprocess.PIPE)
  errors=toString(p.stderr)
  outstr=toString(p.stdout)
  returncode=p.wait()
  #print(returncode,outstr)
  if returncode==0:
    <blat: class: commandMachine: command performed OK 9.8>
    return (returncode,outstr)
  else:
    # returncode!=0, but why?
    # check no connection failures
    if "No route to host" in outstr or 'timed out' in outstr:
      # only update table if no connection
      (machine,ipadr,system,state,aliases)=self.machines[m]
      state='offline'
      self.machines[m]=(machine,ipadr,system,state,aliases)
    # otherwise just return the non-zero return code, with err msg
    return (returncode,errors)

Chunk referenced in 9.6

This method forms the heart of all intermachine communication. It provides a mechanism to run a command on a remote machine. Either this command succeeds (when we know that the machine in question is alive, and we can update its status), or it fails, when we should also check that the status reflects the reason why it failed.

The key component in this method is the call to the subprocess method Popen. I experimented with other methods in subprocess (run, 3.5; and call, 3.3), before settling upon Popen, as this is the most generic of all, and runs on earlier versions of Python 3. But it is also more clunky, and requires more work on the part of the programmer.

The arguments for Popen are assembled, including the remote ssh call if a remote machine is accessed, and the parameters to Popen set up. The universal_newlines flag ensures that the standard out/err channels are returned as strings. The toString call then unpacks these file objects into a single string.

We then have two outcomes, depending on the value of the returncode. If it is non-zero, return the return code and the errors generated. These may be because the machine is not accessible, in which case update the machine state table. If it is zero, then we check that it is a known machine and update the state table if necessary. If not known, then add this machine to the state table, along with dummy information as necessary.

Finally, return the output from the system call.

<blat: class: commandMachine: command performed OK 9.8> =

# update machine state table if necessary
if m in self.machines:
  (machine,ipadr,system,state,aliases)=self.machines[m]
  if state!='online':
    state='online'
    self.machines[m]=(machine,ipadr,system,state,aliases)
else:
  # new machine, check if in aliases
  Found=False
  for k in self.machines:
    (machine,ipadr,system,state,aliases)=self.machines[k]
    if m in aliases:
      Found=True
      break
  if not Found:
    if verbose: print("Machine {} not in state table, adding it".format(m))
    machine=m
    ipadr=''
    system=''
    state='online'
    aliases=[]
    self.machines[m]=(machine,ipadr,system,state,aliases)

Chunk referenced in 9.7

9.4.2 blat: class: define the loadState method

<blat: class: define the loadState method 9.9> =

def loadState(self,filename='/home/ajh/etc/machine.txt'):
  filename=norm.path(filename)
  self.machines={}
  try:
    stateFile=open(filename,'r')
  except IOError:
    print("cannot open state file {}".format(filename))
    sys.exit(1)
  for machineEntry in stateFile.readlines():
    machineEntry=machineEntry.strip()
    if not machineEntry or machineEntry[0]=='#': 
      self.comments.append(machineEntry)
      continue
    res=re.match('([^,]*),([^,]*),([^,]*),([^,]*)(,\[[^]]*\])?$',machineEntry)
    if not res:
      print("Cannot parse machine state entry {}".format(machineEntry))
      sys.exit(2)
    machine=res.group(1)
    ipadr=res.group(2)
    system=res.group(3)
    state=res.group(4)
    aliases=res.group(5)
    if aliases:
      aliases=aliases[1:]
    else:
      aliases=[]
    self.machines[machine]=(machine,ipadr,system,state,aliases)
  return self.machines

Chunk referenced in 9.6

9.4.3 blat: class: define the saveState method

<blat: class: define the saveState method 9.10> =

def saveState(self,filename='/home/ajh/etc/machine.txt'):
  filename=norm.path(filename)
  try:
    stateFile=open(filename,'w')
  except IOError:
    print("cannot open state file {} for writing".format(filename))
    sys.exit(1)
  keys=list(self.machines.keys())
  keys.sort()
  for key in keys:
    (machine,ipadr,system,state,aliases)=self.machines[key]
    #print("saveState with key {0}=>{1}".format(key,self.machines[key]))
    if not ipadr and state=='online':
      # don't know IP address and can look it up!
      eth0=norm.path('/home/ajh/bin/eth0')
      (r,s)=self.commandMachine(machine,[eth0])
      if r==0:
        ipadr=s.strip()
    #print("{}".format((machine,ipadr,system,state,aliases)))
    print("{},{},{},{}".format(machine,ipadr,system,state),end='',file=stateFile)
    if aliases:
      print(",{}".format(aliases),file=stateFile)
    else:
      print("",file=stateFile)
  for c in self.comments:
    print(c,file=stateFile)
  #print("End of saveState")

Chunk referenced in 9.6

9.4.4 blat: class: define the checkMachines method

<blat: class: define the checkMachines method 9.11> =

def checkMachines(self):
  '''
  Probe across network to see what machines are live.
  Build a file containing current accessibilty data 'machine.txt'
  This file should also contain system specific info
  '''
  for k in self.machines:
    (m,i,o,s,a)=self.machines[k]
    test=self.probeMachine(m)
    if test:
      s='online'
    else:
      s='offline'
    self.machines[k]=(m,i,o,s,a)
  return self.machines

Chunk referenced in 9.6

9.4.5 blat: class: define the blatFile method

<blat: class: define the blatFile method 9.12> =

def blatFile(self,f):
  '''
  write file f to all live machines.
  any machines not active, log the attempt for later
  '''
  print("\nBlatting file {}".format(f))
  abspath=os.path.join(os.getcwd(),f)
  <blat: blatFile identify latest copy 9.13>
  <blat: blatFile identify machines with latest copy 9.14>
  <blat: blatFile get latest copy if we don't have it 9.15>
  <blat: blatFile copy out latest copies 9.16>
  <blat: blatFile log unsuccessful attempt 9.17>
  if verbose: print("all done")
  return

Chunk referenced in 9.6

This method is the raison d'etre of the whole sheboodle. The work done in copying (blatting) a file consists of

identifying the latest copy of the file,
identifying which machine(s) hold the latest copy,
getting the latest copy if not on this machine,
copying the file to all other machines, and
finally, noting which machines did not get a copy.

A task (yet to be implemented) is a mechanism to run through (from time to time) this list of un-updated machines, and copy the file if the machine becomes available.

<blat: blatFile identify latest copy 9.13> =

# first find latest copy
latest=0
times={}
for k in self.machines:
  (m,i,o,s,a)=self.machines[k]
  if s=='outofuse': continue # don't even bother!
  print("{: <20}: ".format(m),end='')
  linuxpath=norm.path(abspath,'Linux')
  darwinpath=norm.path(abspath,'Darwin')
  if o=='Linux':
    cmd=['/usr/bin/stat','-c','%Y',linuxpath]
  elif o=='Darwin':
    cmd=['/usr/bin/stat','-f','%m',darwinpath]
  else:
    print("Cannot identify machine {}'s system '{}'".format(m,o))
    cmd=[]
  (res,str)=self.commandMachine(m,cmd)
  if res==0: 
    time=int(str)
    times[m]=time
    time=datetime.datetime.fromtimestamp(time)
    print("{}".format(time))
  else:
    times[m]=0
    if 'No such file' in str:
      print('does not exist')
      s='online' # since we got a response
      self.machines[k]=(m,i,o,s,a)
    else:
      if verbose: 
        print("not available, msg={}".format(str))
      else:
        print("not available")
      s='offline' # since we got no response
      self.machines[k]=(m,i,o,s,a)
      # log this message for later attention
      logf=open(BlatLogFiles,'a')
      print('blat to machine {} returned {}'.format(m,str),file=logf)
      logf.close()
#print(times)
for k in times:
  t=times[k]
  if t>latest: latest=t;
# 'latest' is datetime of latest copy
if latest==0:
  print("Could not find a recent copy of {}.  Are you sure it exists?".format(abspath))

Chunk referenced in 9.12

<blat: blatFile identify machines with latest copy 9.14> =

# now find machines with latest copy
machines=[]
for tk in times.keys():
  if times[tk]==latest:
    machines.append(tk)
# 'machines' is list of machines with latest copy
lateststr=datetime.datetime.fromtimestamp(latest)
if verbose:
  print("\nlatest copy is at {} on machine(s) {}".format(lateststr,machines))
  print("we are on %s" % (thismachine))

Chunk referenced in 9.12

<blat: blatFile get latest copy if we don't have it 9.15> =

# make sure we have latest copy
if thismachine not in times:
  print("This machine ({}) not in machines.txt".format(thismachine))
  sys.exit(1)
localmachine=self.machines[thismachine]
localsystem=localmachine[2]
localpath=norm.path(abspath,localsystem)
#print("localsystem={},localpath={}".format(localsystem,localpath))
if thismachine not in machines and times[thismachine]<latest:
  print("getting latest copy from %s to here ..." % (machines[0]))
  latestmachine=self.machines[machines[0]]
  latestsystem=latestmachine[2]
  print("latestsystem={}".format(latestsystem))
  latestpath=norm.path(abspath,latestsystem)
  # do xfer of f from machine to this machine
  cmd=['/usr/bin/rsync','-auv','%s:%s' % (machines[0],latestpath), localpath]
  if verbose: print(' '.join(cmd),' =>', )
  stat=subprocess.check_output(cmd,universal_newlines=True,stderr=subprocess.PIPE)
  if verbose: print('OK')
  times[thismachine]=latest

Chunk referenced in 9.12

<blat: blatFile copy out latest copies 9.16> =

# now copy the latest version to thise machines with out-of-date copies
print("")
for machine in times.keys():
  (m,i,o,s,a)=self.machines[machine]
  if s!='online':
    # must be up-to-date, since it has a times entry
    # but it is not online, so skip
    if verbose: print("Skipping machine {} from copy".format(m))
    continue
  targetpath=norm.path(abspath,o)
  t=times[machine]
  if t<latest:
    print("copying to machine {: <20} ...".format(machine),end='',flush=True)
    # do xfer of f from this machine to machine    
    # these xfers are dodgy, depending upon several things, so do try
    try:
      # first ensure that target directory exists
      d=os.path.dirname(targetpath)
      cmd=['mkdir','-p',d]
      (res,str)=self.commandMachine(m,cmd)
      if debug: print("mkdir returns:{},{}".format(res,str))
      if res!=0: # log all errors for later analysis
        logf=open(BlatLogFiles,'a')
        print("machine {} mkdir returned msg {}".format(m,str),file=logf)
        logf.close()
      # now do the real copy
      cmd=['/usr/bin/rsync','-auv',localpath,'%s:%s' % (machine,targetpath)]
      #print(' '.join(cmd),' =>', )
      stat=subprocess.check_output(cmd,universal_newlines=True,stderr=subprocess.PIPE)
      #(res,str)=self.commandMachine(m,cmd)
      #print("rsync returns:",stat)
      print('OK')
      copy=(lateststr,m,abspath)
      self.copied.append(copy)
    except subprocess.CalledProcessError as err:
      print("Oops!")
      print("Could not copy %s to machine %s, reason:" % (abspath,machine))
      print(err)

Chunk referenced in 9.12

<blat: blatFile log unsuccessful attempt 9.17> =

for k in self.machines:
  (m,i,o,s,a)=self.machines[k]
  if s!='online':
    # could not/did not copy file to this machine
    synchFails=open(MissedFilesLog,'a')
    synchFails.write('{0}  {1}  {2}\n'.format(lateststr,m,abspath))
    #print("writing {0} {1} {2} to log".format(lateststr,m,abspath))
    synchFails.close()

Chunk referenced in 9.12

9.4.6 blat: class: define the blatDirectory method

<blat: class: define the blatDirectory method 9.18> =

def blatDirectory(self,d,r):
  '''
  write all files in d to all live machines.
  If r True, recursively visit nested directories
  '''
  if verbose: print("\nBlatting directory {}".format(d))
  dirlist=os.scandir(d)
  for direntry in dirlist:
    relpath=direntry.path
    if direntry.is_file():
      self.blatFile(relpath)
    elif r and direntry.is_dir():
      self.blatDirectory(relpath,r)
  return

Chunk referenced in 9.6

9.4.7 blat: class: define the updateToDos method

<blat: class: define the updateToDos method 9.19> =

def updateToDos(self):
  datepat='(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})'
  machinepat='([\w\.]+)'
  filenamepat='([\w/\.]+)'
  pat=re.compile(datepat+' +'+machinepat+' +'+filenamepat)
  updates={}
  f=open(BlatLogFiles,'r')
  for l in f.readlines():
    l=l.strip()
    #print(l)
    res=re.match(pat,l)
    if res:
      year=int(res.group(1))
      month=int(res.group(2))
      day=int(res.group(3))
      hour=int(res.group(4))
      min=int(res.group(5))
      sec=int(res.group(6))
      thisdt=datetime.datetime(year,month,day,hour,min,sec)
      machine=res.group(7)
      filename=res.group(8)
      #print(thisdt,machine,filename)
      if not machine in updates:
        updates[machine]=[]
      if (thisdt,filename) not in updates[machine]:
        updates[machine].append((thisdt,filename))
    else:
      print("could not match {}".format(l))
  # now sort lists for each machine
  for m in updates:
    l=updates[m]
    l.sort()
    updates[m]=l
  return updates
def updateFileRequests(self,updates):
  for mac in updates:
    if mac in self.machines:
      (m,i,o,s,a)=self.machines[mac]
      if s=='online':
        newlist=[]
        for (d,f) in updates[mac]:
          print("copy file {} to machine {}".format(f,mac))
        updates[mac]=[]
    else:
      print("Machine {} still offline".format(mac))
  return updates

Chunk referenced in 9.6

9.5 The scanlogs.py program

"scanlogs.py" 9.20 =

import datetime
import re

LogFileName='/home/ajh/logs/blatmissed.log'
LogFile=open(LogFileName,'r')
dt='(\d{4})-(\d{2})-(\d{2}) +(\d{2}):(\d{2}):(\d{2})'
mc=' +(\w+) +'
name='([A-Za-z0-9/_.-]+)$'
pat=dt+mc+name
machines={}

count=0
<scanlogs: build data structure of files from blatmissing 9.21>
#print(machines)
print(f"{count=}") # note: Python 3.10ff only

<scanlogs: restructure data to identify most recent files 9.22>
# now scan the tosave data, with an escape on keyboard interrupt,
# which stops the saving, and logs a new (truncated) missing log file.
# (code to follow)

The scanlogs.py program is a start on the problem of addressing those files not blatted due to the target machines being unavailable. It builds a directory of machines, where each machine has a (nested) directory of files, and each file has a list of associated attempts to disseminate them. These lists are then sorted by date order, and the most recent attempt is followed up (currently manually).

9.5.1 scanlogs: build data structure of files from blatmissing

<scanlogs: build data structure of files from blatmissing 9.21> =

for l in LogFile.readlines():
  #print(l)
  res=re.match(pat,l)
  if res:
    yr=int(res.group(1))
    mo=int(res.group(2))
    dy=int(res.group(3))
    hr=int(res.group(4))
    mi=int(res.group(5))
    se=int(res.group(6))
    dattim=datetime.datetime(yr,mo,dy,hour=hr,minute=mi,second=se)
    machine=res.group(7)
    filen=res.group(8)
    #print(f"{dattim.strftime('%Y%m%d:%H%M%S')} {machine} {filen}")
    if machine in machines:
      if filen in machines[machine]:
        machines[machine][filen].append(dattim)
      else:
        machines[machine][filen]=[dattim]
    else:
      machines[machine]={}
      machines[machine][filen]=[dattim]
    count+=1
  else:
    print(f"Could not match {l}")

Chunk referenced in 9.20

9.5.2 scanlogs: restructure data to identify most recent files

<scanlogs: restructure data to identify most recent files 9.22> =

mkeys=list(machines.keys())
mkeys.sort()
tosave=[]
for m in mkeys:
  print(f"{m}\n{'-'*len(m)}")
  fkeys=list(machines[m].keys())
  fkeys.sort()
  for f in fkeys:
    times=machines[m][f]
    times.sort()
    t=times[-1]
    print(f"  {t.strftime('%Y%m%d:%H%M%S')}  {f}")
    saveentry=(t,m,f)
    tosave.append(saveentry)

Chunk referenced in 9.20

10. Makefile

This uses my standard literate makefile structure, and relies heavily upon the generic package MakeXLP to build all the tangled and woven parts.

"Makefile-blat" 10.1 =

default=blat.tangle
include ${HOME}/etc/MakeXLP

html: blat.html

blat.py: blat.tangle

blat: blat.py blat.tangle
	cp blat.py blat
	chmod 755 blat

install-blat: blat
	cp blat ${HOME}/bin/
	cp blat.py ${HOME}/lib/python/
	touch install-blat

11. Literate Tests

I used a very simple test script. Since this code is generated using a literate program, I used cycles of the following two commands to test:

        $ make blat
        $ blat blat.xlp

with appropriate checking of the state table after step two.

12. TODOs

Blat does not handle file names with embedded blanks - need to fix!
Add an option to blat all files newer than the time of the last blat.
When blatting a directory, should use scandir to collect file data on all contained files and directories.
List known instances as most recent first

13. Bibliography

Nothing appropriate.

14. Glossary

Nothing appropriate.

15. Indices

15.1 Chunk Names

Chunk Name	Defined in	Used in
blat: blatFile copy out latest copies	9.16	9.12
blat: blatFile get latest copy if we don't have it	9.15	9.12
blat: blatFile identify latest copy	9.13	9.12
blat: blatFile identify machines with latest copy	9.14	9.12
blat: blatFile log unsuccessful attempt	9.17	9.12
blat: class: commandMachine: command performed OK	9.8	9.7
blat: class: define the blatDirectory method	9.18	9.6
blat: class: define the blatFile method	9.12	9.6
blat: class: define the checkMachines method	9.11	9.6
blat: class: define the commandMachine method	9.7	9.6
blat: class: define the loadState method	9.9	9.6
blat: class: define the saveState method	9.10	9.6
blat: class: define the updateToDos method	9.19	9.6
blat: define the blat class	9.6	9.3
blat: define the main routine	9.4	9.3
blat: main cli interpretation	9.5	9.3
current author	16.2
current comment	16.4
current date	16.1
current version	16.3	9.2
scanlogs: build data structure of files from blatmissing	9.21	9.20
scanlogs: restructure data to identify most recent files	9.22	9.20

15.2 Identifiers

Identifier	Defined in	Used in

16. Maintenance History

16.1 Previous status of this document

20160910	ajh	1.0	provide file synchronization primitives
20160912	ajh	1.1	add outline of synchronization process
20160915:111941	ajh	1.2	add actual synchronization calls
20170103:175225	ajh	2.0.0	start conversion to python 3, with new module 'blat'
20170107:131318	ajh	2.1.0	converted subprocess.run to subprocess.call, to preserve backwards compatibility through earlier versions of Python 3
20170109:164708	ajh	2.1.1	fix version option in accordance with argparse semantics
20170116:145456	ajh	2.1.2	add initial update mechanisms
20170211:164410	ajh	2.1.3	revised non-verbose printing
20170215:145950	ajh	2.1.4	further revision of non-verbose printing
20170426:163925	ajh	2.1.5	kludge around utf-8/ascii problems in 'toString'
20200615:130219	ajh	2.1.6	separate logging of missed files in separate log, preparatory to adding some form of "catch-up" when machines become available
<current date 16.1>	ajh	<current version 16.3>	<current comment 16.4>

16.2 Current status of this document:

<current date 16.1> = 20230319:154358

<current author 16.2> = ajh

<current version 16.3> = 2.1.7

Chunk referenced in 9.2

<current comment 16.4> =

        remove reference to obsolete machine hamilton,
        add comments in machine state to end of saved state
      

Global Synchronization of Files

Version 2.1.7

John Hurst

Table of Contents

1. Abstract

2. Introduction

3. More Detailed Description

4. Description of The Environment

5. Program Specification

6. Design Document

7. User Manual

8. Literate Data

9. Literate Code

9.1 The blat module

9.2 blat: define the main routine

9.3 blat: main cli interpretation

9.4 blat: define the blat class

9.4.1 blat: class: define the commandMachine method

9.4.2 blat: class: define the loadState method

9.4.3 blat: class: define the saveState method

9.4.4 blat: class: define the checkMachines method

9.4.5 blat: class: define the blatFile method

9.4.6 blat: class: define the blatDirectory method

9.4.7 blat: class: define the updateToDos method

9.5 The scanlogs.py program

9.5.1 scanlogs: build data structure of files from blatmissing

9.5.2 scanlogs: restructure data to identify most recent files

10. Makefile

11. Literate Tests

12. TODOs

13. Bibliography

14. Glossary

15. Indices

15.1 Chunk Names

15.2 Identifiers

16. Maintenance History

16.1 Previous status of this document

16.2 Current status of this document: