Tardis3

John Hurst

Version 2.1.1

20200825:174425

Related Programs

camera3 - the general collection of camera operations;
download3 - collect image files from camera;
filePics - (this program has been decommissioned);
photo3 - process images in directory to build web pages;
tardis3 - change time and date stamps on processed photos (this document).

1	User Manual
2	The Main Program
	2.1	The Main Routine
	2.2	Setup
	2.3	Define miscellaneous subroutines
	2.4	Collect the Command Line Options
	2.5	Perform the Top Level Visit
	2.6	write list file
	2.7	Plan for new data structure
3	The visit routine
	3.1	check if directory is standard date structure, exit if not
	3.2	initialize variables for visit routine
	3.3	open index.xml and write header
	3.4	get information from album XML file
	3.5	write tree element to index.xml
	3.6	sort fnames into directories and images
	3.7	Visit All Directories
	3.8	scan all images and process them
	3.9	wind up index file
4	Descriptions
	4.1	read descriptions file and create dictionary
	4.2	wind up descriptions file
5	the addToIndex routine definition
6	the addToList routine definition
7	the makeImage routine
8	the makeSubImages routine
	8.1	New SubImage generalized size definitions
9	the retrieve support routines definition
10	Makefile
11	TODOs
12	Indices
13	Document History

1. User Manual

tardis3.py is a Python program that adjusts the time settings on images generated by the photo.py. It is designed to correct errors caused by incorrect time settings on the camera, and after the web pages have been generated by photo.py.

There are four key parameters to the program:

The start time of photos to be corrected
The end time of photos to be corrected
The correction to be applied to each photo so selected
The camera model to be corrected

Starting and ending times are specified in the format YYYYMMDD:HHMMSS where YYYY is the year, MM the month, DD the day, HH the hour, MM the minute, and SS the second respectively.

The correction time has a similar format, but leading and trailing components may be optional. If the ':' is omitted, the first digits are assumed to be hours, followed by minutes and seconds. If minutes or seconds are specified, then each of hours, minutes and seconds must be pairs of digits.

If the ':' is given, then the preceding digits are interpreted as days, days, months and days, and years/months/days respectively, according to the actual number of digits present.

2. The Main Program

"tardis3.py" 2.1 =

<interpreter definition 2.3>
<banner 2.4>
<basic usage information 2.5>
<todos 2.6>
<imports 2.7>
<global variables 2.8>
<initialization 2.9,2.10,2.11,2.12,2.13,2.14,2.15,4.1,8.1>
<define the usage subroutine 2.16>
<define miscellaneous subroutines 2.17,2.18,2.19,2.20,2.25,3.7>
<the addToList routine definition 6.1>
<the makeImage routine definition 7.1>
<the makeSubImages routine definition 8.2>
<the addToIndex routine definition 5.1>
<the visit routine definition 3.1>
<the retrieve routine definition 9.2>
<write list file 2.26> **** Chunk omitted!
<main routine 2.2>

2.1 The Main Routine

<main routine 2.2> =

def main():
  global debug,startDT,endDT,correction,model
  <collect the command line options 2.21>
  <perform the top level visit 2.22>
  pass
if __name__=="__main__":
  main()

Chunk referenced in 2.1

2.2 Setup

<interpreter definition 2.3> = #! /home/ajh/binln/python3

Chunk referenced in 2.1

<banner 2.4> =

########################################################################
#                                                                      #
#                          t a r d i s 3 . p y                         #
#                                                                      #
########################################################################

Chunk referenced in 2.1

<basic usage information 2.5> =

#       A program to correct time stamps on photo album web page images.
#
#       John Hurst
#       version <current version 13.1>, <current date 13.2>
#
# This program recursively scans a set of directories (given as
# command line parameters) for images that comprise a photo album to
# be rendered into a set of web pages.  A file 'index.xml' is created
# in each (sub)directory, containing a list of images and
# subdirectories.  This file can be rendered by an XML stylesheet
# 'album.xsl' into HTML pages.

Chunk referenced in 2.1

<todos 2.6> =

# TODO
#  20150914 none to date
#  20200825:163744 reverse scan through photos when incrementing time
#  20200825:163810 add model "*" to update all photos in range
# 

Chunk referenced in 2.1

<imports 2.7> =

import cgi
import datetime
#import EXIF
from PIL import Image
from PIL.ExifTags import TAGS
import getopt
import hashlib
import os
import os.path
import re
import stat
import subprocess
import sys
import time
import xml.dom
from xml.dom import Node
from xml.dom.minidom import parse
from xml.parsers.expat import ExpatError

Chunk referenced in 2.1

<global variables 2.8> =

startDT=datetime.datetime(datetime.MINYEAR,1,1,0,0,0)
endDT=datetime.datetime(datetime.MAXYEAR,12,31,23,59,59)
correction=None
model=None
forceXmls=thumbsOnly=False; recurse=large=True
debug=False
# define the full model names to allow shortcuts
fullModelNames=["Canon PowerShot SX230 HS","Canon EOS 600D","DSC-HX90V"]

Chunk referenced in 2.1

<initialization 2.9> =

indexdict={}
debug=0

Chunk referenced in 2.1
Chunk defined in 2.9,2.10,2.11,2.12,2.13,2.14,2.15,4.1,8.1

Initialize the debug flag. This is currently set manually, but eventually we expect to make it a command line option.

<initialization 2.10> =

lists = {}

Chunk referenced in 2.1
Chunk defined in 2.9,2.10,2.11,2.12,2.13,2.14,2.15,4.1,8.1

Initialize the variable 'lists'. This is a python directory, where the keys are (os) directories paths rooted at the album subdirectory (the cli parameters), and the entries are lists of image file names within that directory. This is used to build a list of images within each (os) directory traversed. The list/directory is initially empty.

<initialization 2.11> =

descriptions={}
sounds={}
protected={}

Chunk referenced in 2.1
Chunk defined in 2.9,2.10,2.11,2.12,2.13,2.14,2.15,4.1,8.1

Initialize the variable 'descriptions'. This is a python directory, where the keys are image file names, and the entries are descriptions/captions for the image concerned. It is built from the eponymously named file 'descriptions' in the directory currently being scanned.

<initialization 2.12> =

ALBUMXSL="file:///home/ajh/lib/xsl/album.xsl"

Chunk referenced in 2.1
Chunk defined in 2.9,2.10,2.11,2.12,2.13,2.14,2.15,4.1,8.1

Define the XSL script used to render all XML files built by this program.

<initialization 2.13> =

ignorePat=re.compile("(.*(_\d+x\d+)\..*$)|(.xvpics)")
medmatch=re.compile(".*_640x480.JPG")
bigmatch=re.compile(".*_1600x1200.JPG")
thumbmatch=re.compile(".*_128x128.JPG")
identifyPat = re.compile(".* (\d+)x(\d+)( |\+).*")
treepat=re.compile('.*/(200[/\d]+)')

Chunk referenced in 2.1
Chunk defined in 2.9,2.10,2.11,2.12,2.13,2.14,2.15,4.1,8.1

<initialization 2.14> =

maxmissing = 0
maxdir = ''

Chunk referenced in 2.1
Chunk defined in 2.9,2.10,2.11,2.12,2.13,2.14,2.15,4.1,8.1

maxmissing, maxdir are variables that track the directory with the maximum number of missing descriptions. This is purely a convenient device to point the user to where most documenting effort is required!

<initialization 2.15> =

(sysname,nodename,release,version,machine)=os.uname()
if sysname=='Linux':
  EXIV2='/usr/bin/exiv2'
  IDENTIFY='/usr/bin/identify'
  CONVERT='/usr/bin/convert'
elif sysname=='Darwin':
  EXIV2='/usr/local/bin/exiv2'
  IDENTIFY='/sw/bin/identify'
  CONVERT='/sw/bin/convert'
else:
  print("You have not yet programmed this module for use on {} systems".format(sysname))

Chunk referenced in 2.1
Chunk defined in 2.9,2.10,2.11,2.12,2.13,2.14,2.15,4.1,8.1

Setup system-specific constants.

2.3 Define miscellaneous subroutines

<define the usage subroutine 2.16> =

def usage():
  print('''usage:
    tardis3.py [-d] [ -h | -s starttime -e endtime -c correction -m model directory]
    where:
      -d : print debug information
      -h : print usage information (this message)
      -s : set starttime in format YYYMMDD(':'|'-')[HH[MM[SS]]]
      -e : set endtime in format YYYMMDD(':'|'-')[HH[MM[SS]]]
      -c : set correction in format ('+'|'-') [D+(':'|'-')] [HH[MM[SS]]]
      -m : camera model number (as given in the exif file)
      
    In the start and end times, the year/month/day are mandatory, while
    hours/minutes/seconds are optional.  To avoid ambiguity, if
    seconds are given, minutes must be specified, and if minutes
    are given, hours must be specified.  The days/hours separator
    can be either a colon or a minus.

    Example:
    -s 20160803-142351 -> starttime is datetime(2016,8,3,14,23,51)
    -e 20140205:0704 -> endtime is datetime(2014,2,5,7,4)
    -e 20140205:07   -> endtime is datetime(2014,2,5,7,0)
    (Note that seconds are optional in a datetime instance)

    In the correction time, if days are given, then a trailing colon or minus
    must be used.  Again, trailing seconds/minutes can only be specified
    if the preceding quantity is specified.  If both days and hours
    are omitted, no correction is made.

    Example:
      -c +01:     add one day to the exif time stamp and file name
      -c +01-     add one day to the exif time stamp and file name
      -c -01      subtract one hour from the time stamp and file name
      -c +01-0230 add one day and two and a half hours (26.5 hours)
      -c +2630    add one day and two and a half hours (26.5 hours)
  ''')

Chunk referenced in 2.1

<define miscellaneous subroutines 2.17> =

dtPat=re.compile('(\d{4})(\d{2})(\d{2})((:|-)(\d{2})((\d{2})(\d{2})?)?)?')
def str2DT(strDT):
  res=dtPat.match(strDT)
  if res:
    hour=min=sec=0
    year=int(res.group(1))
    month=int(res.group(2))
    day=int(res.group(3))
    if res.group(6):
      hour=int(res.group(6))
      if res.group(8):
        min=int(res.group(8))
        if res.group(9):
          sec=int(res.group(9))
    dt=datetime.datetime(year,month,day,hour,min,sec)
  else:
    return None
  return dt

Chunk referenced in 2.1
Chunk defined in 2.17,2.18,2.19,2.20,2.25,3.7

The str2DT routine converts a string into a date and time. The string must be in the format as described in the <define the usage subroutine > statement above. Note that the date and hours separator can be either a colon or a minus sign.

<define miscellaneous subroutines 2.18> =

deltaPat=re.compile('(\+|\-)(\d+(:|-))?((\d{1,2})((\d{2})(\d{2})?)?)?')
def str2Delta(strDT):
  res=deltaPat.match(strDT)
  dt=datetime.timedelta()
  if res:
    day=hour=min=sec=0
    if res.group(1):
      sign=res.group(1)
    else:
      sign='+'
    if res.group(2):
      day=int(res.group(2)[0:-1])
      if res.group(5):
        hour=int(res.group(5))
        if res.group(7):
          min=int(res.group(7))
          if res.group(8):
            sec=int(res.group(8))
    else:
      # no day field
      if res.group(5):
        hour=int(res.group(5))
        if res.group(7):
          min=int(res.group(7))
          if res.group(8):
            sec=int(res.group(8))
    dt=datetime.timedelta(day,sec,0,0,min,hour)
    if sign=='-':
      dt = -dt
  return dt
    

Chunk referenced in 2.1
Chunk defined in 2.17,2.18,2.19,2.20,2.25,3.7

<define miscellaneous subroutines 2.19> =

jpgPat=re.compile('(.*)\.(j|J)(p|P)(g|G)$')
def trimJPG(name):
    res=jpgPat.match(name)
    if res:
      name=res.group(1)
    return name

Chunk referenced in 2.1
Chunk defined in 2.17,2.18,2.19,2.20,2.25,3.7

<define miscellaneous subroutines 2.20> =

def attrString(el):
  if el.hasAttributes():
    str=''
    nl=el.attributes
    if debug: print("looking at attributes of length %d" % (nl.length))
    for i in range(nl.length):
      attr=nl.item(i)
      str+=' %s="%s"' % (attr.name,attr.value)
    return str
  else:
    return ''
  
def flatString(elem):
  if elem.nodeType==Node.TEXT_NODE:
    return elem.nodeValue
  elif elem.nodeType==Node.ELEMENT_NODE:
    attrs=attrString(elem)
    strng=""
    for el in elem.childNodes:
      strng+=strng+flatString(el)
    print("DEBUG!!: flatString returns {}, is this correct?".format(strng))
    return "<%s%s>%s</%s>" % (elem.tagName,attrs,strng,elem.tagName)
  else:
    return "[unknown nodeType]"

def flatStringNodes(nodelist):
  strng=''
  for node in nodelist:
    strng+=flatString(node)
  return strng

def get_exif(fn):
  ret = {}
  i = Image.open(fn)
  info = i._getexif()
  if info:
    for tag, value in info.items():
      decoded = TAGS.get(tag, tag)
      ret[decoded] = value
  return ret

Chunk referenced in 2.1
Chunk defined in 2.17,2.18,2.19,2.20,2.25,3.7

2.4 Collect the Command Line Options

There are five command line options:

v: show current version
d: turn on Debugging
s: Specify the Starting date and time of the first image to be corrected (inclusive).
e: Specify the Ending date and time of the last image to be corrected (inclusive).
c: Specify the time Correction (+ve to advance time, -ve to retard time).
m: Specify the camera Model involved.

<collect the command line options 2.21> =

(opts,args) = getopt.getopt(sys.argv[1:],'dhs:e:c:m:v')
correction=None
for (option,value) in opts:
  if option == '-h':
    usage()
    sys.exit(0)
  if option == '-d':
    debug=True
  elif option == '-s':
    startDTstr=value
    startDT=str2DT(startDTstr)
    if not startDT:
      startDT=datetime.datetime.now()
    startTuple=(startDT.year,startDT.month,startDT.day,\
                startDT.hour,startDT.minute,startDT.second)
    print("start datetime = %s" % startDT)
  elif option == '-e':
    endDTstr=value
    endDT=str2DT(endDTstr)
    if not endDT:
      endDT=datetime.datetime.now()
    print("end datetime   = %s" % (endDT))
  elif option == '-c':
    correctionstr=value
    correction=str2Delta(correctionstr)
    print("correction datetime=%s" % (correction))
  elif option == '-m':
    modelstr=value; model=''
    count=0
    for ms in fullModelNames:
      if modelstr in ms:
        model=ms
        count+=1
    if count==0:
      model=modelstr
    elif count==1:
      pass # have unique model
    else:
      print("Ambiguous model number {}".format(modelstr))
      sys.exit(1)
    print("Model is {}".format(model))
  elif option == '-v':
    print("<current version 13.1>")
    sys.exit(0)
if correction==None:
  print("You have not specified a correction value!")
  sys.exit(1)

Chunk referenced in 2.2

If a model is specified, we check to see if it is an abbreviation for a known model. This is done against the global variable fullModelNames. We also check that the abbreviation is unique, otherwise there may be some confusion as to what model is actually updated! If no abbreviation is found, the supplied parameter is used verbatim.

2.5 Perform the Top Level Visit

We visit every directory given as arguments to the program invocation. If no arguments are given, then visit the current directory.

All the real work of visiting a directory and computing the index XML files is done by the visit procedure, which returns a tuple of potentially changed values that are of relevance to higher level directory index XML files. These are, respectively: the title of the visited directory; the number of photos or images contained in the directory and its sub-directories; the number of albums ditto; the thumbnail to characterize the directory; the description of the directory; and the number of missing descriptions in the directory and its sub-directories.

These values are then propogated to higher level photo albums (directories) via the chunk <explore next higher level to update counts >, which also recomputes updated values of the tuple described above.

<perform the top level visit 2.22> =

visitdirs = args

# if there are no directories to visit, include at least current directory
if len(visitdirs)==0:
  visitdirs = ['.']

for dir in visitdirs:
  checkdir = os.path.abspath(dir)
  (h,t) = os.path.split(checkdir)
  (titl,nph,nalb,thm,descr,nmiss)=visit(0,checkdir,lists)
  atTop=False
  <explore next higher level to update counts 2.23> **** Chunk omitted!

Chunk referenced in 2.2

<explore next higher level to update counts 2.23> =

while not atTop:
  reldir=os.path.relpath(checkdir)
  msg = "Directory %s has %d images " % (reldir,nph)
  msg += "and %d missing descriptions" % (nmiss)
  print(msg)
  try:
    higherdir=checkdir+'/..'
    highername=higherdir+'/index.xml'
    higherindex=open(highername)
    higherdom=parse(higherindex)
    higherindex.close()
    if debug: print("parsed %s" % (highername))
    direlements=higherdom.getElementsByTagName('directory')
    countalbums=countphotos=countmissing=0
    maxmissing=0;maxdir=""
    <scan all directory elements, updating counts 2.24>
    (titl,nph,nalb,thm,descr,nmiss)=\
        ('',countphotos,countalbums,'','',countmissing)
    try:
      summary=higherdom.getElementsByTagName('summary')[0]
      summary.setAttribute('photos',"%d" % (nph))
      summary.setAttribute('albums',"%d" % (nalb))
      summary.setAttribute('missing',"%d" % (nmiss))
      summary.setAttribute('maxmissing',"%d" % (maxmissing))
      summary.setAttribute('maxdir',maxdir)
    except:
      pass # no summary to worry about
    newxml=higherdom.toxml()
    if debug: print(newxml)
    higherindex=open(highername,'w')
    higherindex.write(newxml)
    higherindex.close()
    checkdir=os.path.abspath(higherdir)
    (h,t) = os.path.split(checkdir)
  except IOError:
    #print("No higher level index.xml file")
    atTop=True
  except ExpatError:
    print("XML error in higher level index.xml file")
    atTop=True

Chunk referenced in 2.22

The next higher directory is identified and the index.xml file parsed. This is used to identify all the directory elements, which are scanned to update the key parameters. The title, thumbnail and descriptions are reset, since these will not change in the higher level summaries.

The summary element is then recomputed, and the index.xml file rewritten.

Note that this rewriting may not be necessary if none of the key variables have changed. This is not currently tested.

<scan all directory elements, updating counts 2.24> =

for direl in direlements:
    if direl.getAttribute('name')==t:
        if debug: print("found directory %s!" % (t))
        if titl:
          direl.setAttribute('title',titl)
        direl.setAttribute('albums',"%d" % nalb)
        direl.setAttribute('images',"%d" % nph)
        if thm:
          direl.setAttribute('thumb',t+'/'+thm)
        if descr:
          direl.setAttribute('description',descr)
        if nmiss==0:
          if direl.hasAttribute('missing'):
            direl.removeAttribute('missing')
        else:
          direl.setAttribute('missing',"%d" % nmiss)
    countalbums+= collectAttribute(direl,'albums')
    countalbums+= 1 # include one album for this directory
    countphotos+= collectAttribute(direl,'images')
    miss=collectAttribute(direl,'missing')
    countmissing+=miss
    if miss>maxmissing:
      maxmissing=miss
      maxdir=direl.getAttribute('name')

Chunk referenced in 2.23

<define miscellaneous subroutines 2.25> =

def collectAttribute(d,a):
    if d.hasAttribute(a):
        v=d.getAttribute(a)
        try:
            return(int(v))
        except:
            print("Cannot convert attribute '%s' with value '%s'" % (a,v))
            return(-1)
    else:
        return 0

Chunk referenced in 2.1
Chunk defined in 2.17,2.18,2.19,2.20,2.25,3.7

2.6 write list file

<write list file 2.26> =

keys=lists.keys()
for k in keys:
    try:
        listn=k+"/list"
        listf=open(listn,"w")
    except:
        print("Cannot open %s" % (listn))
        continue
    for a in lists[k]:
        #listf.write(a+"\n")
    listf.close()

Chunk referenced in 2.1

Open the file list in each of the directories visited, and write a list of images found in that directory to the file. This may not be entirely accurate, and needs to be confirmed that it does work correctly. It has not been used recently, and may be superflous to needs. It has now been commented out (v1.2.3).

2.7 Plan for new data structure

In order to develop this program further, it is suggested that a full data structure for each image and album be developed. This data structure would be populated in some way, and would then provide the data for writing the index.xml file. How is it populated?

from the current index.xml file
from the album.xml file
from the image file
from the descriptions file
from the command line (?)

There would be a list of such data components for each directory visited (hence a local variable to routine visit). This list could be sorted on one or more of its attributes to provide a range of views.

Components of the new data structure:

name	the name of the image or album
title	a title or caption for the image/album
date/time	the date and time of the image
threads	(new theme) a list of thread names
shutter/aperture	shutter speed and aperture opening
film speed
flash used

3. The visit routine

<the visit routine definition 3.1> =

def visit(level,dirname, lists):
    global descriptions, sounds, protected, indexdict
    <check if directory is standard date structure, exit if not 3.2> **** Chunk omitted!
    <initialize variables for visit routine 3.3>
    <read descriptions file and create dictionary 4.2>
    fnames=os.listdir(".")
    fnames.sort()
    dirs=[];
    <sort fnames into directories and images 3.8>
    <open index.xml and write header 3.4>
    <get information from album XML file 3.5>
    reldir=os.path.relpath(dirname)
    indent='  '*level
    absdir=os.path.abspath(reldir)
    print('%sScanning directory %s (relative "%s", title "%s"' % \
        (indent,absdir,reldir,title))
    <write tree element to index.xml 3.6>
    if not thumb:
        if len(images)>0:
            thumb=images[0]
    if thumb[-4:len(thumb)].lower()=='.jpg':
        thumb=thumb[0:-4]
    <visit all directories 3.9>
    indexout+='  <thumbnail>%s</thumbnail>\n' % (thumb)
    #sys.stdout.write("%d" % (len(images)))
    <scan all images and process them 3.10>
    indexout+='  <summary photos="%d" albums="%d" missing="%d" \
        maxmissing="%d" maxdir="%s" minmissing="%d" mindir="%s"/>\n' % \
        (nphotos,nalbums,missingdescrs,maxmissing,maxdir,minmissing,mindir)
    <wind up index file 3.13>
    <wind up descriptions file 4.3>
    os.chdir(saved)
    return (title,nphotos,nalbums,thumb,description,missingdescrs)

Chunk referenced in 2.1

3.1 check if directory is standard date structure, exit if not

<check if directory is standard date structure, exit if not 3.2> =

res=re.match('.*(20\d\d)(/(\d\d)(/(\d\d))?)?.*',dirname)
print("checking dirname %s ..." % (dirname))
if res:
  print(res.groups())
  month=day=0
  year=int(res.group(1))
  if res.group(2):
    month=int(res.group(3))
  else:
    if year < startDT.year:
      return (None,None,None,None,None,None)
    else:
      print("year %d OK, continuing ..." % (year))
  if res.group(4):
    day=int(res.group(5))
  else:
    print(year,startDT.year,month,startDT.month)
    if month > 0 and year==startDT.year and month < startDT.month:
      return (None,None,None,None,None,None)
    else:
      print("month %d OK, continuing ..." % (month))
else:
  print("Directory %s not date format, ignored" % (dirname))
  return (None,None,None,None,None,None)

Chunk referenced in 3.1

3.2 initialize variables for visit routine

<initialize variables for visit routine 3.3> =

# initialize variables for visit routine
nphotos=nalbums=missingdescrs=gotdescrfile=0
listmissingimages=[]
albumdom=None; title=""; thumb=description=""
maxmissing=0; maxdir=""; minmissing=999999; mindir=""
saved = os.getcwd()
try:
  os.chdir(dirname)
except OSError:
  print("*** Cannot open directory %s - do you have the correct path?" % (dirname))
  sys.exit(1)
mdescr=-1
images=[]

Chunk referenced in 3.1

nphotos: the number of image files found in this subdirectory (album)
nalbums: the number of subalbums (subdirectories containing images) found in this subdirectory
missingdescrs: the number of images for which no description was found in the descriptions file
gotdescrfile: set if the descriptions file was succesfully read
listmissingimages: a list of those images for which a description was found, but no image file
albumdom: the XML-parsed album file
title: the title of this album (subdirectory)
thumb: the thumbnail image used to represent this album
description: the description of this album
maxmissing
maxdir
saved
mdescr
images

3.3 open index.xml and write header

<open index.xml and write header 3.4> =

try:
  indexf=open("index.xml","r")
  indexin=indexf.read()
  indexf.close()
  #print("read index.xml")
  indexdom=parse("index.xml")
except:
  indexin=''; indexdom=None
#print("parsed index.xml")
#print(indexdom.toprettyxml())
#print("HELP")
if indexdom:
  indexdict={}
  indexarr=indexdom.getElementsByTagName('image')
  for image in indexarr:
    #print(image.toprettyxml())
    if image.hasAttribute('gps'):
      gps=image.getAttribute('gps')
      name=image.getAttribute('name')
      indexdict[name]=gps
      #print(name,gps)
  #print(indexdict)
else:
  print("could not load indexdom")
  sys.exit(1)
try:
    albumdom=parse("album.xml")
except:
    print("Cannot parse album.xml in directory %s" % dirname)
indexout=''
indexout+='<?xml version="1.0" ?>\n'
indexout+='<?xml-stylesheet type="text/xsl" '
indexout+='href="%s"?>\n' % ALBUMXSL
indexout+='<album>\n'

Chunk referenced in 3.1

3.4 get information from album XML file

<get information from album XML file 3.5> =

if albumdom:
    indexout+='  <title>'
    try:
        titles=albumdom.getElementsByTagName('title')
        title=titles[0].firstChild.nodeValue
    except:
        title=""
    indexout+=title
    indexout+='</title>\n'
    thumbs=albumdom.getElementsByTagName('thumbnail')
    fc=thumbs[0].firstChild
    if fc:
        thumb=fc.nodeValue
    albumdescr=albumdom.getElementsByTagName('description')
    if not albumdescr:
        try:
            albumdescr=albumdom.getElementsByTagName('shortdesc')
        except:
            albumdescr=""
    try:
        description=albumdescr[0].firstChild.nodeValue
        description=description.strip()
    except:
        description=""
    indexout+='  <description>'
    indexout+=description
    indexout+='</description>\n'
    commentary=albumdom.getElementsByTagName('commentary')
    if commentary:
        commentary=commentary[0]
        strng=commentary.toprettyxml()
    else:
        commentary=albumdom.getElementsByTagName('longdesc')
        strng=""; l=commentary.length
        for i in range(l):
          n = commentary.item(i)
          children = n.childNodes
          for c in children:
            if debug: print("Looking at child node %s" % c)
            if c.nodeValue:
              strng+=c.nodeValue
    indexout+='  <commentary>'
    indexout+=strng
    indexout+='</commentary>\n'

Chunk referenced in 3.1

3.5 write tree element to index.xml

<write tree element to index.xml 3.6> =

indexout+="  <tree>"
treepath=os.getcwd()
#(prev,next)=getPrevNext(treepath)
res=treepat.match(treepath)
if res:
    treepath=res.group(1)+'/'
    if debug: print("treepath=%s" % treepath)
    #indexout='  <tree prev="%s" next="%s">' % (prev,next)
    indexout+=treepath
indexout+='</tree>\n'

Chunk referenced in 3.1

<define miscellaneous subroutines 3.7> =

def getPrevNext(treePath):
  return('','')    

Chunk referenced in 2.1
Chunk defined in 2.17,2.18,2.19,2.20,2.25,3.7

3.6 sort fnames into directories and images

<sort fnames into directories and images 3.8> =

dirIgnores=re.compile("(movies)|(sounds)")
if debug: print(images)
for f in fnames:
    res=ignorePat.match(f)
    if res:
        if debug: print("ignoring %s" % (f))
        continue
    if os.path.isdir(f):
      if not dirIgnores.match(f):
        dirs.append(f)
      else:
        print('  '*level,"skipping subdirectory %s" % (f))
      continue
    if os.path.isfile(f):
        (r,e)=os.path.splitext(f)
        if e.lower() in ['.jpg','.avi']:
            if r not in images:
                images.append(r)
                listmissingimages.append(r)
        continue

Chunk referenced in 3.1

Sort the list of all files in this directory into directories and images. We detect the first from an os.path.isdir call, and the second by checking its extension, which must be either .JPG or .jpg (or capitalisations in between).

3.7 Visit All Directories

<visit all directories 3.9> =

for d in dirs:
    missd=0
    if recurse:
        if debug: print("in dir %s, about to visit %s ..." % (dirname,d))
        ad=os.path.abspath(d)
        (dtitle,nimages,ndirs,thumbn,ddescr,missd)=visit(level+1,d,lists)
    else:
        res=retrieve(d)
        if res:
            (dtitle,nimages,ndirs,thumbn,ddescr,missd)=res
        else:
            continue
    nphotos+=nimages;
    nalbums+=ndirs+1
    missingdescrs+=missd
    if missd>maxmissing:
      maxmissing = missd
      maxdir = d
    if missd>0 and missd<minmissing:
      minmissing = missd
      mindir = d
    if not thumb:
        thumb=d+"/"+thumbn
    if thumbn[-4:len(thumbn)].lower()=='.jpg':
        thumbn=thumbn[0:-4]
    indexout+='  <directory title="%s" name="%s"\n' % (dtitle,d)
    indexout+='             albums="%d"\n' % (ndirs)
    indexout+='             images="%d"\n' % (nimages)
    if missd:
        indexout+='             missing="%d"\n' % (missd)
    indexout+='             thumb="%s"\n' % (d+"/"+thumbn)
    indexout+='             description="%s"/>\n' % (ddescr)

Chunk referenced in 3.1

3.8 scan all images and process them

<scan all images and process them 3.10> =

# scan all images and process them
descrNames={}
for i in images:
  istr=re.sub('-',':',i)
  thisdt=str2DT(istr)
  if not thisdt:
    # skip this image
    print("Cannot process image name %s" % (i))
    continue
  omit="include"
  # get image camera model
  args=[EXIV2,"-g","Model","-P","v","pr","%s.JPG" % (i)]
  output=subprocess.Popen(args,stdout=subprocess.PIPE).communicate()[0]
  output=output.decode()
  output=output.split('\n')[0].strip()
  if thisdt<startDT: omit="earlier"
  if thisdt>endDT: omit="later"
  if debug:
    print("%s: dt=%s (%s,%s) *%s*, model=%s (target=%s)" % \
        (i,thisdt,startDT,endDT,omit,output,model))
  if omit=="include" and re.match(model,output):
    suffix=""
    res=re.match('.*(-\d)$',i)
    if res:
      suffix=res.group(1)
    print("%s  Processing image %s (model=%s)" % (indent,i,output),end='')
    newDT=thisdt+correction
    newImageName=newDT.strftime("%Y%m%d-%H%M%S")
    newImageName+=suffix
    #
    # need test here to check if newImageName already exists
    #
    print("%s    new image name %s" % (indent,newImageName))
    <process image to update time 3.11>
    # add this image to descriptions repair list
    descrNames[i]=newImageName
  else:
    if debug: print("%s skipped (include=%s,model=%s)" % (i,omit,output))
<repair descriptions file names 3.12>

Chunk referenced in 3.1

<process image to update time 3.11> =

# process image to update time
cwd=os.getcwd()
if debug: print("current directory=%s" % (cwd))
# exiv2 exif date time data
# get timedelta in exiv format
secs=correction.seconds
days=correction.days
while days < 0:
  days+=1
  secs-=86400
while days > 0:
  days-=1
  secs+=86400
if secs < 0:
  sign = '-'
  secs = -secs
else:
  sign='+'
hours = secs / 3600
secs = secs % 3600
mins = secs / 60
secs = secs % 60
#print("exiv2 -a %s%02d:%02d:%02d adjust %s/%s.JPG" % (sign,hours,mins,secs,cwd,newImageName))
adjustTime="%s%02d:%02d:%02d" % (sign,hours,mins,secs)
cmd=[EXIV2,'-a',adjustTime,'adjust','%s.JPG' % (newImageName)]
#print(cmd)
output=subprocess.Popen(args,stdout=subprocess.PIPE).communicate()[0]
#print(output)

# mv image to new name
suffixes=["_128x128","_384x288","_640x480","_1200x900",""]
suffixes=list(map(lambda a : a+".JPG",suffixes))
suffixes.append(".xml")
for suffix in suffixes:
  src="%s%s" % (i,suffix)
  dst="%s%s" % (newImageName,suffix)
  #print("(cwd=%s) rename %s to %s" % (cwd,src,dst))
  # test to see if dst already exists!
  # try alternative names for destination    
  while os.path.exists(dst):
    if debug:
      print("The file {} needs to be renamed to {},".format(src,dst)+\
          " but the latter file already exixts")
    if newImageName[-2]=='-':
      version=int(newImageName[-1])
      version+=1
      newImageName=newImageName[0:-1]+str(version)
      dst="%s%s" % (newImageName,suffix)
    else:
      newImageName=newImageName+'-1'
      dst="%s%s" % (newImageName,suffix)
    if debug: print("Now trying image name {}".format(dst))
  try:
    os.rename(src,dst)
    print("%s    (renamed to %s)" % (indent,dst))
  except:
    print("Could not rename %s to %s in %s" % (src,dst,cwd))

Chunk referenced in 3.10

This chunk operates in two parts:

Firstly, the image file itself has its internal EXIF data updated to reflect the new time of exposure, and
Secondly, the file itself, and its subordinate associated files, are renamed according to the new time stamp. This latter part is not yet complete - if the new file name conflicts with a preexisting file, it will need to be modified to avoid the clash. An algorithm to do this has not yet been coded.

<repair descriptions file names 3.12> =

desfile=open("descriptions",'r')
descriptions=[]
for l in desfile.readlines():
  l=l.strip()
  res=re.match('([0-9-]+)( +(.*))?$',l)
  if res:
    oldname=res.group(1)
    descr=res.group(3)
    if not descr: descr='' # avoid 'None'
    if oldname in descrNames:
      entry="%s %s" % (descrNames[oldname],descr)
    else:
      entry="%s %s" % (oldname,descr)
    descriptions.append(entry)
  else:
    print("Could not match descriptions file entry %s" % (l))
desfile.close()
# finished with old descriptions file, move it to backup
version=1
newname='descriptions.~'+str(version)+'~'
while os.path.exists(newname):
  version+=1
  newname='descriptions.~'+str(version)+'~'
os.rename('descriptions',newname)
print("old descriptions now saved in {}".format(newname))
desfile=open('descriptions','w')
for d in descriptions:
  desfile.write("%s\n" % (d))
desfile.close()

Chunk referenced in 3.10

3.9 wind up index file

<wind up index file 3.13> =

indexout+='</album>\n'
if indexout != indexin:
  #indexof = open("index.xml","w")
  #indexof.write(indexout)
  #indexof.close()
  #print('  '*level,"... index.xml")
  pass

Chunk referenced in 3.1

4. Descriptions

This is a revision of the literate structure to bring all descriptions related material together.

<initialization 4.1> =

descrpat = re.compile("([^ *]+?)(\.JPG)?( |\*)(.*?)( # (.*))?$")

Chunk referenced in 2.1
Chunk defined in 2.9,2.10,2.11,2.12,2.13,2.14,2.15,4.1,8.1

descrpat is a pattern to match image names. Note that the first pattern must be non-greedy, or the '.JPG' gets swallowed, and the fourth pattern likewise, or the ' # ' gets swallowed.

The third pattern has been added (v1.3.0) to flag protected images that must not be shown publically.

4.1 read descriptions file and create dictionary

<read descriptions file and create dictionary 4.2> =

try:
    descrf=open("descriptions","r")
    mdescr=os.stat("descriptions")[stat.ST_MTIME]
    gotdescrfile=1
    for l in descrf.readlines():
        res=descrpat.match(l)
        if res:
            filen=res.group(1)
            ext=res.group(2)
            if not ext: ext=".JPG"
            protect=(res.group(3)=='*')
            if protect:
              protected[filen]=True
            imgtitle=res.group(4)
            comment=res.group(5)
            soundname=res.group(6)
            if debug: print(filen,ext,comment,soundname)
            descriptions[filen]=imgtitle
            if comment:
              sounds[filen]=soundname
              print('  '*level,"Got a sound file:%s" % (soundname))
            images.append(filen)
        else:
            images.append(l.rstrip())
except IOError:
    print("cannot open descriptions file in directory %s" % dirname)
    descriptions={}
    sounds={}
    protected={}

Chunk referenced in 3.1

Build the python dictionary of descriptions. We attempt to open a file in the current directory called descriptions, and use that to initialize the dictionary. The file has the format of one entry per line for each photo, with the name of the photo at the start of the line (with or without the .JPG extension), followed by at least one blank, followed by the descritive text, terminated by the end of line.

4.2 wind up descriptions file

<wind up descriptions file 4.3> =

if not gotdescrfile:
    if os.path.exists('descriptions'):
        print("OOPS!! attempt to overwrite existing descriptions file")
    else:
        descr=open('descriptions','w')
        for i in images:
            #descr.write(trimJPG(i)+' \n')
            pass
        descr.close()
elif listmissingimages:
    print("There are images not mentioned in the descriptions file:")
    print("appending them to the descriptions file.")
    descr=open('descriptions','a')
    for i in listmissingimages:
        #descr.write(trimJPG(i)+' \n')
        pass
    descr.close()

Chunk referenced in 3.1

We finalise the descriptions file. Two scenarios are relevant: either a) the descriptions file does not exist, in which case we build a new one, with just the image names of those images found in the directory scan, or b) the descriptions file exists, but there are images found not mentioned in it. In this latter case, we append image names to the file. No attempt is made to insert them in their 'correct' position.

If there are no changes required, the descriptions file is untouched. The existence of the descriptions file is determined by whether the flag gotdescrfile is set.

5. the addToIndex routine definition

<the addToIndex routine definition 5.1> =

def addToIndex(lists,imagename,descr,hasGPS):
    (r,e)=os.path.splitext(imagename)
    # this was inserted to try an control the length of captions.  It
    # is dangerous, however, as it can munge the well-formedness of
    # any contained XML.
    #if len(descr)>200:
    #    descr=descr[0:200] + ' ...'
    gps=''
    if hasGPS: gps=' gps="yes"'
    return '  <image name="%s"%s>%s</image>\n' % (r,gps,descr) # was cgi.escape(descr)

Chunk referenced in 2.1

I've changed my mind a couple of times about escaping the description string. Currently the string is not escaped, meaning that any special XML characters (ampersand, less than, etc.) must be entered in escaped form. But this model does allow URLs and the like to appear in the description captions.

6. the addToList routine definition

<the addToList routine definition 6.1> =

# 'addToList' is called to add an image name to the list of images.
# See above for the format of the data structure 'list'.  It is
# assumed that the image name is a full path from the album root ((one
# of) the CLI parameter(s)).  The directory part is removed and used
# to match against the keys of 'lists'.  A new entry is created if
# this path has not previously been seen.  The image name is then
# added to this list, if it is not already there.
def addToList(lists,imagefile):
    (base,tail)=os.path.split(os.path.abspath(imagefile))
    (name,ext)=os.path.splitext(tail)
    if base in lists:
        a = lists[base]
    else:
        a = []
        lists[base] = a
    if not name in a:
        a.append(name)

Chunk referenced in 2.1

7. the makeImage routine

<the makeImage routine definition 7.1> =

def makeImage(level,orgwd,orght,newwd,newht,newname,orgname,orgmod):
    # orgwd: original image width
    # orght: original image height
    # newwd: new image width
    # newht: new image height
    # newname: new image file name
    # orgname: original image file name
    # orgmod: original image modification date/time
    #
    if debug: print("makeImage(%d,%d,%d,%d,%s,%s,%s)" % \
          (orgwd,orght,newwd,newht,newname,orgname,orgmod))
    if orght>orgwd: # portrait mode, swap height and width
        t=newht
        newht=newwd
        newwd=t
    if orght>newht: # check that resolution is large enough
        newmod=0
        if os.path.isfile(newname):
            newmod=os.stat(newname)[stat.ST_MTIME]
        if orgmod>newmod:
            print('  '*level,"Writing %s at %dx%d ..." % (newname,newwd,newht))
            size=" -size %dx%d " % (orgwd,orght)
            resize=" -resize %dx%d " % (newwd,newht)
            if debug: print(size+orgname+resize+newname)
            os.system(CONVERT+size+orgname+resize+newname)
        pass
    return

Chunk referenced in 2.1

8. the makeSubImages routine

This routine is responsible for building all the smaller images from the master image.

8.1 New SubImage generalized size definitions

We define here data structures to handle generalized sub-image size definitions. Not yet active.

<initialization 8.1> =

imageSizeDefs=[(128,128,"thumb"),(384,288,"small"),\
               (640,480,"medium"),(1200,900,"large"),(0,0,"original")]
imageSizes=[]; derivedImageMatch=[]
for (wd,ht,sizeName) in imageSizeDefs:
  imageSizes.append("%dx%d" % (wd,ht))
  derivedImageMatch.append(re.compile(".*_%dx%d.JPG" % (wd,ht)))

Chunk referenced in 2.1
Chunk defined in 2.9,2.10,2.11,2.12,2.13,2.14,2.15,4.1,8.1

Initialize the image size parameters. imageSizeDefs is normative, and defines all required image sizes. The first entry is deemed to be the thumbnail size, and an entry of (0,0) refers to the original size. imageSizes is an equivalent list of strings in the form %dx%d, and derivedImageMatch is an equivalent list of patterns to match image names of those sizes. (Except the last, which fix!)

<the makeSubImages routine definition 8.2> =

def makeSubImages(level,lists,imagename,iprev,inext,mdescr):
    global descriptions,forceXmls
    (r,e)=os.path.splitext(imagename)
    if not e:
      e='JPG'
    imagename="%s.%s" % (r,e)
    if not os.path.isfile(imagename):
      e='jpg'
      imagename="%s.%s" % (r,e)
    if debug: print("making sub images for %s, e=%s, forceXmls=%s" % (imagename,e,forceXmls))
    thumbn=r+"_128x128.JPG"
    smalln=r+"_384x288.JPG"
    medn=r+"_640x480.JPG"
    bign=r+"_1200x900.JPG"
    viewn=r+".xml"
    <return on missing image file 8.3>
    <get all modification times 8.4>
    <makeSubImages: make any sub images required 8.5>
    # check if there is a new description
    olddescr=descr=""
    try:
      oldxmldom=parse(viewn)
      olddescr=oldxmldom.getElementsByTagName('description')
      olddescr=flatStringNodes(olddescr[0].childNodes)
      olddescr=olddescr.strip()
    except:
      print("Could not get old description for %s" % viewn)
      pass
    if r in descriptions:
        descr=descriptions[r].strip()
    # create the new IMAGE.xml if the descriptions file exists
    # (mdescr>0) and is more recent than IMAGE.xml, or the IMAGE.JPG
    # is more recent  than IMAGE.xml, and there is a new description
    if debug: print("mdescr=%d, mview=%d, mfile=%d" % (mdescr,mview,mfile))
    hasGPS=r in indexdict
    #print(indexdict)
    #print("checking %s, it has %s gps" % (r,hasGPS))
    if forceXmls or \
       (mview==0) or \
       (  (olddescr!=descr) and \
          (  (mdescr>0 and mdescr>mview) or \
             (mfile>mview)\
          )\
       ):
        <makeSubImages: write image xml file 8.6>
    return (descr,hasGPS)

Chunk referenced in 2.1

<return on missing image file 8.3> =

if not os.path.isfile(imagename):
    if r in descriptions:
        descr=descriptions[r].strip()
        return (descr,False)
    else:
        print("descriptions entry '%s' : image not found" % (imagename))
        return (None,False)

Chunk referenced in 8.2

Check that the image file exists. If it doesn't, then there are two possible reasons. Firstly, the image file has been removed for space reasons, in which case there will be an entry in the descriptions database, and we return that without attempting to make any sub images.

Alternatively, no such descriptions entry means the image is genuinely missing, and therefore we should print a warning message, and return a missing image flag.

<get all modification times 8.4> =

mfile=os.stat(imagename)[stat.ST_MTIME]
mthumb=msmall=mmed=mbig=mfile-1
mview=0
if os.path.isfile(thumbn):
  mthumb=os.stat(thumbn)[stat.ST_MTIME]
if os.path.isfile(smalln):
  msmall=os.stat(smalln)[stat.ST_MTIME]
if os.path.isfile(medn):
  mmed=os.stat(medn)[stat.ST_MTIME]
if os.path.isfile(bign):
  mbig=os.stat(bign)[stat.ST_MTIME]
if os.path.isfile(viewn):
  mview=os.stat(viewn)[stat.ST_MTIME]
if debug: print("%d > (%d,%d,%d)" % (mfile,mthumb,mmed,mbig))

Chunk referenced in 8.2

Get all the modification times. The default modification times are that mview is the oldest, mfile is the youngest, and all the others are in between.

<makeSubImages: make any sub images required 8.5> =

if mfile>min(mthumb,msmall,mmed,mbig):
  p=subprocess.Popen([IDENTIFY,imagename],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
  (cmd_out,cmd_stderr)=p.communicate(None)
  if cmd_stderr:
    print("error in identify for %s ... (%s)" % (imagename,cmd_stderr))
    return (None,False)
  names=[thumbn,medn,bign,imagename]
  (b,t)=os.path.split(imagename)
  if debug: print("base=%s, tail=%s" % (b,t))
  if debug: print("Making images for %s" % (imagename))
  cmd_out=cmd_out.decode()
  res=identifyPat.match(cmd_out)
  if res:
    wd=int(res.group(1))
    ht=int(res.group(2))
    smallwd=384.0; medwd=640.0; largewd=1200.0
    smallht=288.0; medht=480.0; largeht=900.0
    if wd>ht:
      smallwd=int(smallwd*(float(wd)/float(ht)))
      medwd=int(medwd*(float(wd)/float(ht)))
      largewd=int(largewd*(float(wd)/float(ht)))
    else:
      smallht=int(smallwd*(float(ht)/float(wd)))
      medht=int(medwd*(float(ht)/float(wd)))
      largeht=int(largewd*(float(ht)/float(wd)))
    makeImage(level,wd,ht,128,128,thumbn,imagename,mfile)
    if not thumbsOnly:
      makeImage(level,wd,ht,smallwd,smallht,smalln,imagename,mfile)
      makeImage(level,wd,ht,medwd,medht,medn,imagename,mfile)
      if large:
        makeImage(level,wd,ht,largewd,largeht,bign,imagename,mfile)
    count=1

Chunk referenced in 8.2

(20110717:172812) The variables smallwd, medwd, largeht are introduced to ensure that the resulting images all have a consistent height, if not width. This was introduced because panoramas came out with the same width as convention 4x3s, making the height impossible small.

<makeSubImages: write image xml file 8.6> =

print('  '*level,"Writing %s ..." % (viewn),end='')
imagef=open(imagename,'rb')
exiftags=get_exif(imagename)
for tag in exiftags.keys():
    if debug: print("%s = %s" % (tag,exiftags[tag]))
imagexmlf = open(viewn,"w")
imagexmlf.write('<?xml version="1.0" ?>\n')
imagexmlf.write('<?xml-stylesheet type="text/xsl" ')
imagexmlf.write('href="%s"?>\n' % ALBUMXSL)
imagexmlf.write('<album>\n')
(rp,ep)=os.path.splitext(iprev)
(rn,en)=os.path.splitext(inext)
treepath=os.getcwd()+'/'
res=treepat.match(treepath)
if res:
  imagexmlf.write('  <tree>%s</tree>\n' % (res.group(1)))
imagexmlf.write('  <view name="%s" \n' % (r))
imagexmlf.write('        prev="%s" next="%s"\n' % (rp,rn))
if r in protected:
  imagexmlf.write('        access="localhost"\n')
if r in sounds:
  snd=sounds[r]
  print("  Adding sound: %s" % (snd))
  imagexmlf.write('        audio="sounds/SND_%s.WAV"\n' % (snd))
imagexmlf.write('  >\n')
imagexmlf.write('    <description>\n')
imagexmlf.write('      %s\n' % descr) # was cgi.escape(descr)
imagexmlf.write('    </description>\n')
if 'Image Model' in exiftags:
    camera=exiftags['Image Model'].__str__().strip()
    imagexmlf.write('    <camera>%s</camera>\n' % camera)
hasGPS=False
if 'EXIF DateTimeOriginal' in exiftags:
    dt=exiftags['EXIF DateTimeOriginal']
    imagexmlf.write('    <datetime>%s</datetime>\n' % dt)
if 'EXIF ExposureTime' in exiftags:
    shutter=exiftags['EXIF ExposureTime']
    imagexmlf.write('    <shutter>%s</shutter>\n' % shutter)
if 'EXIF FNumber' in exiftags:
    aperture=exiftags['EXIF FNumber']
    try:
        aperture=str(eval(str(aperture)+'.0'))
    except:
        pass
    imagexmlf.write('    <aperture>%s</aperture>\n' % aperture)
if 'GPS GPSLatitudeRef' in exiftags:
  hasGPS=True
  latRef=exiftags['GPS GPSLatitudeRef']
if 'GPS GPSLatitude' in exiftags:
  latitude=exiftags['GPS GPSLatitude']
  if debug: print(latitude)
  res=re.match('\[(\d+), (\d+), *(\d+)/(\d+).*\]',latitude.__str__())
  if res:
    latdegrees=int(res.group(1))
    latminutes=int(res.group(2))
    secnum =res.group(3)
    secden =res.group(4)
    latseconds=float(secnum)/float(secden)
  else:
    res=re.match('\[(\d+), (\d+), *(\d+).*\]',latitude.__str__())
    if res:
      latdegrees=int(res.group(1))
      latminutes=int(res.group(2))
      latseconds=float(res.group(3))
    else:
      res=re.match('\[(\d+), *(\d+)/(\d+), (\d+).*\]',latitude.__str__())
      if res:
        latdegrees=int(res.group(1))
        latminutes=float(res.group(2))
        if latminutes>60.0:
          latseconds=latminutes % 60
          latminutes=latminutes // 60
        else:
          latseconds=int(res.group(3))
      else:
        latdegrees=0
        latminutes=0
        latseconds=0.0
        print("Cannot decode latitude %s" % (latitude.__str__()))
  imagexmlf.write('    <latitude hemi="%s" degrees="%d" minutes="%d" seconds="%f"/>\n' % \
      (latRef,latdegrees,latminutes,latseconds))
if 'GPS GPSLongitudeRef' in exiftags:
  longRef=exiftags['GPS GPSLongitudeRef']
if 'GPS GPSLongitude' in exiftags:
  longitude=exiftags['GPS GPSLongitude']
  if debug: print(longitude)
  res=re.match('\[(\d+), (\d+), *(\d+)/(\d+).*\]',longitude.__str__())
  if res:
    londegrees=int(res.group(1))
    lonminutes=int(res.group(2))
    secnum =res.group(3)
    secden =res.group(4)
    lonseconds=float(secnum)/float(secden)
  else:
    res=re.match('\[(\d+), (\d+), *(\d+).*\]',longitude.__str__())
    if res:
      londegrees=int(res.group(1))
      lonminutes=int(res.group(2))
      lonseconds=float(res.group(3))
    else:
      res=re.match('\[(\d+), *(\d+)/(\d+), (\d+).*\]',longitude.__str__())
      if res:
        londegrees=int(res.group(1))
        lonminutes=float(res.group(2))
        if lonminutes>60.0:
          lonseconds=lonminutes % 60
          lonminutes=lonminutes // 60
        else:
          lonseconds=int(res.group(3))
      else:
        londegrees=0
        lonminutes=0
        lonseconds=0.0
        print("Cannot decode longitude %s" % (latitude.__str__()))
  imagexmlf.write('    <longitude hemi="%s" degrees="%d" minutes="%d" seconds="%f"/>\n' % \
      (longRef,londegrees,lonminutes,lonseconds))
  print('Lat:%d %d %7.4f %s' % (latdegrees,latminutes,latseconds,latRef),end='')
  print('Long:%d %d %7.4f %s' % (londegrees,lonminutes,lonseconds,longRef),end='')

imagexmlf.write('  </view>\n')
imagexmlf.write('</album>\n')
imagexmlf.close()
print

Chunk referenced in 8.2

See comment under <the addToIndex routine definition 5.1> regarding escaping the description string.

9. the retrieve support routines definition

<the retrieve support routines definition 9.1> =

def getNodeValue(dom,field,missing):
    try:
        val = dom.getElementsByTagName(field).item(0).firstChild.nodeValue
    except:
        val=missing
    return val

def retrieveField(field):
    try:
        val = index.getElementsByTagName(field).item(0).firstChild.nodeValue
    except:
        val="[could not retrieve %s]" % (field)
    return val

Chunk referenced in 9.2

These two support routines for retrieve encapsulate data extraction from the DOM model. They perform the same basic operation, differeing only in the default parameters required.

<the retrieve routine definition 9.2> =

def retrieve(d):
    <the retrieve support routines definition 9.1>
    missing=0
    try:
        index = parse(d+'/index.xml')
    except:
        (etype,eval,etrace)=sys.exc_info()
        print("Not scanning directory %s because no index.xml found" % (d))
        print("    (error type %s)" % (etype))
        return None
    if debug: print("Successfully parsed index.xml in directory %s" % (d))
    try:
        album = parse(d+'/album.xml')
    except:
        (etype,eval,etrace)=sys.exc_info()
        print("Cannot open album.xml in directory %s because %s" % (d,etype))
        album=None
    title = retrieveField('title')
    description = retrieveField('description')
    imageElems=index.getElementsByTagName('image')
    summary=index.getElementsByTagName('summary').item(0)
    photos=int(summary.getAttribute('photos'))
    albums=int(summary.getAttribute('albums'))
    missing=int(summary.getAttribute('missing'))

    # sort out the vexed problem of the thumbnail
    if album:
        albumThumbs=album.getElementsByTagName('thumbnail')
    else:
        albumThumbs=''
    thumbDefault=getNodeValue(index,'thumbnail','')
    thumb=thumbDefault
    if not thumbDefault:
        thumbDefault=getNodeValue(album,'thumbnail','')
    if debug: print("    albumThumb: %s" % (thumbDefault))
    if not thumbDefault:
        thumb=thumbDefault

    miss=index.getElementsByTagName('missingdescriptions')
    if miss:
        missing=int(miss.item(0).firstChild.nodeValue)
    if 0:
        print("retrieved:")
        print("    title: %s" % (title))
        print("    photos: %d" % (photos))
        print("    albums: %d" % (albums))
        print("    thumb: %s" % (thumb))
        print("    description: %s" % (description))
    return (title,photos,albums,thumb,description,missing)

Chunk referenced in 2.1

retrieve is called when we do not wish to recursively visit a subdirectory. Instead, relevant details from the subdirectory are extracted ('retrieved') from an index.xml file, which has been constructed when the relevant subdirectory was visited.

There is a slight problem with identifying a thumbnail for this album, since the definite source is that in the album.xml file. However, this may be blank, in which has we need to promote the first thumbnail within the directory (or subdirectories where the directory has no images of its own).

10. Makefile

This Makefile integrates with the other photo tool Makefiles

"MakefileTardis3" 10.1 =

install-tardis3: tardis3.py
	chmod 755 tardis3.py
	cp -p tardis3.py $(HOME)/bin/
tardis3.py: tardis3.tangle

11. TODOs

Check when renaming file that no synonomous file is affected.

12. Indices

File Name	Defined in
MakefileTardis3	10.1
tardis3.py	2.1

Chunk Name	Defined in	Used in
banner	2.4	2.1
basic usage information	2.5	2.1
check if directory is standard date structure, exit if not	3.2	3.1
collect the command line options	2.21	2.2
current date	13.2	2.5
current version	13.1	2.5, 2.21
define miscellaneous subroutines	2.17, 2.18, 2.19, 2.20, 2.25, 3.7	2.1
define miscellaneous subroutines	2.17, 2.18, 2.19, 2.20, 2.25, 3.7	2.1
define miscellaneous subroutines	2.17, 2.18, 2.19, 2.20, 2.25, 3.7	2.1
define the usage subroutine	2.16	2.1
explore next higher level to update counts	2.23	2.22
get all modification times	8.4	8.2
get information from album XML file	3.5	3.1
global variables	2.8	2.1
imports	2.7	2.1
initialization	2.9, 2.10, 2.11, 2.12, 2.13, 2.14, 2.15, 4.1, 8.1	2.1
initialization	2.9, 2.10, 2.11, 2.12, 2.13, 2.14, 2.15, 4.1, 8.1	2.1
initialization	2.9, 2.10, 2.11, 2.12, 2.13, 2.14, 2.15, 4.1, 8.1	2.1
initialize variables for visit routine	3.3	3.1
interpreter definition	2.3	2.1
main routine	2.2	2.1
makeSubImages: make any sub images required	8.5	8.2
makeSubImages: write image xml file	8.6	8.2
open index.xml and write header	3.4	3.1
perform the top level visit	2.22	2.2
process image to update time	3.11	3.10
read descriptions file and create dictionary	4.2	3.1
repair descriptions file names	3.12	3.10
return on missing image file	8.3	8.2
scan all directory elements, updating counts	2.24	2.23
scan all images and process them	3.10	3.1
sort fnames into directories and images	3.8	3.1
the addToIndex routine definition	5.1	2.1
the addToList routine definition	6.1	2.1
the makeImage routine definition	7.1	2.1
the makeSubImages routine definition	8.2	2.1
the retrieve routine definition	9.2	2.1
the retrieve support routines definition	9.1	9.2
the visit routine definition	3.1	2.1
todos	2.6	2.1
visit all directories	3.9	3.1
wind up descriptions file	4.3	3.1
wind up index file	3.13	3.1
write list file	2.26	2.1
write tree element to index.xml	3.6	3.1

Identifier	Defined in	Used in
addToIndex	5.1
addToList	6.1
debug	2.9
descriptions	2.11	3.1, 4.2, 4.2, 8.2, 8.2, 8.2
descrpat	4.1	4.2
getNodeValue	9.1
ignorePat	2.13	3.8
listmissingimages	3.3	3.8, 4.3, 4.3
lists	2.10
makeImage	7.1	8.5, 8.5, 8.5, 8.5
makeSubImages	8.2
retrieve	9.2	3.9
retrieveField	9.1
visit	3.1	2.22, 3.9

13. Document History

20150914:114636	ajh	1.0.0	first version, drafting skelton using photo.xlp as framework
20160320:144825	ajh	1.0.1	add usage routine
20180506:050356	ajh	1.1.0	add check for model abbreviation
20180512:102259	ajh	1.1.1	add check for non-empty correction
20200517:170755	ajh	1.1.2	last version under python2.7
20200517:171222	ajh	2.0.0	first version under python3
20200821:111951	ajh	2.1.0	fixed some bugs (old v1.2.0, relabelled v2.0.0, was never fully tested), improved collision handling, and old descriptions saved to backup when new descriptions file is created
20200825:174425	ajh	2.1.1	tidy up imagename clash handling

<current version 13.1> = 2.1.1

Chunk referenced in 2.5 2.21

<current date 13.2> = 20200825:174425

Chunk referenced in 2.5

Tardis3

John Hurst

Version 2.1.1

20200825:174425

Table of Contents

1. User Manual

2. The Main Program

2.1 The Main Routine

2.2 Setup

2.3 Define miscellaneous subroutines

2.4 Collect the Command Line Options

2.5 Perform the Top Level Visit

2.6 write list file

2.7 Plan for new data structure

3. The visit routine

3.1 check if directory is standard date structure, exit if not

3.2 initialize variables for visit routine

3.3 open index.xml and write header

3.4 get information from album XML file

3.5 write tree element to index.xml

3.6 sort fnames into directories and images

3.7 Visit All Directories

3.8 scan all images and process them

3.9 wind up index file

4. Descriptions

4.1 read descriptions file and create dictionary

4.2 wind up descriptions file

5. the addToIndex routine definition

6. the addToList routine definition

7. the makeImage routine

8. the makeSubImages routine

8.1 New SubImage generalized size definitions

9. the retrieve support routines definition

10. Makefile

11. TODOs

12. Indices

13. Document History