Web Page Management Software

John Hurst

Version 1.6.7

20240430:151827

Abstract

Table of Contents

1. Introduction

This document describes the files used to manage delivery of my personal web pages, and those that I manage for other organisations. The general form of web page delivery is a) a source file written in XML, b) a translation file written in XSLT, and c) the program described here, a python cgi script that calls the appropriate translator on the source file, and delivers the result. It also handles straight HTML, as well as providing some debug and other maintenance options.

The program is invoked by commands in the .htaccess file associated with each web directory. Different .htaccess files can be used for different directories. If none exist in a given directory, the directory path is searched towards the root until one is found.

The XSLT files used can be specified either in the .htaccess file (default), or in the source XML file, through an explicit xml-stylesheet command. If a stylesheet XSLT file is specified, it overrides the default .htaccess one.

Permission is given to reuse this document, provided that the source is acknowledged, and that any changes are noted in the documentation.

The document is in the form of a literate program, and generates all files necessary to maintain the working environment, including a Makefile.

As of 20240513:174010, there are two URLs now recognized: server/file and server/~ajh/file. The first is the original suite of web pages, and the second is the suite of private pages, protected by an .htpasswd gateway. All the original pages are accessible on the private URL, but only the public pages are accessible through the public (~ajh) path. At some stage, it is intended that these roles will be reversed, but only after a significant period of warning.

2. Literate Data

This message flags the fact that the source code is a derived document, and should not be directly edited.

3. Various IP address facilities

3.1 IP Tools

IPtools provides a range of functions to facilitate handling of IP addresses and networks. This module is intended to be imported by programs in the rest of this suite. The functions are:

str2IP(s)

converts a given IP string to a 32 bit integer value

IP2str(ip)

the complement to str2IP: convert an integer IP address to a decimal string representation.

ipMask(bits)

generate an upper integer mask of bits length, useful for extracting a given size network base

ip2Net(ip,bits)

Convert the IP address ip to the base network address of length bits

thisIPrange

return a tuple that defines the range of (integer) addresses for a given base address and network size

3.2 IP decoding program

Handling IP address between string forms, hexadecimal values, and network ranges can be challenging. This program reads a line, interprets what format it is, and prints the altenatives in an easy to read format, consisting of:

        stringform, hexadecimalform, upperstring, upperhex, networklength
      

where the string form is 'dec.dec.dec.dec'; hexadecimal is the 32 bit hex representation; upper string and upper hex are the maximum addresses for this network size, also printed as network length.

4. The checkCountry Module

checkCountry.py is a module to provide mechanisms to check the origin of an http request. It relies upon a database blackListIPs which stores IPs of known requesters, along with a flag to indicate whether they are white, grey or black listed. Any requester not in the database is regarded as grey listed.

The module provides the class countries; instances of which have the methods load(), to load the database of known IP addresses, and checkIP(ips), to check if the IP string ips is known to the database.

It returns a triple (known, status, country, annotation), where:

known

is True if the IP is indeed known in the database,

status

has a value

• white (address is whitelisted),
• black (address is blacklisted), and
• grey (address has unknown status).

Note that an IP may be known to the database (known is True), but still have status as 'grey'. This will be where not enough about the behaviour of the said Ip has been observed to make a decision about whether it is black or white.

Currently no attempt is made to update the database with unknown grey listings. That has to be done manually through an edit of the blackListIPs file (see later in this literate program).

Define a few useful functions

The class countries is a database of IP addresses derived from the file /home/ajh/local/countryIPs, with entries of the form:

        IP adr in dot form/network size status country annotation
      

checkIP checks the given IP address against the database, and return a tuple (isgood,status,country,annotation) where isgood is a boolean, status is ('white','grey,'black'), country is the country of origin, and annotation is the owner of the IP.

5. The Main Program index.py

This program is written in Python3, and relies upon Python v3.6 or later, as it uses f strings in many of its output statements. /home/ajh/binln is a common directory used to identify where required versions of interpreters are to be found.

It is not compatible with python3.11 or later, as it uses features now deprecated in 3.11 and later. Hence it is restricted to python 3.10. (This may change in future, once the deprecated features are isolated and corrected.)

We insert the usual edit warning to alert any code editor (person) to the dangers of directly edit constructed code. The version number is inserted into rendered HTML text.

The showBegins flag turns on tracing output on the stderr output to assist in identifying where program failures may occur, and linking these back to the literate code. Other global variable are also identified upfront, to further assist debugging the raw code.

Note the convention that where a code fragment is defined in several chunks, a begin n comment is inserted to assist debugging, and back referencing into this literate program.

This script processes all my web page XML files It requires apache to be configured:

1. to allow python files (this file) to run as cgi in user directories
2. to add a handler for XML files that call this program
3. to pass as a cgi parm the XSLT file that translates the XML file

The script relies upon picking up the required file and its XSLT file from a) the REDIRECT environment variables, and b) the script parameter, respectively.

The interpreter required varies according to the target server. This detail is captured by the <Makefile 12.2> script, although not all systems have yet been encoded into the Makefile script. The python script also changes its behaviour depending upon the host on which it is running. This is done by an explicit call to os.environ (chunk <determine the host and server environments 5.11,5.12>).

Gather together all the module and library imports needed for this program.

Start processing. Get a timestamp for recording key events in the log. Set the modification times to year dot.

This first code chunk here identifies who and what is being invoked in this instantiation of the server.

The second part identifies known IP addresses that have a track record of abusing the server. The algorithms used are somewhat heuristic.

Define a useful debug routine.

This fragment simply outputs the required header for flagging the generated content as HTML, and builds a number of string matching patterns for later use.

Collect relevant parameters from the original request.

The major work in rendering the required page is done by the <process file 5.35,5.36> code chunks. After this point, processing is complete, and the program falls through to exit.

5.1 Define Global Variables and Constants

returnXML is set True when the display of the raw untranslated XML is required.

convertXML is set True when a converted copy of the translated XML is required to be saved.

alreadyHTML is set True when the incoming file to be rendered is already in HTML and does not require conversion.

cachedHTML is set True when the incoming file to be rendered has been cached in the HTMLS directory, and does not require conversion.

5.2 Define Various String Patterns

Somewhat self-explanatory string matching patterns.

5.3 Determine the Host and Server Environments

The server is the address to which this request was directed, and is useful in making decisions about what to render to the client. Examples are "localhost", "www.ajh.id.au", "chairsabs.org.au".

The host is the machine upon which the server is running, and may be different from the server. This name is used to determine where to store local data, such as logging information. For example, the server may be "localhost", but this can run on a variety of hosts: "murtoa", "dimboola", dyn-13-194-xx-xx", etc..

Note: This section needs a bit more work to distinguish the new 001-default host.

BASE is set to the path to the web root directory on this server. It should not have a trailing slash!

HOME has its usual Unix meaning.

PRIVATE is set to the path to a working directory on this particular server that is used to store accounting and audit information about this particular access. The path includes a specific reference to the server hostname to uniquely distinguish it. This directory basename is rendered on the web page as the parameter server and is the first of the "server@host" pairs rendered at the top of the web page.

COUNTERS is the path to the directory containing all the web page access counts. Each counter is incremented on page access, whether to the cached HTML, or to the (re)rendered XML file.

HTMLS is the path to a local copy of html versions of the files. These are cached versions, and some mechanism to age and delete needs to be identified. If the corresponding XML file is older than the HTML file found in this subdirectory, the HTML version is used.

XSLTPROC is the path to the xsltproc processor. Without this processor, this entire script (as far as XML files are concerned) is meaningless!

5.4 filter out bad ips to which we do not respond

We maintain a database of IP addresses, categorized three ways:

1. 'white': this IP is allowed.
2. 'grey': no information on this IP as yet. Allow for now.
3. 'black': this IP is filtered out.

The database is described in section <Country Database >.

5.5 Handle JPEGs

From version 1.2.0 onwards, this code implements a form of caching for jpg files. A local check for the request file is made, and if it is not found, an attempt to retrieve it from the dimboola server is made. If that is not successful, the file is reported not found. If it is successful, the file is saved locally. No attempt is made to age files out of the cache.

This code checks to see if the request is for a jpg image file. These are cached, and if not present, are retrieved from the master jpg server for my jpeg images. This is still a bit experimental. The server URL is dimboola.infotech.monash.edu.au/~ajh/Pictures.

It requires that the .htaccess file be modified to refer .jpg requests to this cgi script.

5.5.1 Get JPG File from Remote Server

Generate the URL of the corresponding remote JPG file, and issue read request. By using the urllib.request library, we also get the modification time, which we parse in order to set the correct modification time on the locally cached copy.

Now try to cache a local copy. This can fail for several reasons, the main one being that the permissions in the local directory are likely to be against (write) access by the www user. More work is required to make this a bit more robust.

Note that we set the pair (access time, modification time) on the local file to be the current time and remote file modification time respectively. This ensures that attempts to synchronize the two file systems will see this file as the same file as the remote file, and not attempt to update one or the other (thus leading to spurious modification times).

5.6 Collect HTTP Request

{Note 5.20.1}

initialize the filename of the file to be rendered. Most of the work in computing the value of this variable is done in <get filename from redirect 5.22>

When this script is called, it has gained control by virtue of an .htaccess directive to Apache to use this program to render the source file. The name of that source file has to be recovered somehow, and different systems seem to handle this parameter in different ways. The first parameter to explore is the REDIRECT_QUERY_STRING, which, if it is present in the form request, contains secondary parameters to the rendering operation. If this parameter is not present, initialize the variable form to an empty value.

5.6.1 Handle REDIRECT QUERY STRING

There are several possibilities for secondary parameters. The primary one is the debugFlag parameter, which can be set to true, indicating that debugging information is to be printed along with the rendering. This is intended for administrator access only, but as it is harmless, there is no authentication required.

The other parameter that can be offered at this point is the xml parameter, with values of true or convert. The first of these forces no conversion of the XML, but simply copies it to the browser, substituting escape sequences for any special XML character sequences so that it appears as verbatim XML.

The second choice, convert, allows the use of the rendering engine as an XML-to-HTML converter, in which case a copy of the converted HTML is saved to a temporary file. This file can be used subsequently as a statically converted file as necessary.

5.7 Get Filename from Redirect

5.8 Check for Abbreviated URL

5.9 Make File and Dir Absolute

5.10 Check for HTML Request

We now have a requestedFile name for the document to be rendered. We need to investigate this file to see how it is to be rendered. In particular, it may be an HTML file (indicated by a .html extension), or it may be an XML file previously rendered and cached. In these cases, we do not need to do any XML conversion, and the flag alreadyHTML is set true if it is an HTML file, or the flag cachedHTML is set true if it is a cached converted XML to HTML file.

This code now also checks for a cached version of the XML file, as per the following fragment.

Unless the file being retrieved is already an HTML file, check to see if we have a cached HTML version of this (XML) file. Note that any parameters to th http request (indicated by a non-empty form value) will abort the caching process, and force a reload of the XML file.

5.11 Get Default XSLT File

5.12 Scan for Locally Defined XSLT File

Look at modification time of XSL file. If it is more recent than the cached HTML file, we must re-convert the XML file.

5.13 Determine XSLT File

5.14 Update Counter

Compute the name of an XML counter file which contains a counter element with subelements value and date. The value element contains the current count value, and the date element is the date on which this XML file was initialised. We read the current count from that file, increment it, and update the file. This file is used by most xslt translations to output an access count in the footer. It is also used by the site map program to compute the intensity of accesses to this web page.

It was fortuitous, but this counter also keeps track of HTML accesses, both where an HTML file is the initial request, and where it is a cached version of the corresponding XML file. Since the XML files have their own counters included by the XSLT translator, the count attached to the HTML rendering allows a comparision of how many accesses are to the cached copy (the difference between the two).

For example, suppose the XML rendering gives 986 references, and the HTML rendering cites 993 references. The the cached HTML page has itself been referenced 7 times since it was first cached.

First we process relfile to find the counter name. Remove any extension, and replace all slash path separators with minus signs.

(Strictly speaking, the first sub is not required, but I've left it in, as it does no harm.)

Now try to read the counter XML file. The file may not exist if this is the first time we have accessed this page since this mechanism was set up, so we must capture that error, and any error arising from attempting to parse the XML, and create a new counter file, with value initialised to zero, and date initialised to today's date.

5.15 Process File

Decide what to with the file. There are 3 choices:

1. return the raw XML. This means escaping all the active characters, and printing the file verbatim.
2. The file is HTML, either because of an explicit HTML request, or a cached HTML file previously translated has been found. Again, the file is rendered verbatim, this time without escaping the active characters.
3. It is an XML file, and it needs translation. Call the XSLT processor to do that (chunk <process an XML file 5.38,5.39,5.40>).

Note that each line from the HTML file is printed without additional line breaks.

5.15.1 Process an XML File

Run the pipe to perform the translation. Note that this step requires an inordinate amount of time on some servers (sequoia in particular), and was the prompt for including the caching mechanism.

Note that in copying the rendered HTML version, we retain the lines as is, and make sure that they are rendered without any additional (or deleted) new lines.

5.15.1.1 Deal with any Conversion Errors

6. The Train Image Viewer viewtrains.py

The literate code for this program has been moved to ViewTrains

7. The Rankings Display Program ranktrains.py

The literate code for this program has been moved to ViewTrains

8. The Train Ranking Module rank.py

The literate code for this program has been moved to ViewTrains

9. The Web Server Module webServer.py

The various web server programs all make use of a pair of values, known as host and server. The host value defines the machine on which this program is running, while the server value defines the name by which the program was invoked (effectively the domain name of the server). This is necessary, as each program may be invoked by different domain name paths, and the service rendered may be different for each path.

While different domain names may invoke services on the same machine or host, it is never the case that different machines are invoked to handle a single domain name service.

When invoked as a stand-alone program, a test server parameter may be passed in to see what the program determines. This is for testing purposes only, and serves no useful purpose otherwise.

10. File Caching

10.1 The File Cache Module

This code was adapted from an example given on a web page. Sorry, I have forgotten the reference.

10.2 Clearing the Cache

This program clears the HTML caches created by the previous module. It is called independently, and can clear either the entire cache, or subdirectories of it.

The cache is maintained on a per-machine basis, and the machine being used is identified by a hostname call.

11. Country Database

Addresses in this database come in two forms: CIDR (Classless_Inter-Domain_Routing), and double IPv4 addresses. The CIDR forms are standard for defining a range of addresses; the second form simply gives a range of IP addresses, which may or may not (more usually) represent an address that could be written in CIDR form. This latter form is used where an address in a CIDR form has an address (or addresses) in range that need separate handling from the CIDR forms.

A third form, infrequently used, is a single address. This is used to indicate a single whitelisted (but could be black) address that is separated out from a range of other addresses. Currently there are only 2 such entries, one for my own server at ajhurst.org (Digital Ocean), and Nathan (part of the Newfold Digital network, otherwise grey listed)

Each line of the database is network range in one of the three forms of address or address range, followed by a colour flag ('*',' ','.'; black, grey, white resp.), followed by a 2 character country indicator, followed by a network group name. One or more blanks separate these fields.

Notes:

RIPE Network

Réseaux IP Européens Network Coordination Centre

12. The Makefile

The Makefile handles the nitty-gritty of copying files to the right places, and setting permissions, etc.

install python file is an XLP macro that takes four formal parameters. These are:

machine

defines the machine for which this python script is to be built. (The target machine)

address

defines the domain name of the target machine.

interpreter

defines the python interpreter to be used in running this script.

homedir

defines the home directory on the target machine.

The install-system targets are designed to cater for the variations in interpreters and home directories required for each of the servers installed by the Makefile. Currently, all home directories are the same, but this is not necessarily the case (for example, in MacOS systems).

Note that this has not been updated for setting the filecache module.

The machines listed in the <install python file 12.1> calls are those machines for which we want to run a web serving page (Apache 2).

13. TODOs

20240513:173848

requestedFile traces now need extra infor re private v public

20240504:135256

Need better handling of white listed entries. For example, Nathan's address (66.96.163.139) is in the middle of 66.96.128.0/18, which is otherwise grey listed. Want a white listing to override and otherwise grey listing.

20240211:150655

Check that all machines do indeed serve properly. Note that there are some gotchas, like whether the local files are present, correct, and with proper ownership and permissions.

14. Indices

14.1 Files

14.2 Chunks

14.3 Identifiers

15. Document History