Web Page Management Software

John Hurst

Version 1.6.5

20231124:131647

Abstract

Table of Contents

1. Introduction

This document describes the files used to manage delivery of my personal web pages, and those that I manage for other organisations. The general form of web page delivery is a) a source file written in XML, b) a translation file written in XSLT, and c) the program described here, a python cgi script that calls the appropriate translator on the source file, and delivers the result. It also handles straight HTML, as well as providing some debug and other maintenance options.

The program is invoked by commands in the .htaccess file associated with each web directory. Different .htaccess files can be used for different directories. If none exist in a given directory, the directory path is searched towards the root until one is found.

The XSLT files used can be specified either in the .htaccess file (default), or in the source XML file, through an explicit xml-stylesheet command. If a stylesheet XSLT file is specified, it overrides the default .htaccess one.

Permission is given to reuse this document, provided that the source is acknowledged, and that any changes are noted in the documentation.

The document is in the form of a literate program, and generates all files necessary to maintain the working environment, including a Makefile.

2. Literate Data

This message flags the fact that the source code is a derived document, and should not be directly edited.

3. The checkCountry Module

checkCountry.py is a module that provides the class countries; instances of which have the methods load(), to load the database of known IP addresses, and checkIP(ips), to check if the IP string ips is known to the database.

It returns a triple (known, status, country, annotation), where:

known

is True if the IP is indeed known in the database,

status

has a value

• white (address is whitelisted),
• black (address is blacklisted), and
• grey (address has unknown status.

Define a few useful functions

4. The filterWebBots.py module

filterWebBots is a module that provides two functions for validating client IP addresses for access to this server. These are validateIPaddress and filterOnCountry. The first procedure is parameterless, and collects the client IP address from the server calling URL. It checks this address to see whether it is in a list of black-listed ip addresses. The second procedure takes a parameter which is the validated client IP, and checks it to see whether it is from a validated country (currently only Australia).

These functions provide useful tools for dealing with IP addresses. They are:

str2IP(s)

Converts the dot-decimal IP address string s to a binary value.

ip2Net24(ip)

Mask the ip address to a class A network address.

ipMask(bits)

Mask the ip address to a host identifier, whose size is determined by the parameter bits.

validateIPaddress checks the calling IP address to see if it is authorised to access this web page. It first loads a known group of bad addresses in dot-decimal form, maintained in the list badips, where each entry is a tuple of the form (ipadr,bits,co) and ipadr is the dot-decimal address converted to binary, bits indicates the size of the network prefix, and co is the country of origin (if known).

filterOnCountry does three things"

1. First test to see if client is a local machine; local network, localhost,

5. The Main Program index.py

This program is written in Python3, and relies upon Python v3.6 or later, as it uses f strings in many of its output statements. /home/ajh/binln is a common directory used to identify where required versions of interpreters are to be found.

We insert the usual edit warning to alert any code editor (person) to the dangers of directly edit constructed code. The version number is inserted into rendered HTML text.

The showBegins flag turns on tracing output on the stderr output to assist in identifying where program failures may occur, and linking these back to the literate code. Other global variable are also identified upfront, to further assist debugging the raw code.

This script processes all my web page XML files It requires apache to be configured:

1. to allow python files (this file) to run as cgi in user directories
2. to add a handler for XML files that calls this program
3. to pass as cgi parm the XSLT file that translates the XML file

The script relies upon picking up the required file and its XSLT file from a) the REDIRECT environment variables, and b) the script parameter, respectively.

The interpreter required varies according to the target server. This detail is captured by the <Makefile 12.2,12.3> script, although not all systems have yet been encoded into the Makefile script. The interpeter also changes its behaviour depending upon the host on which it is running. This is done by an explicit call to os.environ.

Gather together all the module and library imports needed for this program.

Start processing. Get a timestamp for recording key events in the log. Set the modification times to year dot.

This first code chunk here identifies who and what is being invoked in this instantiation of the server.

The second part identifies known IP addresses that have a track record of abusing the server. The algorithms used are somewhat heuristic.

Define a useful debug routine.

This fragment simply outputs the required header for flagging the generated content as HTML, and builds a number of string matching patterns for later use.

Collect relevant parameters from the original request.

The major work in rendering the required page is done by the <process file 5.35,5.36> code chunks. After this point, processing is complete, and the program falls through to exit.

5.1 Define Global Variables and Constants

ipVisitFileName defines the file used to collect the IP addresses of client callers.

returnXML is set True when the display of the raw untranslated XML is required.

convertXML is set True when a converted copy of the translated XML is required to be saved.

alreadyHTML is set True when the incoming file to be rendered is already in HTML and does not require conversion.

cachedHTML is set True when the incoming file to be rendered has been cached in the HTMLS directory, and does not require conversion.

5.2 Define Various String Patterns

Somewhat self-explanatory string matching patterns.

5.3 Determine the Host and Server Environments

The server is the address to which this request was directed, and is useful in making decisions about what to render to the client. Examples are "localhost", "www.ajh.id.au", "chairsabs.org.au".

The host is the machine upon which the server is running, and may be different from the server. This name is used to determine where to store local data, such as logging information. For example, the server may be "localhost", but this can run on a variety of hosts: "murtoa", "dimboola", dyn-13-194-xx-xx", etc..

BASE is set to the path to the web root directory on this server. It should not have a trailing slash!

HOME has its usual Unix meaning.

PRIVATE is set to the path to a working directory on this particular server that is used to store accounting and audit information about this particular access. The path includes a specific reference to the server hostname to uniquely distinguish it. This directory basename is rendered on the web page as the parameter server and is the first of the "server@host" pairs rendered at the top of the web page.

COUNTERS is the path to the directory containing all the web page access counts. Each counter is incremented on page access, whether to the cached HTML, or to the (re)rendered XML file.

HTMLS is the path to a local copy of html versions of the files. These are cached versions, and some mechanism to age and delete needs to be identified. If the corresponding XML file is older than the HTML file found in this subdirectory, the HTML version is used.

XSLTPROC is the path to the xsltproc processor. Without this processor, this entire script (as far as XML files are concerned) is meaningless!

5.4 filter out bad ips to which we do not respond

5.5 Handle JPEGs

From version 1.2.0 onwards, this code implements a form of caching for jpg files. A local check for the request file is made, and if it is not found, an attempt to retrieve it from the dimboola server is made. If that is not successful, the file is reported not found. If it is successful, the file is saved locally. No attempt is made to age files out of the cache.

This code checks to see if the request is for a jpg image file. These are cached, and if not present, are retrieved from the master jpg server for my jpeg images. This is still a bit experimental. The server URL is dimboola.infotech.monash.edu.au/~ajh/Pictures.

It requires that the .htaccess file be modified to refer .jpg requests to this cgi script.

5.5.1 Get JPG File from Remote Server

Generate the URL of the corresponding remote JPG file, and issue read request. By using the urllib.request library, we also get the modification time, which we parse in order to set the correct modification time on the locally cached copy.

Now try to cache a local copy. This can fail for several reasons, the main one being that the permissions in the local directory are likely to be against (write) access by the www user. More work is required to make this a bit more robust.

Note that we set the pair (access time, modification time) on the local file to be the current time and remote file modification time respectively. This ensures that attempts to synchronize the two file systems will see this file as the same file as the remote file, and not attempt to update one or the other (thus leading to spurious modification times).

5.6 Collect HTTP Request

{Note 5.20.1}

initialize the filename of the file to be rendered. Most of the work in computing the value of this variable is done in <get filename from redirect 5.22>

When this script is called, it has gained control by virtue of an .htaccess directive to Apache to use this program to render the source file. The name of that source file has to be recovered somehow, and different systems seem to handle this parameter in different ways. The first parameter to explore is the REDIRECT_QUERY_STRING, which, if it is present in the form request, contains secondary parameters to the rendering operation. If this parameter is not present, initialize the variable form to an empty value.

5.6.1 Handle REDIRECT QUERY STRING

There are several possibilities for secondary parameters. The primary one is the debugFlag parameter, which can be set to true, indicating that debugging information is to be printed along with the rendering. This is intended for administrator access only, but as it is harmless, there is no authentication required.

The other parameter that can be offered at this point is the xml parameter, with values of true or convert. The first of these forces no conversion of the XML, but simply copies it to the browser, substituting escape sequences for any special XML character sequences so that it appears as verbatim XML.

The second choice, convert, allows the use of the rendering engine as an XML-to-HTML converter, in which case a copy of the converted HTML is saved to a temporary file. This file can be used subsequently as a statically converted file as necessary.

5.7 Get Filename from Redirect

5.8 Check for Abbreviated URL

5.9 Make File and Dir Absolute

5.10 Check for HTML Request

We now have a requestedFile name for the document to be rendered. We need to investigate this file to see how it is to be rendered. In particular, it may be an HTML file (indicated by a .html extension), or it may be an XML file previously rendered and cached. In these cases, we do not need to do any XML conversion, and the flag alreadyHTML is set true if it is an HTML file, or the flag cachedHTML is set true if it is a cached converted XML to HTML file.

This code now also checks for a cached version of the XML file, as per the following fragment.

Unless the file being retrieved is already an HTML file, check to see if we have a cached HTML version of this (XML) file. Note that any parameters to th http request (indicated by a non-empty form value) will abort the caching process, and force a reload of the XML file.

5.11 Get Default XSLT File

5.12 Scan for Locally Defined XSLT File

Look at modification time of XSL file. If it is more recent than the cached HTML file, we must re-convert the XML file.

5.13 Determine XSLT File

5.14 Update Counter

Compute the name of an XML counter file which contains a counter element with subelements value and date. The value element contains the current count value, and the date element is the date on which this XML file was initialised. We read the current count from that file, increment it, and update the file. This file is used by most xslt translations to output an access count in the footer. It is also used by the site map program to compute the intensity of accesses to this web page.

It was fortuitous, but this counter also keeps track of HTML accesses, both where an HTML file is the initial request, and where it is a cached version of the corresponding XML file. Since the XML files have their own counters included by the XSLT translator, the count attached to the HTML rendering allows a comparision of how many accesses are to the cached copy (the difference between the two).

For example, suppose the XML rendering gives 986 references, and the HTML rendering cites 993 references. The the cached HTML page has itself been referenced 7 times since it was first cached.

First we process relfile to find the counter name. Remove any extension, and replace all slash path separators with minus signs.

(Strictly speaking, the first sub is not required, but I've left it in, as it does no harm.)

Now try to read the counter XML file. The file may not exist if this is the first time we have accessed this page since this mechanism was set up, so we must capture that error, and any error arising from attempting to parse the XML, and create a new counter file, with value initialised to zero, and date initialised to today's date.

5.15 Process File

Decide what to with the file. There are 3 choices:

1. return the raw XML. This means escaping all the active characters, and printing the file verbatim.
2. The file is HTML, either because of an explicit HTML request, or a cached HTML file previously translated has been found. Again, the file is rendered verbatim, this time without escaping the active characters.
3. It is an XML file, and it needs translation. Call the XSLT processor to do that (chunk <process an XML file 5.38,5.39,5.40>).

Note that each line from the HTML file is printed without additional line breaks.

5.15.1 Process an XML File

Run the pipe to perform the translation. Note that this step requires an inordinate amount of time on some servers (sequoia in particular), and was the prompt for including the caching mechanism.

Note that in copying the rendered HTML version, we retain the lines as is, and make sure that they are rendered without any additional (or deleted) new lines.

5.15.1.1 Deal with any Conversion Errors

6. The Train Image Viewer viewtrains.py

The literate code for this program has been moved to ViewTrains

7. The Rankings Display Program ranktrains.py

The literate code for this program has been moved to ViewTrains

8. The Train Ranking Module rank.py

The literate code for this program has been moved to ViewTrains

9. The Web Server Module webServer.py

The various web server programs all make use of a pair of values, known as host and server. The host value defines the machine on which this program is running, while the server value defines the name by which the program was invoked (effectively the domain name of the server). This is necessary, as each program may be invoked by different domain name paths, and the service rendered may be different for each path.

While different domain names may invoke services on the same machine or host, it is never the case that different machines are invoked to handle a single domain name service.

When invoked as a stand-alone program, a test server parameter may be passed in to see what the program determines. This is for testing purposes only, and serves no useful purpose otherwise.

10. File Caching

10.1 The File Cache Module

This code was adapted from an example given on a web page. Sorry, I have forgotten the reference.

10.2 Clearing the Cache

This program clears the HTML caches created by the previous module. It is called independently, and can clear either the entire cache, or subdirectories of it.

The cache is maintained on a per-machine basis, and the machine being used is identified by a hostname call.

11. Country Database

Addresses in this database come in two forms: CIDR, and double IPv4 addresses. The CIDR forms are standard for defining a range of addresses; the second form simply gives a range of IP addresses, which may or may not (more usually) represent an address that could be written in CIDR form. This latter form is used where an address in a CIDR form has an address (or addresses) in range that need separate handling from the CIDR forms.

Notes:

RIPE Network

Réseaux IP Européens Network Coordination Centre

12. The Makefile

The Makefile handles the nitty-gritty of copying files to the right places, and setting permissions, etc.

The install.system targets are designed to cater for the variations in interpreters required for each of the servers installed by the Makefile.

Note that this has not been updated for setting the filecache module.

13. TODOs

20081111:103026

footer links filename is incorrect when a .html URL is used. Need to check calculation of filename.

14. Indices

14.1 Files

14.2 Chunks

14.3 Identifiers

15. Document History