NEW BOT FILTERING!

Start of IP checking

End of IP checking

View and Rank Trains

John Hurst

Version 1.0.0

20161223:102357

Table of Contents

1. Overview

This document describes various python cgi scripts for viewing the author's railway photograph collection. The collection is held in a database (directory plus sub-directories) that can be accessed from a variety of web pages (see for example, my school server page). There are two main scripts, one to view an image at full resolution, and one to view the popularity rankings of the images. Other scripts described here help in maintaining the system.

The first script defined is <viewtrains.py 2.1,2.2>. This script takes a single parameter, which is the short name of an image, and searches the database for that image, and then renders it at full resolution, along with data about the image retrieved from the associated XML file. The original version of this script generated the output HTML directly. This has been re-engineered (20161222) to generate an XML file first, which is them translated to HTML. This is to make the script consistent with all the static files in the train catalogue.

The other main script is <ranktrains.py 3.1>, which delivers a page of thumbnail images in order of popularity. Because of the large number of images, each such page delivers only a few of the total number of images, but buttons are included to navigate around the complete rankings.

<tidytrains.py 4.1> reorganizes the ranking files. It reads two files to determine current rankings. The first file contains a single entry for each image, containing the current voting score, and the second file contains a list of votes since then. This ranking is then used to determine if any images should be removed from the system, depending upon whether or not their rank is less than some threshold.

<ranking.py 5.1,5.2> reads two files to determine current rankings. The first file contains a single entry for each image, containing the current voting score, and the second file contains a list of votes since then. A third file is then constructed, containing the updated rankings.

<rank.py 6.1> provides three procedures (strtotime, rankdata, ranklog) for other programs.

2. viewtrains.py Main Body

Here we do all the definitions. There is a subtlety here, as we cannot define all the global constants until we upon which server we are running. Hence the true globals are defined first, then we identify the server (which also defines globals local to the particular server OS, then we define all the globals relevant to this particular program.

Note that <define global constants 2.4> and <determine server environment 2.9> are also used by other programs defined within this literate program.

The main work of this script is done in one of the two procedures display and visit, depending upon whether the user offered a relative image path or not.

Somewhat contradictory to normal usage, a relative path refers to the location of the image relative to the base trains directory, rather than the root directory, hence the terminology. A non-relative path simply gives the image name, and hence a full directory search must be carried out in order to locate the image.

Note that some global constants are determined only after the server has been identified. For this reason, the <determine server environment 2.9> must be invoked after the other global constants are defined.

2.1 Initialisation

2.1.1 Viewtrains Imports

2.1.2 Define Constants

2.1.3 Define Regular Expression Patterns

2.2 Perform the XML Translation

2.2.1

2.3 Supporting Code

2.3.1 Collect cgi parameters

Use the Python library to retrieve cgi parameters. Currently there only one, image, which is the name of an image in the train library. Two alternatives are available:

1. The parameter starts with "trains/", in which case it is a relative pathname into the trains directory, and no searching is required; or
2. It does not, in which case the name must be searched against the image library to find the required image.

Discard any ".jpg" suffix.

Escape any suspect URL parameter characters.

2.3.2 Collect Previous Rankings

All previous rankings have been reduced to a single vote value for each image. These values are stored in a file RANKINGS, together with the date and time of the rankings. These votes are exponentially decayed, and used as the base values for any additional votes cast since that date.

Call the rank module to collect current ranking data. This consists of a quadruple:

2.3.3 Headers and Trailers

2.4 Define Subroutines

2.4.1 Define Subroutines

These procedures are of sufficient significance that they have been moved to a separate section.

2.4.2 Define function convertIPtoHex

The logic of this function is simple enough: extract the integer (decimal) values of each field in an IP address, and convert each to a two-digit hexadecimal value. Concatenate all these into a single hex string, which is returned.

2.4.3 Define procedure log

Every image access is logged, for recording its popularity. The exceptions are where there is an explicit request not to log (dontlog is true), and where there is some problem in delivering the image.

2.4.4 Determine rank of image

Match the various fields in a logfile entry. These are the year, month, day, hour and minute of the entry (in the format YYYYMMDD:hhmm), followed by the IP address (now stored in hexadecimal, but originally in decimal, and before that, not at all), and then the image address, including the base directory trains/, which is stripped off. Note that any logging of whether the image actually was served, or had already been voted upon, has been discarded previously.

The relative path to the image is the absolute pathname for the image, with all prefix directories upto and including trains/ stripped off. This gives a unique canonical form for the image. Note that the image type .jpg has already been stripped.

2.4.5 Define the XML Dispatch Routines

2.4.6 Define procedure to display the image

This is the procedure that does most of the real work in displaying the full image.

The file to display is accessible, so generate the HTML reference to it, print the name of the XML file, then parse it in order to display the various attributes relating to the image (as defined in the XML file).

2.4.7 Define Procedure to Search Directories

The procedure visit is called when we have an image name, but no path to the image. The procedure recursively visits all directories reachable from the initial parameter dir, and if it finds the image, calls display to perform the actual display of the image. It then returns. Thumbnail directories are skipped. It is assumed that the initial dir parameter contains the substring trains/, indicating where the trains subdirectory begins.

3. ranktrains.py

ranktrains.py is a web script that delivers pages of rankings for the railway database. Each page is limited in size (currently 25 images), and there are navigation buttons to browse forwards and backwards through the rankings. Images are buttons that take the viewer to the full-size image (via viewtrains), while image titles are links that take the viewer to the home page of the image.

3.1 ranktrains: Define constants

This macro is also defined in other sections.

3.2 ranktrains: Collect cgi parameters

3.3 ranktrains: Collect previous ranking information

3.4 ranktrains: Update Rankings with Latest Log Info

3.5 rankings: Print Forward and Backward Buttons

3.6 ranktrains: Generate Image Rankings

3.7 ranktrains: Print rankings table

3.8 ranktrains: Print Ranking Analysis

3.8.1 print header of html page

This code prints the header part of the html page.

print the starting lines, then ...

print the page title,including the "MONASH UNIVERSITY", "INFORMATION TECHNOLOGY" and "Clayton School" parts.

Generate the trains image on the trains page. We do this from the list of available images in web/images/banner (added manually to this list), by choosing one indexed by the low order bits of the current microsecond, that is, pseudo-randomly.

Now complete the final part of the header. A warning message about using Internet Explorer is also added, as that application is not W3C compliant.

3.8.2 Print Trailer of HTML Page

4. tidytrains.py

read two files to determine current rankings. The first file contains a single entry for each image, containing the current voting score, and the second file contains a list of votes since then. This ranking is then used to determine if any images should be removed from the system, depending upon whether or not their rank is less than some threshold.

5. ranking.py

This program brings the rankings data up to date. Normally, the trainrank file is untouched by normal browsing, and trainviews simply log an extra entry in the trainview file. Running ranking.py analyses the votes in trainview, adjusts the rankings from trainrank according, and outputs a new version of trainrank.

It was designed this way to avoid race conditions if two or more users simultaneously vote for any railway images, and to avoid multiple update errors on the rankings file. In practice, I am not sure if this is warranted, or indeed whether it avoids the problem. More thought needed.

Currently, the first two parameters to ranking.py are the current files trainrank and trainview (in that order), and the third parameter to ranking.py is the new file that takes the place of trainrank. It is the user's responsibility to check that these files are processed correctly, and then move the new file in place of the old, and to empty the trainview votes file. Perhaps this will be done automatically sometime in the future?

20100610:143701 Note that the : in the IP field (group 6) has been added to cope with IPv6 naming conventions.

6. rank.py

This module provides three procedures for analysing the rankings of John's railway photographs. Rnaking data is stored in two files, generically known as the trainrank and trainview files. The first stores ranking data for a set of images, as computed at a specified data and time. The second stores access requests for images in the database, together with the time and IP address of the request. Note that there is one entry for each unique image in the first file, whereas there may be multiple entries in the second file for any given image.

6.1 rank: define the strtotime function

6.2 rank: define rankdata procedure

The rankdata procedure reads the train rank file (the path to which is passed as the parameter), containing a date and time when the file was created, together a line for every image in the database. Each line contains the image path, starting at the trains subdirectory, together with the (decayed) vote value. Each vote is decayed by an exponential factor votefactor, where the exponent is proportional to the length of time since the creation date of the file. An updated vote value is entered into an associative table, which is returned, along with various housekeeping values of totalimages (the total number of distinct images processed), and datatime, the elapsed wall time spent in processing this file.

6.3 rank: define ranklog procedure

7. The Makefile

The Makefile handles the nitty-gritty of copying files to the right places, and setting permissions, etc.

8. TO DO

• Make link of "rank n" to anchor point in ranktrains.py. First need to add anchors to ranktrains.py to make this work.

9. Document History

10. Indices

10.1 Identifier Index

10.2 Chunk Index

10.3 File Index

54 accesses since 20 Jul 2020, HTML cache rendered at 20180703:1431