UpDater: A mirroring addon for SREhttp/2 (beta 1.20)

Contents

• Introduction
    CGI version
    Feature list
• Installation
• Defining Packages on a Central Server
• Example of a package file
• Defining a Sitelist on a Remote Server
• Example of SITELIST.IN
• Using encryption
• Appendices
  1. File list
  2. User-configurable parameters
  3. Installing UpClient as a NetScape helper application.
  4. The UpDater API
  5. Disclaimer and terms of use:

To minimize bandwidth requirements, UpDater uses GZIP and Rsync. Rsync is a differencing algorithim that is used to transmit only the fraction of a file that has changed, and GZIP is used to compress this fraction.

What is especially nice about RSYNC is that the central-server does NOT need to maintain old versions of files. Instead, the remote server sends a synopsis of an old version of a file, and the central server uses this synopsis to compute a difference file. Thus, one trades off processor utilization (the need to compute a synopsis, and to compute a difference given this synopsis) to lessen bandwidth requirements and to remove the need for the central server to maintain extensive archives of old versions.

UpDater has two components: a remote component, and the central component.

• A central server can distribute an unlimited number of packages
• Each package can have an unlimited number of files, defined using wildcardable include and exclude file descriptors.
• Access to specific packages can be limited to authorized users
• Packages can be obtained by remote servers or by remote clients
• A remote server can access any number of central servers, and is managed via a www-aware user interface (using HTML forms).
• Two remote clients (a non-GUI stand-alone program, and a NetScape helper app) can access any number of central servers.
• GZIP and RSYNC are used to reduce the size of file transfers (both of these encodings can be suppressed on a file specific basis)
• A package specific installation program can be run on a remote client after all the files have been UpDated.
• Files can be encrypted before being sent to the client.
• To avoid tcp/ip overhead, a single connection is maintained and used for numerous file transfers.

UpDater is installed using an installation program named INSTALL.CMD.

1. Unzip UPDATER.ZIP to an empty temporary directory.
2. From an OS/2 prompt,
   1. Change your default directory to this empty temporary directory
   2. Run INSTALL
   For example, assuming you unzipped UPDATER.ZIP to E:\TEMP

      C:>e:
      E:>cd temp
      E:\TEMP>install 

3. Answer the questions that appear on your screen.
   In particular, you'll be asked to provide the name of an UpDater directory.
   This should be an empty directory (it will be created if it doesn't currently exist).

After installing UpDater on a central and a remote server (or, on single machine if you want to test it out), you'll need to create a few files.

In particular:

1. Central servers use "package files"
2. Remote servers use a SITELIST.IN file.

   If you will be using the "remote client" mode, you will also need to

3. Install UPCLIENT.CMD as a NetScape helper application.

Having completed the preceeding, you can run UpDater. It's easy to do:

Notes:

• UpClient's user interface is non-gui.
• The various .CMD files (UPDATER.CMD, RUPDATER.CMD, UPCLIENT.CMD, UPDATEME.CMD, and UPDAEMON.CMD) contain several user-configurable parameters -- see Appendix 2 and Appendix 3 for the details.
• If there are large files in a package, or if a a package contains many files, , you may need to increase your SREhttp/2 and SRE2003 timeout limits.
• It is possible for a remote client to talk to a central server directly -- so as to review what's there, and to get one file at a time. To do this, point your browser at the central server, and enter /UPDATER?PACKAGE=!LIST in the location line.
• If you need to encrypt files before sending them, please see section V.
• The UpDateME program can be run from anywhere -- you will be prompted for a number of parameters.
• The various UpDater programs write status messages to a PMPRINTF window. To view a PMPRINTF window, get the free PMPRINTF program at http://www2.hursley.ibm.com/goserve

To define a package on a central server, you must create a package file. These are fairly simple, "mime-like" files; they must have an extension of .PKG, and they must be in the PACKAGE subdirectory of the UpDater directory (that you defined when you installed UpDater). For example, E:\SRE2003\SREHTTP2\ADDON\UPDATER\PACKAGE\SAMPLE.PKG.

A package file contains several entries, with each entry seperated by a blank line. Entries must begin with a "name:" (without the quotes), where name is one of the following:

   title:               -- the title of this package
   description:         -- a longer description of this package
   privileges:          -- required privileges (possibly *)
   include:             -- files to include
   exclude:             -- files to exclude 
   no_encode:           -- files for which content-encoding is suppresed
   cache:               -- days to cache a package's "list of files"
   run:                 -- an installation program to run after updating
   encrypt:             -- supported encryption methods

cache: days to retain "list of files"

number of days to retain the "list of files" (in this package).
To reduce the amount of time spent generating "list of files" from the definitions found in a package, you can specify the number of days to use a "cached" version of this list. For rapidly changing packages this is not a good idea, but for rarely changing packages that contain lots of files the use of cache can reduce processing time (hence speed up throughput).

description: description of this package.

A long, typically multi-line, description of this package.

encrypt a space delimited list of supported encryption methods

These are encryption methods that the central server is able to apply to the files in this package.

exclude: a list of files to exclude

A list of files to exclude, with * wildcards. You can use a trailing /S to include subdirectories. Files that match an include: entry are checked against exclude: entries, and if a matching exclude is found, they are not used.

include: a list of files in this package. REQUIRED.

This is a list of files that comprise this package. Each item in the list should appear on its own line, and should have two values: a relative destination directory, and a fully qualified file name.
The fully qualified file name can include wildcards (* characters). In addition, you can also include a /S modifier at the end, to indicate include files in subdirectories.
Basically, UpDater will copy the files on a central server (as identified by these filenames) to a relative (to a base-directory you specify) directory on a remote server.

no_encode: a list of files to NOT content-encode

A list of files, using the same syntax as exclude. Files in no_encode will not be GZIP'ed or RSYNC'ed. This is especially useful when you have .ZIP files, or other compressed files -- since further compression rarely helps, and differences are often difficult to detect.

privileges: SREhttp/2 privileges required to obtain this package.

These are user-specific privileges that are set by SREhttp/2. Note that these are in addition to any privileges that SREHttp/2 may require on a selector specific basis. In other words, you might have a general set of privileges that permit access to the UpDater addon, and a variety of specific privileges required for each of several packages.

Note:	If you do not want to limit access, you should NOT specify a privileges: entry.

run: execute an installation program on a remote client.

You can specify a program to run after the helper application has completed. This will only be run if the client permits it -- which requires BOTH checking a box in a form, and answering Yes to a "do you want to run" question asked by the helper application.

The syntax of run: is ...

run: [relative_path]program [PM] [arg=a sting]

where

• [relative_path]
  -- the path (relative to the destination directory on the client's computer) that contains the program to run. For example /cmds/foo.cmd means "the foo.cmd program in the destdir\cmds directory" If you do NOT specify relative_path, the program must be in the remote computer's PATH.
• [pm]
  -- If PM is specified, then this is a presentation manager program. If you do NOT specify PM and you execute a presentation manager programs, an error will occur.
  For .CMD files, if you specify PM, then the child process (launched when the command file is run) will NOT be automatically closed.
• [arg=]
  -- If arg= is specified, then you can specify a string (immediatly after the = sign) to be fed to this program.
  Note if you specify PM and arg=, the PM must preceed the arg=

title: the title of the package. REQUIRED.

This should be a fairly short (50 character) name for this package. It will be used by "remote servers" when communicating with clients.

• lines that begin with a semi-colon (;) are comments, and are ignored.
• / and \ can be used interchangeably. Leading and trailing \ (and /) are ignored.
• entries are case insensitive

• You can use decimal fractions when specifying cache. For example,

  cache: 1.5

  means 36 hours.

• Not specifying a cache: entry, or using cache: 0, mean do not cache.
• Cache validity (whether it's too old) is determined by looking at the datestamp of the "list of files", and not at the datestamp of every file mentioned in this "list of files".
• Cached versions of "file lists" are stored in the PACKAGE subdirectory of the central server's updater directory. They will have an extension of .PKC.
• Examples of run:

        run: /install.cmd
        run: epm PM READ.ME  
        run: /bin/myinstall PM in\params.1 out\out.2 

  Note that since EPM is NOT preceeded by a /, it will be run from the client's machine PATH.

   ;----------------------------------
   ; example of a package file
   title: The srehttp/2 files

   description: This is the srehttp/2 distribution files, including
   a set of addons. 

   privileges: *

  include:
     \                f:\srehttp\*
     \lib             f:\srehttp\srf\*
     \addons\updater  f:\srehttp\addons\updater\* /S
     \newstuff        e:\work\dec99\*  
     \htmls           g:\users\*.htm* /s

   exclude:
     f:\srehttp\*.ZIP
     f:\addons\*.FAQ /S

   no_encode:
    f:\newstuf\*.zip /S
    f:\srehttp\*.GZ 

   cache: 0.5  

   run: /install.cmd

   encrypt: blowfish des

   ; end of example
   ;----------------------------------

• leading and trailing \ (or /) in the "relative directory" entries are optional
• a * is used as a wildcard. It can be used anywhere in a filename (but not in a directory name).
• * a /s after a wildcard filename means include/exclude/no_encode files in subdirectories

  E:\SRE        -- will contain F:\SREHTTP\* but excluding files that end
                  with .ZIP.
  E:\SRE\LIB    -- will contain F:\SREHTTP\SRF\*, but excluding  files that
                   end with .ZIP
  E:\SRE\ADDONS\UPDATER -- will contain F:\SREHTTP\ADDONS\UPDATER\* and
                           it's subdirectories, but excluding files that 
                           end with .FAQ
  E:\SRE\NEWSTUFF --- will contain E:\WP\DEC99\*.* 
  E:\SRE\HTMLS   --   will contain G:\USES\*.HTM* (and files in subdirectories
                      that match *.HTM*)

A SITELIST.IN file contains "mime-like" records, with each record containing several single-line entries, and with records seperated by blank lines.

Entries have the following structure:

name: value

where

name may be one of address, description, key, password, root, site, or username.

address: Required

The IP address (name or numeric) of a server running the central component of updater.

Example:

address: srehttp2.srehttp.org

description: Site title.

A one line description of the site.

Example:

Description: This is the base version of SREHttp/2.

key: a shared-secret

Key is used to specify a shared-secret. This shared-secret is used as a key for decrypting files from this site, assuming the file has been encrypted due to an Encrypt: option in the package file.

• Key is case-insensitive (it will be capitalized), and should NOT contain embedded spaces (only the first word will be used).
• If key is used, password and username must also be specified -- since the server uses the user specific secret privilege to determine what key to use when encrypting.
• Of course, the key entered in key:, and the key extracted by the server from an SREhttp/2 secret privilege, must be identical!

Example:

   key: wa752a99

password: Optional

A password to send to the central server. Use this with Username: Note that the username/password will be sent to the central server (at the address:) -- it is not used to control access to this remote server.

Example:

username: joe
password: schalbotnik

root: required

A space delimited list of fully qualified directories. These are directories on this (remote) server to which files from a package can be copied to. You can specify one, or many, directories -- when it's time to download a package, you'll specify a "destination directory" that must be under one of these "root" directories.

This is a security option -- it prevents hackers from writing junk all over your machine (or, worse, causing files to be deleted). We recommend that these "root directories" be dedicated to the storage of these "packages", or that you exercise caution in who you grant UpDater access to.

Example:

root: G:\STUFF F:\GOODJUNK

site: Required.

A one word "nickname" for the site that is running the central component of UpDater (as identified by the address:).

Example:

site: SREHttp/2_home

username: A username to send to the site

The remote server will send the username and password are sent to the central server site as part of a basic authorization header. The username and password can be used by a central server to limit access to selected packages.

Note that this username & password are not the username & password that the remote server may require. That is, the remote server may have access restrictions placed on using the remote server component (RUPDATER.CMD; restrictions that require a username & password that have nothing to do with the username & password required by a package on a central server.

In other words, the remote server component (RUPDATER.CMD) has two levels of access controls:

• For a client to use RUPDATER.CMD, she must have access privileges defined for her on the remote site
• For the remote site to talk to a central site, it must have access privileges defined for it on the central site, and an appropriate username and password defined in SITELIST.IN.

Thus, a superuser of both a remote server and also of a central server might not be able to use RUPDATETR.CMD to instruct the remote server to obtain files from a central server -- it depends on whether an appropriate sitelist.in entry is specified (an entry that contains a username and password recognized by the central server).

Example:

username: joe
password: schalbotnik

  ;========================================
  ; This is a sample sitelist.in file for UpDater
  
   site: srehttp
   address: www.srehttp.org
   description: The SREHttp/2 home page
   root: e:\srehttp f:\srehttp

  site: cool_stuff
  address: www.invisible.net
  description: Wonderful stuff,but not for everyone
  username: mutwo
  password: xy321
  root: g:\wonders\set1

 ;========================================

To do this, you need to:

1. specify the supported encryption methods in the package file
2. specify an ?ENCRYPT secret privilege for appropriate users
3. optional install encryption/decryption software on the central server and on the remote client (or server).

Step b is accomplished by modifying user specific privileges (see SREhttp/2 USERS.HTM for the details).

Basically, for users who are granted access to this UpDater package, you should add "privileges" of the form:

?UPDATER:a_shared_secret

Step C can be accomplished in one of several ways:

1. using "encryption" and "decryption programs.
2. using encryption and decyption procedures
3. using the built-in SIMPLE encryption algorithim

a) Using programs.

To tell UpDater how to use this software you must create an ENTRY. "stem" variable in UPDATER.CMD (the central server proram), and in UPCLIENT.CMD (the remote client program) and/or UPDAEMON.CMD (the remote server program).

ENCRYPT. variables should have the following format:

      ENCRYPT.!METHOD=progname opt1 %in  opt2 %out opt3 "%key" ...
  where:
       METHOD - the "method"  (case-insensitive).
     progname - the program name 
      opt1 .. - options understood by the program
         %in  - a placeholder for the input file 
        %out  - a placeholder for the output file
        %key  - a placeholder for the "key". This will replaced by the
                xxx in the ?UPDATER:xxx shared-secret.

  Examples:
     encrypt.!blowfish=e:\apps\bf -q -e %in %out "%key"
         could be used by the UPDATER.CMD to "encrypt", and 
     encrypt.!blowfish=g:\bf -q -i %in %out  "%key"
        could be used by UPCLIENT.CMD to "decrypt".

b) Using a procedure.

       ENCRYPT.!METHOD=proc_name(arg1,%out,arg2,%in)
   where the ( must IMMEDIATELY follow the proc_name (no intervening
   spaces allowed). Otherwise, use arguments exactly as you
   would if you were calling this procedure from your own rexx program.

   For example:
        ENCRYPT.!MYWAY=encfast(%in,%out,%key,"q")

c) Using UpDaters built-in facility.

Notes:

• UpDater will replace %in and %out with the approriate filenames.
• You can freely mix %in, %out, and the options & arguments.
• Be sure to put an ! before the METHOD.
• you can use the UPDATER?SEND= option as a means of distributing decryption software.
• BLOWFISH (available at hobbes.nmsu.edu) is a very effective encryption/decryption tool.
• Do NOT use ' characters when defining an ENCRYPT.! stem variable. You CAN use " characters instead. If ' is necessary, use ''.

 install.cmd  -- The installation program.

 rxzlib.dll   -- A dynamic link library used to gzip (requires EMX)
 rxrsync.dll  -- A dynamic link library containing several useful procedures.

 sample.pkg   -- A sample "package" file
 sitelist.smp -- A sample SITELIST.IN file

 updater.htm  -- This documentation file

 updater.cmd --  The "central server" component

 rupdater.cmd -- The "client interface" portion of the remote server component
 updaemon.cmd -- The "daemon" portion of the "remote server" component

 upclient.cmd -- The Netscape helper app (that is used as a remote 
                 client).
 updateme.cmd -- Stand-alone "remote client" component

UPDATER_DIR: The UpDater directory (RUPDATER.CMD and UPDATER.CMD)
   This is where UpDater stores various files, and where UpDaemon.CMD is
   installed.  Note that UPDATER.CMD and RUPDATER.CMD are installed to
   your SREhttp/2 addon directory.

   Examples:

     updater_dir='F:\UPDATER'
     updater_dir=''

   Notes
     * the second example means use the UPDATER sub-directory
       of the SREhptt/2 addon   directory.
     * UPDATER_DIR (in RUPDATER.CMD and UPDATER.CMD) is set by the UpDater
       installation program.


ADDCRC: Compute an Adler-32 value when constructing a package (UPDATER.CMD)
   The CRC-32 value is used by UpDater's "only update if contents have 
   changed" option. If you do NOT enable ADDCRC, then this option will 
   not work.
 
   Set ADDCRC=1 to enable, ADDCRC=0 to disable.

   Note that computation of an Adler-32 takes a bit of time, but not much
   (typically, less then a second per Mb of file).
  
   Example:
       addcrc=1


ALWAYS_USE_CACHE:  Always use cache entry if it's current (UPDATER.CMD). 
   In a few places, you can "request the latest version" (this causes 
   a &nocache=1 option to be added to a request).  If you do NOT want
   to allow clients to suppress the use of cached "list of files in a 
   package", set ALWAYS_USE_CACHE=1. If you do want to allow this,
   set ALWAYS_USE_CACHE=0.

BACK1: The "body background" (RUPDATER.CMD and UPDATER.CMD)
   BACK1 is used to set the background color (or image) used when UpDater 
   communicates with client.

   Examples:
      back1='background="/imgs/web2.gif"'
      back1='bgcolor="#aa9933"'


CanDoGZIP: Enable GZIP encoding (UPDATER.CMD, UPDAEMON.CMD, UPCLIENT.CMD
           and RUPDATER.CMD)
   Set this to 1 to enable GZIP encoding, 0 to disable.
   GZIP encoding takes a bit of CPU time, but can substantially reduce the 
   size of a file (hence saving bandwidth).

   Note that, as currently built, the rxZlib library requires
   EMX 0.9d (emxrev should report 60 or above).  You can get emx from
   http://hobbes.nmsu.edu (search for emxrt.zip).

   Example:
      candogzip=1


CanDoRsync: Enable RSync differencing (UPDATER.CMD, UPDAEMON.CMD, 
            UPCLIENT.CMD, and RUPDATER.CMD)
   Set this to 1 to enable RSync differencing, 0 to disable.
   RSync differencing takes a bit of CPU time, but can substantially reduce 
   the size of a file (hence saving bandwidth).


   Example:
      candorsync=1

Timeout: number of seconds to wait between socket calls (UPCLIENT.CMD and
         UPDAEMON.CMD)
   If a socket call takes longer then timeout seconds, then UpDater
   will give up on this file.  Large values may be needed over 
   very slow lines. Note that Timeout is NOT the total time required to
   get a file, it's the max seconds for each 1000 byte "sockrecv" call.

  Example:
     timeout=30


Use_abbrev: use numeric abbreviations 
   If Use_abbrev=1, then numeric abbreviations are used to display
   files size (i.e.; 28K instead of 28126).
   Otherwise, the full number is display.

  Example:
     use_abbrev=1 

Assuming this describes you, after downloading UpClient.CMD you should

1. Copy UPCLIENT.CMD to a directory in your PATH (i.e.; to x:\OS2\APPS, where x: is your boot drive)
2. Tell NetScape to use UPCLIENT.CMD with all application/x-updater responses.
   To do this, you should:
   1. Open up the Preferences menu in Netscape (in NS 4.6, Edit-Preferences)
   2. Select Applications
   3. Select the New Type option
   4. Fill in the fields of the New Type screen with:
      1. Description of type: UpDater Client
      2. File Extension: UPD
      3. MIME Type: application/x-updater
      4. Application to use: "cmd.exe /c "x:\os2\apps\upclient.cmd" be sure to include the " characters

Note that the INSTALL program will setup your central server with a page that describes how to install UpClient, and which contains links to UPCLIENT.CMD, RXRSYNC.DLL,and RXGZLIB.DLL.

To see this page...

just hit the central server with a /UPDATER? request, select any package, and select the using the UpClient helper app link.
Or, use a request of /updater?action=UPCLIENTDOC.

Once UpClient is installed, you can use your browser to retrieve a package from a central server (that is running UpDater). After making the initial request (say, to /updater?), and selecting a few options, Netscape (or whatever browser you are using) will run UpClient.CMD in a seperate window. After asking a few more questions, UpClient.CMD will use tcp/ip socket calls to GET files and write them to your hard drive (and possibly run an installation program after all the files have been retrieved).

For details on rxRsync and rxZlib, see http://www.srehttp.org/apps/rxrsync/ and http://www.srehttp.org/apps/rxzlib/

To assist in such endeavours, the following lists the "options" understood by UPDATER.CMD, which is the "central server component" of UpDater. The remote components (RUPDATER.CMD, UPCLIENT.CMD,and UPDAEMON.CMD) will not be discussed -- the notion is that you'll be replacing these with your own tools!

As a "server side program" (running as an SREhttp/2 addon) one communicates with UPDATER.CMD by using web requests that contain options. Although there are several ways of defining options (such as through html forms), for illustrative purposes this documentation will assume that options appear after a request string.

For example, in the URI

http://foo.bar.net/updater?action=display&package=craziness

the request string is

updater?action=display&package=craziness

the resource is

updater

and the options follow the ?, with options seperated by & signs. Thus, in this example there are two options,

action=display
package=craziness

Note that options typically have a varname and a value. For example

varname=action, value=display

4.a) UPDATER.CMD options

action: 
   An action to take. Recognized values are:
  
    display: Display a page describing the package that is identified by 
             the package option.

     file1: Get a file from this package, and return it. Also requires that an
            entry option be defined.

            The file will be returned using one of the standard mime type 
            (such as text/plain), or using a mime type of application/octet-stream 
            (if the file's extension  does not map to a common mimetype).

     file:  Get a file from this package, and return it (possibly after GZIP
            compression and RSYNC differencing). Also requires that
            entry option be defined. 

            The file will be returned with a mimetype of application/x-updater.
            Ax x-updater response header is added, that may contain
            several values (described below).

     getl:  Return a list of all the files in the package. This list
            is described below.
     
            An x-updater response header is also returned. You can
            examine this for "GZIP" and "RSYNC" entries -- they signify
            that the server can handle GZIP and RSYNC.

     getl2: Return an html form used by a client to specify where/how to update.
            Upon submitting this form, UpClient will be called and will be 
            passed a list of files in the package.

    upclientdoc: display a page that describes the UpClient helper app.

    Note: the "GETL3" and "GETL4" actions can be used to obtain specific files
          (and "sections") from a package -- contact the author for details.

ashtml:
    When an ashtml=1 option is specified in combination with a action=GETL, then
    the "list of files" will be returned as an html page containing
    "action=file1" links to the various files in the package.

    In addition, when ashtml=1 is specified with a package=!list option,
    then the "list of packages" will be returned as an html page containing
    "action=display" links to the various packages on the server.


client:
   Used with action=getl.
   When client=1, this request is from a "remote" client.
   This has two effects:
   i)  updater will check for some options (destination directory and 
       possibly username/password), and return an error message 
       if they have not been specified.
   ii) if ashtm<>1, some extra information will be added to the "list of files"
       header (see below for the details)


encrypt:
  A comma delimited list of encryption methods that the client supports. 
  This should be a subset of the server supported methods (as specified 
  in the ENCRYPT option in the "list of files", described in section 
  4.b.ii).
         
  Note: the capitalized shared-secret used by the central-server to 
        encrypt the file is pulled from an ?UPDATER:shared_secret
        "secret privilege".  It is your responsibility to know
        what this value is (UpDater does NOT use public-key codes
        or other SSL-like mechanisms).

entry:
   The "entry number" in the package. The entry number is basically the
   index into the list of "root directory\ies"  specified in the 
   .PKG file of the specified package.  It is combined with a
   file option to define a particular file.

   Example:
         entry=1
   
file:
   A filename; which may contain subdirectory information. The actual
   file this refers to will depend on the value of the entry (and package) 
   options -- the filename is relative to the subdirectory identified
   bye the entry option.

   Examples:
        file=foo.bar
        foo=/programs/math/calc.exe

getuser:
   When getuser=1, and client=1, and action=GETL, then updater will check that
   a username and pwd options have been specified.

gzip
   If gzip=1, then "use gzip, if possible, to compress the response"

md4:
   A 32 hex character MD4 hash of an old version of a requested file.
   If the md4 of the requested file matches this md4 value,
   then a "304 not modified" response will be generated.

nocache:
   If nocache=1, and action=getl, then do NOT return a cached version of
   the "list of files". This is ignored if ALWAYS_USE_CACHE=1 is set 
   in user configurable parameters section of UPDATER.CMD.

package:
   Specifies the name of a package on the central server.  Packages are
   specified in .PKG files in \PACKAGE subdirectory of the UPDATER directory.
   See section III (above) for details on package files

   This is an important option, almost all the other options refer to actions
   to take using this package.

   Example:
     package=sample
     package=my_package
     package=!list
   Note: do NOT included the ,PKG in the value
   
   The last example, value=!list, is used to return "list of packages". 
   See Appendix 4b for the details.

rsync:
   If rsync=1, then "use rsync, if possible, to shorten the response". 
   This MUST be combined with an "rsync-signature_updater" request header
   that contains the "rsync synopsis" of a "prior version" of the requested
   file.

send:  Transfer a file in (or under) the updater directory. This is currently
       used by the UpClient helper app page to allow clients to obtain
       the upclient.cmd (and other files). 

       The syntax is:
          UPDATER.DOC, RXZLIB.DLL, GZIP.EXE, RXRSYNC.DLL, UPCLIENT.CMD

       To add files to this list, you should change the SEND_FILES parameter 
       in UPDATER>CMD.

       Examples:
          send=upclient.cmd
          send=morefile\help.doc

       Note:
          SEND could have been specified using "action=SEND&file=filename". 
          However, since these files (in the updater directory) are NOT 
          identified with a "package" or "entry" option, we decided to use a 
          seperate option.
  
text:
   When text=1, and a send option is specified, then transfer the file as
   a text/plain mimetype.

backupdir dodelete check quiet dorun:
   These options are used when action=getl and client=1 (and asthml<>1). 
   They control what extra "remote client" specific information is
   added to the "list of files header".
   For example, UpClient uses the values:
      backupdir: the relative location of a backup directory
      dodelete: 1 (used to signal "delete all orphaned files from 
                destination directory"  
      check: 0,1,2 or 3. The time of "check" to do before requesting the file. 
                0=none, 1=time, 2=crc32, 3=add an md4 option 
      quiet: 1 (non-verbose reporting in UpClient)
      dorun: Name of an "installation" program 
        

4.b)The "list of files" in a package.

• list of packages
• a "list of files" in a package
• a file, that may have been GZIP'ed and RSYNC'ed.

4.b.i) List of packages.

   When package=!list, then a "list of packages" is returned. You can then
   use this list to choose a package for use in later requests to UPDATER.CMD.

   The format of this list is:
      n                
      yyyy/mm/dd/hr/min  Name1  title  1
          ...
      yyyy/mm/dd/hr/min  NameNthg  title nth

  where:
      yyyy/mm/nn/dd/hh      = timestamp (last revision of package file)
      name1 ... nameNth     =  name of the package
      title                 = a short, mutli word, description of the package
   
  Example:
    4
    1999/12/10/02/29 sample   This is a sample package file
    1999/12/10/00/17 SREHTTP  The SREHttp/2 package
    1999/11/23/21/14 ZIPFIP   The ZIPFIP package

   Note that the timestamp refers to the .PKG file, and NOT to files that may
   be referenced by the .PKG file.

4.b.ii) "List of files" in a package.

      header_line
      yyyy/mm/dd/hr/min  entry File1  relDir no_encode crc32 

      yyyy/mm/nn/dd/hh      = timestamp (last revision of package file)
      entry                 = entry number (index into the .PKG file)
      file1 ... fileN       =  name of file in this package
      relDir                = relative directory to copy file to
                              This is typically relative to a root directory
                              that you provide
      no_encode             = Hint to not attempt to encode this file
                              (RSYNC and GZIP should not be attempted)
      crc32                 = Adler-32 checksum (can be used to see if
                              file has changed)
                              Note that an md4 checksum is NOT send as
                              part of this "list of files"

         var=value  var1=value var2=value

        package         = required -- the package this file belongs to
        server          = requird -- ip address that this package is on
        selector        = recommended -- the "resource", on this server,
                          that will process UpDater requests. 
        GZIP            = optional. If GZIP=1, then the server knows GZIP
        RSYNC           = optional. If RSYNC=1, then the server knows RSYNC
        CREATED         = Timestamp, when this "list of packages" was created. 
                          Format is basedate.decimal_fraction_of_a_day
        PKG_DATE        = Timestamp, when this .PKG file was last modified.
        CURRENT_TIME    = Timestamp, when this "list of files" was sent.
                          Note that CURRENT_TIME is ONLY used when this is a
                          cached "list of files". CURRENT_TIME will always
                          be greater then or equal to CREATED.
        ENCRYPT         = A comma delimited list of encryption methods
                          supported by this server (for this package).                                                   

     Package=sample Server=localhost CREATED=730101.92986 PKG_DATE=730097.1035 
           GZIP=1 RSYNC=1 CURRENT_TIME=730102.02361
           ENCRYPT=BLOWFISH,DES  SELECTOR=UPDATER 
      1999/07/20/20/56 1 /ACCESCFG.CMD / 0 652FD5DA
      1999/11/11/14/33 1 /ACCESCHK.RXX / 0 A2D10DAA
      1999/06/19/10/30 1 /access.cnt / 0 45DD05E8
      1999/06/13/16/14 1 /access.in / 0 A69D859E

4.b.iii) Retrieving a single file

         Content-type: application/x-updater
         X-updater: alist

         Content-encoding: alist
         X-NotMod: 1

    The important options are:
         DIR=/rel_dir
         MD4=md4_value
         ERROR
    where
          rel_dir is a  "relative directory": it can be used
          instead of the "relDIR" specified in the list of files.
        
           md4_value is a 32 hex character md4 value.  It is only
           included when an RSYNC has been used.
           As an error check, this value should be compared against
           the MD4 value of reconstructed file.

           ERROR takes no value -- it signals that  "an error occurred"

     Examples:
         X-updater: dir=/sub1
         X-updater: dir=/home/particles,md4=3234a67abdf2f456ca90155156789012
         X-updater: dir=/secure/stuff1,encryt=blowfish

UpDater supports three kinds of content-encodings: GZIP, RSYNC, and ENCRYPT=method.

Note that the order of appearance of these content-encodings is the reverse order of their application --- you should do the last one first. For example,

       Content-encoding:rsync,gzip,encrypt=method1
     means
          ENCRYPT=method1 -- first deencrypt, using the encryption "method1"
          GZIP    -- then inflate (de-compress) it
          RSYNC   -- then apply this "GDIFF" difference file to
                     the old file 

     Examples:
         Content-encoding: gzip
         Content-encoding: rsync,gzip
         Content-encoding: gzip,Encrypt=blowfish

You may use this (or subsets of this) program as you see fit, including for commercial purposes; so long as proper attribution is made, and so long as such use does not in any way preclude others from making use of this code.