Using SREhttp/2 Common-Log Audit Files

I. Introduction

In keeping with http server standards, SREhttp/2 will maintain common-log format audit files.

Besides offering a compact synopsis, a number of tools exist to help you analyze common-log files. Furthermore, as an adjunct to the common-log audit file, SREhttp/2 can also record the browser used by each request (as well as the referer) in seperate log files.

This document discusses how to modify the LOGS.CFG configuration file. LOGS.CFG (which should be in the SREHTTP2\CFG directory) contains the configuration options for these three log files. These options include the host-specific filenames for these log files, audit-suppression instructions, and a few display options.

LOGS.CFG is a general configuration file -- its values are used for all hosts (that is, there is no differentiation between hosts that are defined as superceding or non-superceding). In addition, it can not be changed on-the-fly; you have to restart SRE2003 for its changes to take effect.

II) Basic configuration

Configuration is accomplished by setting variables in LOGS.CFG. The following lists the variables, with details below:

ADD_HOSTNICKNAMES	add the Host-nickname to common-log entries
APACHE_MODE	Select format for Browser and Referer logs
BROWSERLOG BROWSERLOG.hostn	Default browser log file Browser log file for the "hostn" host.
COMMONLOG COMMONLOG.hostn	Default common-log file Common-log file for the "hostn" host (hostn is a host-nickname)>
COMBINEDLOG COMBINEDLOG.hostn	Default combined (common/referer/browser) log file Cominbedlog file for the "hostn" host.
FORCE_REFERER	Use a " - " when no referer request header is available
LOGFILE_MAXSIZE	rename log files once they get to this size
NOOPTIONS	Do not record the portion of the request selector following the ?
NOHTTP	Do not record the HTTP/1.0 portion of the request string
NOCODES	List of "return codes" and "http methods" to not record.
NOUSER	List of IP addresses, and SREhttp/2 "usernames", to not record
NOREFERER	List of IP addresses to not record in REFERER.LOG (COMMON, COMBINED and BROWSER logs do NOT pay attention to this parameter).
NOURL	Rquest selectors to not record
DISPLAY	Display log entries in PMPRINTF window or SRE2003 status screen
REFERERLOG REFERERLOG.hostn	Default referer log file Referer log file for the "hostn" host.

III) Configuration Details:

COMMONGLOG, COMBINEDLOG, BROWSERLOG and REFERERLOG

The COMMONLOG, COMBINEDLOG, BROWSERLOG, and REFERERLOG "stem" variables all have the same structure. The basic notion is that SREhttp/2 will take the host nickname (of the host to whom the request was directed), and attempt to find matching COMMONLOG (and COMBINEDLOG, BROWSERLOG and REFERERLOG) variables. If a match is not found, or if the request is to the default host, the "default" versions of these four files are used.

You should set these stem variable to equal filenames. If you use a relative file name (i.e.; FOO.BAR or DIR1\FOO.BAR), the file is assumed to be relative to the SREHTTP2\LOG directory.

If you do not want to record information to a common-log file, then set COMMONLOG=0. Or, COMMONLOG.HOSTN=0 to not record common-log information for requests to the HOSTN host.

Similarly, the COMBINEDLOG, BROWSERLOG and REFERERLOG variables are treated the same way. For example, to suppress the REFERERLOG, set REFERERLOG=0.

Example	COMMONLOG=COMMON.LOG COMMONLOG.ZOO=D:\CIRCUS\WWW\COMMON.LOG For requests to the `ZOO` host, record common-log information to `D:\CIRCUS\WWW\COMMON.LOG`. For request to all other hosts (including the default host), record common-log information to `D:\SRE\SREHTTP2\LOG\COMMON.LOG` (assuming that SREhttp/2 was installed to D:\SRE\SREHTTP2).

Note:	if you use `COMMONGLOG=0` syntax to suppress writing to a common-log file (or other log file), the `LOG` permission is ignored.

Example

 COMMONLOG=COMMON.LOG
 COMMONLOG.ZOO=D:\CIRCUS\WWW\COMMON.LOG

For requests to the ZOO host, record common-log information to D:\CIRCUS\WWW\COMMON.LOG.

For request to all other hosts (including the default host), record common-log information to D:\SRE\SREHTTP2\LOG\COMMON.LOG (assuming that SREhttp/2 was installed to D:\SRE\SREHTTP2).

Note:

if you use COMMONGLOG=0 syntax to suppress writing to a common-log file (or other log file), the LOG permission is ignored.

The NOxxx,and other options

The followings options allow you to modify the output to the common-log file, and to suppress the recording of selected requests.

ADD_HOSTNICKNAMES

Add a host-nickname to the request portion of a log entry. Note that this is not standard practice. Also note that if you've specified unique log files for each host (and for each log file), then adding a host-nickname is not particularly useful.

Syntax:

ADD_HOSTNICKNAMES=0 : do not add the host-nickname
ADD_HOSTNICKNAMES=1 : always add, even for DEFAULT host
ADD_HOSTNICKNAMES=2 : add, but not if default host

APACHE_MODE

If APACHE_MODE=1, then the referer and browser log's are written using an "Apache" mode (that is, something similar to what Apache server's write). If APACHE_MODE=0, then a "GoHTTP" mode is used.

For example:

      APACHE_MODE (browser and referer logs):
[14/Mar/1996:17:45:08] Mozilla/2.0 (X11; I; HP-UX A.09.05 9000/735)
[14/Mar/1996:17:48:10] http://guide-p.infoseek.com/Titles -> /~sret1/analog/
       GoHTTP:
[03/Jun/1997:19:35:16 -0500] DAL40-29.PPP.IADFW.NET Mozilla/2.02E (OS/2; I)
[2/Mar/1997:18:23:37 -0500] "/imgs/srefiltr.gif" http://ship.float.net/

FORCE_REFERER If FORCE_REFERER=1, then in cases where there is no referer, a " - " is used.

LOGFILE_MAXSIZE

LOGFILE_MAXSIZE is used to limit the size of the LOG files. Once a log file exceeds this size (as specified in Kbytes), then it will be renamed (using .001, .002, etc.). The renamed files will be moved to the SREHTTP2\LOG\OLD directory.

Syntax:	`LOGFILE_MAXIZE=nnn` where `nnn` is in kilobytes.
Examples:	`LOGFILE_MAXIZE=1000` -- maximum size of 1 Megabyte `LOGFILE_MAXIZE=0` -- no maximum size

NOOPTIONS

If NOOPTIONS=1, then the portion of the request selector following a ? will not be included. Since this can be quite long (as when you are using the GET method in an HTML FORM), inclusion of this information can lead to very long (and hard to read) entries.

For example: given a request for:
/LOOK1?arg1=sub1/&arg2=1&argx=1&xmac=0
if NOOPTIONS=1, the "request" portion of the common-log (and referers log) will be /LOOK1

NOHTTP Technically, a request string should end with the HTTP version. Since this is not necessarily interesting, and since it's not clear that the common-log format requires this information, you can suppress this portion of the request string by setting NOHTTP=1.

NOCODES

NOCODES should contain a space delimited list of 3 digit return status codes, and http-methods, that you do not want to record.

If the response code, or the request method, matches one of these NOCODES, then none of the log files are written to.

Examples:

you may not be interested in requests that generated a "authorization response" (a 401 response). If so, set:: NOCODES=401

If you are also uninterested in "moved" responses, you can set:: NOCODES=401 301 302
If you are not interested in "authorization responses" and responses to HEAD method requests:: NOCODES=401 HEAD
And if you are sort of perverse, and are not interested in normal responses, you can set:: NOCODES=200

NOUSER

NOUSER entries should contain a space delimited list of numeric IP addresses, or domain names. Requests from these clients will not be to logged. You can also contain "usernames" (from SREhttp/2's USERS.CFG username/password database) -- requests from these usernames will not be recorded.

One sensible use of this is to include the webmaster's IP address (since you probably don't care about requests you make to yourself), as well as the "usernames" of the various SUPERUSERs.

Example: NOUSER=joe.bars.net *.wah.com 151.121.55.6 ADMINX

Note that ADMINX is a possible username (possibly of a user with SUPERUSER privileges).

Notes:

you can use * as a wildcard for a field. Thus, ab.wah.com and cd.wah.com would both match *.wah.com, but www.ef.wah.com would not
If a match to any of these NOCODES occurs, then none of the log files are written to.
You can have multiple NOUSER entries (they will all be used). Thus, an alternative to the above example would be:
```
   NOUSER=joe.bars.net 
   NOUSER=*.wah.com 
   NOUSER=151.121.55.6  
   NOUSER=ADMINX 
```
NOUSER entries are not host-specific. In particular, username suppression is not host-specific.

NOREFERER

This should contain a space delimited list of IP addresses and domain names that you do NOT wish to record in the REFERER.LOG file. That is, requests that were refered from these addresses will NOT be recorded in REFERER.LOG. However, these requests WILL be recorded in the common.log, combined.log, and browser.log files.

Example: NOREFERER= *.my.site.net backup.site.org

Notes:

you can use * as a wildcard for a field (see the above example).
You can have multiple NOREFERER entries (they will all be used). Thus, an alternative to the above example would be:
```
   NOREFERER=*.my.site.net
   NOREFEER= backup.site.org 
```
NOREFERER entries are not host-specific.
NOREFERER only looks at the domain name portion of a request-URI. It does not look at the selector portion (the stuff after the http://foo.bar.net/)

NOURL

The NOURL entries (there can be more than one) should contain URLs to not record. That is, if a request selector matches one of these URLs, it will NOT be recorded.

For example:

        NOURL =  IMGS/*
        NOURL =  BORING/TEDIOUS.HTM

The preceding examples will be used for all requests (to any host).
Alternatively, you can include a host-nickname:

        NOURL =  DEFAULT:/IMGS/*
        NOURL =  ZOO:/BORING/TEDIOUS.HTM

Notes	You can include * as a wildcard. Thus, IMGS/HORSE.GIF matches IMGS/* If a request selector matches one of these NOURL variables, then none of the log files are written to.

DISPLAY

Use DISPLAY to also display the common-log entry on screen.

DISPLAY=1 : display common-log entries in the PMPRINTF window (you can use PMPRINTF.EXE to display the PMPRINTF window).
DISPLAY=0 : do not display common-log entries on-screen
DISPLAY=-1 : display common-log entries in the message area of the SRE2003 status screen. If you choose this, it is strongly recommended that you increase the OS/2 box size using the OS/2 MODE command.
For example, to resize the OS/2 box to 95 columns x 35 rows, execute the following before running SRE2003.
d:\sre>MODE CO95,35

IV) Notes

The following parameters, etc. are used to determine if a request is logged:
- If a NO_LOG permission is specified, the request is not logged
- if the SREhttp/2 LOG_SUPPRESS variable equals 1, and a LOG permission is not specified, then the request is not logged
- If COMMONLOG=0 (or COMMONLOG.host-nickname=0), the request is not logged to the COMMON log file. SImilar rules hold for the other LOG files. Note that it is possible (perhaps even typical) to suppress logging to a subset of the LOG files; say, to suppress the combined-log and referer-log files.
If you are running a single-host server, and you are happy to log all requests in a standard fashion, then you can use the default version of LOGS.CFG. Similarly, if you are running multiple-hosts, and are happy to log all requests to a single file, use the default version of LOGS.CFG.
If you choose to use the default LOGS.CFG file, log information will be written to the following log files in the SREHTTP2\LOG directory.

COMMON.LOG -- the common-log audit file
BROWSERS.LOG -- the browser-log file
REFERER.LOG -- the referer-log file
SREhttp/2 will "lazy write" the audit files (about once a minute).
In addition to the common, etc. log files, SREhttp/2 has several other means of auditing requests:
The REQUESTS.CNT file is a running count of the total number of requests for each URI.
You can write audit information to the SRE2003 log file
If you've set DISPLAY=1, you can use PMPRINTF.EXE to display and to save log information.
You can write your own Post-Response procedure

V) Structure of .LOG files

(the following description is adapted from http://www.apache.org/docs/mod/mod_log_common.html)



i) The common-log file format


The common-log file contains a separate line for each request.
A line is composed of several items (tokens) separated by spaces:
     host ident authuser date request status bytes
If a token does not have a value then it is represented by a hyphen (-).

The meanings and values of these tokens are as follows:

client
  The fully-qualified domain name of the client, or its IP number if the
  name is not available.
ident
   The identd identity information: NOT SUPPORTED BY SREhttp/2.
authuser
   The client's username, as provided in the AUTHORIZATION request header.
   Note that the common-log standard suggests that "if the request was for a
   password protected document, then this is the username used in the request".
   However, SREhttp/2 will always include the authuser (if available), even
   if the particular request did not require authorization.
date
   The date and time of the request, in the following format:
      date = [day/month/year:hour:minute:second zone]
   where:
         day = 2 digit
         month = 3 letter
         year = 4 digit
         hour = 2 digit
         minute = 2 digit
         second = 2 digit
         zone = (`+' | `-') 4 digit
request
     The request line from the client, enclosed in double quotes
status
   The three digit status code returned to the client.
bytes
   The number of bytes in the object returned to the client, not including
   any headers.

For example:
 acl.wear.org - - [18/Mar/1996:16:24:08 -0500] "GET /SMP/HLO.HTM HTTP/1.0" 200 1251

In the above example, the HTTP/1.0 substring may not be standard (you can
suppress it by setting the NOHTTP option in SREFLOGS.INI). Also note that
-0500 corresponds to the EST offset from GMT.

ii) The BROWSERS log file 

The browsers log file has the following structure:
   [date] client : user-agent
For example:
  [24/Feb/1997:18:14:30 -0500] 151.121.65.143 : IBM-WebExplorer-DLL/v1.1b

iii) The REFERERS log file

The referers log file has the following structure:
   [date]  "sel"  referer_url
Where SEL is from the request string, and the referer_url is from the
REFERER request header.
For example:
   [24/Feb/1997:18:18:19 -0500] "/SAMPLES/samp1.htm" http://dh.ag.gov/i.htm
(http://dh.ag.gov/i.htm contained a link to /samples/samp1.htm)

Note: if a request does not have a REFERER request header (for example, WebEx
does not provide a REFERER request header):
  if FORCE_REFERER=0, an entry will NOT be added to the   REFERERS log.
     FORCE_REFERER=1, a referer of " - " is used.

iv)The COMBINEDLOG log file


The  combined log file combines the common, referer, and browser logs.
It has the following structure:
  PARROT.HALL.NET - - [04/Jun/1997:22:38:16 -0400] "GET /imgs/justscal.gif"
  200 516 "http://mysite.heights.edu/SAMPLES/guide.htm"
  "Mozilla/2.02E (OS/2; I)"
(but all on one long line).



 Last updated 26 June 2002