SREhttp/2 and Content Negotiation

1) Introduction

In general, content negotiation is used to choose between resources created with one of several languages, or one of several mimetypes. Content negotiation can also be used to choose between resources using alternate character sets, and to choose shorter documents. In the future, content negotiation may also be used to choose documents using certain features (such as documents using different versions of html).

There are basically two forms of content negotiation:

1)Server-side.

Server-side content negotiation (which is defined in http/1.0) is performed by the server -- the server uses ACCEPT request headers (provided by the client) to automatically choose and return one of serveral variants.

2)Client-side.

Client-side content negotiation (which is new to http/1.1) is accomplished by the client automatically requesting one of several variants:

the choice is based on a "variants list" (that contains URI's and descriptive information) contained in response headers returned by the server.

II) Specifying a negotiable resource

1. Create several "variants" of a resource
2. Create a file, in your web-space, containing a variant list
3. Create an entry in ATTRIBS.CFG that contains a NEGOTIATE type redirection to this variant-list file

Once you've accomplished these steps, all you need to do is put make avaialble a URI that matches this entry (in ATTRIBS.CFG); and hope that your client's browser either provides useful ACCEPT headers, or knows how to do client side content negotiation.

There are a few constraints:

1. variants must be on the same server as the "variant list" (discussed below). In addition SREhttp/2 enforces the good practice (from a security standpoint) of requiring that each variant to be in (or under) the same path as the variant list.
2. variants must be retrievable via GET requests. That is, each variant should be accessible with a standard URL; which also means that content negotiation will not work with POST requests.

II.2) Creating a Variant List

Each record in a variant list must specify a URI (a selector), and several pieces of identifying information. This identifying information is used to specify up to six dimensions of negotiation : such as the mimetype, charset, language, encoding, features, and length.

The syntax of these multi-line records is (note that a wildcarded form of these records can also be specified).

   URI: a_selector
   Content-type: type/subtype ; charset=a_charset ; qs=m.mm
   Content-language: l1, l2 
   Content-encoding: enctype
   Content-length: nnnn

II.3) Identifying a negotiable resource.

If a request-selector does not have negotiation redirection attribute associated with it, then SREhttp/2 will not treat the requested resource as a variant list. That is, the request will be treated in the normal fashion.

That is again ... the variant list file would be returned verbatim (say, as a text/plain response).

vntlist should have one of the following forms:

1. blank or .
   If not specified, or if a period (.), then the request-selector is used to determine the location of the variant list file, and to create URI's for variants.
2. new_selector
   new_selector is similar to an local redirection -- SREhttp/2 uses new_selector to determine the location of the variant list file, and to create URI's for variants.
3. new_selector vntlist_file
   vntlist_file is similar to virtual directories. If specified,vntlist_file must be a fully qualified file name. It will be used as the variant list file. Furthermore, all variants (in the variant list file) must be relative to the path of the vntlist_file.
   When vntlist_file is specified, new_selector is only used to form URI's for variants -- it is not used to actually find the variant files. Instead, the path of vntlist_file is used.

• realm: negot1 rule: /VARTEST/VAR1.LST redirect: Negotiate= . /VARTEST/VAR1.LST is a negotiable resource, with a variant list defined in E:\WWW\VARTEST\VAR1.LST. All variants (defined in the variant list) are assumed to be relative to E:\WWW\VARTEST\. URI's to these variants are defined as relative to /VARTEST/

• realm: negot2 rule: /VARtest/VAR2.LST redirect: Negotiate= /ZOO/VARS/VARList.2 /VARtest/VAR2.LST is a negotiable resource, with a variant list defined in E:\WWW\ZOO\VARS\VARLIST.2. All variants (defined in the variant list) are assumed to be relative to E:\WWW\ZOO\VARS\. URI's to these variants are defined as relative to /ZOO/VARS/

• realm: negot3 rule: /VARtest/VAR3.LST redirect: Negotiate= /ZOO/MOREVARS/VARList.3 f:\ZOOSTUFF\VARlist.3 /VARtest/VAR3.LST is a negotiable resource, with a variant list defined in F:\ZOOSTUFF\VARlist.3. All variants (defined in the variant list) are assumed to be relative to F:\ZOOSTUFF\. However, URI's to these variants are defined as relative to /ZOO/MOREVARS/. Thus, you should define (elsewhere in ATTRIBS.CFG) virtual directory entries that map /ZOO/MOREVARS/* to F:\ZOOSTUFF.

II.3.a) Wildcarded form

For example...
    Realm: negot2
    Rule: /MANUALS/*.HTM
    Redirect: Negotiate=/MANUALS/DOCS.LST

In this case, all request selectors that match /MANUALS/*.HTM will use the variant list specified in /MANUALS/DOCS.LST. Please see the next section for details on how variants are resolved when this wildcarded form is used.

II.4) Using wildcarded variants

In recognition of this possibility, SREhttp/2 supports a special form of variant list that supports such multiple sets of negotiable resources. The specification of these sets requires two changes to the simple case.

The first difference is discussed above -- the use of the wildcarded form of sel. The second involves modifications to the variant list file.

Recollect that the wildcarded form directs many possible request selectors to a single variant list. Thus, the variant list should contain information that allows the request selector to influence the value of the URI: field of each record in the variant list. To do this, two steps are required.

1. A (case insensitive)
      PATTERN: wild_sel
   entry should be put at the top of the variant list.

   For example: Pattern: /manuals/*.HTM

   The value of the wild_sel used in a PATTERN: entry should be the same as the wildcarded sel used in the ATTRIBS.CFG entry.

2. The URI: fields may contain * characters.

   SREhttp/2 will replace these * (in the URI: entries) with corresponding portions of the request selector.

III) The content negotiation algorithim.

Alternatively, you can specify your own server side content negotiation algorithim on a selector specific basis

1. First, SREhttp/2 checks for a Negotiate: request header. If no such header exists, then server-side negotiation is always attempted. If this header does exist, then server-side negotiation may be attempted (see the notes for details).

2. If server-side negotiation is to be attempted, by default the following selection algorithim is used.

   If the client allows a "remote variant selection algorithim" (by including a Negotiate: n.n request header), then a custom procedure can be used instead (see Appendix B for the details).

   Note that this is a leave as soon as a definitive answer is found method -- latter steps are only used if earlier steps yield ties.
   Furthermore, variants eliminated in earlier steps are not available -- they are not considered in latter steps.
   Lastly, if all variants are eliminated, a suitable 406 (could not find representation) response is immediately returned.

   1. Accept: headers are read. Accept: headers contains information on acceptable mime-types. This information can contain selection quality (q) information.
      

      If a variant has no Content-Type: entry, assign qs=0.0001 The variant with the best combined quality is used. Combined quality is deterimined by multiplying the variant-list qs (quality) by the accept: header "q: factors".

      If there are ties (i.e.; several mimetypes have a combined quality of 1.0), then move to step b. If there is no accept: header (most browsers send some form of Accept: header) then skip this step.

   2. Accept-language: headers are read. These request headers can contain language specific q modifiers (that indicate which language is prefered). The variant (of those surviving step a) with the highest language q factor is used.

      Note that if a surviving variant has no Content-Language: entry, assign qs=0.0001

      If there are ties, move to step c. If there is no accept-language: request header, this step is skipped. Note that the content-language entries in the variant list should not include q factors.

   3. Accept-Encoding: headers are read. These headers can also contain encoding specific q modifiers. The variant (of those surviving step b) with the highest encoding q factor is used.

      Note that if a surviving variant has no Content-Encoding: entry, assign qs=0.0001

      If there are ties, move to step d. If there is no accept-encoding header, this step is skipped. Note that the content-encoding entries in the variant list should not include "q" factors.

   4. Accept-charset: headers are read. Variants that do not match this charset are dropped (i.e.; default qs=0). If no charset is defined, use the default of ISO-8859-1. Actually, by assumption all requests can honor ISO-8859-1, in addition to other charsets noted in Accept-Charset request headers.

      If there are ties then move to step e. If there is no accept-charset header then skip this step.

   5. Use the variant with the smallest content-length (as pulled from the variant list). If no length is defined for this variant, use 0 (always acceptable).

      If there are ties, move to step f.

   6. Use the first of the remaining variants.

   Note: If all variants are removed at any step in this process (say, no variants have a content-language, and an explicit Accept-language was specified), then the algorithim exits.

   In this case, if a default variant was specified, it will be used. Otherwise, either a 406 (for http/1.1 clients) or a 404 (for http/1.0 clients) response is returned.

   Once a variant has been chosen, two things occur: The chosen variant is used to form a negotiated selector, one that is relative to either the request selector or to the new_selector. Selector-specific attributes associated with this negotiated selector (as read from ATTRIBS.CFG) are then used. Note that redirection attributes are not used, but all other attributes (such as requires) are. Thus, the selector-specific attributes of the request selector are not used to control access, etc. to the negotiated selector. Possibly using the vntlist_file, the variant is mapped to a file. This file will then be returned to the client (or used as a CGI-BIN or a SREhttp/2 addon).

3. If client-side negotiation is to be used, SREhttp/2 returns a special "300" return code. The body of the response contains an UL list containing links to each variant. Furthermore, the variant list (suitably formatted) is returned in an Alternates: response header.

III.a) Using your own server-side content negotiation algorithims

Although client-side negotiation can be quite powerful, it does
require fairly advanced http/1.1 clients. Until this is 
commonplace, server-side negotiation is a good alternative.

Although the Apache-emulating content negotation algorithim 
(described above) is fairly powerful, it is limited. For example,
there is no way that cookies can influence the choice of variant.
Therefore, SREhttp/2 allows you to specify your own "server side"
content negotiation algorithims, on a selector specific basis.

To do this, you must:

 a) Include a 
      PROC: procname
   entry at the top of your variant list file.  
  This PROC: entry must have a blank line before and after it!

 b) Create a procedure with "procname" that can choose a variant.
    This procedure should reside in macrospace.

For example, your variant list could look like:

  URI: bar
  Proc: SREH2_NEGOTIATE_1

  Uri:bar.html
  content-type: text/html ; qs=1.0

  Uri:bar.txt
  content-type: text/plain ; qs=0.7

which would mean "use the SREH2_NEGOTIATE_1 procedure, which has been loaded
into macrospace as SREH2_NEGOTIATE_1, to choose between these two variants".

Note: to load a procedure into macrospacce, you can define an EXEC in PRELOADS.CFG. 
      For example:   LOAD_EXEC= SREH2_NEGOTIATE_1 PROCS\NEGOT.REX


III.a.i) The Custom Procedure

Your custom procedure will be passed four arguments:
   i) the original request selector, 
  ii) the variant list file (that the original selector maps to)
 iii) number of entries in the variant list
  iv) the variant list

For example, you can read these arguments using:

   parse arg  sel0,afile,nentries,varilist

Varilist (the variants list) will have the form:
    n
    spec_1
     ...
    spec_n
where:
   n = # of variants
   spec_j = specifications for alternate j, with structure:
           "a_sel" q  {field1 values1} {field2 values2} ....

  For example:
     4
     "tst.1" 1 {type text/plain } {language en } {charset cyrillic }
     "tst.2" 0.3 {type text/plain } {language fr } {features tables def} 
     "tst.3" 1 {type text/html } {language en } {length 20065 } {charset latin5 }
     "/status?" 1 {type application/octet-stream } {language gr }

  Note that each entry appears on a single line (which may be quite long).


The procedure should return:
    statcode  header_list
where
 statcode:
       '' -- a blank line means "could not find a best match,
             use SREhttp/2's default algorithim instead"
       0 -- 0 means "could not find best match, return a 406 response"
       j -- A number between 1 and n, which points to the best variant.

 header_list (optional):

       only used if statcode=j (j=1..n). This should contain a space delimited
       list of the request headers used to choose the variant. This list
       is returned as a Vary: response header, and will then be used by
       proxy servers to determine the cacachability of this variant.
       If you don't specify header_list, a Vary: * header will be
       added (which implies that this variant is not cachable).  

Example of a return:
    return '3 Accept Accept-Language '
which means: "use the 3rd variant, and use Accept and Accept-Language in a Vary:
             response header"
     

Notes:
  *  '' and 0 differ in that a return of 0 causes SREhttp/2 to stop trying
    to find a best variant (and return a 406 response); wheras as ''
    means "I give up, please try the default algorithim" (which might yield a 
    match).

  * Example of a header_list:  If you used the ACCEPT and ACCEPT-LANGUAGE
    request headers to determine the best variant, header_list should
    be "ACCEPT ACCEPT_LANGUAGE" (without the quotes). Or, if you also
    used a cookie, then header_list should be 
    "ACCEPT, ACCEPT_LANGUAGE, COOKIE".

  * The NEGOT.REX file, that comes with SREhttp/2, provides a well-documented
    example; it basically emulates the default "server side" content
    negotiation algorithim. To use it, you'll need to load it into macrospace
    with a procedure name of NEGOT.

III.a.iii) Useful procedures.

In order to ease the task of writing a custom procedure SREhttp/2 provides 
two procedures that you can use to search and extract information
from the the variant list: SREH2_NEGOTIATE_SEARCH and SREH2_NEGOTIATE_EXTRACT.
See Appendix D for the details.

In addition:

   * the SRE2003 sre_reqfield(header_name) function can be used to 
     lookup other specific request headers (see SRE2PRC.HTM for the
     details).
   * The SREhttp/2 SREH2_GET_COOKIE(cookie_name) to read
     the value of the cookie_name cookie (see SREH2PRC.HTM for the details).

For an example of how to a custom "server side" content negotiation
procedure, please see NEGOT.REX (in SREHTTP2\PROCS).

Appendix A) Sample Variants file.

        ---------------------- Start example -----------
; this is sample variant file

URI: foo
PROXY-RVSA: 1.3

Uri:foo.fr.de.html
content-type: text/html 
content-language: sp,fr-ca,de
content-length: 2005

Uri:foo.txt
content-type: text/plain ; qs=0.5
content-length: 2005
content-language: en 
description: this is the english text version

Uri:foo.gz
content-type: text/plain ; qs=0.5
content-length: 2005
content-encoding: gzip
content-language: en 

uri: /status?
content-type: application/octet-stream

uri: foo.en.html
content-type: text/html ; charset=iso-8859-1 ; qs=1
content-language: en
features: tables frames

uri:foo.def0


        ---------------------- End of example -----------

Notes:
  * Lines beginning with ; are comments, and are ignored

  * Each, typically multi-line record, contains a
    URI: entry, and (optionally) a combination of content-type, 
    content-language, and content-length entries.

  * Blank lines are treated as "record delimiters"

  * The first "one line record" URI: entry is optional -- it's skipped

  * An (optional) Proxy-RVSA: entry (at the top of the file, 
    before the start of the first non-one line  record) is used to signal 
    to proxies what  RVSAs they can use. 

  * Note that PROC:, if used, should also be placed before the 
    start of the first record (i.e.; next to the first "one line record" URI).

  * Content-length is the estimated size of the file, 
    it does NOT have to be the actual size of the file.
    All else equal, smaller "lengths" are preferentially returned.  

  * Content-language is a comma delimited set of languages 

  * Content-encoding is an encoding type, with "identity" meaning
    "no encoding". Only one encoding-type can be specified.

  * Content-type contains 3 fields. The type/subtype field is required,
    the charset= field is optional (iso-8859-1 is the default), and qs
    is the (optional) "quality" measure used to weight the variants.

  * Note the last entry is a "fall back" variant (the client's user-agent
    can choose this if all else fails).

Appendix B: Specifying a Remote Variant Selection Algorithim

SREhttp/2 provides a simple hook by which a custom remote variant
selection algorithims (RVSA) can be implemented.  The process is:

a) obtain the RVSA. 

b) write a rexx procedure that calls this rvsa

c) save this procedure into macrospace using a name of
   SREH2_RVSA_n, where "n" is the major version number
   of the rvsa.

   For example, if the request contains:
       Negotiate: 2.3,vlist
   then SREH2_RVSA_2 is appropriate "macrospace" procedure name.

In the simplest case, the rvsa procedure will be a rexx procedure;
in which step a and b are combined.

The rvsa procedure (say, SREH2_RVSA_2) will be called with
two arguments, the version number, and the list of alternates.
For example, to read these arguments the procedure could use:
  parse arg version,altlist

In the above example (Negotiate: 2.3,vlist) the version number would 
be "2.3" (without the quotes). The list of alternates is simply a 
copy of the Alternates: header (that would be returned to the client).

The rvsa procedure should use this information, along with request
specific information (such as the various Accept headers, which 
can be read using the SRE2003 SRE_reqfield function) to determine which
variant is best.   The number of this best variant should be
returned; or a 0 should be returned if there is no best match.

For example, if the second variant (in the list of alternates) is
best, then SREH2_RVSA_2 should return a "2" (without the quotes).

Notes:

  * You can use RXU, REXXLIB, or other DLLs to create macrospace
    procedures. 
    Or, you can define an EXEC in PRELOADS.CFG. 
    For example:   LOAD_EXEC= SREH2_RVSA_2 PROCS\TCN1.RXX 

  * For details on the structure of the alternates list, see RFC2295.
    (see Appendix C for an example).

Appendix C: Example of a TCN Request and Response

The following is a simple example of TCN request, and the response
from SREhttp/2.

Assume that:
  TSTHTM/TSTHTM.NEG
is a negotiable resource, and TSTHTM.NEG has the following structure:

   ;---- TSTHTM.NEG is a negotiable resources
   uri:tsthtm

   uri:tst.1
   content-type: text/plain; qs=0.8
   content-language: en 

   uri: tst.2
   content-type: text/plain ; qs=0.3
   content-language: fr 
   description: The French Version
   features: tables [abc def] 

   uri: /gene_test?
   content-type: application/octet-stream; charset=cyrillic
   content-language: ru
   ;---- End of TSTHTM.NEG


Assume that a client requests:
  http://foo.bar.net/tsthtm/tsthtm.neg

which leads to the following request:

   GET /tsthtm/tsthtm.neg  HTTP/1.1
   HOST:foo.bar.net
   Negotiate: vlist,*
   Accept: text/plain
   Accept-language: fr

This would yield the response:
---------------
  HTTP/1.1 200 Ok
  Date: Sat, 26 Jun 1999 13:49:10 GMT
  Accept-Ranges: bytes
  Connection: close
  Last-Modified: Thu, 24 Jun 1999 05:52:38 GMT
  ETag: "ABDY3S;7964E38D"
  Server: SRE2003 for OS/2, version 1.12d
  Content-Type: text/plain
  Tcn: choice
  Content-location: tsthtm/tst.1
  Vary: negotiate,accept
  Alternates: {"tst.1" 0.8 {type text/plain } {language en }},
  {"tst.2" 0.3 {type text/plain } {language fr } {features tables [abc def] }
  {description "The French Version"}},
  {"/gene_test?" 1 {type application/octet-stream } {language ru } 
  {charset cyrillic }}
  Content-Language: en
  Content-length: 18
  Cache-control: public

  [contents of tsthtm/tst.1]
---------------

Notes:
  * Since * appears in the Negotiate: header, SREhttp/2 will 
    attempt the default server side negotiation.  The Accept: text/plain
    is the only information used, and the explicit qs=0.8 of the first
    entry beats the explicit 0.3 of the second entry.  The implicit
    qs=1.0 of the third entry is irrelevant, since the third entry's
    content-type is not text/plain
  * the Alternates: header has been reformatted (it is actually on one 
    long line).
  * The contents of tsthtm/tst.1, which (in this example) has a size of 
    18 characters, is sent as the body of the response. 


Alternatively, if the client does not want the server to resolve the
variant:

   GET /tsthtm/tsthtm.neg  HTTP/1.1
   HOST:foo.bar.net
   Negotiate: vlist
   Accept: text/plain
   Accept-language: fr

which yields:
-----------
   HTTP/1.1 300 Multiple Choices
   Date: Sat, 26 Jun 1999 14:01:20 GMT
   Server: SRE2003 ver 1.12d
   Tcn: list
   Etag: 9XXGE;7964E38D
   Vary: *
   Alternates: {"tst.1" 0.8 {type text/plain } {language en }},
   {"tst.2" 0.3 {type text/plain } {language fr } {features tables [abc def] }
   {description "The French Version"}},
   {"/gene_test?" 1 {type application/octet-stream } {language ru } 
   {charset cyrillic }}
   Cache-Control: public

   [alternates as a UL list]
-----------

Note that the portion of the etag following the ; (the 7964E38D)
is the same for both requests -- it's the "variant list validator".
And since the Alternates: header is the same in both responses,
this "variant list validator" is also the same.