MAMA: HTTP headers report

By Brian Wilson

Introduction

Before we start discussing the main part of MAMA's results on markup, CSS, and script, we need to start at the logical beginning of our story. Any page that a browser displays begins with a HTTP transaction. The requester asks a Web server for a document, and the server responds with important meta-information about the HTTP transaction response before moving forward and delivering the bulk of the response—the document itself. This HTTP response contains many items worth examining. Some may consider this topic dull— it is an easy claim to make that most authors will never have a need to know about the details of the HTTP transport layer in the process of writing their documents. However, it all starts with the HTTP headers— it is the basis for all the rest that follows and is critical for us to look at to produce a cohesive exploration of Web documents.

Read the first report in the series - MAMA: Markup validation report - if you haven't already done so. For more on MAMA, check out the MAMA homepage. For more on HTTP headers, check out the full HTTP header results.

MAMA's HTTP request headers

We must first look at MAMA's HTTP request headers before looking at how Web servers responded to MAMA. The initial HTTP request is the first link in the chain, and it shapes much of what follows. For this study, MAMA tried to experience the Web as closely as possible to how Opera would experience it. Of particular interest is MAMA's User-Agent string:

Note: The Accept-Language and Accept-Charset values chosen reflect the author's own particular language bias. This can have an affect on what is served in the HTTP response.

The HTTP Response— general anatomy

The HTTP response block consists of a "status line" followed by any number of newline-separated header field/value pairs. HTTP/1.1 (RFC 2616) provides considerable detail about the anatomy of an HTTP response, but in this overview we will only examine some of its most popular field/value components. The format that HTTP header name/value pairs follows is generally:

[Field Name (case-insensitive)]: [Field Value]

HTTP header response fields

First we will take a look at the most popular HTTP header fields in MAMA's URL set. Three fields were found to be nearly universal: Content-Type (the MIME type of the document), Date (the date of the HTTP transaction) and Server (information about the Web server sending the HTTP response).

Although many of the other HTTP header fields are not as ubiquitous as the top 3, we can see from the top 10 that many other headers are used VERY frequently. The average is 9 fields for an HTTP header.

The average length of a typical HTTP header in MAMA's URL set is 381 characters. The smallest HTTP header block encountered was 66 characters, while the longest found was 9,725 characters.

Content-Type HTTP header "charset" value

While the purpose of the Content-Type header to describe a document's MIME type is naturally ubiquitous, MAMA's URL selection process eliminated MIME types that it could not analyze. The URLs that MAMA surveyed were almost all "text/html", but other values were also encountered.

One optional portion of the Content-Type header field does deserve some extra scrutiny. It is the "Charset" parameter, found in 688,819 of MAMA's URLs (19.6%):

Ex: Content-Type: text/html; charset=utf-8

The charset field describes the encoding used for the document. Two values are dominant and roughly equal here—"utf-8" and "iso-8859-1", with the former value having the slight edge.

The Server HTTP header

This is an interesting value to examine. The main story we see here illustrates a long-standing competition between the two most popular Web servers, Apache and Microsoft's IIS. MAMA's numbers show the distribution heavily skewed in Apache's favor (67.7% as versus 25.9% for IIS). This balance does not match Netcraft's numbers for the same time period: Netcraft's data shows Apache as the dominant Web server, but less so than in MAMA's case (Apache: 50.76%, IIS: 35.84%).

The URL set used in Rene Saarsoo's previous study of "Coding practices of Web pages" shares a great deal of overlap with MAMA, and its HTTP header Server field usage ratio is very similar to MAMA's. The main conclusion we can draw from this is that the Dmoz URL set itself (which is MAMA's main URL source in this study) has a slight bias towards Apache over the Web-at-large that Netcraft appears to cover.

The Content-Length HTTP header

This header field, used in ~70% of MAMA's URLs, tells the requester exactly how long the incoming document is. Does it tell the truth? Luckily, we have a way to measure this. MAMA assessed the length of every document it analyzed by using Perl's length function against the received content body. Comparing these two values, MAMA finds that the Content-Length values are rarely incorrect. If they are, they are only off by 1. Out of 2,533,890 URLs using the Content-Length HTTP header, all were the same as MAMA's measured length value, except:

Summary

It is very likely that the typical Web page author will have never seen the HTTP headers with which their pages are served. Many of the common HTTP headers are the moral equivalent of boring paperwork that has little effect on the author. However, authors should not make the mistake of simply dismissing them entirely— they lay the groundwork for the document that follows. Some of the popular header fields (like those influencing a document's caching and encoding behavior) directly influence the end-user's experience of the pages they've spent so much effort creating.

This article is licensed under a Creative Commons Attribution, Non Commercial - Share Alike 2.5 license.