HTTPush Plugin API v0.9
-----------------------

1. Introduction
---------------

This document describes the functionality of plugins, giving a reference of
the HTTPush plugin API messaging protocol as well as implementation ideas
and guidelines.

Plugins are the force behind HTTPush. The program takes care of ordinary
tasks, such as handling connections and recording the data, while plugins
provide the security knowledge, the whole idea behind the project and what 
makes HTTPush different from anything else.

2. Events
---------

Plugins are event driven, that is they register to receive notification of
certain events on startup, and the program takes care to let them know when
any of the events they are waiting for has occured. There are a few events
that HTTPush knows about, specifically:

- FQDN
- Port
- Protocol
- Directory
- Page
- Request
- Sent
- Received

They are discussed in detail in the ssection 4, "Message specification".

Most of them are associated with an XML tag and occur when a new XML tag
corresponding with the event is created, e.g. whenever a new page is
retrieved (one that hasn't still been recorded), the "page" event occurs.
After this, requests for the same page will NOT produce an event.

In the following section we will discuss how plugins fit in HTTPush, how do
they communicate and the different types of plugins:

- Event processing plugins
- Modification plugins
- Report plugins

3. Plugins interface
--------------------

When HTTPush is started it invokes all plugins and asks them to register.
Communication between the main program and the plugins is through STDIN and
STDOUT. HTTPush sends data to the plugin through its STDIN file handle and 
receives data from it through its STDOUT file handle:

        -----------                  ----------
        -         - >>>>>>>>>> STDIN -        -
        - HTTPush -                  - plugin -
        -         - <<<<<<<<< STDOUT -        -
        -----------                  ----------

This is all it is needed for a plugin to communicate with HTTPush, so the
plugin can be written in any language, not just Perl, as long as it can read
from and write to file handles (C, Python, Java, shell script, etc.).

In order to communicate, a messaging protocol has been defined that, while
still allowing the plugins to be powerful, is easy to understand and easy to
implement. In order to describe the protocol syntax, we will use a familiar
augmented Backus-Naur Form (BNF) similar to that used to describe HTTP/1.1 in
RFC 2616. Please refer to the RFC for a full description if you are not
familiar with the notation. The basic constructs we will use are:

 CTL = <any US-ASCII control character
       (octets 0 - 31) and DEL (127)>

 CR             = <US-ASCII CR, carriage return (13)>
 LF             = <US-ASCII LF, linefeed (10)>
 SP             = <US-ASCII SP, space (32)>
 HT             = <US-ASCII HT, horizontal-tab (9)>

 separators = CR | LF | SP | HT | ":"

The protocol is message-based. The generic syntax of a message is:

 request-line = event SP version CR *( header CRLF ) CRLF

 event = httpush-event | plugin-event

 httpush-event = "IDENTIFY"
               | "RES"
               | "FQDN"
               | "PORT"
               | "PROTOCOL"
               | "DIRECTORY"
               | "PAGE"
               | "REQUEST"
               | "SENT"
               | "RECEIVED"

 plugin-event = "REGISTER" ; Section 1.1
              | "REQ"
              | "VULN"

 version = 1*DIGIT "." 1*DIGIT

 header = header-name ":" header-value

 header-name = 1*<any CHAR except CTLs and separators>

 header-value = <base64 of content>

 An example of a FQDN event message could be:


A plugin should read messages sent to it by HTTPush and once it has enough 
data process them and, if needed send a batch of messages back to HTTPush. 
Messages traveling in one direction are independent from those traveling 
in the opposite direction, so no concept of ACK is needed.

A message transaction could be:

> FQDN 1.0
> id:MzM3Mg==
> content:aHR0cHVzaC5zb3VyY2Vmb3JnZS5uZXQ=
> uri:aHR0cDovL2h0dHB1c2guc291cmNlZm9yZ2UubmV0
>

< VULN 1.0
< id:MzM3Mg==
< content:aHR0cHVzaC5zb3VyY2Vmb3JnZS5uZXQ=
<

Libraries which implement protocol handling as well as some of the more common 
tasks are included with the distribution, for more information see Section XXX 
("Plugin SDKs").

4 Message specification
-----------------------

Each type of message has its own headers, described in this section.

4.1 Generic headers
-------------------

There are a few headers which are common to all messages, except for the
"IDENTIFY" message which sends no headers:

4.1.1 id header
---------------

Details: required
An item reference, helps HTTPush correlate messages from the plugins to
elements in the XML tree. This header is used by HTTPush to inform the
plugin of the element ID the event occured on, and used by the plugins to
associate a plugin response (such as a "VULN" message), with the element.

Syntax: 1*DIGIT "/" 1*DIGIT

4.1.2 uri header
----------------

Details: required
The full URI that was being processed when the event occured.

Syntax: ( "http" | "https" ) "://" host [ ":" port ] [ path [ "?" query ]]                                                      

Notes:
The URI may not be the one associated with the current event. For instance,
during the processing of a request to
"http://www.example.org/cgi-bin/test.cgi", there could be a "DIRECTORY"
event refering to the "cgi-bin" directory, and the URI will still be the
whole request URI, not just that of the directory.

4.2 Message specific headers
----------------------------

4.2.3 Messages from HTTPush
---------------------------

4.2.3.1 IDENTIFY
----------------

When HTTPush first starts, it executes every plugin and sends an "IDENTIFY"
message to them. In order for the plugin to be successfully loaded, it MUST
respond with a "REGISTER" message.

This message sends no headers.

4.2.3.2 FQDN
------------

This message is sent whenever a new fully qualified domain name is added to the server records.

4.2.3.2.1 content header
------------------------
Details: required
FQDN added, e.g. "www.example.com".

4.2.3.3 PORT
------------

This message is sent whenever a new port for a known server is added to the server records.

4.2.3.3.1 content header
------------------------
Details: required
Port number added, e.g. "80"

4.2.3.4 PROTOCOL
----------------

This message is sent whenever a new protocol for a known server is added to the server records.

4.2.3.4.1 content header
------------------------
Details: required
Protocol name added, e.g. "https"

4.2.3.5 DIRECTORY
-----------------

This message is sent whenever a new directory for a known server is added to the server records.

4.2.3.5.1 content header
------------------------
Details: required
Directory path added, e.g. "cgi-bin/"

4.2.3.6 PAGE
------------

This message is sent whenever a new page for a known server is added to the server records.

4.2.3.6.1 content header
------------------------
Details: required
Page name added, e.g. "index.html"

4.2.3.7 SENT
------------

This message is sent wherever a request content has been sent to the remote
host.

4.2.3.7.1 content header
------------------------
Details: required
Content of the request to the remote server, including HTTP request line,
headers and body.

4.2.3.8 RECEIVED
----------------

This message is sent wherever a response content has been received from the remote
host.

4.2.3.8.1 content header
------------------------
Details: required
Content of the response from the remote server, including HTTP status line,
headers and body.

4.2.3.9 REQUEST
---------------

This message is sent whenever the server has finished processing a request.

4.2.3.9.1 who header
--------------------
Details: required
The IP address and port of the user making the request.

Syntax: <client address, DIGIT or "."> ":" <client port, DIGIT>

4.2.3.9.2 sent header
---------------------
Details: required
See the SENT message "content header", on section 4.2.3.7.1.

4.2.3.9.3 received header
-------------------------
Details: required
See the RECEIVED message "content header", on section 4.2.3.8.1

4.2.3.9.4 content header
------------------------
Details: required
Page name added, e.g. "index.html"


4.2.3.10 RES
------------

In response to a "REQ" message, HTTPush (potentially) sends the plugin a
reponse containing the content of the request, by using a "RES" message.

4.2.3.10.1 content header
-------------------------
Details: required
Content of the response from the server, including HTTP server status line,
headers and body.

4.2.4 Messages from plugins
---------------------------

4.2.4.1 REGISTER
----------------

In response to an "IDENTIFY" message from HTTPush, the plugin MUST send a
"REGISTER" message, which details its capabilities as well as events it
wants to register, identification, etc.

4.2.4.1.1 name header
--------------------
Details: required
A one-word, short name of the plugin.

4.2.4.1.2 description
-------------------
Details: optional
A one-line, concise description of the plugin, intended for human
readability.

4.2.4.1.3 version
-----------------
Details: required
Version of the protocol soported by the plugin, as of thie writing MUST be
"1.0".

4.2.4.1.4 event 
---------------
Details: required,multiple
Event the plugin wants to receive notifications about.

Syntax: "SENT" | "RECEIVED" | "REQUEST" | "FQDN" | "PORT" | "PROTOCOL"
        | "DIRECTORY" | "PAGE"

4.2.4.1.5 noise
---------------
Details: optional
Noise Level associated with the plugin. It's an estimation of the
probabilities the plugin actions will be detected. Must be determined by
the plugin writer, in a worst case scenario and based on the following general 
scale:

- 100 - Really noisy, the plugin action will be noticed, such as a DoS.
- 75  - Very noisy, there is a high probability the actions will be noticed,
        such as a plugin that generates a lot of requests.
- 50  - Noisy, there is a 50% probability it will be detected.
- 25  - Somewhat noisy, the actions could be noticed but it's not very
        likely, such as modifying parameters to an application.
- 0   - Really stealth, its actions will never be noticed on the remote
        host, such as performing a WHOIS request on the domain.

If not present a default value of 100 will be assumed.

The noise threshold is one of the program options, any plugins beyond the
threshold will be ignored during the processing.

4.2.4.1.6 intrusive
-------------------
Details: optional
An estimation of how intrusive is the plugin, or what are the changes that
running it could disrupt normal operation of the server, be it a DoS or a
plugin that will get Administrative privileges on the remote server.

As the NL, it must be estimated in a worst case scenario and based on the 
following general scale:

- 100 - Really intrusive, a DoS or something really bad for the server.
- 75  - Very intrusive, there is a high probability the server operation
        will be disrupted.
- 50  - Intrusive, e.g. an exploit will be run on the remote server.
- 25  - Somewhat intrusive, e.g. it will exploit a minor vulnerability.
- 0   - Non-intrusive, no action will be performed on the remote server. A
        CGI scanner plugin could be classified as level 0.

If not present a default value of 100 will be assumed.

The intrusive threshold is one of the program options, any plugins beyond the
threshold will be ignored during the processing.

4.2.4.3 REQ
-----------

A plugin can ask HTTPush to retrieve a remote URL on its behalf, by using
the "REQ" message. It is the basic message for those plugins that need to
make further HTTP requests.

4.2.4.3.1 id header
-------------------
Details: required
In a "REQ" message, the ID field has a special meaning, when HTTPush has 
finished retrieving the request, it will send back a "RES" message, with 
the ID set to that of the original "REQ" message, so it is useful for 
plugins to correlate REQ/RES messages. The ID field won't be interpreted 
by HTTPush, so it could be set to anything, not just the usual ID syntax.

4.2.4.3.2 record header
-----------------------
Details: required
Whether the request will be passed on to the HTTPush recording module, so
the user will be able to browse it as a normal request. Useful when there is
an interesiting URL found and the plugin wants to note it in the final
report.

Syntax: "0" | "1"

4.2.4.3.3 method header
-----------------------
Details: required
Method to use in the HTTP request.

4.2.4.3.4 uri header
--------------------
Details: required
The URI to retrieve, including protocol, hostname and port.

Syntax: ( "http" | "https" ) "://" host [ ":" port ] [ path [ "?" query ]]                                                      

4.2.4.3.5 protocol header
-------------------------
Details: required
Protocol to use in the HTTP request.

4.2.4.3.6 header header
-----------------------
Details: optional, multiple
HTTP Header to set in the request.

Syntax: <header name> ":" <header value>

4.2.4.3.7 content header
------------------------
Details: optional
Body to send to the remote server.

4.2.4.4 VULN
------------

A plugin informs HTTPush that a vulnerability has been found by sending a
"VULN" message. HTTPush will insert the vulnerability inside the element
referenced by the ID header, and it will be accessible through the internal
webserver.

4.2.4.4.1 level header
----------------------
Details: required

This header quantifies the risk preented by the vulnerability. It ranges
from 0 to 5: a level of 0 will mean minimal level/not quantified, while a
level of 5 will mean a huge vulnerability has been found. The internal
webserver uses this level indicators to group and present the information in
a useful way, so using them carefully assures a good experience while
auditing HTTP server.

4.2.4.4.2 title header
----------------------
Details: required

A generic description of the vulnerability found, which allows for grouping
and nicer presentation. It's not meant to include specific data of a found
vulnerability.

4.2.4.4.3 bid header
--------------------
Details: optional

Bugtraq ID, if any, associated with the vulnerability.

4.2.4.4.4 cve header
--------------------
Details: optional

Common Vulnerability Exposure ID, if any, associated with the vulnerability.

4.2.4.4.5 content header
------------------------
Details: required

Concise description of the vulnerability. Proper, minimalistic HTML
formatting formatting is allowed and encouraged, as this information will
probably end up in the final report "as is".

5 Modification plugins
----------------------
XXX To be completed XXX

6 Reporting plugins
-------------------

7 Plugin example
----------------

XXX To be completed XXX

8 Plugin SDKs
-------------

XXX To be completed XXX

$Id: plugins.txt,v 1.4 2001/10/15 21:17:24 jfs Exp $
