First, traffic is acquired from the network link via libpcap. Packets are passed through a series of decoder routines that first fill out the packet structure for link level protocols then are further decoded for things like TCP and UDP ports.
Packets are then sent through the registered set of preprocessors. Each preprocessor checks to see if this packet is something it should look at.
Packets are then sent through the detection engine. The detection engine checks each packet against the various options listed in the Snort config files. Each of the keyword options is a plugin. This allows this to be easily extensible.
For example, a TCP analysis preprocessor could simply return if the packet does not have a TCP header. It can do this by checking:
if (p->tcph==null) return;
Similarly, there are a lot of packet_flags available that can be used to mark a packet as “reassembled” or logged. Check out src/decode.h for the list of pkt_* constants.
Basically, look at an existing output plugin and copy it to a new item and change a few things. Later, we'll document what these few things are.
Generally, new output plugins should go into the barnyard project rather than the Snort project. We are currently cleaning house on the available output options.