ULOGD 2.x - the Netfilter Userspace Logging Daemon
Harald Welte <laforge@netfilter.org>, Eric Leblond
<eric@inl.fr>
Revision 2009/04/18
This is the documentation for ulogd-2.x, the second generation
Netfilter Userspace logging daemon. ulogd makes use of the Linux >=
2.6.14 nfnetlink_log and nfnetlink_conntrack subsystems, but also
provides backwards compatibility for Linux >= 2.4.0 ipt_ULOG.
______________________________________________________________________
Table of Contents
1. DESIGN
1.1 CONCEPT
1.2 DETAILS
2. INSTALLATION
2.1 Linux kernel
2.2 Userspace libraries
2.3 ulogd
2.3.1 Recompiling the source
3. Configuration
3.1 iptables NFLOG target
3.1.1 Quick Setup
3.1.2 NFLOG target reference
3.2 iptables ULOG target
3.2.1 Quick Setup
3.2.2 ULOG target reference
3.2.3 ipt_ULOG module parameters
3.3 ulogd
3.3.1 ulogd configfile syntax reference
3.3.2 ulogd commandline option reference
4. Signals / Logrotate
5. Available plugins
5.1 Input plugins
5.1.1 ulogd_inppkt_NFLOG.so
5.1.2 ulogd_inpflow_NFCT.so
5.1.3 ulogd_inppkt_ULOG.so
5.2 Interpreter plugins
5.2.1 ulogd_raw2packet_BASE.so
5.2.2 ulogd_filter_PWSNIFF.so
5.2.3 ulogd_filter_IFINDEX.so
5.2.4 ulogd_LOCAL.so
5.2.5 ulogd_filter_HWHDR.so
5.2.6 ulogd_filter_IP2BIN.so
5.2.7 ulogd_filter_IP2HBIN.so
5.2.8 ulogd_filter_IP2STR.so
5.2.9 ulogd_filter_PRINTFLOW.so
5.2.10 ulogd_filter_PRINTPKT.so
5.2.11 ulogd_filter_MARK.so
5.3 Output plugins
5.3.1 ulogd_output_OPRINT.so
5.3.2 ulogd_output_LOGEMU.so
5.3.3 ulogd_output_MYSQL.so
5.3.4 ulogd_output_PGSQL.so
5.3.5 ulogd_output_PCAP.so
5.3.6 ulogd_output_SQLITE3.so
5.3.7 ulogd_output_SYSLOG.so
6. QUESTIONS / COMMENTS
______________________________________________________________________
1. DESIGN
1.1. CONCEPT
ulogd-2.x wants to provide a flexible, almost universal logging daemon
for netfilter logging. This encompasses both packet-based logging
(logging of policy violations) and flow-based logging, e.g. for
accounting purpose.
ulogd consists of a small core and a number of plugins. All the real
power lies in the plugins, and in the user who configures the
interactions between those plugins.
Input Plugins
Input plugins acts data source. They get data from somewhere
outside of ulogd, and convert it into a list of ulogd keys.
Filter Plugins
Filter plugins interpret and/or filter data that was received
from the Input Plugin. A good example is parsing a raw packet
into IPv4 / TCP / ... header information.
Output Plugins
Output plugins describe how and where to put the information
gained by the Input Plugin and processed by one or more Filter
Plugins. The easiest way is to build a line per packet and
fprint it to a file. Some people might want to log into a SQL
database or want an output conforming to the IETF IPFIX
language.
By means of the configuration file, the administrator can build any
number of Plugin Stacks. A plugin stack is a series of plugins,
starting with an Input plugin, none, one or multiple filter plugins,
and one output plugin on top.
1.2. DETAILS
The major clue is providing a framework which is as flexible as
possible. Nobody knows what strange network protocols are out there
:) But at the same time, logging of a packet filter is often very
performance critical. Like all ulogd releases since 0.3.x, the
ulogd-2.x core doesn't do any dynamic allocations at runtime. Yes,
obviously, at startup time the config file is parsed, and allocations
are made. But after that, everything is pre-allocated. As an
additional improvement over ulogd-1.x, there are also no hashtable
lookups for key resolval. All input/output keys of plugins within
every stack are resolved at config file parsing time, and directly
interconnected by pointers.
2. INSTALLATION
2.1. Linux kernel
To use the NFCT or NFLOG input plugin, you will need a 2.6.14 or later
kernel. For old-style ULOG logging, you need a kernel >= 2.4.18.
2.2. Userspace libraries
If you plan to use NFCT and NFLOG input plugin, you will need to
compile libnfnetlink, libnetfilter_conntrack and libnetfilter_log
libraries which can be downloaded from
<http://ftp.netfilter.org/pub/>. A simple './configure; make; make
install' will be enough to have library installed on your system.
2.3. ulogd
2.3.1. Recompiling the source
Download the ulogd package from <http://ftp.netfilter.org/pub/ulogd/>
and untar it.
If you want to build ulogd with MySQL support, type './configure
--with-mysql'. You may also have to specify the path of the mysql
libraries using '--with-mysql=path'. To build ulogd without MySQL
support, just use './configure'.
To compile and install the program, call 'make install'.
3. Configuration
3.1. iptables NFLOG target
3.1.1. Quick Setup
Just add rules using the NFLOG target to your firewalling chain. A
very basic example:
iptables -A FORWARD -j NFLOG --nflog-group 32 --nflog-prefix foo
To increase logging performance, try to use the
--nflog-qthreshold N
option (where 1 < N <= 50). The number you specify is the amount of
packets batched together in one multipart netlink message. If you set
this to 20, the kernel schedules ulogd only once every 20 packets. All
20 packets are then processed by ulogd. This reduces the number of
context switches between kernel and userspace.
Of course you can combine the NFLOG target with the different
netfilter match modules. For a more detailed description, have a look
at the netfilter HOWTO's, available on the netfilter homepage.
3.1.2. NFLOG target reference
--nflog-group N
The number of the netlink multicast group to which NFLOG'ed
packets are sent. You will have to use the same group number in
the NFLOG target and ulogd in order to make logging work.
--nflog-range N
Copyrange. This works like the 'snaplen' parameter of tcpdump.
You can specify a number of bytes up to which the packet is
copied. If you say '40', you will receive the first fourty bytes
of every packet. Leave it to 0 to dump the whole packet.
--nflog-threshold N
Queue threshold. If a packet is matched by the iptables rule,
and already N packets are in the queue, the queue is flushed to
userspace. You can use this to implement a policy like: Use a
big queue in order to gain high performance, but still have
certain packets logged immediately to userspace.
--nflog-prefix STRING
A string that is associated with every packet logged by this
rule. You can use this option to later tell from which rule the
packet was logged.
3.2. iptables ULOG target
3.2.1. Quick Setup
Just add rules using the ULOG target to your firewalling chain. A very
basic example:
iptables -A FORWARD -j ULOG --ulog-nlgroup 32 --ulog-prefix foo
To increase logging performance, try to use the
--ulog-qthreshold N
option (where 1 < N <= 50). The number you specify is the amount of
packets batched together in one multipart netlink message. If you set
this to 20, the kernel schedules ulogd only once every 20 packets. All
20 packets are then processed by ulogd. This reduces the number of
context switches between kernel and userspace.
Of course you can combine the ULOG target with the different netfilter
match modules. For a more detailed description, have a look at the
netfilter HOWTO's, available on the netfilter homepage.
3.2.2. ULOG target reference
--ulog-nlgroup N
The number of the netlink multicast group to which ULOG'ed
packets are sent. You will have to use the same group number in
the ULOG target and ulogd in order to make logging work.
--ulog-cprange N
Copyrange. This works like the 'snaplen' parameter of tcpdump.
You can specify a number of bytes up to which the packet is
copied. If you say '40', you will receive the first fourty bytes
of every packet. Leave it to 0
--ulog-qthreshold N
Queue threshold. If a packet is matched by the iptables rule,
and already N packets are in the queue, the queue is flushed to
userspace. You can use this to implement a policy like: Use a
big queue in order to gain high performance, but still have
certain packets logged immediately to userspace.
--ulog-prefix STRING
A string that is associated with every packet logged by this
rule. You can use this option to later tell from which rule the
packet was logged.
3.2.3. ipt_ULOG module parameters
The ipt_ULOG kernel module has a couple of module loadtime parameters
which can (and should) be tuned to accomodate the needs of the
application:
nlbufsiz N
Netlink buffer size. A buffer of the specified size N is
allocated for every netlink group that is used. Please note that
due to restrictions of the kernel memory allocator, we cannot
have a buffer size > 128kBytes. Larger buffer sizes increase the
performance, since less kernel/userspace context switches are
needed for the same amount of packets. The backside of this
performance gain is a potentially larger delay. The default
value is 4096 bytes, which is quite small.
flushtimeout N
The flushtimeout determines, after how many clock ticks (on
alpha: 1ms, on x86 and most other platforms: 10ms time units)
the buffer/queue is to be flushed, even if it is not full. This
can be used to have the advantage of a large buffer, but still a
finite maximum delay introduced. The default value is set to 10
seconds.
Example:
modprobe ipt_ULOG nlbufsiz=65535 flushtimeout=100
This would use a buffer size of 64k and a flushtimeout of 100
clockticks (1 second on x86).
3.3. ulogd
ulogd is what this is all about, so let's describe it's
configuration...
3.3.1. ulogd configfile syntax reference
All configurable parameters of ulogd are in the configfile, typically
located at '/etc/ulogd.conf'.
The following configuration parameters are available:
logfile
The main logfile, where ulogd reports any errors, warnings and
other unexpected conditions. Apart from a regular filename, the
following special values can be used; ``syslog'' to log via the
unix syslog(3) mechanism. ``stdout'' to log to stdout.
loglevel
This specifies, how verbose the logging to logfile is. Currently
defined loglevels are: 1=debug information, 3=informational
messages, 5=noticable exceptional conditions, 7=error
conditions, 8=fatal errors, program abort.
plugin
This option is followed by a filename of a ulogd plugin, which
ulogd should load upon initialization. This option may appear
more than once.
stack
This option is followed by a list of plugin instances which will
start with an input plugin, contains optionnal filtering plugin
and finish by an output plugin. This option may appear more than
once.
3.3.2. ulogd commandline option reference
Apart from the configfile, there are a couple of commandline options
to ulogd:
-h --help
Print a help message about the commandline options.
-V --version
Print version information about ulogd.
-d --daemon
For off into daemon mode. Unless you are debugging, you will
want to use this most of the time.
-c --configfile
Using this commandline option, an alternate config file can be
used. This is important if multiple instances of ulogd are to be
run on a single machine.
-i --info
Display informations about the plugin whom filename is given as
argument.
4. Signals / Logrotate
ulogd understands two kinds of signals:
SIGHUP
Close and re-open all logfiles. This is mainly intended for
logrotate scripts. Also closes and re-opens database
connections.
SIGUSR1
Reload configuration file. This is not fully implemented yet.
SIGUSR2
Dump the whole conntrack table and flush counters afterwards.
Only Plugin ulogd_inpflow_NFCT.so uses this signal.
5. Available plugins
It is important to understand that ulogd without plugins does nothing.
It will receive packets, and do nothing with them.
There are two kinds of plugins, interpreter and output plugins.
Interpreter plugins parse the packet, output plugins write the
interpreted information to some logfile/database/...
You can get information about plugins by running
______________________________________________________________________
ulogd -i path/to/plugin/file.so
______________________________________________________________________
5.1. Input plugins
ulogd comes with the following input plugins:
5.1.1. ulogd_inppkt_NFLOG.so
This interfaces the new nfnetlink_log interface. To compile, you need
libnetfilter_log installed in your system.
group
The number of the netlink multicast group to which NFLOG'ed
packets are sent. You will have to use the same group number in
the NFLOG target (--nflog-group) and in the input plugin.
addressfamily
You will need to specify the value of the protocol if you are
not loging IPv4 packet. addressfamily is 7 to bridged packet
and 10 for IPv6 packet.
numeric_label
You can use this label to store information relative to the
logging. The administrator can define a convention which can be
used later to differenciate packet. For example, it can store
the severity of the logged event.
netlink_socket_buffer_size
Specify the base socket buffer size. This start value will be
increased if needed up to netlink_socket_buffer_maxsize.
netlink_socket_buffer_maxsize
Specify the base socket buffer maximum size.
5.1.2. ulogd_inpflow_NFCT.so
This interfaces the nfnetlink_conntrack kernel subsystem, and provides
flow-based logging. To compile, you need libnetfilter_conntrack
installed on your system.
pollinterval
Change connection tracking dump interval.
hash_enable
If set to 1 (default) a internal hash will be stored and only
destroy event will reach the output plugin. It set to 0, all
events are reveived by the output plugin.
hash_buckets
Size of the internal hash bucket.
hash_max_entries
Maximum number of entries in the internal connection hash.
event_mask
Select event received from kernel based on a mask. Event types
are defined as follows:
o Creation event: 0x00000001
o Update event: 0x00000002
o Destroy event: 0x00000004
netlink_socket_buffer_size
Specify the base socket buffer size. This start value will be
increased if needed up to netlink_socket_buffer_maxsize.
netlink_socket_buffer_maxsize
Specify the base socket buffer maximum size.
5.1.3. ulogd_inppkt_ULOG.so
The good old ipt_ULOG input plugin. This basically emulates ulogd-1.x
which didn't have input plugins.
nlgroup
The number of the netlink multicast group to which ULOG'ed
packets are sent. You will have to use the same group number in
the ULOG target and nin the input plugin.
numeric_label
You can use this label to store information relative to the
logging. The administrator can define a convention which can be
used later to differenciate packet. For example, it can store
the severity of the logged event.
5.2. Interpreter plugins
ulogd comes with the following interpreter plugins:
5.2.1. ulogd_raw2packet_BASE.so
Basic interpreter plugin for nfmark, timestamp, mac address, ip
header, tcp header, udp header, icmp header, ah/esp header... Most
people will want to load this very important plugin.
5.2.2. ulogd_filter_PWSNIFF.so
Example interpreter plugin to log plaintext passwords as used with FTP
and POP3. Don't blame me for writing this plugin! The protocols are
inherently insecure, and there are a lot of other tools for sniffing
passwords... it's just an example.
5.2.3. ulogd_filter_IFINDEX.so
Filter plugin that provides translation from the numerical ifindex
(e.g. '1') to the network interface name (e.g. 'eth4').
5.2.4. ulogd_LOCAL.so
This is a 'virtual interpreter'. It doesn't really return any
information on the packet itself, rather the local system time and
hostname. Please note that the time is the time at the time of
logging, not the packets receive time.
5.2.5. ulogd_filter_HWHDR.so
This plugin convert hardware header to string. In the case of ethernet
packet, it basically convert mac address to a string represetation.
5.2.6. ulogd_filter_IP2BIN.so
This plugin convert IP addresses to a binary form usable by databases
like MySQL.
5.2.7. ulogd_filter_IP2HBIN.so
This plugin convert IP addresses to a binary form in host order usable
by databases like MySQL.
5.2.8. ulogd_filter_IP2STR.so
This plugin convert IP addresses to string.
5.2.9. ulogd_filter_PRINTFLOW.so
Convert the keys relative to a flow in a string readable by human.
5.2.10. ulogd_filter_PRINTPKT.so
Convert the keys relative to a packet in a string readable by human.
This plugin has to be used to print packet in the format similar to
the LOG target format.
5.2.11. ulogd_filter_MARK.so
When this plugin is put in a stack, only messages were the mark
(packet mark or connection mark) matches the given mark/mask will be
logged.
mark
Define the mark which will be used to check packet or flow.
mask
Define the mask which will be used to check packet or flow.
5.3. Output plugins
ulogd comes with the following output plugins:
5.3.1. ulogd_output_OPRINT.so
A very simple output module, dumping all packets in the format
===>PACKET BOUNDARY
key=value
key=value
...
===>PACKET BOUNDARY
...
to a file. The only useful application is debugging.
The module defines the following configuration directives:
dumpfile
The filename where it should log to. The default is
/var/log/ulogd_oprint.log
5.3.2. ulogd_output_LOGEMU.so
An output module which tries to emulate the old syslog-based LOG
targed as far as possible. Logging is done to a seperate textfile
instead of syslog, though.
The module defines the following configuration directives:
file
The filename where it should log to. The default is
/var/log/ulogd_syslogemu.log
sync
Set this to 1 if you want to have your logfile written
synchronously. This may reduce performance, but makes your log-
lines appear immediately. The default is 0
5.3.3. ulogd_output_MYSQL.so
An output plugin for logging into a mysql database. This is only
compiled if you have the mysql libraries installed, and the configure
script was able to detect them. (that is: --with-mysql was specified
for ./configure)
The plugin automagically runs a procedure with arguments taken from a
the configurable table; It connects to mysql during the startup phase
of ulogd and obtains a list of the columns in the table. Then it tries
to resolve the column names against keys of interpreter plugins. This
way you can easily select which information you want to log - just by
the layout of the table.
If, for example, your table contains a field called 'ip_saddr', ulogd
will resolve this against the key 'ip.saddr' and put the ip address as
32bit unsigned integer into the corresponding argument of table.
The file 'doc/mysql-ulogd2.sql' contains a schema for both packet and
flow logging.
The module defines the following configuration directives:
table
Name of the table which ulogd will use to build arguments list.
procedure
Stored procedure that will be run with the argument specified in
the table variable. Behaviour of the procedure option can be
twitted by using specific name. If procedure name is:
o "INSERT": A classic INSERT SQL query is done in the table
pointed by the "table" variable.
o start with "INSERT ": Configuration has to specify the start
of the INSERT query that will be used. For example, a typical
value is "INSERT INTO ulog2".
o start with "CALL": the named stored procedure is executed
with the "CALL" MySQL command.
o Otherwise the named stored function is executed with the
"SELECT" MySQL command.
db Name of the mysql database.
host
Name of the mysql database host.
port
TCP port number of mysql database server.
user
Name of the mysql user.
pass
Password for mysql.
reconnect
Number of reconnection attempt before declaring the output
plugin as dead.
connect_timeout
Database connection timeout.
5.3.4. ulogd_output_PGSQL.so
An output plugin for logging into a postgresql database. This is only
compiled if you have the pgsql libraries installed, and the configure
script was able to detect them. (that is: --with-pgsql was specified
for ./configure)
The plugin automagically runs a procedure with arguments taken from a
the configurable table; It connects to pgsql during the startup phase
of ulogd and obtains a list of the columns in the table. Then it tries
to resolve the column names against keys of interpreter plugins. This
way you can easily build your own procedure and select it arguments
just by modifying the layout of the table.
If, for example, your table contains a field called 'ip_saddr', ulogd
will resolve this against the key 'ip.saddr' and put the ip address as
32bit unsigned integer into the table.
The file 'doc/pgsql-ulogd2.sql' contains a schema for both packet and
flow logging.
The module defines the following configuration directives:
table
Name of the table which ulogd will use to build arguments list.
procedure
Stored procedure that will be run with the argument specified in
the table variable.
schema
PGSQL schema to use.
db Name of the database.
host
Name of the mysql database host.
port
TCP port number of database server.
user
Name of the sql user.
pass
Password for sql user.
reconnect
Number of reconnection attempt before declaring the output
plugin as dead.
connect_timeout
Database connection timeout.
5.3.5. ulogd_output_PCAP.so
An output plugin that can be used to generate libpcap-style packet
logfiles. This can be useful for later analysing the packet log with
tools like tcpdump or ethereal.
The module defines the following configuration directives:
file
The filename where it should log to. The default is:
/var/log/ulogd.pcap
sync
Set this to 1 if you want to have your pcap logfile written
synchronously. This may reduce performance, but makes your
packets appear immediately in the file on disk. The default is 0
5.3.6. ulogd_output_SQLITE3.so
An output plugin for logging into a SQLITE v3 database. This is only
compiled if you have the sqlite libraries installed, and the configure
script was able to detect them. (that is: --with-sqlite3 was specified
for ./configure)
The plugin automagically inserts the data into the configured table;
It opens the sqlite db during the startup phase of ulogd and obtains a
list of the columns in the table. Then it tries to resolve the column
names against keys of interpreter plugins. This way you can easily
select which information you want to log - just by the layout of the
table.
If, for example, your table contains a field called 'ip_saddr', ulogd
will resolve this against the key 'ip.saddr' and put the ip address as
32bit unsigned integer into the table.
You may want to have a look at the file 'doc/sqlite3.table' as an
example table including fields to log all keys from ulogd_BASE.so.
Just delete the fields you are not interested in, and create the
table. This file contains two tables, one for packet-based logging and
another for flow-based logging.
To create the database file with the tables, you have to invoke the
following command: sqlite3 ulogd.sqlite3db < sqlite3.table
To check that we are logging stuff into it correctly: sqlite3
ulogd.sqlite3db "SELECT * from ulog_ct"
The module defines the following configuration directives:
table
Name of the table to which ulogd should log.
db Name of the database.
5.3.7. ulogd_output_SYSLOG.so
An output plugin that really logs via syslogd. Lines will look exactly
like printed with traditional LOG target.
The module defines the following configuration directives:
facility
The syslog facility (LOG_DAEMON, LOG_KERN, LOG_LOCAL0 ..
LOG_LOCAL7, LOG_USER)
level
The syslog level (LOG_EMERG, LOG_ALERT, LOG_CRIT, LOG_ERR,
LOG_WARNING, LOG_NOTICE, LOG_INFO, LOG_DEBUG)
6. QUESTIONS / COMMENTS
All comments / questions / ... are appreciated.
Just drop a note to netfilter-devel@vger.kernel.org.
The preferred method for reporting bugs is the netfilter bugzilla
system, available at <http://bugzilla.netfilter.org/>.
Generated by dwww version 1.15 on Thu Aug 28 22:51:58 CEST 2025.