Universal Logging System

A small proxy that records tcp network connections and/or any network traffic.

The ipprotd daemon listens on a port for all configured ip addresses or only for one specific ip address and connects to a destination port and ip address. All traffic is transparently forwarded to the destination and returned back to the source.

The ipprotd is part of the [[uls:agents:linux]].

The ipprotd may collect all connection information like ip addresses or timestamp of begin and end of the connection is written to one log file (option [[ipprot#-c <connection_log_file>]]).

Network traffic (option [[ipprot#-f <data_log_file>]]) can be recorded in connection specific log files for incoming, for outgoing or for data in both directions. The traffic for each connection is written to its own log file. The log file name will contain date and time, ip addresses and ports to make it unique. After the connection is closed, a post-processing command may be executed (option [[ipprot#-u <cmd>]]). That receives among other things the log file name as a parameter.

Each connection has its own ipprotd process which terminates when the connection ends.

Use the correct operating system user to start the ipprot.

ipprotd [-p inport[@listen_host]] [-h out_host] [-P out_port[@out_host]] [-b buffsize]
        [-f data_log_file] [-l|-L] [-s] [-z|-j] [-t secs] [-M] [-u[T] cmd]
        [-c connection_log_file] [-C connection_log_format] 
        [-r client_ip1 [-r client_ip2 ...]]
        [-D|-Dp pid_file]
        [-?|-v]

Have also a look at the [[ipprot#Examples]].

The default buffer size is 300000 Bytes, that may be to large for some szenarios, e.g. a client may expect a specific or maximum amount of bytes. Then specify a smaller <buffsize> in Bytes (the buffer is always placed in memory).

Write connection infos to the <connection_log_file>. All entries will be appended to the <connection_log_file> when the connection is closed. Use option [[ipprot#-C <connection_log_format>]] to specify what is written to the <connection_log_file>. The network traffic is not recorded.

Specify the format of a text expression that is appended to the <connection_log_file>. Any characters are copied unchanged into the output string, except the following special expressions, which are replaced by gathered connection metrics. Only in combination with option [[ipprot#-c <connection_log_file>]].

-C "Connected: %s, disconnected after %S secs at: %e"

Start ipprotd as daemon.

Start as daemon and write its pid to the specified full path of <pid_file>.

Specify the path and file prefix as the <data_log_file> which will contain the logged data. The timestamp of the start of the connection in the format yyyy-mm-dd_HHMMSS, a consecutive number <n> and the client ip <client_ip> is appended to the final log file name to make it unique.

Format: <data_log_file>_<yyyy-mm-dd_HHMMSS>_<n>_<client_ip>

Forward all incoming data to this <out_host>. Default is localhost.

Compress the <data_log_file> by using bzip2 while it is written. Only in combination with option [[ipprot#-f <data_log_file>]].

Log all network traffic in both directions from connection establisher, if neither option [[ipprot#-l]], nor option [[ipprot#-L]] is given, only network traffic to the connection establisher is recorded. Only in combination with option [[ipprot#-f <data_log_file>]].

Log all incoming network traffic from connection establisher, if neither option [[ipprot#-l]], nor option [[ipprot#-L]] is given, only network traffic **to** the connection establisher is recorded. Only in combination with option [[ipprot#-f <data_log_file>]].

(New in version 2.5)

Each logged block is preceded by a timestamp. The timestamp is written as microseconds ($\mu sec$) since Unix epoch preceded by a magic number enclosed in '#'. Only in combination with option [[ipprot#-f <data_log_file>]].

The timestamp is preceded by magic number

• #848367# for incoming data blocks and
• #948366# for outgoing data blocks.

Examples:

#848367#1386169176610141
#948366#1386169176610343

You may convert that to a human readable form on most linuxes (remember that the timestamp is in microseconds):

$ date -d@1386169176 '+%F %T'
2013-12-04 15:59:36

Remember that magic number #748369# indicates an iso timestamp which is inserted at regular intervals, see [[ipprot#-t <secs>]].

example.log

#848367#1386170322361458

#748369#2013-12-04 16:18:42-01:00

#948366#1386170322404186
HTTP/1.1 200 OK
Date: Wed, 04 Dec 2013 15:18:42 GMT
Server: unix2web/9.1.1
Expires: Wed, 04 Dec 2013 15:18:42 GMT
Last-Modified: Wed, 04 Dec 2013 15:18:42 GMT
Content-Type: text/html; charset=UTF-8

#948366#1386170322406047
Connection: Keep-Alive
Content-Length: 535

<!DOCTYPE html>
<HTML><HEAD><meta http-equiv="Pragma" content="no-cache"><meta http-equiv="Expires" content="now"><TITLE>ULS - lulsed001</TITLE>
<link rel="stylesheet" type="text/css" href="stylesheet_uls.css">
<frameset rows="90,*"><frame name="mtop" src="kopf.u2w"><frameset cols="235, *"><frame src="index.u2w?u645342me=on847629" name="mleft"><frame src="index.u2w?u645342me=mr847629" name="mright"></frameset><noframes><body>Diese Seite verwendet Frames, der Browser unterstützt keine Frames!</body></noframes></frameset>
</HTML>

#848367#1386170322763181

#948366#1386170322767464
HTTP/1.1 200 OK
Date: Wed, 04 Dec 2013 15:18:42 GMT
Server: unix2web/9.1.1
Expires: Wed, 04 Dec 2013 15:18:42 GMT
Last-Modified: Wed, 04 Dec 2013 15:18:42 GMT
Content-Type: text/html; charset=UTF-8

#948366#1386170322790097
Connection: Keep-Alive
Content-Length: 1061

<!DOCTYPE html>
<HTML><HEAD><meta http-equiv="Pragma" content="no-cache"><meta http-equiv="Expires" content="now"><TITLE></TITLE>
<style type="text/css">
a img
{ border: 0;
}
</style>
<link rel="stylesheet" type="text/css" href="stylesheet_uls.css">
</HEAD>
<BODY>
<div><div style="float:left; text-align:left;">
<div id="mainmenue">
<A href="index.u2w?u645342me=on847629" target="mleft">Hauptmenü</A>
<A href="list_favoriten.u2w" target="mleft">Favoriten</A>
<A href="list_reports.u2w" target="mleft">Berichte</A>
<A href="index_admin.u2w" target="mleft">Verwalten</A>
<A href="operator/index.u2w" target="_top">Operator Menü</A>
<A href="admin/index.u2w" target="_top">Admin-Menü</A>
<A href="search_server.u2w?sdat=sinceyesterday" target="mleft">Server suchen</A>
</div>
<BR>
<BR>
<div id="breadcrumbs">

</div>
</div><div class="logo" style="float:right">
<A href="http://www.universal-logging-system.org/" target="_blank"><img src="ULS-0003DE-02W_ULS_Logo.png"></A><font size="4"><sup>@</sup></font><img src="Head_Logo.gif">
</div></div>
</BODY></HTML>
/dev/shm>

ipprot listens on this port on all ip addresses or only for <listen_host> for incoming connections and data. Default port is 8080 on all currently defined ip addresses and on all additionally defined ip addresses during the runtime of ipprot.

Forward all incoming data to this <out_port> and <out_host>, which must be given as direct parameter or must be set as option [[ipprot#-h <out_host>]]. Default is port 80 on localhost.

Restrict a connection for this allowed ip address. You may repeat this option for up to 30 times.

Log only printable characters. CR, LF, members of the character classes [:blank:] and [:graph:] are logged, one blank is logged for any number of successive non-printable characters. Only in combination with option [[ipprot#-f <data_log_file>]].

Specify the number of seconds after which a timestamp is inserted into the <data_log_file> when new data arrives from the connection establisher. Only in combination with option [[ipprot#-f <data_log_file>]].

Call <cmd> for post-processing as soon as the connection is closed. The <cmd> will receive as parameters:

1. the full path and file name of the <data_log_file>
2. start timestamp of the connection in seconds since Unix epoch
3. end timestamp of the connection in seconds since Unix epoch
4. the number of bytes received from the connection establisher
5. the number of bytes sent to the connection establisher
6. the ip address of the connection establisher

Only in combination with option [[ipprot#-f <data_log_file>]].

(New in version 2.4)

Call <cmd> for post-processing as soon as the connection is closed. The <cmd> will receive as parameters:

1. the full path and file name of the <data_log_file>
2. start timestamp of the connection in the RFC 3339 format yyyy-mm-dd HH:MM:SS+hh:mm
3. end timestamp of the connection in the RFC 3339 format yyyy-mm-dd HH:MM:SS+hh:mm
4. the number of bytes received from the connection establisher
5. the number of bytes sent to the connection establisher
6. the ip address of the connection establisher

Only in combination with option [[ipprot#-f <data_log_file>]].

Show version information.

Compress the <data_log_file> by using gzip while it is written. Only in combination with option [[ipprot#-f <data_log_file>]].

Show this help.

Here are two examples of ipprot's usage and their explanations.

Start ipprotd as daemon and

• listen on port 5432 for all locally configured ip addresses and all possibly later dynamically configured ip addresses,
• forward the connection to ip 127.0.0.1, port 3333,
• record network traffic in both directions,
• write recorded network traffic to file /var/log/conn_rec/prot6543_<dest_ip>_<yyyy-mm-dd_HHMMSS>_<listen_ip>, the <values> will be replaced by the actual values of the connection or timestamp,
• gzip the file during writing,
• write the pid of the ipprotd master process to file /var/log/conn_rec/ipprot_pid_5432

$ ipprotd -p 5432 -h 127.0.0.1 -P 3333 
          -l -f /var/log/conn_rec/prot -z 
          -Dp /var/log/conn_rec/ipprot_pid_5432

Stop thru kill:

$ kill `cat /var/log/conn_rec/ipprot_pid_5432`

Currently active ipprotd processes will continue to execute until the connection of each respective process is closed normally.

A typical security requirement is the recording of all sql commands that are executed by a database administrator usind e.g. the SQL Developer from his work station. (Opposed to a local connection to the database as sysdba, which does not use the listener and therefor cannot be captured by ipprot. See “Logging a Terminal Session” in the "ULS-client for Linux“ documentation to find out how to do that.

Start ipprotd as daemon and

• listen on ip 10.1.2.33, port 7777
• forward the connection to ip 10.1.2.34, port 5678,
• record only printable incoming network traffic,
• write recorded network traffic to file /u01/oradata/orcl/connection_protocol/prot_7777_10.1.2.34_<yyyy-mm-dd_HHMMSS>_10.1.2.33,
• bzip2 that file during it is written,
• call post-processing script /u01/oradata/orcl/oracle_tools/send_ipprot after the connection has been closed and bzip2 has terminated,
• write the pid of the ipprotd master process to file /u01/oradata/orcl/connection_protocol/pid_7777

$ ipprotd -p 7777@10.1.2.33 -P 5678@10.1.2.34 
          -L -s -f /u01/oradata/orcl/connection_protocol/prot -j  
          -u /u01/oradata/orcl/oracle_tools/send_ipprot 
          -Dp /u01/oradata/orcl/connection_protocol/pid_7777

Stop thru kill:

$ kill `cat /u01/oradata/orcl/connection_protocol/pid_7777`

Currently active ipprotd processes will continue to execute until the connection of each respective process is closed normally.

The ipprotd is part of the [[uls:agents:linux]].