Universal Logging System

This value file format was defined to make the development of agents easy. An agent just has to create a file that contains the acquired values in the here defined format and call a script of the ULS-Agent for Linux to transfer the file contents to the ULS-server.

The encoding of the file is Latin-1 (ISO 8859-1) by default, use L: Set Defaults and Encoding to specify UTF-8 as content encoding.

This file format can easily be generated from arbitrary scripts or executables and is therefore especially useful for independent development of customized ULS-agents. See the Example.

The ULS-Agent for Linux as well as the executable 'send2uls.exe' for W*ndows allow the transfer of ULS-values to the ULS-server as files which conform to the ULS Value File Format.

If you are on a host that already runs the ULS-Agent for Linux, then you just need to create a file (file name extension .uls is mandatory) containing your values in the ULS Value File Format and place the file in the /var/tmp/uls directory. A cron job of the ULS-Agent for Linux will grab that directory regularly and send the complete contents to the ULS-server.

This is a list of placeholders which are used in the following description of the ULS Value File Format.

Remember the data structure: domain - source - section - teststep - detail - value [unit]

The ULS value file contains one value per line. Each line begins with a character indicating a remark or it begins with an identifier which defines the further processing of the rest of the line, all elements are separated by semi colons:

<ID>;<element>[;<element>][;<element>][;<element>]...
#<remark>
;<remark>

Text expressions as values can span more than one line but must be enclosed in double quotes. Contained double quotes must be preceded by a backslash '\\'.

You may leave out <element> at most positions, the last <element> from previous lines within the file will be used. The separator “;” must always be present. See also D: Set Defaults.

See the Example for how to generate similar contents:

V;2018-08-15 15:04:30;SourceA3;My Section;My Teststep;start-stop;Start 2018-08-15 15:04:30;{T}
V;2018-08-15 15:04:30;SourceA3;My Section;My Teststep;file count;66;#
V;2018-08-15 15:04:30;SourceA3;My Section;My Teststep;listener;"Lorem ipsum dolor sit amet";
E4:00;2018-08-15 15:04:30;SourceA3;My Section;My Teststep;kernel release;4.9.0-8-amd64; 
V;2018-08-15 15:04:30;SourceA3;My Section;My Teststep;start-stop;Stop 2018-08-15 15:04:36;{T}

Lines starting with '#' or ';' and empty lines are ignored:

# remark, line is ignored
; remark

# empty lines as above are ignored

(new in version 1.8.0)

A text that is saved as a documentation to a detail:

A;<ignored>;<source>;<section>;<teststep>;<detail>;"<text>";<unit>

The ULS-server can operate a counter, you can set, reset, increment or decrement it:

C;<datetime>;<source>;<section>;<teststep>;<detail>;[<cmode>];[<value>][;<access>]
  
C;2018-11-11 11:11:11;myhost;section;event entries;serious;inc

<cmode> := { add | inc | sub | dec | set | reset }

<value> := { beliebige Zahl }

The defaults are:

<cmode> = inc
<value> = 1

For inc and dec <value> is always 1 independent of what is specified as <value>.

Set one or many values to default expressions for all following lines. They will be used until re-specified to other values:

D;<datetime>;<source>;<section>;<teststep>;<detail>;<ignored>;<ignored>[;<access>]

D;2018-04-19 11:13:31;myhost;my_section;my_teststep;my_detail;;;CHEF_ONLY

Like N: No Duplicate Value but it is definitely saved when the time <elapsed> has elapsed:

E<elapsed>;<datetime>;<source>;<section>;<teststep>;<detail>;<value>;<unit>[;<access>]

E3:45;2018-08-13 14:14:00;host032;system;information;kernel release;4.9.0-8-amd64;

Assuming that you run a monitoring script every 10 minutes and send the above line to the ULS-server, the kernel release will be saved whether:

• the kernel release has changed or
• the last value concerning the kernel release is older than 03:45 hours

Transfer a whole file as a value to the ULS-server:

F;<datetime>;<source>;<section>;<teststep>;<detail>;<local filename>;<uls filename>[;<access>]

F;;;mysection;myteststep;perl script;datetime.pl;datetime.pl

You must copy the file to the same directory as the ULS value file. That is the <local filename>. The <uls filename> must be specified, that is the filename as shown to the user during interactive analysis in ULS. Both filenames can be equal, but you also can specify a differing <uls filename>, e.g. each night you use a tool that generates an output file, always with the same filename “OUTPUT.txt”. In your script you generate the ULS value file:

F;2018-08-13 00:42:13;WINBOX5;nightly jobs;output tool;script result;OUTPUT.txt;OUTPUT.txt

and copy “OUTPUT.txt” to the same directory as the ULS value file. Everything works fine, but what if the ULS value file and the “OUTPUT.txt” could not be transfered to the ULS-server? The next night, “OUTPUT.txt” would be overwritten!

Better use e.g. a timestamp (or pid) in the filename “OUTPUT_2013-08-13.txt”::

F;2018-08-13 00:42:13;WINBOX5;nightly jobs;output tool;script result;OUTPUT_2018-08-13.txt;OUTPUT.txt

but use always the same <uls filename>.

Transfer of file in chunks. You must use exactly the same <datetime> for each file chunk:

FA;<datetime>;<source>;<section>;<teststep>;<detail>;<local filename>[;;<access>]

Transfers an image file. It is the same as F: File, but the file contents (the image itself) is shown when using the browser-based interactive analysis in ULS:

I;<datetime>;<source>;<section>;<teststep>;<detail>;<local filename>;<uls filename>[;<access>]

I;2018-09-03 11:51:37;PNGHOST;paleontologists;Germany;Georg August Goldfuss;"C:\TMP\Georg_August_Goldfuss.jpg";"GAG.jpg"

You must copy the file <local filename> to the same transfer directory as the ULS value file. The <uls filename> must be specified, that is the filename, additionally to the image itself, as shown to the user during interactive analysis in ULS. Both filenames can be equal, but you also may specify a different <uls filename>.

(new in version 1.8.0)

Set one or many values to default expressions for all following lines, including the encoding of the file contents. They will be used until re-specified to other values:

L;[<datetime>];[<source>];[<section>];[<teststep>];[<detail>];[<encoding>];[<unit>];[<access>]

Empty expressions are ignored, the previously set expressions will be used for further value lines.

The <encoding> may be one of:

• latin1
• utf8

Like V: Normal Value but is only saved in ULS if this <value> with this <unit> is different from the previous value in the ULS database (it will only be saved if the value has changed):

N;<datetime>;<source>;<section>;<teststep>;<detail>;<value>;<unit>[;<access>]

N;;lxtest1;OneLine Format;NoDups;IP-Adresse;10.10.10.5;

Set a monitoring pause in ULS to prevent the generation of notifications, directly from the monitored system, without the need to manually set the monitoring pause in the ULS web interface. It can, e.g. be integrated in maintenance scripts.

This is useful to avoid notifications for planned outages.

To be able to use this, the monitored system must be enabled in ULS (administration - client suspension) to do so.

P;<datetime>;<source>;<section>;<teststep>;<detail>;<end_datetime>;<unit>[;<access>[;<remark>]]

P;2018-01-18 07:00:00;host010;System;;;2018-01-19 07:59:59;;;automated sunday patching

alternatively (new in version 1.8.1), you can use a time interval as

P;<datetime>;<source>;<section>;<teststep>;<detail>;<time interval>;<unit>[;<access>[;<remark>]]

P;2018-11-11 11:11:11;host012;System;;;+4h;;;planned outage of system for carnival

All expressions except <source> may contain wildcard pattern. That wildcard pattern must match the in ULS defined pattern for the possible monitoring pause (client suspension). The <source> expression of the monitoring pause definition in ULS, though, may contain wildcards to allow more than one <source> to set monitoring pauses.

Change a monitoring pause. All placeholders must exactly match the ones used when setting the monitoring pause! Only the <end_datetime> ist the <new_end_datetime>.

The monitoring pause must have been created by the same <source>.

PC;<datetime>;<source>;<section>;<teststep>;<detail>;<new_end_datetime>;<unit>[;<access>[;<remark>]]

PC;2018-01-18 07:00:00;host010;System;;;2018-01-18 12:59:59;;;automated sunday patching

alternatively (new in version 1.8.1), if a time interval was used when setting the pause:

PC;<datetime>;<source>;<section>;<teststep>;<detail>;<time interval>;<unit>[;<access>[;<remark>]]

PC;2018-11-11 11:11:11;host012;System;;;+10m;;;planned outage of system for carnival

See the P: Pause for the explanation of expressions.

Delete a monitoring pause. All placeholders must exactly match the ones used when setting the monitoring pause!

The monitoring pause must have been created by the same <source>.

PD;<datetime>;<source>;<section>;<teststep>;<detail>;<end_datetime>;<unit>[;<access>[;<remark>]]

PD;2018-01-18 07:00:00;host010;System;;;2018-01-18 23:59:59;;;automated sunday patching

alternatively (new in version 1.8.1), if a time interval was used when setting the pause:

PD;<datetime>;<source>;<section>;<teststep>;<detail>;<time interval>;<unit>[;<access>[;<remark>]]

PD;2018-11-11 11:11:11;host012;System;;;+4h;;;planned outage of system for carnival

See the P: Pause for the explanation of expressions.

Remember that ULS automatically removes monitoring pauses when they are expired (next night), hence, an explicit deletion may perhaps be unnecessary.

(new in version 1.8.0)

A text that is saved as a documentation to a section::

R;<ignored>;<source>;<section>;<ignored>;<ignored>;"<text>"

A document (as file) is saved related to the source (server etc):

S;<datetime>;<source>;<name>;<description>;<ignored>;<local filename>;[<download>]

S;2018-01-17 12:01:02;lxtest1;Name;"ULS-client for Linux configuration file";;/etc/uls/uls.conf;uls.conf

Use <download> to specify a text expression that is shown when using the browser-based interactive analysis in ULS. The default is the basename of the <local filename>.

A text that is saved as a documentation to a teststep:

T;<ignored>;<source>;<section>;<teststep>;<ignored>;"<text>"

T;;lxtest1;System;System Load;;"System Load shows CPU usage"

T;;host003;"Oracle DB";"tablespace:TS4";;"Lorem ipsum dolor sit amet, 
consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et 
dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo 
dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem 
ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, 
sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, 
sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. 
Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet."

A developer of a script can put the complete documentation of all teststeps and their details into her/his script and transfer that documentation to the ULS-server (e.g. once a day). The user of the interactive analysis can then access that documentation.

The teststep documentation can be accessed via '<domain> –> <server> –> <section> –> <teststep>'. and then click on the headline in the right frame. A is shown, if a teststep documentation is present.

Like V: Normal Value but is only saved in ULS if exactly this <value> with this <unit> does not exist in the ULS database for the (exactly) specified <datetime>:

U;<datetime>;<source>;<section>;<teststep>;<detail>;<value>;<unit>[;<access>]

U;2018-04-18 12:00:00;lxtest1;OneLine Format;Einmalig;Unique;47;#

Line for a normal value, numeric or text::

V;<datetime>;<source>;<section>;<teststep>;<detail>;<value>;<unit>[;<access>]

V;2018-03-26 15:47:00;lxhost;"Oracle DB";"tablespace:TS3";"size";"28842.5";"MB";dba

V;2018-11-10 10:10:00;moorea;"Oracle DB";"tablespace:TS4";"size";"25.6";"MB"

V;2018-05-13 13:17:00;herkules;"Infoboard";"Introduction";"Text";"Lorem ipsum dolor sit amet, 
consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et 
dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo 
dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem 
ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, 
sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, 
sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. 
Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet."; 

To represent time intervals (like execution times of actions) specific value tuples have been defined. They consist of the both values:

• Start YYYY-MM-DD hh:mm:ss
• Stop YYYY-MM-DD hh:mm:ss

where the 'YYYY-MM-DD hh:mm:ss' must be replaced by the actual date and time of the start and the stop of the action. Both values must be sent referring to the same <datetime>. A system specific unit {T} is used to indicate time tuples to the ULS-server, they are processed particularly:

V;<datetime>;<source>;<section>;<teststep>;<detail>;Start YYYY-MM-DD hh:mm:ss;{T}[;<access>]
V;<datetime>;<source>;<section>;<teststep>;<detail>;Stop YYYY-MM-DD hh:mm:ss;{T}[;<access>]

V;2018-09-08 08:32:47;myhost;section;"backup";"start-stop";Start 2018-09-08 08:32:47;{T}
V;2018-09-08 08:32:47;myhost;section;"backup";"start-stop";Stop 2018-09-08 10:13:28;{T}

Important: the start/stop time tuple MUST have the SAME <datetime>. <detail> is often: “start-stop”, but may be any expression.

If a user wants to send data to the ULS-server, she/he must

1. create a script or program that gathers the wanted values.
2. write these values to a file, use the ULS value file format. The file must have .uls as extension!
3. call the command script (part of the ULS-client) flush_test_values (Un*x) or send2uls.exe (W*ndows)

An example:

example.sh

#!/bin/sh
#
# Example of generation of a file in ULS-value file format
#
 
# Path where the ULS-value file are placed.
VDIR="/tmp/myprivateuls"
 
mkdir $VDIR
 
# Filename of the ULS-value file.
# NOTE: it MUST have the extension '.uls'
 
VFILE=$VDIR/uls_$$.uls
 
echo "Write values to file: $VFILE"
echo > $VFILE
 
# -----
# Current date and time, that is the ULS timestamp
DT=`date +"%Y-%m-%d %H:%M:%S"`
 
# Take the hostname of the system
# (Remember that hostname and ip-address must be known to ULS to accept values!)
HOST=`uname -n`
SECTION="My Section"
TESTSTEP="first values"
 
# -----
# Start time of the script
T1=`date +"%Y-%m-%d %H:%M:%S"`
 
# V;<datetime>;<source>;<section>;<teststep>;<detail>;Start 2018-08-13 13:45:04;{T}
echo "V;$DT;$HOST;$SECTION;$TESTSTEP;start-stop;Start $T1;{T}" >> $VFILE
 
# -----
# a value
# V;<datetime>;<source>;<section>;<teststep>;<detail>;<value>;<unit>[;<access>]
 
# Get some value
V=`ls /tmp/ | wc -l`
 
echo "V;$DT;$HOST;$SECTION;$TESTSTEP;file count;$V;#" >> $VFILE
 
# -----
# Get the kernel release
K=`uname -r`
 
echo "E4:00;$DT;$HOST;$SECTION;$TESTSTEP;kernel release;$K; " >> $VFILE
# SEND it every time, but SAVE it at the ULS-server only if 
# kernel release has changed or the last kernel release value
# is older than 4 hours (E4:00;...).
 
# -----
# Simulate an action
 
echo "Pause for 5s"
sleep 5
 
# -----
# a text expression
# It may span several lines, so enclose it in double quotes.
 
T=`ps -ef | fgrep sbin/init | grep -v grep`
 
echo "V;$DT;$HOST;$SECTION;$TESTSTEP;init process;\"$T\"; " >> $VFILE
# ' ' (blank) is the unit for normal text expressions
 
# -----
# Script ends at
 
T2=`date +"%Y-%m-%d %H:%M:%S"`
 
# V;<datetime>;<source>;<section>;<teststep>;<detail>;Stop 2018-08-13 13:45:04;{T}
echo "V;$DT;$HOST;$SECTION;$TESTSTEP;start-stop;Stop $T2;{T}" >> $VFILE
 
# -----
# flush_test_values is found in the ULS-Client for Unix.
# Must be added to PATH.
 
echo "Transfer to ULS-server"
flush_test_values $VDIR
 
# The directory is completely removed after the successful transfer
 
echo "FINISHED"

The resulting ULS-value file looks like:

  V;2018-08-15 15:04:30;SourceA3;My Section;first values;start-stop;Start 2018-08-15 15:04:30;{T}
  V;2018-08-15 15:04:30;SourceA3;My Section;first values;file count;66;#
  V;2018-08-15 15:04:30;SourceA3;My Section;first values;init process;"root         1     0  0 10:09 ?        00:00:00 /sbin/init";
  E4:00;2018-08-15 15:04:30;SourceA3;My Section;first values;kernel release;4.9.0-8-amd64; 
  V;2018-08-15 15:04:30;SourceA3;My Section;first values;start-stop;Stop 2018-08-15 15:04:36;{T}

note the equal <datetime> for all lines, but the different time stamps for Start and Stop.