===== ULS Value File Format =====

This value file format was defined to make the development of agents easy.
An agent just has to create a file that contains the acquired values in the 
here defined format and call a script of the [[uls:agents:linux]] to transfer
the file contents to the ULS-server.

The encoding of the file is Latin-1 (ISO 8859-1) by default, 
use [[value_file_format#L: Set Defaults and Encoding]] to specify UTF-8 as content encoding.

This file format can easily be generated from arbitrary scripts or executables 
and is therefore especially useful for independent development of customized  
ULS-agents. See the [[value_file_format#Example]].

The [[uls:agents:linux]] as well as the 
executable 'send2uls.exe' for W*ndows  
allow the transfer of ULS-values to the ULS-server as files 
which conform to the [[value_file_format]].

If you are on a host that already runs the [[uls:agents:linux]], 
then you just need to create a file (file name extension ''.uls'' is mandatory) 
containing your values in the [[value_file_format]] and
place the file in the ''/var/tmp/uls'' directory.
A cron job of the [[uls:agents:linux]] will grab that directory regularly 
and send the complete contents to the ULS-server.

-----

==== Place Holders ====

This is a list of placeholders which are used in the 
following description of the [[value_file_format]].

Remember the data structure: domain - source - section - teststep - detail - value [unit]

{|
! place holder 
! length
! description
|-
| <access>
|
|this must be one of the defined access codes which are defined
 in the ULS administration (:menuselection:`edit access`). 
 Only users who are assigned to that <access> can view 
 the such tagged values in the ULS interactive analysis.
 Details with equal name and unit but different <access> 
 **are** distinguished!
|-
| <date>
| 10
| date in the format "YYYY-MM-DD"
|-
| <datetime>
| 19
| date and time in the format: YYYY-MM-DD HH:MM:SS.
 It may NOT be enclosed in quotes!
|-
| <detail>
| 0..100
| user-defined text expression for a detail.
|-
| <elapsed>
| 5..7
| time period in the format [h[h]]hh:mm
|-
| < filename>
| 4GB
| maximum size of files/LOBs, the max size that YOU will send to the ULS-server is **at least** needed as /tmp
|-
| <source>
| 0..30
| name of the source (of the values), typically the 
 server name or hostname, but can also be an arbitrary 
 text expression e.g. when using a cluster.
|-
| <ignored>
| 
| the value at this position is ignored, it can be omitted
|-
| <local filename>
|
| complete path of a file
|-
| <section>
| 0..50
| arbitrary text expression for the section to which the value belongs.
|-
| <teststep>
| 0..200
| arbitrary text expression for the teststep, 
 to which the value belongs.
 Use the separator ':' within the text expression to 
 build recursively more levels of sub-teststeps.
|-
| <time>
| 8
| time in the format "HH:MM:SS"
|-
| <unit>
| 0..100
| Generally arbitrary text expression as unit of the value.
 Specific units allow graphical representations, 
 others only tabular.
|-
| <value>
| 0..20 | 0..32000
| arbitrary text expression for a value. Can be textual (0..32000)
 or numeric (0..20, integer, float also scientific notation).
 double quotes (") contained in text expressions 
 must be quoted with back slash (\\").
|}

-----

==== Format Description ====

The ULS value file contains one value per line. Each line begins 
with a character indicating a remark or 
it begins with an identifier which defines the further processing 
of the rest of the line, all elements are separated by semi colons:

  <ID>;<element>[;<element>][;<element>][;<element>]...
  #<remark>
  ;<remark>


Text expressions as values can span more than one line 
but must be enclosed in double quotes. Contained double 
quotes must be preceded by a backslash '\\'.

You may leave out <element> at most positions, the last 
<element> from previous lines within the file will be used.
The separator ";" must always be present. 
See also [[value_file_format#D: Set Defaults]].

See the [[value_file_format#Example]] for how to generate similar contents:

  V;2018-08-15 15:04:30;SourceA3;My Section;My Teststep;start-stop;Start 2018-08-15 15:04:30;{T}
  V;2018-08-15 15:04:30;SourceA3;My Section;My Teststep;file count;66;#
  V;2018-08-15 15:04:30;SourceA3;My Section;My Teststep;listener;"Lorem ipsum dolor sit amet";
  E4:00;2018-08-15 15:04:30;SourceA3;My Section;My Teststep;kernel release;4.9.0-8-amd64; 
  V;2018-08-15 15:04:30;SourceA3;My Section;My Teststep;start-stop;Stop 2018-08-15 15:04:36;{T}

-----

=== Remarks ===

Lines starting with '#' or ';' and empty lines are ignored:

  # remark, line is ignored
  ; remark
  
  # empty lines as above are ignored

-----

=== A: Detail Documentation ===

(new in version 1.8.0)

A text that is saved as a documentation to a detail:

  A;<ignored>;<source>;<section>;<teststep>;<detail>;"<text>";<unit>

-----

=== C: Counter ===

The ULS-server can operate a counter, you can set, reset, 
increment or decrement it:

  C;<datetime>;<source>;<section>;<teststep>;<detail>;[<cmode>];[<value>][;<access>]
    
  C;2018-11-11 11:11:11;myhost;section;event entries;serious;inc

<cmode> := { add | inc | sub | dec | set | reset }

<value> := { beliebige Zahl }

The defaults are:

  <cmode> = inc
  <value> = 1

For inc and dec <value> is always 1 independent of what is specified as <value>.

-----

=== D: Set Defaults ===

Set one or many values to default expressions for all following lines. 
They will be used until re-specified to other values:

  D;<datetime>;<source>;<section>;<teststep>;<detail>;<ignored>;<ignored>[;<access>]
  
  D;2018-04-19 11:13:31;myhost;my_section;my_teststep;my_detail;;;CHEF_ONLY

-----

=== E: No Duplicate with Elapsed ===

Like [[value_file_format#N: No Duplicate Value]] but it is definitely saved when 
the time <elapsed> has elapsed:

  E<elapsed>;<datetime>;<source>;<section>;<teststep>;<detail>;<value>;<unit>[;<access>]
  
  E3:45;2018-08-13 14:14:00;host032;system;information;kernel release;4.9.0-8-amd64;

Assuming that you run a monitoring script every 10 minutes 
and send the above line to the ULS-server, the kernel release 
will be saved whether:

  * the kernel release has changed or
  * the last value concerning the kernel release is older than 03:45 hours

-----

=== F: File ===

Transfer a whole file as a value to the ULS-server:

  F;<datetime>;<source>;<section>;<teststep>;<detail>;<local filename>;<uls filename>[;<access>]
  
  F;;;mysection;myteststep;perl script;datetime.pl;datetime.pl

You must copy the file to the same directory as the ULS value file. 
That is the <local filename>. The <uls filename> must be specified, 
that is the filename as shown to the user during interactive analysis in ULS.
Both filenames can be equal, but you also can specify a differing
<uls filename>, e.g. each night you use a tool that generates 
an output file, always with the same filename "OUTPUT.txt".
In your script you generate the ULS value file:

  F;2018-08-13 00:42:13;WINBOX5;nightly jobs;output tool;script result;OUTPUT.txt;OUTPUT.txt

and copy "OUTPUT.txt" to the same directory as the ULS value file.
Everything works fine, but what if the ULS value file and the "OUTPUT.txt"
could not be transfered to the ULS-server? The next night, "OUTPUT.txt" 
would be overwritten!

Better use e.g. a timestamp (or pid) in the filename "OUTPUT_2013-08-13.txt"::

  F;2018-08-13 00:42:13;WINBOX5;nightly jobs;output tool;script result;OUTPUT_2018-08-13.txt;OUTPUT.txt

but use always the same <uls filename>.

-----

=== FA: File Chunks with Append ===

Transfer of file in chunks. You must use exactly the same <datetime> for 
each file chunk:

  FA;<datetime>;<source>;<section>;<teststep>;<detail>;<local filename>[;;<access>]

-----

=== I: Image File ===

Transfers an image file. It is the same as [[value_file_format#F: File]], 
but the file **contents** (the image itself) is shown 
when using the browser-based interactive analysis in ULS:

  I;<datetime>;<source>;<section>;<teststep>;<detail>;<local filename>;<uls filename>[;<access>]
  
  I;2018-09-03 11:51:37;PNGHOST;paleontologists;Germany;Georg August Goldfuss;"C:\TMP\Georg_August_Goldfuss.jpg";"GAG.jpg"

You must copy the file <local filename> to the same transfer directory as the ULS value file. 
The <uls filename> must be specified, that is the filename, additionally to the image itself, 
as shown to the user during interactive analysis in ULS.
Both filenames can be equal, but you also may specify a different <uls filename>.

-----

=== L: Set Defaults and Encoding ===

(new in version 1.8.0)

Set one or many values to default expressions for all following lines, 
including the encoding of the file contents. 
They will be used until re-specified to other values:

  L;[<datetime>];[<source>];[<section>];[<teststep>];[<detail>];[<encoding>];[<unit>];[<access>]

Empty expressions are ignored, the previously set expressions will be used for further value lines.

The <encoding> may be one of:

  * latin1
  * utf8

-----

=== N: No Duplicate Value ===

Like [[value_file_format#V: Normal Value]] but is only saved in ULS if this <value> 
with this <unit> is different from the previous value in the ULS database
(it will only be saved if the value has changed):

  N;<datetime>;<source>;<section>;<teststep>;<detail>;<value>;<unit>[;<access>]
  
  N;;lxtest1;OneLine Format;NoDups;IP-Adresse;10.10.10.5;

-----

=== P: Pause ===

Set a monitoring pause in ULS to prevent the generation of notifications, 
directly from the monitored system, without the need to manually set the monitoring pause
in the ULS web interface. It can, e.g. be integrated in maintenance scripts.

This is useful to avoid notifications for **planned** outages.

To be able to use this, the monitored system must be enabled in ULS (administration - client suspension) to do so.

  P;<datetime>;<source>;<section>;<teststep>;<detail>;<end_datetime>;<unit>[;<access>[;<remark>]]
  
  P;2018-01-18 07:00:00;host010;System;;;2018-01-19 07:59:59;;;automated sunday patching

alternatively (new in version 1.8.1), you can use a time interval as 

  P;<datetime>;<source>;<section>;<teststep>;<detail>;<time interval>;<unit>[;<access>[;<remark>]]
  
  P;2018-11-11 11:11:11;host012;System;;;+4h;;;planned outage of system for carnival


All expressions except <source> may contain wildcard pattern. 
That wildcard pattern must match the in ULS defined pattern for the 
possible monitoring pause (client suspension). The <source> expression 
of the monitoring pause definition in ULS, though, may contain wildcards 
to allow more than one <source> to set monitoring pauses.

{|
! expression
! explanation
|-
| <datetime> 
| sets the begin of the monitoring pause. May also lie in the past.
|-
| <end_datetime>
| sets the end of the monitoring pause. Should obviously lie in the future.
|-
| <time interval>
| +<n>[mhd], use the time interval given in number of minutes, hours or days for the monitoring pause. **The '+' is mandatory!**
|-
| <source> 
| must not contain any wildcards!
|}

-----

=== PC: Pause Change ===

Change a monitoring pause. **All** placeholders must **exactly** match the ones
used when setting the monitoring pause! Only the <end_datetime> ist the <new_end_datetime>.

The monitoring pause must have been created by the same <source>.

  PC;<datetime>;<source>;<section>;<teststep>;<detail>;<new_end_datetime>;<unit>[;<access>[;<remark>]]
  
  PC;2018-01-18 07:00:00;host010;System;;;2018-01-18 12:59:59;;;automated sunday patching

alternatively (new in version 1.8.1), if a time interval was used when setting the pause:

  PC;<datetime>;<source>;<section>;<teststep>;<detail>;<time interval>;<unit>[;<access>[;<remark>]]
  
  PC;2018-11-11 11:11:11;host012;System;;;+10m;;;planned outage of system for carnival

See the [[value_file_format#P: Pause]] for the explanation of expressions.

-----
=== PD: Pause Delete ===

Delete a monitoring pause. **All** placeholders must **exactly** match the ones
used when setting the monitoring pause!

The monitoring pause must have been created by the same <source>.

  PD;<datetime>;<source>;<section>;<teststep>;<detail>;<end_datetime>;<unit>[;<access>[;<remark>]]
  
  PD;2018-01-18 07:00:00;host010;System;;;2018-01-18 23:59:59;;;automated sunday patching

alternatively (new in version 1.8.1), if a time interval was used when setting the pause:

  PD;<datetime>;<source>;<section>;<teststep>;<detail>;<time interval>;<unit>[;<access>[;<remark>]]
  
  PD;2018-11-11 11:11:11;host012;System;;;+4h;;;planned outage of system for carnival

See the [[value_file_format#P: Pause]] for the explanation of expressions.

Remember that ULS automatically removes monitoring pauses when they are expired (next night), 
hence, an explicit deletion may perhaps be unnecessary.

-----
=== R: Section Documentation ===

(new in version 1.8.0)

A text that is saved as a documentation to a section::

  R;<ignored>;<source>;<section>;<ignored>;<ignored>;"<text>"

-----

=== S: Source Documentation ===

A document (as file) is saved related to the source (server etc):

  S;<datetime>;<source>;<name>;<description>;<ignored>;<local filename>;[<download>]
  
  S;2018-01-17 12:01:02;lxtest1;Name;"ULS-client for Linux configuration file";;/etc/uls/uls.conf;uls.conf

Use <download> to specify a text expression that is shown
when using the browser-based interactive analysis in ULS. The default is the 
basename of the <local filename>.

-----

=== T: Teststep Documentation ===

A text that is saved as a documentation to a teststep:

  T;<ignored>;<source>;<section>;<teststep>;<ignored>;"<text>"
  
  T;;lxtest1;System;System Load;;"System Load shows CPU usage"
  
  T;;host003;"Oracle DB";"tablespace:TS4";;"Lorem ipsum dolor sit amet, 
  consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et 
  dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo 
  dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem 
  ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, 
  sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, 
  sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. 
  Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet."

A developer of a script can put the complete documentation of all 
teststeps and their details into her/his script and transfer that 
documentation to the ULS-server (e.g. once a day). 
The user of the interactive analysis can then access that documentation.

The teststep documentation can be accessed via 
'<domain> --> <server> --> <section> --> <teststep>'.
and then click on the headline in the right frame. 
A {{:uls:agents:memo.gif}} is shown, if a teststep documentation is present.

-----

=== U: Unique Value ===

Like [[value_file_format#V: Normal Value]] but is only saved in ULS if exactly this <value> 
with this <unit> does not exist in the ULS database for the (exactly) specified <datetime>:

  U;<datetime>;<source>;<section>;<teststep>;<detail>;<value>;<unit>[;<access>]
  
  U;2018-04-18 12:00:00;lxtest1;OneLine Format;Einmalig;Unique;47;#

-----

=== V: Normal Value ===

Line for a normal value, numeric or text::

  V;<datetime>;<source>;<section>;<teststep>;<detail>;<value>;<unit>[;<access>]
  
  V;2018-03-26 15:47:00;lxhost;"Oracle DB";"tablespace:TS3";"size";"28842.5";"MB";dba
  
  V;2018-11-10 10:10:00;moorea;"Oracle DB";"tablespace:TS4";"size";"25.6";"MB"
  
  V;2018-05-13 13:17:00;herkules;"Infoboard";"Introduction";"Text";"Lorem ipsum dolor sit amet, 
  consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et 
  dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo 
  dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem 
  ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, 
  sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, 
  sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. 
  Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet."; 

-----

=== Start/Stop Time Tuple ===

To represent time intervals (like execution times of actions) 
specific value tuples have been defined. They consist of the 
both values:

  * Start YYYY-MM-DD hh:mm:ss
  * Stop YYYY-MM-DD hh:mm:ss

where the 'YYYY-MM-DD hh:mm:ss' must be replaced by the actual 
date and time of the start and the stop of the action.
Both values must be sent referring to the same <datetime>. A system specific
unit ''{T}'' is used to indicate time tuples to the ULS-server, 
they are processed particularly:

  V;<datetime>;<source>;<section>;<teststep>;<detail>;Start YYYY-MM-DD hh:mm:ss;{T}[;<access>]
  V;<datetime>;<source>;<section>;<teststep>;<detail>;Stop YYYY-MM-DD hh:mm:ss;{T}[;<access>]

  V;2018-09-08 08:32:47;myhost;section;"backup";"start-stop";Start 2018-09-08 08:32:47;{T}
  V;2018-09-08 08:32:47;myhost;section;"backup";"start-stop";Stop 2018-09-08 10:13:28;{T}

Important: the start/stop time tuple **MUST** have the **SAME** <datetime>. 
<detail> is often: "start-stop", but may be any expression.

-----

==== Example ====

If a user wants to send data to the ULS-server, she/he must 

  - create a script or program that gathers the wanted values.
  - write these values to a file, use the ULS value file format. The file must have .uls as extension!
  - call the command script (part of the ULS-client) ''flush_test_values'' (Un*x) or ''send2uls.exe'' (W*ndows)

An example:

<code bash example.sh>

#!/bin/sh
#
# Example of generation of a file in ULS-value file format
#

# Path where the ULS-value file are placed.
VDIR="/tmp/myprivateuls"

mkdir $VDIR

# Filename of the ULS-value file.
# NOTE: it MUST have the extension '.uls'

VFILE=$VDIR/uls_$$.uls

echo "Write values to file: $VFILE"
echo > $VFILE

# -----
# Current date and time, that is the ULS timestamp
DT=`date +"%Y-%m-%d %H:%M:%S"`

# Take the hostname of the system
# (Remember that hostname and ip-address must be known to ULS to accept values!)
HOST=`uname -n`
SECTION="My Section"
TESTSTEP="first values"

# -----
# Start time of the script
T1=`date +"%Y-%m-%d %H:%M:%S"`

# V;<datetime>;<source>;<section>;<teststep>;<detail>;Start 2018-08-13 13:45:04;{T}
echo "V;$DT;$HOST;$SECTION;$TESTSTEP;start-stop;Start $T1;{T}" >> $VFILE

# -----
# a value
# V;<datetime>;<source>;<section>;<teststep>;<detail>;<value>;<unit>[;<access>]

# Get some value
V=`ls /tmp/ | wc -l`

echo "V;$DT;$HOST;$SECTION;$TESTSTEP;file count;$V;#" >> $VFILE

# -----
# Get the kernel release
K=`uname -r`

echo "E4:00;$DT;$HOST;$SECTION;$TESTSTEP;kernel release;$K; " >> $VFILE
# SEND it every time, but SAVE it at the ULS-server only if 
# kernel release has changed or the last kernel release value
# is older than 4 hours (E4:00;...).

# -----
# Simulate an action

echo "Pause for 5s"
sleep 5

# -----
# a text expression
# It may span several lines, so enclose it in double quotes.

T=`ps -ef | fgrep sbin/init | grep -v grep`

echo "V;$DT;$HOST;$SECTION;$TESTSTEP;init process;\"$T\"; " >> $VFILE
# ' ' (blank) is the unit for normal text expressions

# -----
# Script ends at

T2=`date +"%Y-%m-%d %H:%M:%S"`

# V;<datetime>;<source>;<section>;<teststep>;<detail>;Stop 2018-08-13 13:45:04;{T}
echo "V;$DT;$HOST;$SECTION;$TESTSTEP;start-stop;Stop $T2;{T}" >> $VFILE

# -----
# flush_test_values is found in the ULS-Client for Unix.
# Must be added to PATH.

echo "Transfer to ULS-server"
flush_test_values $VDIR

# The directory is completely removed after the successful transfer

echo "FINISHED"
</code>

The resulting ULS-value file looks like:

<file>
  V;2018-08-15 15:04:30;SourceA3;My Section;first values;start-stop;Start 2018-08-15 15:04:30;{T}
  V;2018-08-15 15:04:30;SourceA3;My Section;first values;file count;66;#
  V;2018-08-15 15:04:30;SourceA3;My Section;first values;init process;"root         1     0  0 10:09 ?        00:00:00 /sbin/init";
  E4:00;2018-08-15 15:04:30;SourceA3;My Section;first values;kernel release;4.9.0-8-amd64; 
  V;2018-08-15 15:04:30;SourceA3;My Section;first values;start-stop;Stop 2018-08-15 15:04:36;{T}
</file>

:!: note the equal <datetime> for all lines, 
but the different time stamps for Start and Stop.

-----