===== ULS-Agent for Linux =====

That is the ULS-agent for
checking runtime metrics and status information of
several Linux and Unix-like operating systems, 
searching logfiles, checking audit relevant settings
and logging administrative access.

==== Overview ====

The ULS-Client for Linux consists of several compiled executables, 
bash scripts and configuration files.

  * the base software parts to generate ULS value files in the [[uls:agents:value_file_format]], from simple values to complete files
  * scripts to transfer the ULS value files to the ULS-server
  * scripts to gather runtime metrics of the operating system
  * a logfile adapter to search logfiles for text pattern or to extract values from text files
  * provides a user-related logging mechanism for all entered commands and its output during a terminal session (good for audits)

It requires the package [[:unix2web]].

It is available for SLES11, Kubuntu xy, Solaris x and HP-UX B11.31, although 
the latest version is not always and at once available for all operating systems.

-----

=== Features ===

A part from the core ULS-related functions the ULS-Client for Linux provides:

  * metrics about cpu, disk and network performance
  * the current usage of memory and disks
  * detailed process monitoring
  * storing of files, plain ascii/binary (e.g. /etc/postfix/master.cf, /etc/sudoers, ...)
  * scripts to test web service responses (GET, POST, ...)
  * inventory information in csv format

-----

==== Download & Installation ====

-----
=== openSUSE Leap 42.3 ===

Download the the current version of the [[:unix2web]] and the [[:uls:agents:linux]]
  * {{:unix2web:u2webtools-9.5-3_opensuse_leap_42.3.x86_64.rpm}}
  * {{:uls:agents:uls-client-3.10-8_opensuse_leap_42.3.x86_64.rpm}}
and copy these files as root to a temporary directory.

Checksums:
  # sha1sum u2webtools*
  b3bf67a71b0e3c34b58a0553030e9ec9a767ee9e  u2webtools-9.5-3_openSUSE_Leap_42.3.x86_64.rpm
  
  # sha1sum uls-client*.rpm
  9b457d752f5e4cb8266e413b28a4b8c195f58079  uls-client-3.10-8_openSUSE_Leap_42.3.x86_64.rpm

Install the package:
  # rpm --install u2webtools-9.5-3_openSUSE_Leap_42.3.x86_64.rpm
  # rpm --install uls-client-3.10-8_openSUSE_Leap_42.3.x86_64.rpm


-----

=== SLES12 ===

Download the the current version of the [[:unix2web]] and the [[:uls:agents:linux]]
  * {{:unix2web:u2webtools-9.5-2_sles12.x86_64.rpm}}
  * {{:uls:agents:uls-client-3.10-7_sles12.x86_64.rpm}}
and copy these files as root to a temporary directory.

Checksums:
  # sha1sum u2webtools*
  33be6ec70a4362167daf82cb3e8b9a5a35af168d  u2webtools-9.5-2_sles12.x86_64.rpm
  
  # sha1sum uls-client*.rpm
  d109085f7219a3691803fe3fdb06c44c661a0931  uls-client-3.10-7_sles12.x86_64.rpm

Install the package:
  # rpm --install u2webtools-9.5-2_sles12.x86_64.rpm
  # rpm --install uls-client-3.10-7_sles12.x86_64.rpm


-----

=== RedHat 9 based ===

Download the the current version of the [[:unix2web]] and the [[:uls:agents:linux]]
  * {{:unix2web:u2webtools-9.9-5.el9.x86_64.rpm}}
  * {{:unix2web:u2webtools_mariadb_18-9.9-5.el9.x86_64.rpm}}
  * {{:uls:agents:uls-client-3.15-8.el9.x86_64.rpm}}
and copy these files as root to a temporary directory.

# Checksums: sha256sum
  72fdb9db611d4c18e95eaa89acb72861574c068c2f10827f34bc01f6d8d93a0e  u2webtools-9.9-5.el9.x86_64.rpm
  0fc702734ad6beffda1293cd20adb84a2cfc417a3cf2e4c406d70a9ca63fb263  u2webtools_mariadb_18-9.9-5.el9.x86_64.rpm
  da833ebe7d906f9da31866eb57ac040a03dba7f5116fa8e39b54ef7f6006c6ab  uls-client-3.15-8.el9.x86_64.rpm


-----

=== CentOS 7 (RedHat 7) ===

Download the the current version of the [[:unix2web]] and the [[:uls:agents:linux]]
  * {{:unix2web:u2webtools-9.5-2.el7.x86_64.rpm}}
  * {{:uls:agents:uls-client-3.10-7.el7.x86_64.rpm}}
and copy these files as root to a temporary directory.

Checksums:
  # sha1sum u2webtools*
  c299402729185837755f00a7e2ac17339ff7388f  u2webtools-9.5-2.el7.x86_64.rpm
  
  # sha1sum uls-client*.rpm
  19b55bbf55ec2f7ce1bf9424e6eb5157af8ef3ac  uls-client-3.10-7.el7.x86_64.rpm

Install the package:
  # rpm --install u2webtools-9.5-2.el7.x86_64.rpm
  # rpm --install uls-client-3.10-7.el7.x86_64.rpm


-----

=== CentOS 6 (RedHat 6) ===

Download the the current version of the [[:unix2web]] and the [[:uls:agents:linux]]
  * {{:unix2web:u2webtools-9.5-2.el6.x86_64.rpm}}
  * {{:uls:agents:uls-client-3.10-7.el6.x86_64.rpm}}
and copy these files as root to a temporary directory.

Checksums:
  # sha1sum u2webtools*
  033dbd4d2107f58c999166ced866b927b1f06983  u2webtools-9.5-2.el6.x86_64.rpm
  
  # sha1sum uls-client*.rpm
  d272f4e0fb3b042f6023f2f0ba5eef913e2210fe  uls-client-3.10-7.el6.x86_64.rpm

Install the package:
  # rpm --install u2webtools-9.5-2.el6.x86_64.rpm
  # rpm --install uls-client-3.10-7.el6.x86_64.rpm


-----
=== Debian based ===

Download the the current version of the [[:unix2web]] and the [[:uls:agents:linux]]
  * {{:unix2web:u2webtools_9.6-11debian9_amd64.deb}}
  * {{:unix2web:u2webtools_9.6-11debian10_amd64.deb}}
  * {{:uls:agents:uls-client_3.10-8_amd64.deb}}
and copy these files as root to a temporary directory.

Checksums:
  # sha1sum u2webtools*
  f1753b8e815bf9202c8ab74f63fe9c62  u2webtools_9.6-11debian9_amd64.deb
  795f8121d9201936730b906a152e3e29  u2webtools_9.6-11debian10_amd64.deb
  
  # sha1sum uls-client*.rpm
  2de5da6331c57e48d32d21d0fe7b6575c4260e67  uls-client_3.10-8_amd64.deb

Install the package:
  # dpkg -i u2webtools_9.5-2_amd64.deb
  # dpkg -i uls-client_3.10-7_amd64.deb


-----

=== HP-UX ===

Download the the current version of the [[:unix2web]] and the [[:uls:agents:linux]]
  * {{:uls:agents:unix2web-hpux-8.5.6-3.tgz}}
  * {{:uls:agents:uls-client-hp-ux.2.1-1.tgz}}
and copy these files as root to a temporary directory.

Checksums:
  # sha1sum unix2web*.tgz
  
  # sha1sum uls-client*.tgz

:TODO:

Install the package:

:TODO:


:?: What about Ubuntu and Solaris

-----

==== Configuration ====

All configuration is done in directory ''/etc/uls''.
Define the ip-address of your ULS-server:

  # vi /etc/uls/uls.conf
  ...
  ULSSERVER=10.1.2.3:11975

-----

==== Regular Executions ====

A number of scripts is executed in regular time intervals to gather 
all metrics. That is defined in the crontab:

  $ cat /etc/cron.d/ulsclient
  
  # ULS Monitoring
  @reboot root /usr/share/ulsclient/do_tests -f /etc/uls/tests_reboot.conf >/dev/null 2>&1
  */5 * * * * root /usr/share/ulsclient/do_tests /etc/uls/tests5.conf >/dev/null 2>&1
  */15 * * * * root /usr/share/ulsclient/do_tests -f /etc/uls/tests15.conf >/dev/null 2>&1
  36 5 * * * root /usr/share/ulsclient/do_tests -f /etc/uls/tests_daily.conf >/dev/null 2>&1
  35 12 * * * root /usr/share/ulsclient/do_tests -f /etc/uls/tests_daily12.conf >/dev/null 2>&1

{|
! tests5.conf
| Contains all calls to metric scripts that are executed every 5 minutes
|-
! tests15.conf
| Contains all calls to metric scripts that are executed every 15 minutes
|-
! tests_daily12.conf
| Contains all calls to metric scripts that are executed at around 12:00 h
|-
! tests_daily.conf
| Contains all calls to metric scripts that are executed at around 24:00 h
|}

-----

==== Metric Scripts ====

These scripts are provided to acquire metrics about 
the current status and load of the server (=source).

-----

=== test_aide ===

:TODO:

-----

=== test_checkrc ===

(SLES11)

Configure the services in the file ''/etc/uls/checkrc.conf'', that you want to have checked.

<file - /etc/uls/checkrc.conf>
# checkrc.conf
#
# Configuration file for test_checkrc
#
# List all system services that you want to have checked if they are running.
#
# Name           Path to rc-script, it is called with parameter "status"
###########################################################################

cron             /etc/init.d/cron

</file>

The return value of the rc-script is transferred to the ULS-server. 
Normally, 0 indicates no error, the service is running, any other value indicates error.
In the ULS user interface (interactive webbrowser-based analysis), 
it is displayed as (matching to the above example):

<file>
<domain>
  │
  └──►<source>
      │
      └──►System
            │
            └──►Checkrc   | time stamp           | cron |
                          | ---------------------|------|
                          | 2013-11-12 13:05:00  |   0  |
                          | 2013-11-12 13:00:00  |   0  |
                          | 2013-11-12 12:55:00  |   0  |
                          ...
</file>

-----

=== test_conf_files ===

Do the configuration in ''/etc/uls/conf_files.conf'', 
define the name pattern for configuration files
and send the matching files, each compressed with bzip2, to the ULS-server.

<file - /etc/uls/conf_files.conf>
# conf_files.conf
#
# Configuration file for test_conf_files
#
# Files will be compressed by using bzip2 before being sent to ULS
#
# Format:
#
# section teststep        list of pattern for file names including path
###########################################################################

apache    configuration   /etc/apache2/httpd.conf /etc/apache2/conf.d/*

</file>

In the ULS user interface (interactive webbrowser-based analysis), 
it is displayed as (matching to the above example):

<file>
<domain>
  │
  └──►<source>
      │
      └──►apache
            │
            └──►configuration

                | time stamp          | httpd.conf/    | php5.conf/    | dokuwiki.conf.aus/    |
                | --------------------|----------------|---------------|-----------------------|
                | 2013-11-12 14:30:00 | httpd.conf.bz2 | php5.conf.bz2 | dokuwiki.conf.aus.bz2 |
                ...
</file> 

You may check for changes in any of the files by monitoring for differences between consecutive files.

-----

=== test_crontabs ===

That script sends all crontabs (except /etc/crontab) as text to the ULS-server.
In the ULS user interface (interactive webbrowser-based analysis), 
it is displayed as:

<file>
<domain>
  │
  └──►<source>
      │
      └──►System
            │
            └──►Crondir 
              │
              └──►cron.d

                      | time stamp           |  oracle                                                                        |
                      | ---------------------|--------------------------------------------------------------------------------|
                      |                      | #                                                                              |
                      | 2013-11-11 22:06:00  | # 2013-06-05 16:39:05, oracle, make_scripts.pl (0.31)                          |
                      |                      | #                                                                              |
                      |                      | # crontab for Oracle database orcl                                             |
                      |                      | #                                                                              |
                      |                      | # -----                                                                        |
                      |                      | # Monitoring script for Oracle                                                 |
                      |                      | 1,11,21,31,41,51 * * * * oracle /oracle/admin/orcl/oracle_tools/watch_oracle   |
                      |                      | #                                                                              |
                      |                      | # -----                                                                        |
                      |                      | # Hourly actions (e.g. backup of archived redo logs)                           |
                      |                      | 2 * * * * oracle /oracle/admin/orcl/oracle_tools/hourly                        |
                      |                      | #                                                                              |
                      |                      | # -----                                                                        |
                      |                      | # Nightly jobs (database backup, removal of trace files, etc)                  |
                      |                      | 2 22 * * * oracle /oracle/admin/orcl/oracle_tools/nightly                      |
</file> 

-----

=== test_disk_load ===

Gathers metrics about the load of each disk found to be in use on the system.
In the ULS user interface (interactive webbrowser-based analysis), it is displayed as:

<file>
<domain>
  │
  └──►<source>
      │
      └──►System
            │
            └──►Dev
              │
              └──►dm-0

                | time stamp          | rdps | wrps | rd_sec | wr_sec | 
                |                     | 1/s  |  1/s |   1/s  |   1/s  | 
                | --------------------|------|------|--------|--------|
                | 2013-11-12 00:00:00 | 0.06 | 4.80 |   0.61 |  38.43 |
                | 2013-11-12 00:15:00 | 0.01 | 4.72 |   0.08 |  37.79 |
                | 2013-11-12 00:30:00 | 0.00 | 3.90 |   0.00 |  31.20 |
              │ ...
              └──►dm-1
              └──►sda2
              ...
</file>

-----

=== test_disk_space ===

Gathers the disk space usage of all partitions and of all nfs mounted 
partitions if option **-n** is specified.

In the ULS user interface (interactive webbrowser-based analysis), it is displayed as:

<file>
<domain>
  │
  └──►<source>
      │
      └──►System
            │
            └──►Disk Space /data

                | time stamp          | Groesse | belegt | frei  | %belegt | 
                |                     |  GByte  |  GByte | GByte |    %    | 
                | 2013-11-12 00:00:00 |   19.93 |  13.72 |  6.21 |      69 |
                | 2013-11-12 00:15:00 |   19.93 |  13.72 |  6.21 |      69 |
                | 2013-11-12 00:30:00 |   19.93 |  13.72 |  6.21 |      69 |
                | 2013-11-12 00:45:00 |   19.93 |  13.72 |  6.21 |      69 |
                | 2013-11-12 01:00:00 |   19.93 |  13.72 |  6.21 |      69 |
            │   ...
            └──►Disk Space /home
            └──►Disk Space /tmp
            ...
</file>

-----

=== test_eth_load ===

Acquire the network throughput on all found interfaces. Use option **-v** for more details.

In the ULS user interface (interactive webbrowser-based analysis), it is displayed as:

<file>
<domain>
  │
  └──►<source>
      │
      └──►System
            │
            └──►Net
              │
              └──►eth0

                | time stamp          | rxpck | txpck | rxbyt | txbyt | 
                |                     | 1/s  |  1/s |   1/s  |   1/s  | 
                | --------------------|------|------|--------|--------|
                | 2013-11-12 15:35:00 | 0.80 | 0.20 |  48.00 |  10.80 |
                | 2013-11-12 15:40:00 | 1.12 | 0.57 | 509.03 |  67.71 |
                | 2013-11-12 15:45:00 | 1.07 | 0.45 | 598.91 |  34.05 |
                | 2013-11-12 15:50:00 | 1.23 | 0.53 | 699.30 |  39.95 |
                | 2013-11-12 15:55:00 | 1.68 | 0.98 | 659.11 | 123.32 |
              └──►eth1
              ...
</file>

-----

=== test_md ===

Check the status of multiple disks (md).

-----

=== test_mem ===

Gather metrics about the memory and swap usage. Use option **-v** for more details.

In the ULS user interface (interactive webbrowser-based analysis), it is displayed as:

<file>
<domain>
  └──►<source>
      └──►System
            └──►Memory Swap

                | time stamp          | Mem total | Mem used | Mem free | Swap total | Swap used | Swap free | 
                |                     |   MByte   |   MByte  |   MByte  |    MByte   |   MByte   |    MByte  |
                | --------------------|-----------|----------|----------|------------|-----------|-----------|
                | 2013-11-12 00:00:00 |      3834 |      733 |     3100 |       4095 |        37 |      4058 |
                | 2013-11-12 00:15:00 |      3834 |      733 |     3101 |       4095 |        37 |      4058 |
                | 2013-11-12 00:30:00 |      3834 |      733 |     3100 |       4095 |        37 |      4058 |
                ...
</file>

-----

=== test_ping ===

ping a list of ip-addresses or hosts, send the measured response time to ULS.

Do the configuration in ''/etc/uls/test_ping.conf'', 
define the ping destinations and expressions for the ULS logical structure.

<file - /etc/uls/test_ping.conf>
# test_ping.conf
#
# Configuration for test_ping
#
# All <pingdest> are pinged and the response time 
# is sent as value to ULS.
#
# pingdest    : hostname or ip-address to be pinged
# ULS host    : hostname that is used as source, 
#               `hostname` is used if a '.' is set.
#               If nothing is set, <pingdest> is used.
# ULS detail  : detail in ULS, default is <pingdest>
# ULS teststep: teststep in ULS, default: "Ping"
# ULS section : section in ULS, default: "System"
#
#
# pingdest   [ULS host] [ULS detail] [ULS teststep] [ULS section]
# ---------------------------------------------------------------------

host001      .
host002      .
host003      .

</file>

In the ULS user interface (interactive webbrowser-based analysis), 
it is displayed as (matching to the above example):

<file>
<domain>
  └──► <source>
      └──► System
            └──► Ping

                | time stamp          | host001 | host002 | host003 | 
                |                     |    ms   |    ms   |    ms   |
                |---------------------|---------|---------|---------|
                | 2013-11-12 16:15:00 |    3.09 |   0.021 |    3.52 |
                | 2013-11-12 16:10:00 |   0.675 |   0.608 |   0.620 |
                | 2013-11-12 16:05:00 |   0.710 |   0.652 |    7.98 |
                ...
</file>

-----

=== test_proc ===

Determines the number of processes on the system.

In the ULS user interface (interactive webbrowser-based analysis), it is displayed as:

<file>
<domain>
  └──► <source>
      └──► System
            └──► Process

                | time stamp          | Total | running | sleeping | stopped | zombie | 
                | --------------------|-------|---------|----------|---------|--------|
                | 2016-11-12 16:15:00 |   189 |       0 |      189 |       0 |      0 |
                | 2016-11-12 16:10:00 |   187 |       1 |      186 |       0 |      0 |
                | 2016-11-12 16:05:00 |   186 |       0 |      186 |       0 |      0 |
                ...
</file>

-----

=== test_procmon ===

This is an advanced feature to monitor single processes, their used cpu time, memory and much more.

Do the configuration in ''/etc/uls/procmon.conf'', 
you find a detailed description of what can be configured in the 
configuration file. Be sure to start with a simple example :!:

<file - /etc/uls/procmon.conf>
...
#
# Name                 Sum-FLAGS       PID-Flags        [Line-Filter]  [Teststep]  [Section]
#                      (.|[cCdDeEfhM   ([bCdDeEfhlLmMN
#                        RSTuVwWy])    RSsTtUuVwWy])
################################################################################################

.*kswapd0.*            RSVdDeE
.*backup_redologs.sh.* bCtU
.*winbindd.*           .               cCo
.*winbindd.*           acCmMsy

</file>

There will be a summary for all winbind processes
and specific metrics for each winbind process.

In the ULS user interface (interactive webbrowser-based analysis), it will be displayed as: 
<file>
<domain>
  └──► <source>
      └──► System
            └──► Process =>
              └──► kswapd0

                  | time stamp          | RES | Shared-Mem | Virt | io-Read | io-write | 
                  |                     | MB  |     MB     |  MB  | kB/s    |  kB/s    | 
                  | --------------------|-----|------------|------|---------|----------|
                  | 2017-04-04 16:15:00 |  0  |     0      |   0  |       0 |      0   |
                  | 2017-04-04 16:10:00 |  0  |     0      |   0  |       0 |      0   |
                  | 2017-04-04 16:05:00 |  0  |     0      |   0  |       0 |      0   |
                ...

              └──► winbind
              └──► winbind =>

                  | time stamp          | Cpu | Num-Open-Files |
                  |                     |  %  |                |
                  | --------------------|-----|----------------|
                  | 2017-04-04 16:15:00 |  0  |       29       |
                  | 2017-04-04 16:10:00 |  0  |       29       |
                  | 2017-04-04 16:05:00 |  0  |       29       |
                ...

</file>

-----

=== test_security ===

:TODO:

-----

=== test_tcp_connect ===

Similar to [[linux#test_ping]], but uses netcat to just establish a tcp 
or udp connection to destination hosts.

Do the configuration in ''/etc/uls/test_tcp_connect.conf'', 
define the ping destinations and expressions for the ULS logical structure.

-----

=== test_time_diff ===

Check time differences between the local time of the server compared to the time of another server.
You need to set up a reference time server.

:TODO:

Do the configuration in ''/etc/uls/time_diff.conf''.

-----

=== test_system_load ===

:TODO:

-----

==== Bash Scripts ====

These bash scripts are used in the [[linux#Metric Scripts]] and can also be 
used to send arbitrary other values to the ULS-server. You can use it 
to develop your own ULS-agent.

See the header information within the scripts for a detailed 
explanation of the possible command line parameters. The general 
command line parameters are explained [[linux#General Command Line Parameters|below]].

-----

=== General Command Line Parameters ===

See also [[uls:structure]]

{|
! parameter
! description
|-
| -c <charset>
| the character set of the values, <charset> can be "latin1" or "utf8"
|-
| <domain>
| domain, to which the source is related
|-
| <hostname>
| hostname of the server/source that sends values to the ULS
|-
| <section>
| section in the ULS logical data structure
|-
| <teststep>
| teststep in the ULS logical data structure
|-
| <detail>
| detail (headers of the tables containing values) in the ULS logical data structure
|-
| <value>
| value
|-
| <unit>
| unit of the value
|-
| <date>
| date the value belongs to (YYYY-MM-DD)
|-
| <time>
| time the value belongs to (hh:mm[:ss])
|-
| %%<file>%%
| complete path to a file
|-
| <access>
| a privilege text expression that rules the access to the related value, e.g. 'adm' or 'sec'.
 Only users that are granted that privilege can view these such marked values 
 in the ULS user interface (interactive webbrowser-based analysis).
|}

-----

=== flush_test_values ===

Flushes all ULS value files in a directory (default is ULS_TMP_PATH ((as configured in /etc/uls/uls.conf))) as .fpz to ULS and removes the directory if successful.

   flush_test_values [-s <secs>] [-u <ulsserver:port>] [<path>]

If there is no connection to the ULS-server the ULS value files are temporarily stored to the 
LOKALER_TEST_PFAD ((as configured in /etc/uls/uls.conf)). 
Regularly, the transmission of all temporarily stored ULS value files 
is retried (normally induced by crontab entries).

Placing all ULS value files in a defined directory can be used in DMZ environments 
where, because of security reasons, not the source (=server in the DMZ) **sends** the ULS value files **to** the ULS-server, 
but the ULS-server or a proxy **gets** the ULS value files **from** the source (=server in the DMZ).

-----

=== get_server_value ===

Request values from the ULS-server. You may use simple wildcards.

-----

=== jobmonitor2uls ===

Execute a script and send
  * the catched the output to stdout and stderr
  * the result of the script execution (return value)
  * the execution time of the script

to ULS.

  jobmonitor2uls [-d <storepath>] [-S] [-u <timeunit>] [-h <hostname>] [-s <section>]
                 [-t <teststep>] [-f|-z|-j] <script> [parameters...]

{|
! Parameter
! Description
|-
| -S
| Send at once to ULS. Normally, the resulting file is placed in the directory ULS_TMP_PATH
 which is configured in the /etc/uls/uls.conf. It is then sent to ULS after a delay.
|-
| -u <timeunit>
| define the timeunit for the script runtime, use ''m'' for minutes, ''h'' for hours, 
 any other value or not set will calculate the runtime in seconds
|- 
| -f -z -j
| the output of stdout and stderr is send as file **-f**, as gzipped file **-z**
 or as bzip2-ed file **-j**, default is as text value (which is directly readable 
 in the ULS user interface (interactive webbrowser-based analysis))
|-
| <script>
| full path to an executable script
|-
| <parameter>
| appended parameters will be forwarded to the <script>
|}

-----

=== send_file2uls ===

Transfers the data stream from stdin to ULS as value (not as file/LOB).

  send_file2uls [-c <charset>] [-d <storepath>] [-S] [-h <hostname>] [-n|-N HH:MM|-u] [-s acc] [-r <rettime>]
                <section> <teststep> <detail> <date> <time> <unit> <file>


{|
! option
! description
|-
| -n
| (no duplicate) the <value> is only stored in ULS if it differs from the previous value
|-
| -u
| (unique) the <value> is only stored in ULS if exactly this <value> does not exist with the same timestamp
|-
| -N HH:MM
| (no duplicate or time elapsed) the <value> is only stored in ULS if it differs from the previous value or 
 more time than HH:MM has elapsed
|}

-----

=== send_file_value ===

Transfers a file as LOB to ULS, it will be shown as weblink for download.

  send_file_value [-d <storepath>] [-h <hostname>] [-s <access>] 
                  <section> <teststep> <detail> <date> <time> <file>

-----

=== send_image_value ===

Transfers a file as image to ULS, the image will be shown in the ULS user interface.

  send_image_value [-d <storepath>] [-S] [-h <hostname>] [-s <access>] <section> <teststep> <detail> <date> <time> <file>

-----

=== send_script2uls ===

:TODO:

-----

=== send_section_remark ===

Stores a text expression as explanation for a section (of a source/hostname).
That allows agents to contain their own documentation and transmit it 
to ULS in regular time intervals. 

  send_section_remark [-d <storepath>] [-S] [-h <hostname>] <section> <text>

{|
! parameter
! description
|-
| <text>
| a text expression up to 50000 characters (less for UTF8)
|}

-----

=== send_server_doku ===

Saves a document related to a source (server), not as a timestamped value but as "main menu -> administration -> documentation -> server documentation -> <domain> -> <source>".

Any type of document can be transferred, already existing documents with the same name will be replaced.

  send_server_doku [-d <storepath>] [-S] [-h <hostname>] <name> <description> <date> <time> <document>

{|
! Parameter
! Description
|-
| <name>
| short name of the document
|-
| <description>
| long description of the document content
|-
| <document>
| path of the document to be transferred
|}

<file>
main menu
└─► administration
    └─► documentation
        └─► server documentation
            └─► <domain>
                └─► <source>

     | name          |      date           |           description              |   download     |
     | ------------------------------------|------------------------------------|----------------|
     | _ora_dbinfo   | 2013-11-16 22:13:00 | Oracle Database Information Report | ora_dbinfo.txt |
     ...
</file>

-----

=== send_stdin2uls ===

This is just a link to [[linux#send_file2uls]].

=== send_stdin_file ===

Transfers the data stream from stdin to ULS as file/LOB.

  send_file_value [-d <storepath>] [-S] [-h <hostname>] [-s <access>] [-a] [-b <bufflen>]
                  <section> <teststep> <detail> <date> <time> <filename>

{|
! option
! description
|-
| -a
| %%<filename>%% will be appended to already existing filename
|-
| -b <bufflen>
| a buffer file of size <bufflen> will be used
|-
| %%<filename>%%
| the data stream is saved by using this filename
|}

-----

=== send_test_counter ===

Maintain a counter in ULS for that source-section-teststep-detail. It can be set, reset, decremented or incremented.

  send_test_counter [-h <hostname>] [-s <access>] <section> <teststep> <detail> <date> <time> [<mode> [<value>]]

{|
! <mode>
! action
|-
| inc
| increment counter by 1 (is default action, if <mode> is not given)
|-
| reset
| reset counter to zero
|-
| set <value>
| set counter to <value>
|-
| add <value>
| add <value> to counter
|-
| sub <value>
| subtract <value> from counter
|- 
| dec
| decrement counter by 1
|}

-----

=== send_test_doku ===

Stores a text expression as explanation for a source-section-teststep.
That allows agents to contain their own documentation and transmit it 
to ULS in regular time intervals. 

  send_test_doku [-d <storepath>] [-S] [-h <hostname>] <section> <teststep> <text>

{|
! parameter
! description
|-
| <text>
| a text expression up to 50000 characters (less for UTF8)
|}

-----

=== send_test_tab ===

Send a file which is created after the [[uls:agents:value_file_format]] to ULS. 

No images, no files (line IDs: I, F and FA) can be processed.

  send_test_tab [-d <storepath>] [-S] [<filename>]

If you do not specify a %%<filename>%%, the script will read its input from STDIN.

{|
! parameter
! description
|-
| -d <storepath>
| Use explicitly the directory specified as <storepath>. 
|-
| -S
| Send at once to ULS. Normally, the resulting file is placed in the directory ULS_TMP_PATH which is configured in the /etc/uls/uls.conf
|-
| %%<filename>%%
| You may use any filename, you may also use the same filename in subsequent executions of ''send_test_tab''. 
|}

Usage example: call ''send_test_tab -d <storepath>'' for several [[uls:agents:value_file_format]] files, use the same temporary directory %%<storepath>%% for all files. 
Then call ''flush_test_values <storepath>'' to transfer all [[uls:agents:value_file_format]] files to ULS. Remember: no images, no files.

Usage example: If your executable generates UVFF output to stdout you may redirect that output to 
an intermediate file and call ''send_test_tab <intermediate file>'' or you 
pipe the output directly like: ''your_exec | send_test_tab''

-----


=== send_test_value===

Sends a value to ULS. Not finally, but it generates an ULS value file, 
that will be sent to ULS when [[linux#flush_test_values]] is executed.

  send_test_value [-d <storepath>] [-S] [-h <hostname>] [-n|-N HH:MM|-u] [-s <access>] [-r <rettime>]
                  <section> <teststep> <detail> <date> <time> <value> <unit>

{|
! parameter
! description
|-
| -n
| <value> will only be saved in ULS if it differs compared to the previous value
|-
| -u
| <value> will only be saved in ULS if this <value> does not already exists for the same timestamp
|-
| -N HH:MM
| <value> will only be saved in ULS if it differs compared to the previous value or 
 more than the time interval specified as HH:MM has eleapsed compared to the timestamp of 
 the last saved value.
|-
| -r <rettime>
| :TODO:
|}

-----

==== Logfile Adapter ====

The [[uls:agents:log_file_adapter|Logfileadapter]] can read a file contents and continues 
to read the contents when it grows, when lines are appended.
It offers sophisticated filtering rules and aggregation functions.

-----

==== Monitoring Settings ====

Many metrics will be gathered and sent to the ULS-server. 
All these metrics can be monitored for violations of thresholds. 
Here are some threshold definitions that we normally use and 
which should be usable from the beginning on.

Please check the User's Manual for more information about 
monitoring, thresholds, notifications and notification destinations.

Recommendations:

<file ini monitoring_settings.txt>

[limits]

server=
section=System
teststep=Disk*,!Disk Space /mnt,!Disk Space /media*
detail=%belegt
unit=
function=last
over=1
relation=>
infolevel=80
warnlevel=90
errorlevel=
notifymode=24
message=On $SERVER disk $TESTSTEP is used for $VALUE $UNIT.
destination1=<your_destination>
dest1levels=IWF
destination2=
dest2levels=IWF
remark=
access=all

server=
section=System
teststep=Logfileadapter
detail=Log
unit=
function=last
over=1
relation=~|
infolevel=
warnlevel=EXT.*-fs error
errorlevel=
notifymode=S10
message=Filesystem error:\n$VALUE
destination1=<your_destination>
dest1levels=IWF
destination2=
dest2levels=IWF
remark=
access=all

server=lx*,!*nes
section=System
teststep=Logfileadapter
detail=Log
unit=_
function=last
over=1
relation=~|
infolevel=
warnlevel=multipath.*fail
errorlevel=
notifymode=S10
message=Multipath error:\n$VALUE
destination1=<your_destination>
dest1levels=IWF
destination2=
dest2levels=IWF
remark=
access=all

server=
section=System
teststep=Crontab:*
detail=
unit=
function=differ
over=2
relation==
infolevel=
warnlevel=1
errorlevel=
notifymode=24
message=A crontab has changed: \n$DIFF_OF(Crontab)
destination1=<your_destination>
dest1levels=IWF
destination2=
dest2levels=IWF
remark=
access=all

server=
section=Security
teststep=User
detail=Idle
unit=h
function=last
over=1
relation=>
infolevel=
warnlevel=10
errorlevel=
notifymode=24
message=User $VALUE_OF(Name) is logged in at $VALUE_OF(TTY), but no action for $VALUE hours.
destination1=<your_destination>
dest1levels=IWF
destination2=
dest2levels=IWF
remark=
access=all

server=
section=Security
teststep=Aide
detail=DB
unit=
function=differ
over=2
relation==
infolevel=
warnlevel=1
errorlevel=
notifymode=24
message=$DIFF_OF(DB)
destination1=<your_destination>
dest1levels=IWF
destination2=
dest2levels=IWF
remark=
access=all

# -----------------------------------------------
[isalive]

server=
section=System
teststep=System Load
detail=%idle
unit=%
period=30
periodunit=min
notifymode=24
message=No values from $SERVER for $TIMEDIFF mins.
destination1=<your_destination>
destination2=
remark=
access=all 


# -----------------------------------------------
[compression]

server=
section=System
teststep=System Load
detail=%idle
unit=
hourly=avg
daily=avg,count
weekly=
monthly=
access=all

server=
section=System
teststep=Disk Space*,!Disk Space /mnt,!Disk Space /media*
detail=Groesse,belegt
unit=
hourly=avg
daily=max
weekly=
monthly=
access=all

</file>