Universal Logging System

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
===== Overview & Features =====

==== Overview ====

The Universal Logging System (ULS) is an open-source solution 
for logging, monitoring, reporting and interactive analysis 
of values of different data types.

Agents running on clients acquire values, transfer them to the 
ULS-server which stores them as timestamp-related, structured 
data into a database. 

Users can navigate and inspect this data as tables or graphs 
by using a web browser. Displayed results can be strung 
together to form reports. Thresholds can be defined which induce 
the ULS to send notifications when violated. Data can be aggregated 
by numerous functions to hourly, daily, weekly or monthly figures, 
which can e.g. be kept for long term analysis.

{{:uls:uls_architecture.jpg|ULS Architecture Overview}}

-----

==== Values ====

You can store:

  * numerical values with units, e.g. disk usage ''75 %''
  * utf8 based text expressions up to 32000 characters, e.g. ''no error found''
  * tuple-timestamps, e.g. ''Start 2013-08-22 12:39:01'' and ''Stop 2013-08-22 12:44:17''
  * files, e.g. images, PDF documents or text files

Each value **must** belong to a timestamp (and some more parameters). 
The default retention time of 14 days for values can be individually defined for each detail.

-----

==== Logical Data Structure ====

The logical data structure is hierarchical:

<code>
<domain>
  │
  └──►<source>
      │
      └──►<section>
            │
            ├──►<teststep>  | time stamp           |  detail  |  detail  |
            ├──►<teststep>  |                      |  unit    |  unit    |
            ├──►...         | ---------------------|----------|----------|
            ├──►...         | yyyy-mm-dd hh:mi:ss  |  value   |  value   |
            ├──►...         | yyyy-mm-dd hh:mi:ss  |  value   |  value   |
            ├──►...         | yyyy-mm-dd hh:mi:ss  |  value   |  value   |
</code>

^data structure elements ^description   ^
|domain |a pool of one or more logically related sources (or servers) |
|source |the source of the values, e.g. a 'server', a 'car park' or a 'marathon' |
|section |comprises related teststeps, e.g. 'system performance', a 'car' or a 'marathon participant' |
|teststep |comprises related details, e.g. 'system load', 'refueling' or 'key body indicators' |
|detail   |the attribute to which the value belongs, e.g. '%idle', 'gallons' or 'heartbeat' |
|value    |that is the (measured) value, e.g. '73.8', '13.234', '82' |
|unit     |the unit of the value, e.g. '%', 'gal', '1/s' |
|time stamp |the date and time to which the value relates, always in (nearly) ISO format, max down to the second |

See also these [[uls:hierarchical_data_structure_examples|examples]].

Names and relations of sections to teststeps and details 
can freely be chosen by the agent creator. Only domains, sources (servers) and 
their relations must be defined in ULS (might be automated thru an interface). 
The interpretation of her/his values is obviously up to her-/himself.

The hierarchical structure allows a drill-down style analysis of the values.

-----

==== Features ====

=== Data Communication ===

Datasources (agents on monitored systems) **transfer** their acquired values 
**to** the central ULS-server. A dedicated port is used for the HTTP
or HTTPS connection to the ULS-server. 
This allows dedicated connections between separated network segments.



=== Interactive Analysis ===

The user can use any browser to:

  * navigate through the online and aggregated data
  * choose from a list of predefined time intervals to restrict the data quantity
  * enter an arbitrary time interval to restrict the data quantity
  * select **any** accessible data and make side-by-side comparisons in tables or graphs
  * pick from a list of aggregation functions to get calculated key figures
  * save composed results as a favorite
  * append composed results as a report detail to an existing or new report


=== Reports ===

Reports consist of sequences of report details. 

  * a report has an optional header and text description
  * there is no limit of the number of report details per report
  * report details show data as table, graph or calculated figures
  * a report detail can optionally contain a separator, a heading and a description
  * use logical time intervals like 'today', 'last week' or 'since yesterday' 

  * reports can be shared to all users who have access to the domain(s) of the referenced values
  * a PDF document can be generated from a report which then can be sent regularly by e-mail


=== Thresholds ===

The user may set up threshold definitions:

  * on numeric values (e.g. value > 90%, avg(value) <= 3s)
  * on text values (e.g. value is not "OK", value matches "EXT.*-fs error")
  * on values compared to the previous value (e.g. difference between value and previous value > 2GB)
  * on values of different details for the same teststep by using comparisons, mathematical functions and/or MySQL functions (e.g. ABS($VALUE_OF(average_delay) - $VALUE_OF(current_delay)) >= 1.5)
  * define e-mail addresses as destination for notifications on threshold violations
  * domain-wide threshold pattern with automatic applying
  * define ULS-internal ticket tracking destinations for notifications on threshold violations

All stored data is periodically checked:

  * for violations of defined thresholds (limits, combi-limits)
  * if values have **not** arrived within a certain period of time (isAlive)


=== Ticket Tracking ===

Notifications of violated thresholds from ULS can be directed to the ULS Ticket Tracking (UTT). 
A ticket is issued and assigned to a ticket tracking destination, 
which is assigned to a group of ULS users. 
Group members can view, take over and work on the new ticket and close it after having solved it.

Users may:

  * take over tickets to work on it
  * add textual solution descriptions
  * append files
  * pass tickets to other groups
  * close the ticket

You **must** use the "group" feature of ULS to take full advantage of the UTT.
Groups combine domains, users, notification destinations and ticket tracking destinations.

All changes to the ticket are saved in the ticket's history and cannot be reverted.


=== What Else ===

Aggregation:

  * use grouping functions like min, max, sum, avg and count to calculate key figures
  * aggregate to hourly, daily, weekly or monthly figures
  * aggregated and original data can be intermixed in interactive analysis and reports

Retention of values:

  * the data retention time for each value as needed (typical 14 days)

Derived details:

  * show derived or calculated details in interactive analysis (e.g. cpu usage = 100 - $VALUE_OF(%idle) )


=== Administration ===

The adminstration uses an own web application which is only available 
for ULS-administrators and the ULS-master-admin.

As ULS-master-admin you need to:

  * create ULS-administrators

As ULS-administrator you need to:

  * specify domains
  * maintain a list of source names (server)
  * add notification destinations
  * create groups
  * relate domains, users and notification destinations to groups
  * manage the units of the incoming values (many are already pre-defined)

Everything else will be dynamically created as the values are stored.

=== Agents ===

Agents deliver values to the ULS-server. There are several possibilities to 
transfer values:
  * any programming environment capable of http(s) requests can be used for agent development
  * wget will work, too
  * use the simple ULS value file format. Your programming environment can output to a file? Then use this file format to generate ULS value files and transfer them to the ULS-server by transfer clients provided for most Un*xes (flush_test_values) and W*ndows (send2uls.exe) operating systems.
  * perl modules to generate ULS value files
  * basic shell tools to send any data for Linux, HP-UX, Solaris
  * Java classes to send data to the ULS-server
  * SuSE Linux system agent
  * HP-UX 11 system agent
  * Solaris 8 system agent
  * Powershell scripts for W*ndows monitoring and event log scanning
  * Oracle 11/12 (no RAC specifics) database monitoring
  * Informix 11/12 database monitoring

See the [[uls:agents]].

=== Requirements ===

To run the ULS-server you need:
  * a linux-like operating system
  * a MySQL database
  * the ULS-server package provided in [[uls:install]]
Universal Logging System

User Tools

Site Tools

Page Tools
