===== Test Suite =====

This test suite is used to check the proper functionality of a complete working ULS environment.

==== Overview ====

The test suite contains the test plan, the test case specifications, their execution instructions and scripts. 
All used data is explained, as well as the test report definitions and general information about the current version of the ULS-server.

The test suite was developed and is constantly evolving with each version 
to check the functionality of the ULS-server. The tests are grouped into 
functional sections.

  * Generally, well-defined values are sent to the ULS-server and checked for correctness by using the interactive analysis via web browser.
  * Boundary tests are made to assure the covered value ranges.
  * Threshold violations are checked for the most common scenarios, including notifications containing message bodies with replaced static and dynamically calculated embedded placeholders.
  * Aggregation of values.
  * Reports are used to check the expected results and to check the report functionality itself.
  * Administrative actions are also part of the test.
  * Tests of the effectiveness of security based separations on users.
  * ULS ticket tracking: threshold violations generate notifications and tickets. ULS users can work on those tickets and document their actions.

-----
==== Test Plan ====

The ULS stage environment is used to perform the test.

The ULS-server runs in a production-like mode, 
doing all the processing, aggregation and checks in 
the same manner as the production, unless otherwise specified for particular tests.

The approach is: 
  * use the administration section to define new groups, users and notification destinations
  * run scripts on operating system level to generate values of all supported value types and send them to ULS
  * use two different users to perform the interactive analysis on those values
  * create several reports to show the correct results
  * define various possible notification scenarios and verify the correctly generated notifications
  * define aggregation of values and verify the results
  * define various possible notification scenarios on aggregated values and verify the correctly generated notifications

The schedule is:
  * some notification scenarios must be defined BEFORE the first test script is run
  * all test scripts are executed each 5 minutes by cron, until the end of the tests
  * some test scripts only produce a limited number of values, they end when all values have been transferred
  * the results and test examinations must be done within 14 days
  * the results and test examinations on aggregated values must be done within 120 days


The test environment is:

:TODO: an image to describe the test environment (test script, test server, directories, interactive analysis...)

-----

==== Test Suites ====

The test suites for the ULS versions:
  * [[uls:testsuite:version_180]]


-----

==== General structure of test sequences ====

:NOTE: This is of informational use for testers and has no general relevance for ULS.

General structure of test sequences:

  * manually / explicitly specified
  * domain-based specifications
  * group-based specifications

Subsections:
  * === (A) Specifications ===
    * define specifications
    * specifications successfully defined? applied correctly? properly propagated?
    * correct operation?

  * === (B) Change ===
    * change specifications
    * applied correctly? properly propagated?
    * correct operation?

  * === (C) Deactivation === or === (C) Removal ===
    * deactivate specifications
    * applied correctly? properly propagated?
    * correct operation?

Typical table for test results, use 4 char (abbreviation of test section) and 3 digits to give a unique expression.

{|
! Test
! Description
! Result
! Notes
|-
| XXXX dddd
| <section header>: <what> <result>
| style="color:green" | passed or :TODO: or FIXME
| <remark>
|}


RTIM Retention TIMe
AGGR Aggregation
NTFY Notification
SECU Security
UTTR ULS Ticket Tracking