Universal Logging System

This is the test suite for version 1.8.0 of ULS.

A number of administrative tasks and entries must be done to set up the necessary settings for all tests. This is also a test of the ULS administration itself.

A number of well-defined basic values containing all supported datatypes are sent to the ULS and the correct transmission and presentation is checked thru a manual basic analysis.

boundary tests are made to verify the correct processing of e.g. very small and very large values.

The general tests are based on values derived from the current number of seconds

• of the current hour
• of the current day and
• of the current week (since 00:00 of last monday)

These values are specifically converted and calculated to cover the most common scenarios. In combination with threshold definitions they are used to test the firing of notifications for threshold violations The notifications will contain replaced static and dynamically calculated embedded placeholders in their message bodies.

These values will also be used for the check of the aggregation of values.

Reports are used to check the different expected results and to check the report functionality itself.

Throughout the documentation, the term “stage environment” will be used to identify the staging or test or whatever environment is used for the tests.

Set up the stage environment as a new ULS-server, you do not need any data from the production environment.

Remove all defined limits, aggregations and value retentions. Login to the ULS-server as user “uls” and issue the command:

u2w_interpreter /home/uls/testsuite/del_all_limits.s2w

Check that the regular ULS jobs are correctly executed by cron:

cd /etc/cron.d/
vi ulsserver

Should look like:

ulsserver

# delete old values
3 0 * * * uls /usr/local/bin/jobmonitor2uls -s ULS -t Delete-Old-Loggings /srv/uls/clean/mysql_delete_old_loggings >/dev/null 2>&1
#
# test limits
4-59/5 * * * * uls /usr/local/bin/jobmonitor2uls -s ULS -t Test-Limits /srv/uls/ulsmeld/do_ulsmeld test_isalive test_limits test_kombilimits >/dev/null 2>&1
#
# send mail reports
# 3 * * * * uls /usr/local/bin/jobmonitor2uls -s ULS -t Mailreports /srv/uls/jobs/send_mailreports >/dev/null 2>&1
#
# aggregation
1 5 * * * uls /usr/local/bin/jobmonitor2uls -s ULS -t Compression /srv/uls/jobs/komprimierung >/dev/null 2>&1

The ULS-server always marks the last checked value to continue with the next execution of the test scripts. So you need to reset these marks before starting the test scripts (or wait until it is executed by cron):

/usr/local/uls/ulsmeld/do_ulsmeld -f test_limits
/usr/local/uls/ulsmeld/do_ulsmeld -f test_kombilimits
/usr/local/uls/ulsmeld/do_ulsmeld -f test_isalive

That will perhaps run for a long time depending on the number of unprocessed values

Clear the log file for the notification messages:

echo > /tmp/ulsserver.meldungen.out
chmod 664 /tmp/ulsserver.meldungen.out

This section describes all test cases.

Test specific domains must be defined.

Test specific sources (servers) must be defined.

Add the sources, all sources must be defined with the ip-address of the ULS stage server, all sources are mocked. Use the ip-address of your server!

Define test specific groups. Be sure to show all domains, even those freshly created by clicking .

Use show groups to check the result.

Define test specific users. Use the username as password. Do not assign any domains or domain privileges.

Use notification » edit notification destinations » Mail to add an e-mail notification destination. Use a correct e-mail address instead of “your@emailaddress”.

Assign the new defined notification destination to group “Hotline”. Try the test link to send a test e-mail to your@emailaddress.

Use groups » group ticket targets to define ULS ticket targets, which are destination groups in the ULS internal ticket tracking (UTT) system.

Edit the configuration file and set the time between two values. There is a mximum of 10 values to be sent. So chose a matching wait time.

test_suite.conf

# ===================================================================
[BASIC_VALUES]
 
IDENTIFIER = _basic_values
 
# Wait time in seconds before generating/sending the next value
WAIT = 5

Execute a script once to check that values of all data types and from all sources are transferred correctly to ULS.

cd testsuite/test_suite_1_8_0/
./basic_values

Verify, that the generated ULS value files are transferred to the ULS-server and that their content is stored correctly.

No threshold violations are generated in this test

Use a webbrowser, login as ULS user “TheSpyder”, chose the correct time interval (today or explicit iso time interval) and check all values for domain “ULS”:

ULS
└─► testsrv11
   └─► VAL 0010: Basic Values
      └─► 010 Integer Values          (6 values)
      └─► 020 Float Values            (11 values, [E]/m )
      └─► 040 Files                   (two real images, one gif (wind and solar energy), one png (sand dunes))
└─► testsrv12
   └─► VAL 0010: Basic Values
      └─► 010 Integer Values          (6 values)
      └─► 030 Text Expressions        (11 expressions, from "00 AA" to "10 KKKKKKKKKKKKKKKKKKKKKK", proportional)
      └─► 040 Files                   (one html, one odt, one pdf, one txt (as web links to open them))

and for domain “Woodlark”:

Woodlark
└─► testsrv21
   └─► VAL 0010: Basic Values
      └─► 020 Float Values          (6 values [%])
      └─► 030 Text Expressions      (11 expressions, from '00 aaaaa' to '10 kkkkkkkkkkkkkkk', mono spaced)
└─► testsrv22
   └─► VAL 0010: Basic Values
      └─► 020 Float Values          (7 values [kg])
      └─► 040 Files                 (4 real images (LAN LEDs, jpeg))

Generate a report, administration » edit reports and enter “VAL 0010: Basic Values” as report name and press .

Use edit and to create a report detail containing:

All other specifications remain unchanged.

Click and to show the report. Click on and save the report to a path of your choice. It should look like this.

Create a crontab to make the “continuous_values” script execute every minute. The script

cd /etc/cron.d/
vi uls_test_suite

uls_test_suite

* * * * * uls /home/uls/testsuite/test_suite_1_8_0/continuous_values > /home/uls/testsuite/test_suite_1_8_0/continuous_values.log 2>&1

Verify, that the generated ULS value files are transferred to the ULS-server and that their content is stored correctly.

Use a webbrowser, login as ULS user “AmieAction” (full rights on “ULS”, read-only on “Woodlark”), chose the correct time interval (yesterday if the script runs already for more than a day or explicit iso time interval) and check all values for domain “ULS”:

ULS
└─► testsrv11
   └─► VAL 0110: Continuous Hourly Values 
      └─► 110 Integer Values     (1 value)
      └─► 120 Float Values       (1 value)
      └─► 130 Text Expressions   (2 values)
└─► testsrv12
   └─► VAL 0110: Continuous Hourly Values
      └─► 110 Integer Values     (2 values)
      └─► 120 Float Values       (1 value)
      └─► 140 Files              (1 image, 1 file "spiro.png")

and for domain “Woodlark”

Woodlark
└─► testsrv21
   └─► VAL 0110: Continuous Hourly Values
      └─► 120 Float Values       (1 value)
      └─► 130 Text Expressions   (2 expressions, 3 words, each 20 characters)
      └─► 140 Files              (1 image with different contents, 1 file "trw.jpg"')
└─► testsrv22
   └─► VAL 0110: Continuous Hourly Values
      └─► 120 Float Values       (1 value)

Generate a report, administration » edit reports and enter “VAL 0110: Continuous Hourly Values” as report name and press .

Use edit and to create a report detail.

All other specifications remain unchanged.

Click and to show the report.

Go to administration » edit mail reports and send the “VAL 0110: Continuous Hourly Values” report to your e-mail address. Use

• (x) landscape
• [x] send now

The report should look like this

Aggregations are defined on continuously generated values. Connect as “AmieAction” to ULS and go to administration » compression » set auto-compression and chose domain “ULS” (AmieAction has enough rights only for domain “ULS”).

• set section to “VAL 0110: Continuous Hourly Values”
• activate all checkboxes of “hourly” and
• continue with and
• to export the definitions to a new browser tab.
• copy the definition and paste it into the textbox of previous browser tab.
• copy the “hourly” entry to the other lines

It should look like:

create auto-compression for ULS

server=
section=VAL 0110: Continuous Hourly Values
teststep=
detail=
unit=
hourly=last,avg,sum,dev,min,max,max-min,last-first,count,grad,first,accel,differ,firstsize,last/avg,lastsize
daily=last,avg,sum,dev,min,max,max-min,last-first,count,grad,first,accel,differ,firstsize,last/avg,lastsize
weekly=last,avg,sum,dev,min,max,max-min,last-first,count,grad,first,accel,differ,firstsize,last/avg,lastsize
monthly=last,avg,sum,dev,min,max,max-min,last-first,count,grad,first,accel,differ,firstsize,last/avg,lastsize
special=
access=all 

Click to save that. A new entry was added, delete the entry created in the first step.

Click on changes to verify that the initial change has been logged.

Any compression/aggregation action will be executed in the following night.

Verify, that the above rules are propagated correctly. Check:

• administration » compression » set compression » ULS » testsrv11 » 110 Integer Values

does not work currently (2014-01-22)

Threshold violations

Define various possible notification scenarios and verify the correctly generated notifications

ULS ticket tracking: threshold violations generate notifications and tickets. ULS users can work on those tickets and document their actions.

Threshold violations

Define various possible notification scenarios on aggregated values and verify the correctly generated notifications

ULS ticket tracking: threshold violations generate notifications and tickets. ULS users can work on those tickets and document their actions.

Continue here

the correct storage of all value types:

• including boundary tests to assure the covered value ranges
• checked by interactive analysis
• checked by reports
• deletion of values after the retention time has elapsed

values, thresholds, aggregation, …

• threshold violations

• numeric, string and file-like values
• different functions combined with number of values and/or time range
• including message bodies of notifications with replaced static and dynamically calculated embedded placeholders

• aggregation of values

• check the different possible aggregation functions on different data types

• reports

• check the expected results
• check the creation and modification of reports
• user-defined headers, separators, descriptions and logo
• PDF generation

detailing how to run each test, including any set-up preconditions and the steps that need to be followed

Test Script is a program written to test the functionality of the application. It is a set of system readable instructions to automate the testing with the advantage of doing repeatable and regression testing easily.

It is the Hardware and Software Environment where is the testing is going to be done. It also explains whether the software under test interacts with Stubs and Drivers.

Create a mock business. Treat it as real and process its data.

Test Procedure is a document with the detailed instruction for step by step execution of one or more test cases. Test procedure is used in Test Scenario and Test Scripts.

Test Log contains the details of test case execution and the output information.

recording which tests cases were run, who ran them, in what order, and whether each test passed or failed

detailing, for any test that failed, the actual versus expected result, and other information intended to throw light on why a test has failed. This document is deliberately named as an incident report, and not a fault report. The reason is that a discrepancy between expected and actual results can occur for a number of reasons other than a fault in the system. These include the expected results being wrong, the test being run wrongly, or inconsistency in the requirements meaning that more than one interpretation could be made. The report consists of all details of the incident such as actual and expected results, when it failed, and any supporting evidence that will help in its resolution. The report will also include, if possible, an assessment of the impact of an incident upon testing.

A management report providing any important information uncovered by the tests accomplished, and including assessments of the quality of the testing effort, the quality of the software system under test, and statistics derived from Incident Reports. The report also records what testing was done and how long it took, in order to improve any future test planning. This final document is used to indicate whether the software system under test is fit for purpose according to whether or not it has met acceptance criteria defined by project stakeholders.