Differences

This shows you the differences between two versions of the page.

--- uls:testsuite:version_180 [2014-01-23 09:03]
uls [VAL 0200: Aggregations]
+++ uls:testsuite:version_180 [2014-01-29 08:14] (current)
uls [Test Scenarios]
@@ Line 2: / Line 2: @@
 This is the test suite for version 1.8.0 of ULS.
+==== Test Environment ====
+The stage environment is used for all tests.
@@ Line 14: / Line 19: @@
 **boundary tests** are made to verify the correct processing of e.g. very small and very large values.
-The **general tests** are based on values derived from the current number of seconds
+**continuous tests** are based on values derived from the current number of seconds
-  * of the current hour
+  * within the current hour
-  * of the current day and
+  * within the current day
-  * of the current week (since 00:00 of last monday)
+  * within the current week (since 00:00 of last monday) and
+  * within the current month
-These values are specifically converted and calculated to cover the most common scenarios.
+These values are specifically prepared and calculated to cover the most common scenarios.
 In combination with **threshold** definitions they are used to test the firing of **notifications** for threshold violations
 The notifications will contain replaced static and dynamically calculated embedded **placeholders** in their message bodies.
-These values will also be used for the check of the **aggregation** of values.
+These values will also be used for the check of the **aggregation** or **compression** of values.
 Reports are used to check the different expected results and to check the **report** functionality itself.
-----
+  * do the necessary [[uls:testsuite:version_180:preparations]]
-==== Preparations ====
-Throughout the documentation, the term "stage environment" will be
+The test cases are separated by function in different sections:
-used to identify the staging or test or whatever environment
-is used for the tests.
-Set up the stage environment as a new ULS-server, you do not need any data from the production environment.
+  * administrative settings and tests in [[uls:testsuite:version_180:admin_settings]]
+  * basic values, one execution in [[uls:testsuite:version_180:basic_values]]
+  * continuous values generation in [[uls:testsuite:version_180:continuous_values]]
+  * special values generation in [[uls:testsuite:version_180:special_values]]
+  * aggregations in [[uls:testsuite:version_180:aggregations]]
+  * notifications in [[uls:testsuite:version_180:notifications]]
+  * deletion of values after elapsed [[uls:testsuite:version_180:retention]]
-Remove all defined limits, aggregations and value retentions.
-Login to the ULS-server as user "uls" and issue the command:
-<code bash>
-u2w_interpreter /home/uls/testsuite/del_all_limits.s2w
-</code>
-Check that the regular ULS jobs are correctly executed by cron:
-<code bash>
-cd /etc/cron.d/
-vi ulsserver
-</code>
-Should look like:
-<file crontab ulsserver>
-# delete old values
-0 * * * uls /usr/local/bin/jobmonitor2uls -s ULS -t Delete-Old-Loggings /srv/uls/clean/mysql_delete_old_loggings >/dev/null 2>&1
-#
-# test limits
--59/5 * * * * uls /usr/local/bin/jobmonitor2uls -s ULS -t Test-Limits /srv/uls/ulsmeld/do_ulsmeld test_isalive test_limits test_kombilimits >/dev/null 2>&1
-#
-# send mail reports
-# 3 * * * * uls /usr/local/bin/jobmonitor2uls -s ULS -t Mailreports /srv/uls/jobs/send_mailreports >/dev/null 2>&1
-#
-# aggregation
-5 * * * uls /usr/local/bin/jobmonitor2uls -s ULS -t Compression /srv/uls/jobs/komprimierung >/dev/null 2>&1
-</file>
-:!: The ULS-server always marks the last checked value to continue
-with the next execution of the test scripts.
-So you need to reset these marks before starting the test scripts
-(or wait until it is executed by cron):
-<code bash>
-/usr/local/uls/ulsmeld/do_ulsmeld -f test_limits
-/usr/local/uls/ulsmeld/do_ulsmeld -f test_kombilimits
-/usr/local/uls/ulsmeld/do_ulsmeld -f test_isalive
-</code>
-:!: That will perhaps run for a long time depending on the number of unprocessed values :!:
-Clear the log file for the notification messages:
-<code bash>
-echo > /tmp/ulsserver.meldungen.out
-chmod 664 /tmp/ulsserver.meldungen.out
-</code>
 ----
-==== Test Cases ====
-This section describes all test cases.
-----
-=== ADM 0010: Defining domains and sources ===
-Test specific domains must be defined.
-{|
-! domain
-! description
-|-
-| Woodlark
-| create this as new domain
-|-
-| ULS
-| use this existing domain
-|}
-Test specific sources (servers) must be defined.
-Add the sources, all sources must be defined with the ip-address of the ULS stage server, all sources are mocked.
-:!: Use the ip-address of **your** server!
-{|
-! serverid
-! servername
-! domain
-! ip-address
-! secondary ip-address
-! accept ip-addresses
-|-
-| ulstest1
-| testsrv11
-| ULS
-| 10.1.2.3
-|
-|
-|-
-| ulstest2
-| testsrv12
-| ULS
-|
-| 10.1.2.3
-|
-|-
-| ulstest3
-| testsrv21
-| Woodlark
-|
-|
-| 10.1.2.3
-|-
-| ulstest4
-| testsrv22
-| Woodlark
-|
-|
-| 10.1.2.3/32 ((only exactly this ip-address))
-|}
-Check that all domains and sources have been created correctly.
-----
-=== ADM 0020: Groups and Users ===
-Define test specific groups.
-Be sure to show all domains, even those freshly created by clicking {{:uls:testsuite:version_180:all_domains.png?nolink|}}.
-{|
-! group
-! rights on domain "ULS"
-! rights on domain "Woodlark"
-|-
-| Hotline
-| none
-| read-only
-|-
-| System
-| read-only
-| full rights
-|-
-| Database
-| full rights
-| read-only
-|}
-Use //show groups// to check the result.
-Define test specific users.
-Use the username as password.
-Do not assign any domains or domain privileges.
-{|
-! user
-! group "Hotline"
-! group "System"
-! group "Database"
-! admin
-! password never expires
-|-
-| AmieAction
-|
-|
-| style="text-align:center" | x
-|
-| style="text-align:center" | x
-|-
-| HeyJoe
-| style="text-align:center" | x
-|
-|
-|
-| style="text-align:center" | x
-|-
-| SystematicGuy
-|
-| style="text-align:center" | x
-|
-|
-| style="text-align:center" | x
-|-
-| TheSpyder
-| style="text-align:center" | x
-| style="text-align:center" | x
-| style="text-align:center" | x
-| style="text-align:center" | x
-| style="text-align:center" | x
-|}
-Use //show user// to show the list of all users and verify all users for correct group membership.
-Verify especially that "TheSpyder" has full rights on both domains.
-----
-=== ADM 0030: Notification Destinations ===
-Use
-//notification >> edit notification destinations >> Mail//
-to add an e-mail notification destination.
-Use a correct e-mail address instead of "your@emailaddress".
-{|
-! type
-! name
-! address
-|-
-| Mail
-| ULS Test
-| your@emailaddress
-|}
-Assign the new defined notification destination to group "Hotline".
-Try the //test// link to send a test e-mail to your@emailaddress.
-Use //groups >> group ticket targets// to define ULS ticket targets,
-which are destination groups in the ULS internal ticket tracking (UTT) system.
-{|
-! group
-! name
-! escalation ((is currently not checked))
-! remark
-|-
-| Database
-| UTT Database
-|
-| The ticket target for "Database" monitoring.
-|-
-| System
-| UTT System
-|
-| The ticket target for "System" monitoring.
-|}
-----
-=== VAL 0010: Basic Values ===
-Some basic values of all supported data types are transferred to ULS.
-A check is made whether all have arrived correctly.
-Edit the configuration file and set the time between two values.
-There is a mximum of 10 values to be sent. So chose a matching wait time.
-<file ini test_suite.conf>
-# ===================================================================
-[BASIC_VALUES]
-IDENTIFIER = _basic_values
-# Wait time in seconds before generating/sending the next value
-WAIT = 5
-</file>
-Execute a script once to check that values of all data types
-and from all sources are transferred correctly to ULS.
-<code bash>
-cd testsuite/test_suite_1_8_0/
-./basic_values
-</code>
-Verify, that the generated ULS value files are transferred to the ULS-server
-and that their content is stored correctly.
-:!: No threshold violations are generated in this test
-Use a webbrowser, login as ULS user "TheSpyder",
-chose the correct time interval (today or explicit iso time interval)
-and check all values for domain "ULS":
-<file>
-ULS
-└─► testsrv11
-   └─► VAL 0010: Basic Values
-      └─► 010 Integer Values          (6 values)
-      └─► 020 Float Values            (11 values, [E]/m )
-      └─► 040 Files                   (two real images, one gif (wind and solar energy), one png (sand dunes))
-└─► testsrv12
-   └─► VAL 0010: Basic Values
-      └─► 010 Integer Values          (6 values)
-      └─► 030 Text Expressions        (11 expressions, from "00 AA" to "10 KKKKKKKKKKKKKKKKKKKKKK", proportional)
-      └─► 040 Files                   (one html, one odt, one pdf, one txt (as web links to open them))
-</file>
-and for domain "Woodlark":
-<file>
-Woodlark
-└─► testsrv21
-   └─► VAL 0010: Basic Values
-      └─► 020 Float Values          (6 values [%])
-      └─► 030 Text Expressions      (11 expressions, from '00 aaaaa' to '10 kkkkkkkkkkkkkkk', mono spaced)
-└─► testsrv22
-   └─► VAL 0010: Basic Values
-      └─► 020 Float Values          (7 values [kg])
-      └─► 040 Files                 (4 real images (LAN LEDs, jpeg))
-</file>
-Generate a report, //administration >> edit reports// and enter "VAL 0010: Basic Values" as report name
-and press {{:uls:testsuite:version_180:ok_button.png?nolink}}.
-Use //edit// and {{:uls:testsuite:version_180:append_autodetails.png?nolink}} to create a report detail containing:
-{|
-| name:
-| Show All Values
-|-
-| domains:
-| ULS,Woodlark
-|-
-| server:
-| testsrv*
-|-
-| section:
-| VAL 0010: Basic Values
-|-
-| separate by:
-| [x] server, [x] teststep
-|-
-| period:
-| today
-|}
-or use a specific time interval like, especially if you made several executions of the script:
-{|
-| from:
-| 2014-01-14 10:09:00
-|-
-| to:
-| 2014-01-14 10:11:00
-|}
-All other specifications remain unchanged.
-Click {{:uls:testsuite:version_180:ok_button.png?nolink}} and {{:uls:testsuite:version_180:show_report.png?nolink}}
-to show the report. Click on {{:uls:testsuite:version_180:generate_pdf.png?nolink}} and save the report
-to a path of your choice. It should look like {{:uls:testsuite:version_180:val_0010_basic_values.pdf|this}}.
-----
-=== VAL 0100: Continuous Values ===
-Several values of all supported value type are created and transferred to ULS
-in regular time intervals (each minute)
-Create a crontab to make the "continuous_values" script execute every minute. The script
-<code bash>
-cd /etc/cron.d/
-vi uls_test_suite
-</code>
-<file cron uls_test_suite>
-* * * * * uls /home/uls/testsuite/test_suite_1_8_0/continuous_values > /home/uls/testsuite/test_suite_1_8_0/continuous_values.log 2>&1
-</file>
-Verify, that the generated ULS value files are transferred to the ULS-server
-and that their content is stored correctly.
-Use a webbrowser, login as ULS user "AmieAction" (full rights on "ULS", read-only on "Woodlark"),
-chose the correct time interval (yesterday if the script runs already for more than a day
-or explicit iso time interval) and check all values for domain "ULS":
-<file>
-ULS
-└─► testsrv11
-   └─► VAL 0110: Continuous Hourly Values
-      └─► 110 Integer Values     (1 value)
-      └─► 120 Float Values       (1 value)
-      └─► 130 Text Expressions   (2 values)
-└─► testsrv12
-   └─► VAL 0110: Continuous Hourly Values
-      └─► 110 Integer Values     (2 values)
-      └─► 120 Float Values       (1 value)
-      └─► 140 Files              (1 image, 1 file "spiro.png")
-</file>
-and for domain "Woodlark"
-<file>
-Woodlark
-└─► testsrv21
-   └─► VAL 0110: Continuous Hourly Values
-      └─► 120 Float Values       (1 value)
-      └─► 130 Text Expressions   (2 expressions, 3 words, each 20 characters)
-      └─► 140 Files              (1 image with different contents, 1 file "trw.jpg"')
-└─► testsrv22
-   └─► VAL 0110: Continuous Hourly Values
-      └─► 120 Float Values       (1 value)
-</file>
-Generate a report, //administration >> edit reports// and enter "VAL 0110: Continuous Hourly Values" as report name
-and press {{:uls:testsuite:version_180:ok_button.png?nolink}}.
-Use //edit// and {{:uls:testsuite:version_180:append_autodetails.png?nolink}} to create a report detail.
-{|
-| name:
-| Show Last Values
-|-
-| groups:
-| Database
-|-
-| server:
-| testsrv*
-|-
-| section:
-| VAL 0110: Continuous Hourly Values
-|-
-| separate by:
-| [x] server, [x] teststep
-|-
-| period:
-| yesterday ((be sure, that the script is already running longer than a day!))
-|-
-| display:
-| last values
-|}
-or use a specific time interval like:
-{|
-| from:
-| 2014-01-15 00:00:00
-|-
-| to:
-| 2014-01-15 23:59:59
-|}
-All other specifications remain unchanged.
-Click {{:uls:testsuite:version_180:ok_button.png?nolink}} and
-{{:uls:testsuite:version_180:show_report.png?nolink}} to show the report.
-Go to //administration >> edit mail reports// and send the
-"VAL 0110: Continuous Hourly Values" report to your e-mail address.
-Use
-  * (x) landscape
-  * [x] send now
-The report should look like {{:uls:testsuite:version_180:val_0110_continuous_hourly_values_2014-01-22.pdf|this}}
------
-=== VAL 0200: Aggregations ===
-Aggregations are defined on continuously generated values.
-Connect as "AmieAction" to ULS and go to
-//administration >> compression >> set auto-compression// and chose domain "ULS"
-(AmieAction has enough rights **only** for domain "ULS").
-  * set section to "VAL 0110: Continuous Hourly Values"
-  * activate all checkboxes of "hourly" and {{:uls:testsuite:version_180:ok_button.png?nolink|}}
-  * continue with {{:uls:testsuite:version_180:textmode_button.png?nolink|}} and
-  * {{:uls:testsuite:version_180:export_button.png?nolink|}} to export the definitions to a new browser tab.
-  * copy the definition and paste it into the textbox of previous browser tab.
-  * copy the "hourly" entry to the other lines
-It should look like:
-<file ini create auto-compression for ULS>
-server=
-section=VAL 0110: Continuous Hourly Values
-teststep=
-detail=
-unit=
-hourly=last,avg,sum,dev,min,max,max-min,last-first,count,grad,first,accel,differ,firstsize,last/avg,lastsize
-daily=last,avg,sum,dev,min,max,max-min,last-first,count,grad,first,accel,differ,firstsize,last/avg,lastsize
-weekly=last,avg,sum,dev,min,max,max-min,last-first,count,grad,first,accel,differ,firstsize,last/avg,lastsize
-monthly=last,avg,sum,dev,min,max,max-min,last-first,count,grad,first,accel,differ,firstsize,last/avg,lastsize
-special=
-access=all
-</file>
-Click {{:uls:testsuite:version_180:ok_button.png?nolink|}} to save that.
-A new entry was added, delete the entry created in the first step.
-Click on //changes// to verify that the initial change has been logged.
-Any compression/aggregation action will be executed in the following night.
-== VAL 0210: Propagation of Aggregation Rules ==
-Verify, that the above rules are propagated correctly. Check:
-  * //administration >> compression >> set compression >> ULS >> testsrv11 >> 110 Integer Values//
-:TODO: does not work currently (2014-01-22)
-:NOTE:  When the [E] values are compressed/aggregated, the effective unit with the maximum numbers is used to process the aggregations.
-That may (in extreme cases) vary from day to day!
-Nevertheless, you may use combi-compressed-details to convert it into your favorite unit.
------
-=== VAL 0300: Special Values ===
-== VAL 0310: Timing Tuples with Additional Attributes ==
-== VAL 0330: Boundary Values ==
------
-=== NFY 0100: Notification Scenarios ===
-Threshold violations
-Define various possible notification scenarios and verify the correctly generated notifications
-ULS ticket tracking: threshold violations generate notifications and tickets. ULS users can work on those tickets and document their actions.
------
-=== NFY 0300: Notification Scenarios on Aggregated Values ===
-Threshold violations
-Define various possible notification scenarios on aggregated values and verify the correctly generated notifications
-ULS ticket tracking: threshold violations generate notifications and tickets. ULS users can work on those tickets and document their actions.
-=== Continue Here ===
-:NOTE: Continue here
-the correct storage of all value types:
-  * including boundary tests to assure the covered value ranges
-  * checked by interactive analysis
-  * checked by reports
-  * deletion of values after the retention time has elapsed
-=== Boundary Tests ===
-=== General Tests ===
-values, thresholds, aggregation, ...
-  * threshold violations
-    * numeric, string and file-like values
-    * different functions combined with number of values and/or time range
-    * including message bodies of notifications with replaced static and dynamically calculated embedded placeholders
-  * aggregation of values
-    * check the different possible aggregation functions on different data types
-  * reports
-    * check the expected results
-    * check the creation and modification of reports
-    * user-defined headers, separators, descriptions and logo
-    * PDF generation
-==== Test Procedure Specification ====
-detailing how to run each test, including any set-up preconditions and the steps that need to be followed
-Test Script is a program written to test the functionality of the application. It is a set of system readable instructions to automate the testing with the advantage of doing repeatable and regression testing easily.
-==== Test Environment ====
-It is the Hardware and Software Environment where is the testing is going to be done. It also explains whether the software under test interacts with Stubs and Drivers.
-Create a mock business. Treat it as real and process its data.
-==== Test Procedure ====
-Test Procedure is a document with the detailed instruction for step by step execution of one or more test cases. Test procedure is used in Test Scenario and Test Scripts.
-==== Test Log ====
-Test Log contains the details of test case execution and the output information.
-recording which tests cases were run, who ran them, in what order, and whether each test passed or failed
-==== Test Incident Report ====
-detailing, for any test that failed, the actual versus expected result, and other information intended to throw light on why a test has failed. This document is deliberately named as an incident report, and not a fault report. The reason is that a discrepancy between expected and actual results can occur for a number of reasons other than a fault in the system. These include the expected results being wrong, the test being run wrongly, or inconsistency in the requirements meaning that more than one interpretation could be made. The report consists of all details of the incident such as actual and expected results, when it failed, and any supporting evidence that will help in its resolution. The report will also include, if possible, an assessment of the impact of an incident upon testing.
-==== Test Summary Report ====
-A management report providing any important information uncovered by the tests accomplished, and including assessments of the quality of the testing effort, the quality of the software system under test, and statistics derived from Incident Reports. The report also records what testing was done and how long it took, in order to improve any future test planning. This final document is used to indicate whether the software system under test is fit for purpose according to whether or not it has met acceptance criteria defined by project stakeholders.

Universal Logging System

User Tools

Site Tools

Differences

Page Tools