Test Suite Guide

HPC Gateway include a test suite mechanism to test basic functionalities are working, as well as more complete functionalities and even workload tests. This test suite mechanism can be customized to build any test suite, according to the system. By default, 2 tests suite are provided:

HPC Gateway include a basic test suite that launch 4 tasks in sequence. This test suite is used to quickly check the basic operations of submitting jobs through HPC Gateway is working well.

- Start the basic test suite

Use the command hpcg_tests_run.py with configuration file tsuite.base.cfg.

$ source /opt/hpcg/core/etc/profile.sh
$ $HPCG_HOME/core/tsuite/bin/hpcg_tests_run.py -c $HPCG_HOME/core/tsuite/config/tsuite.base.cfg 

2015/11/27 16:04:36 - INFO  - Initialize configuration with /opt/hpcg/core/tsuite/config/tsuite.base.cfg
2015/11/27 16:04:36 - INFO  - Test suite directory = /home/hpcgadmin/hpcgateway/tsuite/base
2015/11/27 16:04:36 - INFO  - Start create_small_inputs
2015/11/27 16:04:36 - INFO  - Start populate_applications
2015/11/27 16:04:36 - INFO  - Import application TS_App_Basic_003 (5638e702a3a5893d5315db22)
...
2015/11/27 16:04:36 - INFO  - Start execute_test_plans
2015/11/27 16:04:36 - INFO  - Load test plan = testplan.base.json (/opt/hpcg/core/tsuite/config/testplans/testplan.base.json)
2015/11/27 16:04:36 - INFO  - Create task: 1 x (hpcgadmin - TS_App_Basic_001 - ts_001)
2015/11/27 16:04:47 - INFO  - Create task: 1 x (hpcgadmin - TS_App_Basic_002 - ts_001)
2015/11/27 16:04:57 - INFO  - Create task: 1 x (hpcgadmin - TS_App_Basic_003 - ts_001)
2015/11/27 16:05:07 - INFO  - Create task: 1 x (hpcgadmin - TS_App_Basic_004 - ts_001)
2015/11/27 16:05:17 - INFO  - Number of tasks created = 4

- Monitor the basic test suite

The best way to monitor the progress of the test suite is to connect to HPC Gateway desktop and open the Task Monitor tool.

You can also go in the test suite working directory located in $HOME/hpcgateway/tsuite/base, and check the log file tsuite.log

$ cd $HOME/hpcgateway/tsuite/base
$ tail -f tsuite.log

- Build report for basic test suite

Use the command hpcg_tests_report.py.

$ source /opt/hpcg/core/etc/profile.sh
$ $HPCG_HOME/core/tsuite/bin/hpcg_tests_report.py -c $HPCG_HOME/core/tsuite/config/tsuite.base.cfg
2015/11/27 17:51:34 - INFO  - Initialize configuration with /opt/hpcg/core/tsuite/config/tsuite.base.cfg
2015/11/27 17:51:34 - INFO  - Test suite directory = /home/hpcgadmin/hpcgateway/tsuite/base
2015/11/27 17:51:34 - INFO  - total: 4 ; pass: 4 ; fail: 0 ; notindb: 0 ; unkn: 0

A CSV file is generated in $HOME/hpcgateway/tsuite/base/tsuite.report.csv, with the status of all tests.

The test suite read a configuration file, that drive the execution of one or more test plans. Here is an example of the base test suite configuration file:

$ cat $HPCG_HOME/core/tsuite/config/tsuite.base.cfg
[install]
mode = silent

[tsuite]
debug = false
base_url = http://localhost:8080/torii
directory = /home/hpcgadmin/hpcgateway/tsuite/base
max_jobs = 15
list = create_small_inputs, populate_applications, execute_test_plans

[testplans]
list = testplan.base.json
  • tsuite.list = the steps of the test suite
  • tsuite.max_jobs = maximum number of batch jobs in the system. This is a best effort and this number may be exceeded during the time of the loop of the check. This value can be changed dynamically during a test suite to increase or decrease the batch system load on the fly.
  • testplans.list = the list of test plans that will be executed. The test plans are located in testplans directory.

Here is an example of test plan base:

$ cd $HPCG_HOME/core/tsuite/config/testplans
$ cat testplan.base.json
{
  "name": "tp_base",
  "description": "This test plan is to check basic execution",
  "users": ["hpcgadmin"],
  "delay": 10,
  "tests": [
    {"type": "submit", "inst": 1, "application": "TS_App_Basic_001", "name": "ts_001", "parameters": {"sleep": 10}},
    {"type": "submit", "inst": 1, "application": "TS_App_Basic_002", "name": "ts_001", "parameters": {"sleep": 10}},
    {"type": "submit", "inst": 1, "application": "TS_App_Basic_003", "name": "ts_001", "parameters": {"sleep": 10}},
    {"type": "submit", "inst": 1, "application": "TS_App_Basic_004", "name": "ts_001", "parameters": {"sleep": 10}}
  ]
}
  • users = list of users who will execute each test. Each user is launched in a separate thread, so tests are executed in parallel
  • delay = number of seconds between each test submission
  • inst = number of instance the test is submitted
  • rand = random number (between 1 to rand) of instance the test is submitted

HPC Gateway include a functional test suite that launch several tests. This test suite is used to check some functionalities of HPC Gateway are working well.

- Start the functional test suite

Use the command hpcg_tests_run.py with tsuite.functional.cfg configuration file.

$ source /opt/hpcg/core/etc/profile.sh
$ $HPCG_HOME/core/tsuite/bin/hpcg_tests_run.py -c $HPCG_HOME/core/tsuite/config/tsuite.functional.cfg 

2015/11/27 16:04:36 - INFO  - Initialize configuration with /opt/hpcg/core/tsuite/config/tsuite.functional.cfg
2015/11/27 16:04:36 - INFO  - Test suite directory = /home/hpcgadmin/hpcgateway/tsuite/run001
...

- Monitor the functional test suite

The best way to monitor the progress of the test suite is to connect to HPC Gateway desktop and open the Task Monitor tool.

You can also go in the test suite working directory located in $HOME/hpcgateway/tsuite/run001, and check the log file tsuite.log

$ cd $HOME/hpcgateway/tsuite/run001
$ tail -f tsuite.log

- Build report for functional test suite

Use the command hpcg_tests_report.py.

$ source /opt/hpcg/core/etc/profile.sh
$ $HPCG_HOME/core/tsuite/bin/hpcg_tests_report.py -c $HPCG_HOME/core/tsuite/config/tsuite.functional.cfg
2015/11/27 17:51:34 - INFO  - Initialize configuration with /opt/hpcg/core/tsuite/config/tsuite.functional.cfg
2015/11/27 17:51:34 - INFO  - Test suite directory = /home/hpcgadmin/hpcgateway/tsuite/run001
2015/11/27 17:51:34 - INFO  - total: xx ; pass: xx ; fail: 0 ; notindb: 0 ; unkn: 0

A CSV file is generated in $HOME/hpcgateway/tsuite/run001/tsuite.report.csv, with the status of all tests.

HPC Gateway include a complete test suite that launch hundreds of tasks. This test suite is used to check the behavior of HPC Gateway with high load.

- Start the QA test suite

Use the command hpcg_tests_run.py with tsuite.qa.cfg configuration file.

$ source /opt/hpcg/core/etc/profile.sh
$ $HPCG_HOME/core/tsuite/bin/hpcg_tests_run.py -c $HPCG_HOME/core/tsuite/config/tsuite.qa.cfg 

2015/11/27 16:04:36 - INFO  - Initialize configuration with /opt/hpcg/core/tsuite/config/tsuite.qa.cfg
2015/11/27 16:04:36 - INFO  - Test suite directory = /home/hpcgadmin/hpcgateway/tsuite/run001
...

- Monitor the QA test suite

The best way to monitor the progress of the test suite is to connect to HPC Gateway desktop and open the Task Monitor tool.

You can also go in the test suite working directory located in $HOME/hpcgateway/tsuite/run001, and check the log file tsuite.log

$ cd $HOME/hpcgateway/tsuite/run001
$ tail -f tsuite.log

- Build report for QA test suite

Use the command hpcg_tests_report.py.

$ source /opt/hpcg/core/etc/profile.sh
$ $HPCG_HOME/core/tsuite/bin/hpcg_tests_report.py -c $HPCG_HOME/core/tsuite/config/tsuite.qa.cfg
2015/11/27 17:51:34 - INFO  - Initialize configuration with /opt/hpcg/core/tsuite/config/tsuite.qa.cfg
2015/11/27 17:51:34 - INFO  - Test suite directory = /home/hpcgadmin/hpcgateway/tsuite/run001
2015/11/27 17:51:34 - INFO  - total: xxx ; pass: xxx ; fail: 0 ; notindb: 0 ; unkn: 0

A CSV file is generated in $HOME/hpcgateway/tsuite/run001/tsuite.report.csv, with the status of all tests.