Application Template Editor

The Application Template Editor is a simple and practical utility to onboard an application method (template) or procedure into the Gateway's local application catalogue. New methods are then accessed through the Application Management interface.


To use the Application Editor you must have a role editor in at least one team. Refer to the HPC Gateway User's Management guide for details.

Open the Application Editor from the start menu or a right-click in open space within the Application Management left pane.


The Application Editor is organized by panels. All panels must be treated in order to obtain a working application method.

Panel Purpose
Main Set the general attributes of application – name, version, icon, team, tags
Inputs Declare and annotatethe input variables of the application.
Outputs Declare and annotate the output variables of the application.
Phases Write the scripts that will be run in the different phases – prolog, execute, epilog, monitor - and set their active status.
Environment Write the script to define the environment of the application on each target cluster.

Main panel

Name is the label displayed for this application in the Application Management window. It can contain spaces.

Version is an index for the editor to track evolution of the development. There is no prescription for this format.

Description is where the editor should provide the basic level of detail for end-users to understand the purpose and general function of this application method. More extensive documentation should be provided in the Wiki that can be attached to this application.

Tags provides a means to later search and order this application within the Application Management window. This is a free-form attribute.

Teams are the preset groups that this application will belong to. Only users who are members of one or more of these teams will be able to run this application. Clicking in the entry field will display the available teams for selection.

Owners includes the initial editor by default. Other identities can also be assigned as owners.

NOTE:
An application under development is in DRAFT state. It should be independely tested before switching to RELEASED state. Defining multiple owners is a means to allow such independent testing.

Input and Output panels

This is a critical panel since it will shape the application parameter interface for the end-user.

The set of parameters to be presented must therefore be carefully designed.

  • Consider whether unique files or directories should be expected as input data. It is possible for users to select multiple input files during job preparation, but if this list might become too large a directory input may be more practical.
  • Alternatively, the run location may be set by its execution directory. In which case text file labels may be sufficient.
  • Consider the contraints to apply on the inputs. The method will become more robust if the user needs only to select from predefined values, or can only enter numbers within an accepted range.

Each input or output variable if characterised by the following attributes.

Name This will be the variable name used within the runnable scripts for this application method.
Type Type of the input/output that can be boolean, file, float, int, script, text
Label Short but meaningful label that is displayed to the end-user in the graphical user interface before execution
Description Longer description of the variable that is used in a tooltip.
Required Check if this parameter must be set by the end-user.
Automatic check Check if this input must trigger the validation script after each change of value.
Hidden Check if this input has to be hidden in the user interface.

A complete input parameter definition for a basic method would look something like the following.


Inputs / Outputs files

Input and output files have a particular handling.

Inputs files are copied into the run directory during the Prolog phase. Therefore, whenever an input file is defined a Prolog phase must be defined to ensure the copy takes place. A single character is sufficient to define the Prolog, eg. a comment statement. This transfer takes place at the Prolog phase to avoid large file copies during an Execute phase which has reserved multiple nodes.

The respective input file variable contains the names of all files transferred. Multiple file names are separated by a “:”. The copy mechanism use the scp command initiated from the compute node. It is possible to change the scp command by cp command in /home/hpcgadmin/hpcgateway/utilities/hpcg_copy.py. Please refer to the Fujitsu support staff to make this change.

In summary, automated file movement currently uses the following procedure.

  1. Input files are copied in the run directory during the prolog phase (this is to avoid large copies during the execute phase that might allocate a lot of nodes).
  2. As of now, output variables are not used because there is no workflow. We just set the value and it is displayed in the Task Details panel.
  3. Fetch in / fetch out are done only during the prolog / epilog phase, so you must define these phases (even empty) whenever input files are defined and you want to copy in / out some files.

Save some result files to a specific location after execution

Outputs files are not treated as of now. They will be used when workflow is available to copy some defined output files from the current task to the following ones.

There is still a mechanism to specify that some files must be stored to a specific location after the execution of the application. The application developer should set these specific variables in the Epilog phase.

HPCG_STORE_DIR Path to the store directory.
HPCG_STORE_PATTERN

or

HPCG_STORE_PATTERN="*.log  result.out"

Usually, the application developer would declare these two variables as an input of the application. Note that their type must be text, not file. In this case the end-user can decide what to store and where in the application parameter profile before execution. Equally, the application developer can also set a default in the Epilog to force a storage action.

For a comprehensive understanding of this save to location mechanism, here is the python code that is executed during the epilog. This is an extract of the full file transfer python script that is located by default in /home/hpcgadmin/hpcgateway/utilities/hpcg_copy.py.

        # On demand, copy files to an output directory that can be set by the user
        # By convention, the output files are stored under HPCG_STORE_DIR if set in env
        if copy_type == "outputs":
            cluster = os.getenv("HPCG_HEAD_NODE", "unknown")
            dst_path = os.getenv("HPCG_STORE_DIR")
            user_pattern = os.getenv("HPCG_STORE_PATTERN")
            if cluster != "unknown" and dst_path is not None and user_pattern is not None:
                action = "COPY"
                cmd = "ssh " + cluster + " mkdir -p " + dst_path
                ierr += exec_command(cmd)
                for src_name in user_pattern.split():
                    src_path = os.path.join(run_directory, src_name)
                    cmd = "scp -pr " + src_path + " " + cluster + ":" + dst_path
                    ierr += exec_command(cmd)
                    file_json = {}
                    for loc_path in glob.glob(src_path):
                        name = os.path.basename(loc_path)
                        size = get_size(loc_path)
                        size_of_files_transferred += size / (1024*1024)

The phases panels are used for entering the scripts that will be used during the execution. If the script is empty, the phase is skipped.
Scripts can be written in shell, python, or any interpreted language by using the shebang syntax on top of the script.

#!/bin/env python

Usage of variables within phase script

The HPC Gateway back-end execution system sets all defined inputs and outputs as environment variables for the run phases scripts. The editor does not have to further declare these as environment variables or process them further.

All such variables can thus be referenced by their name directly within the phase scripts. For example, if the input variable model_name is defined, it can be used in the phases script as:

  • bash
echo ${model_name}
  • python
print os.getenv(model_name)

HPC Gateway also sets numerous variables that can be used by the application if needed. These use capitalised labels incorporating the prefex HPCG_.

**HPCG_BATCH_HOST_FILE**=/var/spool/torque/aux/3709.hpcgdemo.private.cluster.zone
**HPCG_BATCH_ID**=3709.hpcgdemo.private.cluster.zone
**HPCG_BATCH_MPI_PROCS**=16
**HPCG_BATCH_OMP_THREADS**=4
**HPCG_BATCH_QUEUE**=long
**HPCG_HEAD_NODE**=hpcgdemo
**HPCG_INFO_APP**=/home/hpcgadmin/hpcgateway/runlog/hpcgadmin/2015/11/18/task_92/.hpcg/hpcg_info_app.out
**HPCG_INFO_SYS**=/home/hpcgadmin/hpcgateway/runlog/hpcgadmin/2015/11/18/task_92/.hpcg/hpcg_info_sys.out
**HPCG_JOB_NAME**=neon.pre
**HPCG_LOG_DIR**=/home/hpcgadmin/hpcgateway/runlog/hpcgadmin/2015/11/18/task_92
**HPCG_PATH_DATE**=2015/11/18
**HPCG_RUN_DIR**=/home/hpcgadmin/tmp/lsdyna
**HPCG_STATUS_FILE**=/home/hpcgadmin/hpcgateway/runlog/hpcgadmin/2015/11/18/task_92/neon.pre.status
**HPCG_STOP_FILE**=/home/hpcgadmin/hpcgateway/runlog/hpcgadmin/2015/11/18/task_92/neon.pre.stop
**HPCG_TASK_NAME**=neon
**HPCG_TASK_NUM**=92
**HPCG_TASK_PHASE**=prolog

Output variables

It is possible to set output variable value that is then handled by HPC Gateway. This is achieved by creating the file ${HPCG_INFO_APP} with content according to the fixed convention:

name=value

Some examples:

$ echo "energy=404">> ${HPCG_INFO_APP}
$ echo "mass=80"  >> ${HPCG_INFO_APP}
$ env | grep "app_">> ${HPCG_INFO_APP}

Exit codes and messages

It is possible to set a specific exit code and exit message. that is then handled by HPC Gateway. This is achieved by creating the file ${HPCG_STATUS_FILE} with content according to the fixed convention:

exit_code:exit_message

Some examples:

$ echo "99:cannot find model"> ${HPCG_STATUS_FILE}
$ echo "0:normal termination"> ${HPCG_STATUS_FILE}
ATTENTION:
When a phase exit code is different from 0 HPC Gateway considers this is an error and does not execute any of the subsequent phases of the task.

Prolog phase

This phase can be used for pre-processing, check input variables, etc …
It is executed in batch, using a single core.


Execute phase

This phase is used to run the simulation. It is executed in batch, using the batch resources set by the user.


Epilog phase

This phase can be used for for-processing, check output variables, etc …
It is executed in batch, using a single core.


Important note for the phases

By design, the prolog and epilog phases do not use the scheduler options. So, prolog and epilog use batch defaults. The main reason is that we do not want to allocate 32 procs (if requested by the user) in the prolog and epilog, but only in the execute. We did not want to add in the interface specific scheduler options for prolog and epilog, because this would make the tool more and more complex to use. Therefore, prolog and epilog should be used only for small and short checks or computations. We also use the prolog phase for data transfer to avoid it during the execute phase.

If prolog and epilog has to become “big”, it is recommended to build a specific applications, then (may be) use the process mapper to link the 3 applications.

Nevertheless, we understand that this can be a limitation for some specific cases.

Recently (in the current development - release target June 2017 - core 1.4) we allow the application developer (not the end-user) to set specific (or default) batch options for prolog, epilog and execute phase in the environment of an application (specific to a cluster then). The syntax is the following:

# batch option set for all phases
#HPCG_BATCH -l mem=1024m

# batch option set for phase prolog
#HPCG_BATCH_prolog -q short  -l nodes=1:ppn=1

# batch option set for phase execute
#HPCG_BATCH_execute -q batch

# batch option set for phase epilog
#HPCG_BATCH_epilog -q post-proc  -l nodes=1:ppn=2

Monitor script

This script is used during the data monitoring.
It has to extract data from application specific file and print them in standard output.
First line must output name of parameters that will be displayed.

By convention, first column must be called “step” and second column (optional) can be called “time”.
Step and time column must contain integer. Other columns can contain any type of numbers.


Then, values must be written line by line with | separator.

Example:

step | time | val 1 | val 2 | val 3 | val 4
1 | 10 | 0.2 | 1.1e2 | 4.2e2 | 11.23e2
2 | 15 | 0.3 | 3.3e2 | 4.2e2 | 11.23e2
3 | 20 | 0.4 | 4.4e2 | 4.2e2 | 11.23e2
4 | 25 | 0.5 | 5.5e2 | 4.2e2 | 11.23e2

Typical monitoring script

Monitoring script can be written in any scripting language.
It is mandatory to set the right shebang, otherwise there is no output.

#!/bin/bash

export HPCG_LOG_DIR="@@__HPCG_LOG_DIR__@@"
export HPCG_RUN_DIR="@@__HPCG_RUN_DIR__@@"

cd ${HPCG_RUN_DIR}

cat <<'eof'> monitor.awk
BEGIN { i=0
print "step | time | time step | kinetic energy | internal energy | total energy |  X velocity | Y velocity | Z velocity"
}
/^ dt of cycle/ {
if(i>0) { printf("%d | %s | %s | %s | %s | %s | %s | %s | %s\n",cycle,time,ts,ke,ie,tote,xvel,yvel,zvel) }
i=NR; cycle=$4
}
/^ time /              { time=$2 }
/^ time step /         { ts=$3 }
/^ kinetic energy /    { ke=$3 }
/^ internal energy /   { ie=$3 }
/^ total energy /      { tote=$3 }
/^ global x velocity / { xvel=$4 }
/^ global y velocity / { yvel=$4 }
/^ global z velocity / { zvel=$4 }
eof

awk -f monitor.awk  filetomonitor

Internal monitoring mechanism

When data monitoring tool is opened, the Cluster Agent starts the script mediators/batch_monitor (python). The lease (by default 120s) is automatically renewed until the data monitoring tool is closed. The script mediators/batch_monitor stops only when a lease is expired.

The script mediators/batch_monitor executes the application script app_monitor in loop (by default every 5s) and get the standard output to write hpcg_monitoring.csv. Then, the Cluster Agent reads hpcg_monitoring.csv and write records in mongo DB. These records are displayed in the data monitoring tool.

The command line that is started by the Cluster Agent is like the following: /opt/hpcg/live/clusters/esteban/scheduler/batch_monitor /home/hpcgadmin/hpcgateway/runlog/hpcgadmin/2015/10/10/task_53/.hpcg/bench5_monitor /home/hpcgadmin/hpcgateway/runlog/hpcgadmin/2015/10/10/task_53/.hpcg/hpcg_monitoring.csv /home/hpcgadmin/hpcgateway/runlog/hpcgadmin/2015/10/10/task_53/.hpcg/hpcg_monitoring.lease


Set the environment needed to run the application, specific to a cluster. If there is no environment set for one cluster, the application is not available for this cluster. Therefore, you must set an environment (even with 1 blank line) to publish the application to a cluster.

Here is an example of classical environment file to set PATH and specific variables to run an application. It is wise to set environment variables that then will be used in the execution part (script panel). This allow to develop very portable application wrapper that can be installed on any cluster and configured through this environment mechanism.

##
## Environment for DL_POLY_Classic
##

## DL_POLY_Classic

export DL_POLY_HOME=/dx90/bmt/applications/dlpoly2_stfc/dl_class_1.9
export DL_POLY_BIN=${DL_POLY_HOME}/execute
export EXEC_CMD=${DL_POLY_BIN}/DLPOLY.X
export PATH=${DL_POLY_BIN}:${PATH}

export R_EXEC=/dx90/bmt/applications/R/R-2.15.3/bin/R

## MPI

# module load batch/torque
export MPI_ROOT=/dx90/bmt/compilers/intel.2013/impi/4.1.1.036/intel64
export MPI_BIN=/dx90/bmt/compilers/intel.2013/impi/4.1.1.036/intel64/bin
export MPI_LIB=/dx90/bmt/compilers/intel.2013/impi/4.1.1.036/intel64/lib
export MPI_CMD=${MPI_BIN}/mpirun

export PATH=${MPI_BIN}:${PATH}
export LD_LIBRARY_PATH=${MPI_LIB}:${LD_LIBRARY_PATH}

##
## END OF ENVIRONMENT
##

Recently (in the current development - release target June 2017 - core 1.4) we allow the application developer (not the end-user) to set specific (or default) batch options for prolog, epilog and execute phase in the environment of an application (specific to a cluster then). The syntax is the following:

# batch option set for all phases
#HPCG_BATCH -l mem=1024m

# batch option set for phase prolog
#HPCG_BATCH_prolog -q short  -l nodes=1:ppn=1

# batch option set for phase execute
#HPCG_BATCH_execute -q batch

# batch option set for phase epilog
#HPCG_BATCH_epilog -q post-proc  -l nodes=1:ppn=2



The validation panel of the Application Editor enables the application method (template) developer the possibility to write a script which checks the validity of the application parameters set by the end-user during the submission of a task. The validation script is either uploaded via this panel or input directly into the provided editing window.

The script can be used to perform the following:

  • Validate the user provided values for the Application inputs as shown on the “Application submission” panel and modify, if necessary, the attributes of those parameters

The Validation panel inside the Application Editor is shown below:

Validation script processing

When the application input parameter validation process is started the validation script receives, as a parameter, a description of the Application input parameters in the form of a JSON object. This JSON object is known as the Task definition.

The application parameter validation script can be invoked as a consequence of 3 actions:

  1. User clicks the “Submit” button on the “Application submission” panel
  2. User clicks the “Validate” button on the “Application submission” panel
  3. An input parameter with the “Auto check” option set to “True” is updated by the user and then the user clicks anywhere outside this field on the screen (i.e. the field loses the focus)

Note 1: Even if the validation process is triggered via the “Auto check” mechanism the JSON object passed to the validation script contains the current state information of ALL application input parameters and their values. Thus the script has the possibility to parse, process and update attributes for any or all of the application input parameters.

Note 2: The validation process may be called several times before submission of the application takes place.

Each time the validation process is invoked the following basic processing will occur:

  1. The validation script is called by the Gateway framework and is passed the task definition (in JSON format) containing all Application input parameters and values
  2. The validation script uses the task definition to perform some form of validation on the values supplied by the user
  3. The validation script MUST write a task definition to stdout (in JSON format). This Task definition will most likely be updated based on the validation performed but can remain unchanged if no validation errors are found or no changes are necessary for the Application input definitions

NOTE: Currently only a Python script can be used as the validation script

Validation script processing options

The validation script is able to modify certain aspects of the Application submission interface based on the user supplied values to the application inputs. This provides application method (template) developers with the ability to dynamically check and customize the interface based on supplied values.

The main processing options of the script include:

  • Invalidate an input parameter so the user must modify the supplied content to something else
  • Hide certain parameters from the application submission panel (useful when the value of one parameter means another parameter is not needed or may no longer be a valid input parameter, i.e. the combination of the two is not valid)
  • Change the value of an input after checking the user supplied value (useful if you want to automatically correct an incorrectly entered value with a valid value)
  • Based on the value of one input parameter change the value of one or more other input parameters (useful when one parameter can affect the choices for other parameters)
  • Change the value types of a input based on some other input value. (for example you may wish to change a free format integer field to a restricted choice list of integers)
  • change attributes of input values from not required to required (useful when the specification of one input value requires the specification of another input parameter)
  • set the reason the input parameter validation failed. This message is displayed to the end-user.
  • modify other aspects of an input parameter such as its Label and Description (used as the Tooltip)

Basic properties

The following table shows the main attributes, type and description provided in the Task definition JSON structure.

Task attributes defined in the JSON structure:

Attribute name Type Description
name string The task name.
description string The description of the task.
application object The application reference.
cluster object The cluster reference.
project object The project reference.
accountingCode object The accounting object (referencing an accounting code).
schedOptions array The scheduler options given to the task.
runDir string The run directory (where all the files needed to run the task will be put).
inputs array Array of inputs (the parameters to run the task).
tags array Array of strings, which constitute tags that are useful in easily finding a task.

The most common purpose of the validation script is to check the validity of the application inputs. As previously mentioned the validation script has the possibility to make adjustments to most of the application inputs' attributes.

It's important to know which attributes can be changed for the defined inputs (and therefore how you can affect the application submission graphical interface).

The next table highlights those attributes which can be changed via the validation script. Any updates must be made by creating an updated Task definition in JSON format and echoing that to stdout.

Input's attributes:

Attribute name Type Description
label string The label of the input. Changes to this attribute will change the field label displayed on the application submission panel.
description string The description of the input. Changes to this attribute will change the field tooltip.
hidden boolean Hide the input field, if set to True (by default set to False). Use this to hide the Input on the application submission panel
valid boolean Indicates the validity of the input. By setting to “False” the Application Submitter will prevent the submission of the task as it will understand this input has been incorrectly specified (by default this is set to True).
reason string Comment to explain why the input is invalid (will be shown as an error message to the user).
value ? The value depends of the input type (text, int, float, boolean, script). NOTE: for a “File” input parameter there is no value.
required boolean Set this to “True” if the field is mandatory or not (by default this is set to “False”).
autoCheck boolean Set this to “True” if the validation script will be triggered automatically when this input parameter is changed.

Validation script sample

To show how the validation script works, we have provided the following example:

#!/usr/bin/env python
import json
import sys

## Set an input from a task to invalid with a reason
def invalidInput(json, index, reason, value):
    json["inputs"][index]["valid"] = False
    json["inputs"][index]["reason"] = reason
    json["inputs"][index]["value"] = value
    return json

## Set an input from a task to valid
def validInput(json, index, value):
    json["inputs"][index]["valid"] = True
    json["inputs"][index]["value"] = value
    return json

## Set True to the hidden attribute of an input
def hideInput(json):
    json["inputs"][1]["hidden"] = True
    return json

## Set False to the hidden attribute on an input
def unhideInput(json):
    json["inputs"][1]["hidden"] = False
    return json

## Set some values to the execution environment of the task
def prefillExecutionEnvironment(json):
    ## Set a value to the task name in case there is no name set already
    if not json["name"]:
        json["name"] = "test_task"

    ## Set a value to the task description
    json["description"] = "TEST APPLICATION"

    ## Set a value to the task run directory
    json["runDir"] = "/home/bob"

    ## Set a value to the scheduler options
    json["schedOptions"] = [
        {"num": 0,
         "type": "ADVANCED",
         "syntax": {
             "opt": "-q ",
             "pattern": "[^\\s]+",
             "values": [
                 "batch",
                 "short",
                 "long",
                 "night",
                 "oper"
             ]
         },
         "label": "Queue",
         "description": None,
         "value": "short"
         }, {
            "num": 0,
            "type": "ADVANCED",
            "syntax": {
                "opt": "-l walltime=",
                "pattern": "[0-9]+:[0-5][0-9]:[0-5][0-9]",
                "values": None
            },
            "label": "Wall Clock Limit",
            "description": None,
            "value": "5:30:21"
        }]
    return json
##Starting point of the script, it takes in parameter a file path of a task in JSON format.
if __name__ == '__main__':
  if len(sys.argv)> 1:
      with open(sys.argv[1]) as data_file:
         task = json.loads(data_file.read())
  else:
      task = json.loads(sys.stdin.read())
  errorMessage = "This input wasn't validated by the validation script. Please change the text variable with an other name"
  task = prefillExecutionEnvironment(task)
  if task["inputs"][0]["value"] is not None:
      if task["inputs"][0]["value"] != "error":
          task["inputs"][0]["valid"] = True
          task["inputs"][4]["description"] = "test"
          task["inputs"][1]["valid"] = True
          task = unhideInput(task)
          if task["inputs"][0]["value"] == "hide":
              task = hideInput(task)
          task = validInput(task, 2, 0.5)
          task["inputs"][2]["min"] = 0
          task["inputs"][2]["min"] = 0
          task["inputs"][2]["max"] = 5
          task["inputs"][2]["values"] = [0, 2.5, 5]
          task["inputs"][2]["label"] = "Scale (0-5)"
          task["inputs"][2]["description"] = "Scale to 0-5"
          if task["inputs"][0]["value"] == "reset":
              task["inputs"][4]["description"] = "reset"
              task["inputs"][2]["label"] = "Scale (0-10)"
              task["inputs"][2]["values"] = []
              task["inputs"][2]["description"] = "Scale to 0-10"
              task["inputs"][2]["min"] = 0
              task["inputs"][2]["max"] = 10
          task = validInput(task, 3, 2)
          task = validInput(task, 4, "sleep 5")
          task = validInput(task, 5, True)
      else:
          task = invalidInput(task, 0, errorMessage, task["inputs"][0]["value"])
          task["inputs"][1]["valid"] = False
          task["inputs"][1]["reason"] = errorMessage
          task = invalidInput(task, 2, errorMessage, 666)
          task = invalidInput(task, 3, errorMessage, 66)
          task = invalidInput(task, 4, errorMessage, "Error mode")
          task = invalidInput(task, 5, errorMessage, False)
  ##This will return the task definition to the Gateway backend. (DO NOT USE PRINT IN THE CODE OTHERWISE IT WILL CRASH DURING EXECUTION)
  print(json.dumps(task, indent=4))

Which gives in the Application Editor:

Important tips and properties

In the JSON task definition, the order of the inputs is the same as that configured in the application editor, for example if an application has a text, float and file parameter defined in this exact order, then the result in the inputs array will be [{input_text}, {input_float}, {input_file}]. It's important to know this in order to manage the inputs array properly.

As shown in the sample validation script, there are some special properties for the inputs because depending on their types the inputs have some specific attributes.

Values

This attribute can only be used for an input of type text, float or int.

An input parameter can have multiple values based on the values attribute (it's an array of number or string). If the input beforehand only had a single value set, and this is changed to an array of values then the interface will automatically update the text or number parameter into a combo box rather than a single entry field.

File

File definitions are themselves a complex structure and therefore their values cannot be contained in teh simple value attribute that other input parameters use. For this reason an input parameter of type file does not have a value attribute but instead uses a files attribute,which is itself an array of type file. Each entry in this array describes a specified file.

Note: All attributes of the file structure are mandatory.

File's attributes:

Name Type Description
name string Name of the file
server object Server reference “server” : {“id”: id, “name”: name}
path string Absolute path of the file
action string Enum type choosen between: COPY or LINK
size long Must be set to 0
mods string File permission can be set to “———-“.

Pattern

This attribute can only be used for an input of type text and file.

It's simply an attribute to set a regular expression to those inputs type.


Return to topics page if followed as part of the training programme.

Return to Quickstart page.


It is also possible to advanced applications by directly editing the JSON of the application.
The simplest way to do it is to export and re-import the application using the commands hpcg_app_import.py and hpcg_app_export.py.

Then you can work on the application exposed a small file hierarchy.

Test_App_007
├── app_execute
├── app_validate
├── clusters
│   ├── cpu247
│   ├── head
│   ├── k5_cloud
│   ├── k5_vm57
│   └── ssf
├── definition.json
└── README

Multi phases

The advanced edition allows an editor to have more complex phases.

You can define the following elements :

  • name: The name of the phase
  • shortName: The postfix of the phase jobs
  • cardinality: The number of jobs to execute in parallel for one phase
  • dependency: The phase(s) after which this one is executed
  • udso: (User Defined Scheduler Options) Does the scheduler options defined by the user concern this phase
    "phases": [
        {
            "name": "divautils", 
            "shortName": "diva", 
            "udso": false
        }, 
        {
            "dependency": "divautils", 
            "name": "prolog", 
            "shortName": "pre", 
            "udso": false
        }, 
        {
            "dependency": "prolog", 
            "name": "master", 
            "shortName": "mstr", 
            "udso": true
        }, 
        {
            "cardinality": "@@__HPCG_INPUT_nb_slaves__@@", 
            "dependency": "prolog", 
            "name": "slave", 
            "shortName": "slav", 
            "udso": false
        }, 
        {
            "cardinality": "@@__HPCG_INPUT_optional__@@", 
            "dependency": "prolog", 
            "name": "optional", 
            "shortName": "opt", 
            "udso": false
        }, 
        {
            "dependency": "master", 
            "name": "epilog", 
            "shortName": "post", 
            "udso": false
        }
    ], 

As shown in the example, one can use the task inputs in the phases options.
For example, here, the cardinality of the phase slave is controled by the input nb_slaves of the application.

Custom Layout

The advanced edition allows an editor to have more complex layout.
You can use the ExtJS properties for your layouts:
ExtJS documentation

You can use:

  • Layout type: The type of panel layout (vbox, hbox, …)
  • Properties: The panel properties (style, title, headerPosition, tabPosition, bodyStyle, …)

This is an example of application with a custom layout:

This is the corresponding JSON application:

"customLayout" : {
        "panels" : [ 
            {
                "name" : "main",
                "panels" : [ 
                    {
                        "name" : "Variables",
                        "panels" : [],
                        "layout" : "vbox",
                        "properties" : {}
                    }, 
                    {
                        "name" : "Analysis",
                        "panels" : [ 
                            {
                                "name" : "Energies",
                                "panels" : [ 
                                    {
                                        "name" : "Energies_p1",
                                        "panels" : [],
                                        "layout" : "panel_hbox",
                                        "properties" : {}
                                    }, 
                                    {
                                        "name" : "Energies_p2",
                                        "panels" : [],
                                        "layout" : "panel_hbox",
                                        "properties" : {}
                                    }
                                ],
                                "layout" : "vbox",
                                "properties" : {}
                            }, 
                            {
                                "name" : "Thermodynamics",
                                "panels" : [ 
                                    {
                                        "name" : "Temperature",
                                        "panels" : [ 
                                            {
                                                "name" : "Temperature_p1",
                                                "panels" : [],
                                                "layout" : "panel_hbox",
                                                "properties" : {}
                                            }, 
                                            {
                                                "name" : "Temperature_p2",
                                                "panels" : [],
                                                "layout" : "panel_hbox",
                                                "properties" : {}
                                            }, 
                                            {
                                                "name" : "Temperature_p3",
                                                "panels" : [],
                                                "layout" : "panel_hbox",
                                                "properties" : {}
                                            }
                                        ],
                                        "layout" : "vbox",
                                        "properties" : {}
                                    }, 
                                    {
                                        "name" : "Volume",
                                        "panels" : [ 
                                            {
                                                "name" : "Volume_p1",
                                                "panels" : [],
                                                "layout" : "panel_hbox",
                                                "properties" : {}
                                            }, 
                                            {
                                                "name" : "Volume_p2",
                                                "panels" : [],
                                                "layout" : "panel_hbox",
                                                "properties" : {}
                                            }, 
                                            {
                                                "name" : "Volume_p3",
                                                "panels" : [],
                                                "layout" : "panel_hbox",
                                                "properties" : {}
                                            }
                                        ],
                                        "layout" : "vbox",
                                        "properties" : {}
                                    }
                                ],
                                "layout" : "hbox",
                                "properties" : {}
                            }
                        ],
                        "layout" : "vbox",
                        "properties" : {}
                    }, 
                    {
                        "name" : "Tab_test",
                        "panels" : [ 
                            {
                                "name" : "Tab1",
                                "panels" : [ 
                                    {
                                        "name" : "Group_A",
                                        "panels" : [ 
                                            {
                                                "name" : "Group_01",
                                                "panels" : [],
                                                "layout" : "vbox",
                                                "properties" : {
                                                    "style" : {
                                                        "backgroundColor" : "#D8D8D8"
                                                    },
                                                    "title" : "Group 1"
                                                }
                                            }, 
                                            {
                                                "name" : "Group_02",
                                                "panels" : [],
                                                "layout" : "vbox",
                                                "properties" : {
                                                    "style" : {
                                                        "fontSize" : 20,
                                                        "backgroundColor" : "#A9F5E1"
                                                    },
                                                    "collapsible" : false,
                                                    "html" : "<p>Hello World!</p>",
                                                    "title" : "Group 2"
                                                }
                                            }
                                        ],
                                        "layout" : "hbox",
                                        "properties" : {
                                            "style" : {
                                                "backgroundColor" : "#F8E0F7"
                                            },
                                            "title" : "Group A"
                                        }
                                    }, 
                                    {
                                        "name" : "Group_B",
                                        "panels" : [],
                                        "layout" : "vbox",
                                        "properties" : {
                                            "border" : false,
                                            "title" : "Group B"
                                        }
                                    }
                                ],
                                "layout" : "vbox",
                                "properties" : {
                                    "margin" : 20,
                                    "border" : 5,
                                    "title" : "World"
                                }
                            }, 
                            {
                                "name" : "Tab2",
                                "panels" : [],
                                "layout" : "vbox",
                                "properties" : {
                                    "title" : "Hello",
                                    "icon" : "../../resources/images/applicationManager/application-default-48.png"
                                }
                            }, 
                            {
                                "name" : "Tab3",
                                "panels" : [],
                                "layout" : "vbox",
                                "properties" : {
                                    "hidden" : true
                                }
                            }
                        ],
                        "layout" : "tab",
                        "properties" : {
                            "title" : "Testing",
                            "plain" : true,
                            "tabPosition" : "left",
                            "headerPosition" : "left",
                            "bodyStyle" : {
                                "backgroundColor" : "#D8F781"
                            },
                            "height" : 600
                        }
                    }
                ],
                "layout" : "tab",
                "properties" : {}
            }
        ]
    }

The link between inputs and custom layout is in the JSON application too. Each input contain the layout information as attribute. For the previous example, this is thecorresponding inputs:

"inputs" : [ 
        ...
        
        {
            "type" : "boolean",
            "name" : "total_a",
            "order" : 6,
            "label" : "Total",
            "required" : false,
            "valid" : false,
            "autoCheck" : false,
            "hidden" : false,
            "panel" : "Energies_p1"
        }, 
        {
            "type" : "boolean",
            "name" : "kinetic_b",
            "order" : 7,
            "label" : "Kinetic",
            "required" : false,
            "valid" : false,
            "autoCheck" : false,
            "hidden" : false,
            "panel" : "Energies_p1"
        }, 
        {
            "type" : "boolean",
            "name" : "potential_c",
            "order" : 8,
            "label" : "Potential",
            "required" : false,
            "valid" : false,
            "autoCheck" : false,
            "hidden" : false,
            "panel" : "Energies_p1"
        }, 
        {
            "type" : "boolean",
            "name" : "angle_d",
            "order" : 9,
            "label" : "Angle",
            "required" : false,
            "valid" : false,
            "autoCheck" : false,
            "hidden" : false,
            "panel" : "Energies_p2"
        }, 
        {
            "type" : "boolean",
            "name" : "hendral_e",
            "order" : 10,
            "label" : "D-Hendral",
            "required" : false,
            "valid" : false,
            "autoCheck" : false,
            "hidden" : false,
            "panel" : "Energies_p2"
        },
        
        ...
]