Create a Module
To create a new module on the 4P platform please follow closely the following steps.
Note
Users with no PHIS account cannot create modules.
1. Develop a 4P compatible module
A computation module may be developed in any language (Java, C, C++, Python, etc.) but with the following conditions:
the module is embedded in a Docker Unix image including the complete execution environment
the module is executable in command line mode without any user interaction
all the necessary parameters for the module execution are provided as arguments of the command line
Important
All the technical and operational requirements and recommendations for developing 4P compatible modules are presented in the following document: 4P Modules Operational Conventions. Please read these requirements carefully.
4P already contains a collection of data and modules with which new modules should make connections. To do this, new modules may use the existing parameters on 4P. Browse the List of 4P parameters to design a new module and its entrypoints.
Hint
A template of a compatible module in Python is available on the ForgeMIA: 4P Module Model in Python. Please make sure to use or at least acknowledge this template whenever a module in Python has to be developed.
2. Configure the module integration
The 4P platform uses its own internal continuous integration of modules. Here are the requirements for integrating a module in 4P.
Git Repository
Drop the module source code on a reachable Git repository (Gitlab, Bitbucket, Github, etc.). Developers must include in the sources a compilation script for building the Docker images (Dockerfile, makefile, etc.). It is also recommended to include a valid testing dataset that could be executed during the integration attempt.
Important
Make sure each Docker image is tagged with a specific version tag. This is mandatory for an integration in 4P. See the section Appendix A: Manage Module Versions for more details.
SSH Authentication
4P must have access to the repository in a read-only mode, at least, to be able to perform the module integration. Please add the following SSH public key to the module repository so that 4P can connect to it:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAb35d/7o3OSZoUPuQZgBq/O/WGFfEtvsQEVCvXc6we4S/cFBH/DNQh2y+g68HFdnFGTxIfYUlYWbOhkyduGlVGG5sluXC0MSWlcuJoyO1HyZ0unSOqDT9LwYVx+eQ7kFIig4yVsJTDxeReVNgck1IEqhlA5qWXYRMJNZQqMhCxhzFDeRqyFUidIyLzXIKNbTs0iBfmleJzfblqElAVpoSM09Ndl4SaXlmrPTekae87CS8b5tXQjLs6FwIKxu7TOlHOTFA6FExngv5C61DfRD4C05WqtyqNMHH16eYlXbDZc4rUZPlXgC1rBxqQ6NLL9RxCzyWiqbVlljSgMwdYQ0z jenkins@inra4pweb-qualification.novalocal
Hint
Read the following tutorials on how to add a public SSH key:
On Bibucket: SSH access keys for system use
On Gitlab: Deploy Keys
More specifically, on the GitLab solution of the ForgeMIA server, make sure to activate SSH key named 4P modules integration to the current module’s project. To do this, go to the section Settings > Repository > Deploy keys of the module’s project then enable the deploy key from the Publicly accessible deploy keys tab (as shown below).
Groovy Script
In order to integrate the module, 4P needs to know how to build it. This is done through a file named inra4p.build.groovy that must be put in the root folder of the repository.
Note
This file must use the Groovy syntax for Jenkins and must describe the Docker image building steps of the module. Here is an example of a inra4p.build.groovy file.
3. Add the module on 4P
To add a new module on 4P, click the Create Module button from the list of modules.
Fill in General information
When the page for creating a new module shows up, fill in the general information.
Name: Enter the name of the module. It must be unique, explicit but as short as possible. A Version may be included.
Description: Add an effective description so that macro creators know rapidly what the module does.
Compatibles vectors: Select the vectors which the module is compatible with. All by default.
Compatibles species: Select the species which the module is compatible with. All by default.
Type: The type of module (Calculation or Validation). Almost all modules are calculation modules. Validation modules generates validation reports when a process is complete.
License: If the module has specific runtime requirements (such as a Photoscan license or a GPU capability), use this field to indicate it. This will allow 4P to know the kind of processing node required for executing this module.
Previous version: If the module is a major upgrade of another module, link them together so that the concerned macros can be updated (See page Update modules to newer versions).
Docker image: Enter the name and version (tag) of the Docker image as they are defined in the inra4p.build.groovy file available in the module source code (see section 2 Groovy Script).
Source repository: Enter the URL of the source code repository using the SSH format (ex: git@server-address:namespace/project.git). Only Git repositories can be used for now. If necessary, indicate the name of the branch if different than master (it is also possible to indicate a specific Git tag but it must be an annotated tag)
Technical guide: Select a file or an online documentation to describe the module.
Important
Make sure that the Git repository contains the public SSH key to grant 4P access to it (see part 2 SSH Authentication).
Configure module inputs
Once the general information are set, configure the module inputs (i.e. module input parameters) while honoring the arguments order and potential prefixes used in the module command line.
Example: As an example, let’s consider a module with the following command line:
module-test [--min-max true|false] -i indices-file -e estimations-file output-folder
Inputs are of 2 different types:
existing inputs are data already existing in 4P either from datasets and their metadata or from other modules;
custom inputs are not existing in 4P and are used to configure the module or alter its execution (for example booleans or numeric values).
Add an existing input
Click the button Add an existing input and the list of 4P parameters shows up.
Browse the list of parameters using the filters to refine the list. Then select any parameter by clicking the button Add. If necessary enter a prefix and check the Required box. Set the parameter to the right position using the Move up and Move down buttons.
Example: In our example the indices-file and estimations-file parameters are existing data coming from the Phenoscript - Inversion module and they must be prefixed by -i and -e. These are required inputs.
Note
Existing outputs already have an ID and their type and description are not editable.
Hint
The input Output Folder is always required for any module on the 4P platform.
Add a custom input
Click the button Add a custom input then fill in the fields ID, Label, Description and Type. If necessary enter a prefix, check the Required box and enter a default value.
Hint
The ID of an input is a 4P internal technical code to point to the actual parameter. It must contain no space, nor any special character. We recommend to use the Camel Case naming convention. The ID of a boolean parameter should start with a verb of action such as export, compute, extract, etc.
Example: In our example, the min-max parameter is an optional boolean which doesn’t exist on 4P. We then choose to add a custom input with the prefix --min-max and with a boolean type.
Configure module outputs
Outputs refer to module generated data. Proceed as for the input parameters.
Example: In our example the module generates a file containing statistics data on the indices and estimations inputs. We then add a custom output with a file type.
Conditional parameter: Sometimes an output is generated only when a certain input parameter is present. It is possible to add such a condition by using the conditional parameter that must be one of the boolean input parameters. When it is true the output is generated.
Value: The value field accepts the following special syntaxes in order to refer to input parameters:
${param_id}: this syntax refers to the value of an input parameter (ex:
${outputFolder}/statistics.csv)“*”: the “*” (star) character is a multiplicity mask for the list-type outputs. For example,
${outputFolder}/*.pngrefer to all the png files in the module output folder.
Important
Outputs are saved and referenced in 4P so that they can be used as inputs for other modules. We recommend to carefully define the ID and label of these inputs. IDs must be unique and effective.
Save the module
Save the module configuration by clicking the button Add Module. This triggers the continuous integration workflow and the module details page is displayed, as well as the status of the integration process.
Warning
For some reason, the integration process may fail. In that case, check out the available log file, make any necessary adjustment and restart the integration by clicking the Restart Integration button.
Once integrated the status is updated and the date of the integration is displayed.
Appendices
This section contains additional information for specific use cases.
Appendix A: Manage Module Versions
As it was previously said, Docker images must be specifically tagged for a complete 4P integration. This ensures the results traceability and reproducibility. The tagging operation is set in the groovy file during the Docker image build stage.
There are at least 2 ways to manage module versioning.
Use manual version number
The Docker image tag can be manually entered in the groovy file, using Semantic Versioning. For example, for a 1.0.0 version of the module, simply enter “1.0.0” in the groovy file in the build section:
Warning
In case of changes in the source code the new commits will be included in the Docker image with the same tag and will therefore override the previous version of the image. Make sure to change the Docker image tag in the groovy file if changes have a major impact on the module behavior.
Use Git tags and Git describe
One of the best means to manage module versions in a traceable way is to use Git tags. For example, when a version “1.0.0” is released, simply add a Git tag 1.0.0 on your repository. Then, refer to that tag in the branch / tag field in 4P when creating the module:
Then, in the groovy file make sure to get the tag name using the git describe command:
Note
By using the git describe command, we make sure that the Docker image used by 4P is never overridden in case of new commits. If new commits must be integrated in 4P, simply create and refer to a new tag.