Process Data

In addition to storing datasets, 4P main purpose is to process raw and complex data (images, binary files, etc.) into readable and interpretable data (biophysical and biological variables).

To process data, 4P hosts a library of computation modules developed by data processing specialists. These modules are chained in processing workflows called macros. For example, 4P contains macros for:

  • Extracting Phenomobile data

  • Pre-processing Drone or Phenomobile data

  • Computing biological variables from raw or pre-processed Drone and Phenomobile data

Note

This section describes how to use existing macros to process datasets. Please refer to the Macros and Modules sections of the current guide, to know more about how to create computation workflows.

Start a process

4P offers several entrypoints to process data:

  • from the Home page using the Run a process shortcut

  • from a Macro details page using the Process dataset toolbox button

  • from a specific Dataset details page using the Process dataset button (see the image below)

  • and from a specific data item details page using the Process data button.

../_images/process-data-button.png

This opens the Process data page on which the process will be configured.

../_images/icon-select-document.png

First, select the macro to be executed by clicking the button Select a macro. The list of available valid macros shows up. Then click the Select button of the chosen macro.

../_images/process-data-page.png

Then, select the input dataset to be processed by clicking the button Select a dataset. The list of valid datasets shows up. Then click the Select button of the chosen dataset.

../_images/icon-clear-document.png

You may click this button to clear the fields from the seleted item.

Note

The Macro selector automatically lists the macros that are compatible with the selected input dataset. Similarly, the Input dataset selector automatically lists the datasets that are compatible with the selected macro.

Once the Macro and Input dataset are selected, a review of the input and output data types is displayed.

../_images/process-data-page-selected.png

Fill up the input process parameters if any, then click the Process data button to start the process. If all parameters are correct, the process is started and the user is directed back to the processing detail page.

Note

Process parameters can be numeric values, string values, or files. To select a file click the Select button to open the documents catalog, then choose the desired file and version.

Monitor a process status

The process status can be monitored from the process detail page accessible from the Task History box of the current dataset. The user is automatically directed to this page after starting a process.

../_images/process-data-status-running.png

The box Description displays a few information to monitor the process:

  • Macro: The name and link toward the macro that is used for this process

  • Input data: The names and links toward data used as process inputs

  • Output data: The names and links toward data generated by the current process

  • Status / Progress: Shows the current status of the process (Waiting, Running, Complete, Failed or Cancelled) as well as the progression in steps.

  • Steps: List of steps composing the process workflow and their status

  • Also Start Date, Duration and Author

When a process is running, use the button Stop to abort the process. If a process has failed, use the Restart button to restart it from the beginning, or the Resume process to restart it from where it’s stopped.

In addition, the logs provide useful information to follow the process progression more closely or to investigate potential issues.

Note

It is possible to monitor all the processes currently running on the platform in the section Monitoring.

Report a process issue

Process failures may occur due to macro or module issues, or because of an internal error in 4P. If this happens, use the button Report a bug and fill the form to report an issue linked to that process. Platform administrators will consider the request rapidly.

../_images/process-data-report-issue.png

Process temporary data destruction

Because the capacity of the 4P runtime disk is limited, data of failed processes are automatically destroyed after 21 days of inactivity. In such cases the following indicator, showing the number of days left before destruction, is displayed in the process description box:

../_images/process-destroyed-in.png

Hint

To prevent process data to be destroyed, resume or restart the process.