Add Raw Data

4P is an application for data processing and data storage. The user can upload in 4P the raw datasets that he wishes to process. To do so, click on the Data section in the main menu on the left, then on the button Load Data.

../_images/add-raw-dataset.png

Hint

The Dashboard page also features a shortcut “Upload raw data” to add a new dataset to 4P.

Note

For the moment, 4P only accepts data acquired by Drone, Phenomobile or PhenoMan (Literal) but the system has been made to welcome any type of data.

Fill in metadata

Enter general information

../_images/load-data-general-information.png

The Load Data button opens a new page asking to indicate a few general information:

  • Vector: Select the type of vector (Drone, Phenomobile or PhenoMan) of the dataset to be loaded.

  • Campaign / Experiments: Select the year of the campaign and the experiment(s) that the dataset is associated with.

  • Measuring session: Enter the date and the time of the measuring session.

  • Datatype: Select the type of data to be loaded.

Caution

Make sure that the PHIS instance associated with the selected installation is up-to-date and defines the experiments related to the current dataset. Sometimes it takes a while to load the available campains and experiments. In such a case simply wait for the list to be available or refresh the page.

Case of users with no PHIS account

Users with no PHIS account can load datasets of any type on 4P and will use experiments shared by all users of the current organization. The “+” button allows to create new experiments as needed.

../_images/load-data-add-experiment.png

Enter type-specific metadata

Once the Datatype is selected (here the Drone RGB Images), the list of type-specific metadata appears:

../_images/load-data-metadata.png

Enter - or select - the correct value for each field. Note that some fields are disabled: they are mainly relative paths of data to be found in the dataset archive to be loaded. Then push the Validate button.

Hint

If a metadata is an editable file (like the GCP coordinates in the example above) you can either:

  • leave the field locked and add the file to the TAR archive respecting the given relative path

  • or use a document from the Catalog (see more details in the section Catalog > Use a Document)

These files will then be editable anytime using the Edit metadata option from the data page.

Caution

Make sure that the PHIS instance associated with the selected installation is up-to-date and defines the vectors, the sensors and/or the radiometric targets related to the current dataset.

Prepare the dataset archive

Once the metadata are filled in and valid, a popup window shows up giving a few guidelines on how to structure the archive file to be loaded.

../_images/load-data-archive.png

The raw data coming from the acquisition must be organized in a TAR archive respecting these guidelines.

Drone dataset archive

../_images/drone-archive-format.png

The archive for a Drone dataset must contain the following elements from the root of the archive:

  • a dataset directory containing one or several sub-directories (one per flight) each containing the acquired images

  • the label-marker-file.csv file containing the Geometric Targets (GCPs) coordinates (in meters and 7 decimals at least in the WGS84 reference datum.)

  • the photoscan_settings.json file (optional) containing specific parameters to alter the Photoscan process

For the specific case of multispectral data, make sure to add the following files:

  • the radio-targets.csv file containing the radio targets coordinates (in meters and 7 decimals at least in the WGS84 reference datum.)

  • the infos.json file containing camera settings, informations of the first trigger (dates, times, GPS coordinates, altitude) and informations of the last trigger (dates, times, number of images recorded)

Hint

Here are downloadable examples of certain files:

Notice that these files can be either added to the archive or selected from the documents Catalog.

Caution

  • Make sure the archive files organization is correct, or processes might not be able to run successfully.

  • Beware of hidden files and folders (such as ‘Thumb.db’ automatically added by Windows in some cases) and remove them before uploading the data to 4P.

  • CSV files must be in standard format, using “;” as separators and “.” as decimal separator.

Phenomobile / PhenoMan dataset archive

../_images/pheno-archive-format.png

The archive for a Phenomobile dataset (same as for PhenoMan LITERAL dataset) must contain the following elements from the root of the archive:

  • One or several uplot_X_X folder(s) containing the *.h5 file as well as preview images if any

  • Optional additional files regarding the acquisition mission such as

    • a file microplot_states.txt

    • a file missionId.txt

    • a file mission.zip

    • roslogs.csv

    • etc.

Caution

  • Make sure the archive files organization is correct, or processes might not be able to run successfully.

  • Make sure that the Experiment(s) and the Measuring session provided are consistent with the HDF5 files information.

Upload data to 4P

Finally, select the TAR archive containing the dataset and push the button Load on 4P to start the transfer process.

../_images/drone-data-loading.png

Caution

The loading process may take a while, depending on the dataset size, so beware not to close the loading window before the transfer is complete.

Note

4P loads a dataset in 2 stages:

  • First the archive is loaded on the 4P drive. The content of the archive is then checked according to the metadata.

  • Second, if the archive is valid, it is sent to its permanent storage drive on iRODS.

When the loading is complete, the new dataset appears in the list of raw datasets, and if valid it is possible to run a process on it.

../_images/drone-data-new.png

Troubleshooting

Upload stopped before completion

For some reason, it may happen that a transfer is interrupted before being completed. If that happens it is possible to resume the transfer.

To do so, go to the list of Data and click the dataset being uploaded (it should still be in running state).

../_images/raw-data-uncomplete.png

Unfold the Task History box, and click the button Resume data transfer for the loading task.

../_images/raw-dataset-uncomplete-resume-button.png

In the upload dialog that shows up, just select the same files and click the Load on 4P button. After an archive checking, the transfer will resume from where it stopped.

Validation failed

When loading an archive with the wrong files structure, validation errors might be raised and the dataset will be considered invalid. Invalid datasets are highlighted in the Data list:

../_images/invalid-data-list.png

First of all, determine why the validation has failed by checking the logs in the loading task details page (to access this page, open the dataset then in the task history open the task detail page).

../_images/raw-dataset-unvalid-log.png

Fix the errors in the archive files structure and upload a new archive.

Note

It is not possible to upload a fixed archive for a invalid dataset for now. Delete the invalid dataset and load a new one with the fixed archive.