How to run Apps one after another

Since we decided to create small Apps corresponding to some MEEG processing steps instead of an App that encompasses all these steps (see the Apps created by Guiomar Niso based on mne-bids-template), we have to make sure that these small Apps can produce a good result by their own but also that they can be run in a pipeline of Apps (see Description of existing Apps).

Apps communicate thanks to datatypes: the output of app-A that has the datatype-1 can be the input of app-B if its datatype is datatype-1. This Brainlife specificity can create some issues (see Error when running a pipeline with optional files) and has to be taken into account when developing apps (see Create a helper.py file).

Steps to run a pipeline

The steps are pretty much the same as when you run a single App (see How to run an App on Brainlife):

First the data you want to use must be staged
Submit the first App: you will be proposed to apply Apps that take as input the datatype corresponding to the datatype of the data you have just staged
Submit the second App: you will be proposed to apply Apps that take as input the datatype corresponding to the datatype of the data you have staged and Apps that take as input the datatype corresponding to the datatype of the output data of the first App
an so on

Example with the existing Apps

We want to apply Maxwell filter on our MEG signals. But first we want to compute the destination file and detect bad channels, so we will run two different apps to get these files before applying Maxwell filter:

1.The data is staged

2. Destination file is computed thanks to app-mean-transformation-matrix

This App takes two meg files in input (datatype: meg/fif) and outputs a destination.fif file (datatype: meg/fif-outdir) that will be used later in the app-maxwell-filter, so it is important that this file is given to the next App (app-bad-channels, which is run before app-maxwell-filter) even if this App has no use of this file.

The output file is saved in out_dir folder. The next app will search for its input files in this folder and if the datatype is correct (i.e. output datatype of app-mean-transformation-matrix is one of the input datatype of app-bad-channels) the file will be used.

The name of the output file must correspond to the name of this file defined in the datatype: in meg/fif-override the file is called destination.fif, so app-mean-transformation-matrix must save it under that name, so that the next App with meg/fif-override input can find it. If the name is not right the FileNotFoundError occurs.

The name to use to save the output files of your App are specified in the file mapping section when you register your App on Brainlife.

3. Write bad channels in channels.tsv thanks to app-bad-channels

As we can see, the app-bad-channels takes input files from two datatypes (see How to use the meg/fif-override datatype):

the meg.fif and the optional files from the datatype meg/fif
optionally the files from the meg/fif-override datatype

In this example, we have run the app-mean-transformation-matrix beforehand, so we put the result of this app in input for the meg/fif-override datatype (even if the destination.fif is not used by this App because we need this file to be an output of this App to be used in the next App).

We have to do this because the output of app-bad-channels is also a meg/fif-override datatype: when we will enter the inputs for app-maxwell-filter (see next section), for the datatype meg/fif-override input we will have to choose between the outputs of app-mean-transformation-matrix and app-bad-channels. If we choose the former, we will have only the destination.fif, and if we choose the latter, we will have both destination.fif and channels.tsv, because we copy the destination.fif into the output directory of app-bad-channels (see Error when running a pipeline with optional files).

For the input of the meg/fif datatype, we select one of the meg.fif staged at the first step.

We run the app-bad-channels: its outputs are files from the meg/fif-override datatype and are saved in out_dir_channels

As we can see, even if destination.fif was not used in this App, it is saved in out_dir_bad_channels.

Besides in this folder we have also files that corresponds only to meg/fif datatype ( calibration_meg.dat and crosstalk_meg.fif): usually an output directory contains files from one datatype, but since meg/fif and meg/fif_override are very similar datatypes, we decided that they can share the same output directory (besides this App has only the meg/fif-override as output).

Jun 17, 2021

Maybe it will be cleaner not to save these files in the output directory of app-bad-channels. This step is performed in the helper.py (see Create a helper.py file), so maybe it can be udpated to save only meg/fif-override datatype’s file in this output directory.

4. Apply app-maxwell-filter

The app-maxwell-filter takes as input files from meg/fif and meg/fif-override datatypes.

In the meg/fif datatype, corresponding to data staged at step 1, we have the meg.fif file but also optional files like calibration and cross-talk files. In the meg/fif-override datatatype, we have the destination.fif and channels.tsv.

We run the app-maxwell-filter and in the output directory (corresponding to meg/fif datatype) we have:

Even if we won’t need the files calibration_meg.dat, crosstalk_meg.fif, and destination.fif in the next Apps we will run in this pipeline (such as app-temporal-filtering for instance), they are still present in the output directory (it is not an issue because they are small files).