How to use the meg/fif-override datatype

At the beginning, the Apps that are being developed used mainly one Brainlife datatype: meg/fif (see the documentation on datatypes on BL). A datatype enables Apps to communicate with each other: the output of App A is the input of App B (see Error when running a pipeline with optional files, How to run Apps one after another).

Here we choose to illustrate this datatype mainly with the channels.tsv file, but the principle is the same for the other files in the meg/fif-override datatype.

Why the meg/fif-override datatype was created

At first the app-bad-channels takes as input the meg/fif datatype and its output was also this datatype. In this datatype, the meg.fif must be present and so at the end of app-bad-channel, the meg.fif was saved in out_dir_bad_channels even if the only difference was its field raw.info['bads'] (the signals were the same between the input and output). This was a problem as described here Datatype meeting 22.04.2021, so we decided that the App will only returned a channels.tsv (see Où stocker l'information "bad channels" and How to create a BIDS compliant channels.tsv and events.tsv files).

To do so a new datatype was created: the meg/fif-override datatype that will be used when an App doesn’t change the meg.fif but writes a file that will be used later in another App.

Examples where this datatype is used

Not all Apps modify the MEG signals, some Apps just write files that will be used in other Apps.

The App that writes those files are (May 21, 2021) :

app-head-pos: it outputs the headshape.pos that will be used in app-bad-channels and app-maxwell-filter
app-mean-transformation-matrix: it outputs the destination.fif that will be used in app-maxwell-filter
app-bad-channels: it outputs the channels.tsv that will be used in app-maxwell-filter
app-get-events: it outputs the events.tsv that will be used in app-make-epochs and app-resampling

events.tsv and channels.tsv are BIDS compliant, see How to create a BIDS compliant channels.tsv and events.tsv files.
An updated events.tsv is also returned by app-resampling when this app is applied to continuous data and when an events.tsv is given in input. But the output of this app has not the meg/fif-override datatype because it returns also a meg.fif.

Files written in meg/fif-override

The files written by the Apps whose output is a meg/fif-override datatype can also be present in the meg/fif datatype.

For instance, the app-maxwell-filter takes as inputs the meg/fif datatype (for the meg.fif) and also optionaly the meg/fif-override datatype. The inputs from the meg/fif-override datatype are optional because we don’t know if an App user will run the whole pipeline on Brainlife (i.e. compute the destination.fif, then the headshape.pos, detect bad channels, and eventually run Maxwell filter), maybe he will want to run only the app-maxwell-filter and he will provide the headshape.pos and channels.tsv without computing them with the BL Apps, and these files will be directly into the data staged under the meg/fif datatype.

But, maybe the user will run the pipeline from the beginning and still provide the headshape.pos and channels.tsv when the data is staged under the meg/fif datatype. So, for app-maxwell-filter we will have two versions of each file. So, in this case, only the files computed by the BL Apps will be taken into account.

We deal with this situation while reading the optional files in helper.py (see Create a helper.py file):

first we read the files from the meg/fif datatype:

    # Read destination file
    # We don't copy this file in outdir yet because this file can be given in fif-override 
    # and we take the fif override file by default
    if 'destination' in config.keys():
        destination = config.pop('destination')
        if destination is not None:
            if os.path.exists(destination) is False:
                destination = None
    else:
        destination = None

then, we read the files from the meg/fif-override datatype and copy the file:

    # Read the destination file
    if 'destination_override' in config.keys():  # if the App has no meg/fif-override input this key doesn't exist
        destination_override = config.pop('destination_override')
        # No need to test if destination_override is None, this key is only present when the app runs on BL
        if os.path.exists(destination_override) is False: 
            if destination is not None:
                # If destination from meg/fif exists but destination_override from meg/fif-override doesn't,
                # we copy it in out_dir 
                shutil.copy2(destination, os.path.join(out_dir_name, 'destination.fif'))  # required to run a pipeline on BL
        else:
            # If destination_override from meg/fif-override exists, we copy it in out_dir
            # By default we copy the files given in input of meg/fif-override 
            shutil.copy2(destination_override, os.path.join(out_dir_name, 'destination.fif'))  # required to run a pipeline on BL 
            destination = destination_override  # we overwrite the value of destination
    else: 
        # If the App has no meg/override datatype (or if the App is run locally) 
        if destination is not None:
            # If destination file from meg/fif is not None, we copy it in outdir
            shutil.copy2(destination, os.path.join(out_dir_name, 'destination.fif'))

We decided not to inform the App user that the optional file previously computed by an App (so a file from the meg/fif-override datatype) will overwrite the optional file linked to the meg/fif datatype because we supposed that, if the user computes a new file he doesn’t want to use the file he gave at first when the data was staged.

Example for app-maxwell-filter

In the file mapping section (displayed when registering or editing an App on Brainlife) of app-maxwell-filter, we have:

For meg/fif datatype input:

For meg/fif-overrde datatype input:

The config.json keys for the meg/fif-override datatype don’t correspond to the name of the files they point to: it’s not an issue henceforth the file is saved with the good name when the App is run (shutil.copy2(destination_override, os.path.join(out_dir_name, 'destination.fif')), see How to run Apps one after another)

In the config.json created by BL we will have the keys linked to the meg/fif-override datatype, but these keys won’t be present if the App is run locally because the datatype issue is specific to BL (that’s why we check if the key is present).