At the beginning, the Apps that are being developed used mainly one Brainlife datatype: meg/fif (see the documentation on datatypes on BL). A datatype enables Apps to communicate with each other: the output of App A is the input of App B (see . Error when running a pipeline with optional files, How to run Apps one after another).
Here we choose to illustrate this datatype mainly with the channels.tsv file, but the principle is the same for the other files in the meg/fif-override datatype.
...
To do so a new datatype was created: the meg/fif-override datatype that will be used when an App doesn’t change the meg.fif but writes a file that will be used later in another App.
...
app-head-pos: it outputs the headshape.pos that will be used in app-bad-channels and app-maxwell-filter
app-mean-transformation-matrix: it outputs the destination.fif that will be used in app-maxwell-filter
app-bad-channels: it outputs the channels.tsv that will be used in app-maxwell-filter
app-get-events: it outputs the events.tsv that will be used in app-make-epochs and app-resampling
| Info |
|---|
|
Files written in meg/fif-override
The files written by the Apps whose output is a meg/fif-override datatype are can also be present in the meg/fif datatype.
For instance, the app-maxwell-filter takes as inputs the meg/fif datatype (for the meg.fif) and also optionnaly optionaly the meg/fif-override datatype. The inputs from the meg/fif-override datatype are optional because we don’t know if an App user will run the whole pipeline on Brainlife (i.e. compute the destination.fif, then the headshape.pos, detect bad channels, and eventually run Maxwell filter), maybe he will want to run only the app-maxwell-filter and he will provide the headshape.pos and channels.tsv without computing them with the BL Apps, and these files will be directly into the data staged under the meg/fif datatype.
But, maybe the user will run the pipeline from the beginning and still provide the headshape.pos and channels.tsv in when the data is staged under the meg/fif datatype. So, for app-maxwell-filter we will have two versions of each file. So, in this case, only the files computed by the BL Apps will be taken into account.
destination.fif is quite specific, we think that it will only be obtained with the app-mean-transformation-matrix, so we won’t have two versions of this file.
In the Python code, we first We deal with this situation while reading the optional files in helper.py (see Create a helper.py file):
first we read the files from the
meg/fifdatatype:
| Code Block |
|---|
# Read channelsdestination file channels_file# = config.pop('channels') channels_file_exists = False if channels_file is not None: We don't copy this file in outdir yet because this file can be given in fif-override # and if os.path.exists(channels_file): channels_file_exists = True df_channels = pd.read_csv(channels_file, sep='\t') we take the fif override file by default # Select bad channels' name bad_channels = df_channels[df_channels["status"] == "bad"]['name'] bad_channels = list(bad_channels.values) # Put channels.tsv bad channels in raw.info['bads'] raw.info['bads'].sort()if 'destination' in config.keys(): destination bad_channels.sort(= config.pop('destination') if destination is # Warning messagenot None: if raw.info['bads'] != bad_channelsos.path.exists(destination) is False: user_warning_message_channelsdestination = f'BadNone channels from the info of your MEG file are different from ' \ f'those in the channels.tsv file. By default, only bad channels from channels.tsv ' \ f'are considered as bad: the info of your MEG file is updated with those channels.' warnings.warn(user_warning_message_channels) dict_json_product['brainlife'].append({'type': 'warning', 'msg': user_warning_message_channels}) raw.info['bads'] = bad_channels |
...
else:
destination = None |
then, we read the files from the
meg/fif-overridedatatype and copy the file:
| Code Block |
|---|
# Read channelsthe file channels_file_override_exists = Falsedestination file if 'channelsdestination_override' in config.keys(): # if the App has no channels_filemeg/fif-override input this key doesn't exist destination_override = config.pop('channelsdestination_override') # No need to test if channelsdestination_override is None, this key is only present when the app runs on BL if os.path.exists(channels_filedestination_override) is False: channels_file_override =if Nonedestination is elsenot None: if channels_file_exists: # If destination from meg/fif exists user_warning_message_channels_file = f"You provided two channels files: by default, the file written by " \but destination_override from meg/fif-override doesn't, # we copy it in out_dir f"the App detecting bad channels will be used." warnings.warn(user_warning_message_channels_file) shutil.copy2(destination, os.path.join(out_dir_name, 'destination.fif')) # required to run a pipeline on BL dict_json_product['brainlife'].append({'type': 'warning', 'msg': user_warning_message_channels_file}) else: channels_file_override_exists = True # If destination_override from meg/fif-override exists, we copy it in out_dir df_channels = pd.read_csv(channels_file_override, sep='\t') # SelectBy baddefault channels'we namecopy the files given in input of bad_channels_override = df_channels[df_channels["status"] == "bad"]['name']meg/fif-override bad_channels_override = list(bad_channels_override.values) # Put channels.tsv bad channels in raw.info['bads'] raw.info['bads'].sort() bad_channels_override.sort() # Warning message if raw.info['bads'] != bad_channels_override:shutil.copy2(destination_override, os.path.join(out_dir_name, 'destination.fif')) # required to run a pipeline on BL destination = user_warning_message_channels_destination_override = f'Bad# channelswe fromoverwrite the infovalue of yourdestination MEG file are different fromelse: ' \ # If the App has no meg/override datatype (or if the App is run locally) if destination is not None: # f'those in the channels.tsvIf destination file. By default, only bad channels from channels.tsv ' \ from meg/fif is not None, we copy it in outdir f'are considered as bad: the info of your MEG file is updated with those channels.' warnings.warn(user_warning_message_channels_override) dict_json_product['brainlife'].append({'type': 'warning', 'msg': user_warning_message_channels_override}) raw.info['bads'] = bad_channels_override |
...
shutil.copy2(destination, os.path.join(out_dir_name, 'destination.fif'))
|
We decided not to inform the App user that the optional file previously computed by an App (so a file from the meg/fif-override datatype exists, we inform the user that we will use it and we overwrite the raw.info['bads'] with this file. So, in the file mapping ) will overwrite the optional file linked to the meg/fif datatype because we supposed that, if the user computes a new file he doesn’t want to use the file he gave at first when the data was staged.
Example for app-maxwell-filter
In the file mapping section (displayed when registering or editing an App on Brainlife) of app-maxwell-filter, we have:
...
For
meg/fifdatatype input:
...
For
meg/fif-overrdedatatype input:
...
| Info |
|---|
The |
In the config.json created by BL we will have two more keys (channels_override and headshape_override) that the keys linked to the meg/fif-override datatype, but these keys won’t be present if the App is run locally because the datatype issue is specific to BL (that’s why we check if the key is present).