TO UPDATE
At the beginning, the Apps that are being developed used mainly one Brainlife datatype: meg/fif (see the documentation on datatypes on BL). A datatype enables Apps to communicate with each other: the output of App A is the input of App B (see .Error when running a pipeline with optional files).
Here we choose to illustrate this datatype mainly with the channels.tsv file, but the principle is the same for the other files in the meg/fif-override datatype.
Why the meg/fif-override datatype was created
At first the app-bad-channels takes as input the meg/fif datatype and its output was also this datatype. In this datatype, the meg.fif must be present and so at the end of app-bad-channel, the meg.fif was saved in out_dir_bad_channels even if the only difference was its field raw.info['bads'] (the signals were the same between the input and output). This was a problem as described here Datatype meeting 22.04.2021, so we decided that the App will only returned a channels.tsv (see Où stocker l'information "bad channels" and How to create a BIDS compliant channels.tsv and events.tsv files).
To do so a new datatype was created: the meg/fif-override datatype that will be used when an App writes a file that will be used later in another App.
Examples where this datatype is used
Not all Apps modify the MEG signals, some Apps just write files that will be used in other Apps.
The App that writes those files are () :
app-head-pos: it outputs the headshape.pos that will be used in app-bad-channels and app-maxwell-filter
app-mean-transformation-matrix: it outputs the destination.fif that will be used in app-maxwell-filter
app-bad-channels: it outputs the channels.tsv that will be used in app-maxwell-filter
app-get-events: it outputs the events.tsv that will be used in app-make-epochs and app-resampling
events.tsv and channels.tsv are BIDS compliant, see How to create a BIDS compliant channels.tsv and events.tsv files.
An updated events.tsv is also returned by app-resampling when this app is applied to continuous data, and when an events.tsv is given in input. But the output of this app has not the
meg/fif-overridedatatype because it returns also a meg.fif.
Files written in meg/fif-override
The files written by the Apps whose output is a meg/fif-override datatype are also present in the meg/fif datatype.
For instance, the app-maxwell-filter takes as inputs the meg/fif datatype (for the meg.fif) and also optionnaly the meg/fif-override datatype. The inputs from the meg/fif-override datatype are optional because we don’t know if an App user will run the whole pipeline on Brainlife (i.e. compute the destination.fif, then the headshape.pos, detect bad channels, and eventually run Maxwell filter), maybe he will want to run only the app-maxwell-filter and he will provide the headshape.pos and channels.tsv without computing them with the BL Apps, and these files will be directly into the meg/fif datatype.
But, maybe the user will run the pipeline from the beginning and still provide the headshape.pos and channels.tsv in the meg/fif datatype. So, for app-maxwell-filter we will have two versions of each file. So, in this case, only the files computed by the BL Apps will be taken into account.
In the Python code, we first read the files from the meg/fif datatype:
# Read channels file
channels_file = config.pop('channels')
channels_file_exists = False
if channels_file is not None:
if os.path.exists(channels_file):
channels_file_exists = True
df_channels = pd.read_csv(channels_file, sep='\t')
# Select bad channels' name
bad_channels = df_channels[df_channels["status"] == "bad"]['name']
bad_channels = list(bad_channels.values)
# Put channels.tsv bad channels in raw.info['bads']
raw.info['bads'].sort()
bad_channels.sort()
# Warning message
if raw.info['bads'] != bad_channels:
user_warning_message_channels = f'Bad channels from the info of your MEG file are different from ' \
f'those in the channels.tsv file. By default, only bad channels from channels.tsv ' \
f'are considered as bad: the info of your MEG file is updated with those channels.'
warnings.warn(user_warning_message_channels)
dict_json_product['brainlife'].append({'type': 'warning', 'msg': user_warning_message_channels})
raw.info['bads'] = bad_channels
and then, we read the files from the meg/fif-override datatype:
# Read channels file
channels_file_override_exists = False
if 'channels_override' in config.keys():
channels_file_override = config.pop('channels_override')
# No need to test if channels_override, this key is only present when the app runs on BL
if os.path.exists(channels_file_override) is False:
channels_file_override = None
else:
if channels_file_exists:
user_warning_message_channels_file = f"You provided two channels files: by default, the file written by " \
f"the App detecting bad channels will be used."
warnings.warn(user_warning_message_channels_file)
dict_json_product['brainlife'].append({'type': 'warning', 'msg': user_warning_message_channels_file})
channels_file_override_exists = True
df_channels = pd.read_csv(channels_file_override, sep='\t')
# Select bad channels' name
bad_channels_override = df_channels[df_channels["status"] == "bad"]['name']
bad_channels_override = list(bad_channels_override.values)
# Put channels.tsv bad channels in raw.info['bads']
raw.info['bads'].sort()
bad_channels_override.sort()
# Warning message
if raw.info['bads'] != bad_channels_override:
user_warning_message_channels_override = f'Bad channels from the info of your MEG file are different from ' \
f'those in the channels.tsv file. By default, only bad channels from channels.tsv ' \
f'are considered as bad: the info of your MEG file is updated with those channels.'
warnings.warn(user_warning_message_channels_override)
dict_json_product['brainlife'].append({'type': 'warning', 'msg': user_warning_message_channels_override})
raw.info['bads'] = bad_channels_override
If channels.tsv from the meg/fif-override datatype exists, we inform the user that we will use it and we overwrite the raw.info['bads'] with this file.
So, in the file mapping of app-maxwell-filter, we have:
In the config.json created by BL we will have two more keys (channels_override and headshape_override) that won’t be present if the App is run locally.
Add Comment