| Status | ||||
|---|---|---|---|---|
|
At the beginning, the Apps that are being developed used mainly one Brainlife datatype: meg/fif (see the documentation on datatypes on BL). A datatype enables Apps to communicate with each other: the output of App A is the input of App B (see . Error when running a pipeline with optional files, How to run Apps one after another).
Here we choose to illustrate this datatype mainly with the channels.tsv file, but the principle is the same for the other files in the meg/fif-override datatype.
...
To do so a new datatype was created: the meg/fif-override datatype that will be used when an App doesn’t change the meg.fif but writes a file that will be used later in another App.
...
| Info |
|---|
|
...
The files written by the Apps whose output is a meg/fif-override datatype are can also be present in the meg/fif datatype.
For instance, the app-maxwell-filter takes as inputs the meg/fif datatype (for the meg.fif) and also optionnaly optionaly the meg/fif-override datatype. The inputs from the meg/fif-override datatype are optional because we don’t know if an App user will run the whole pipeline on Brainlife (i.e. compute the destination.fif, then the headshape.pos, detect bad channels, and eventually run Maxwell filter), maybe he will want to run only the app-maxwell-filter and he will provide the headshape.pos and channels.tsv without computing them with the BL Apps, and these files will be directly into the data staged under the meg/fif datatype.
But, maybe the user will run the pipeline from the beginning and still provide the headshape.pos and channels.tsv in when the data is staged under the meg/fif datatype. So, for app-maxwell-filter we will have two versions of each file. So, in this case, only the files computed by the BL Apps will be taken into account.In the Python code, we first
We deal with this situation while reading the optional files in helper.py (see Create a helper.py file):
first we read the files from the
meg/fifdatatype:
| Code Block |
|---|
# Read channelsdestination file channels_file# = config.pop('channels') channels_file_exists = False if channels_file is not None: We don't copy this file in outdir yet because this file can be given in fif-override # and if os.path.exists(channels_file): channels_file_exists = Truewe take the fif override file by default if 'destination' df_channels = pd.read_csv(channels_file, sep='\t')in config.keys(): # Select bad channels' name bad_channels = df_channels[df_channels["status"] == "bad"]['name'] bad_channels = list(bad_channels.valuesdestination = config.pop('destination') if destination is # Put channels.tsv bad channels in raw.info['bads']not None: if raw.info['bads'].sort() bad_channels.sort() # Warning message if raw.info['bads'] != bad_channelsos.path.exists(destination) is False: user_warning_message_channelsdestination = f'Bad channelsNone from the info of your MEG file are different from ' \ f'those in the channels.tsv file. By default, only bad channels from channels.tsv ' \ f'are considered as bad: the info of your MEG file is updated with those channels.' warnings.warn(user_warning_message_channels) dict_json_product['brainlife'].append({'type': 'warning', 'msg': user_warning_message_channels}) raw.info['bads'] = bad_channels |
...
else:
destination = None |
then, we read the files from the
meg/fif-overridedatatype and copy the file:
| Code Block |
|---|
# Read channelsthe file channels_file_override_exists = False destination file if 'channelsdestination_override' in config.keys(): # if the App has no meg/fif-override input this key doesn't exist channels_file destination_override = config.pop('channelsdestination_override') # No need to test if channelsdestination_override is None, this key is only present when the app runs on BL if os.path.exists(channels_filedestination_override) is False: channels_file_override =if Nonedestination is elsenot None: if channels_file_exists: # If destination from meg/fif exists user_warning_message_channels_file = f"You provided two channels files: by default, the file written by " \but destination_override from meg/fif-override doesn't, # we copy it in out_dir f"the App detecting bad channels will be used." warnings.warn(user_warning_message_channels_file) shutil.copy2(destination, os.path.join(out_dir_name, 'destination.fif')) # required to run a pipeline on BL dict_json_product['brainlife'].append({'type': 'warning', 'msg': user_warning_message_channels_file}) else: channels_file_override_exists = True # If destination_override from meg/fif-override exists, we copy it in dfout_channelsdir = pd.read_csv(channels_file_override, sep='\t') # SelectBy baddefault channels'we namecopy the files given in input of meg/fif-override bad_channels_override = df_channels[df_channels["status"] == "bad"]['name'] bad_channelsshutil.copy2(destination_override = list(bad_channels_override.values) # Put channels.tsv bad channels in raw.info['bads'] raw.info['bads'].sort() bad_channels_override.sort() # Warning message if raw.info['bads'] != bad_channels_override:, os.path.join(out_dir_name, 'destination.fif')) # required to run a pipeline on BL destination user_warning_message_channels= destination_override = f'Bad# channelswe fromoverwrite the infovalue of yourdestination MEG file are different fromelse: ' \ # If the App has no meg/override datatype (or if the App is run locally) if destination is not None: # f'those in the channels.tsv file. By default, only bad channels from channels.tsv ' \ If destination file from meg/fif is not None, we copy it in outdir f'are considered as bad: the info of your MEG file is updated with those channels.' warnings.warn(user_warning_message_channels_override) dict_json_product['brainlife'].append({'type': 'warning', 'msg': user_warning_message_channels_override}) raw.info['bads'] = bad_channels_override |
...
shutil.copy2(destination, os.path.join(out_dir_name, 'destination.fif'))
|
We decided not to inform the App user that the optional file previously computed by an App (so a file from the meg/fif-override datatype exists, we inform the user that we will use it and we overwrite the raw.info['bads'] with this file. So, in the file mapping datatype) will overwrite the optional file linked to the meg/fif datatype because we supposed that, if the user computes a new file he doesn’t want to use the file he gave at first when the data was staged.
Example for app-maxwell-filter
In the file mapping section (displayed when registering or editing an App on Brainlife) of app-maxwell-filter, we have:
...
:
For
meg/fifdatatype input:
...
For
meg/fif-overrdedatatype input:
...
| Info |
|---|
The |
In the config.json created by BL we will have two more keys (channels_override and headshape_override) that the keys linked to the meg/fif-override datatype, but these keys won’t be present if the App is run locally because the datatype issue is specific to BL (that’s why we check if the key is present).