commit 18213fd56a4f37707e722618ab3960e42e24e0b6
Author: Étienne Simon <esimon@esimon.eu>
Date: Sat, 26 Mar 2022 17:16:59 +0100
Initial commit
Diffstat:
66 files changed, 5833 insertions(+), 0 deletions(-)
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1,19 @@
+*.png
+*.zip
+create_training_data.py
+*.txt
+__pycache__
+*.pyc
+.pytype
+*.pkl
+.ipynb_checkpoints
+save
+*stderr
+*stdout
+*.bin
+*.json
+*.jsonl
+*.zip
+models/
+images
+data
diff --git a/.gitmodules b/.gitmodules
@@ -0,0 +1,3 @@
+[submodule "OFA"]
+ path = OFA
+ url = https://github.com/OFA-Sys/OFA
diff --git a/CONFIG.md b/CONFIG.md
@@ -0,0 +1,102 @@
+# Configuration
+
+## How-to Use
+
+Train the model described in the `configs/baseline.py` file:
+```
+python -m mmsrl.train configs/baseline.py
+```
+
+Same but changing the value of the `batch_size` parameter:
+```
+python -m mmsrl.train configs/baseline.py --batch_size=42
+```
+
+When a command line argument starts with `--`, the right hand side of the `=` is `eval()`uated, to set a string parameter, the quotes should be escaped:
+```
+python -m mmsrl.train configs/baseline.py --dataname=\"covid19\"
+```
+
+Indeed, the right hand side is first evaluated in the context of the config dictionary, and can be resolved if it refers to a variable, in the following example, the value of modelname is set to the value of the config variable dataname (which is set to `all` in `configs/baseline.py`).
+```
+python -m mmsrl.train configs/baseline.py --modelname=dataname
+```
+
+This allows for some on-the-fly manipulation of config variables:
+```
+python -m mmsrl.train configs/ofa_vqa.py '--ofa_path=ofa_path.replace("vqa_large_best", "ofa_base")'
+```
+
+However, if the right hand side of `=` can't be `eval()`uated, it is cast to a string. In the following example `modelname` is set to `"datanamex"`. (This feature was asked for by Syrielle, blame her for the confusion.)
+```
+python -m mmsrl.train configs/baseline.py --modelname=datanamex
+```
+
+
+## Available Config Parameters
+
+- `dataname` Choose dataset, either 'uspolitics' or 'covid19', or 'all' to use both.
+- `model` Which model to use: ofa or perceiver or baseline.
+- `multilabel` Multi-label classification or not.
+- `output_val` Path to the validation output file.
+- `output_train` Path to the train output file.
+- `output_test` Path to the test output file.
+- `save_model` Save best model there.
+
+### Multimodel
+- `unfreeze` Part of the model to train (supported: "classifier", "none", "all")
+
+### OFA parameters
+- `ofa_task` Task for ofa: vqa_gen or snli_ve.
+- `ofa_path` Path to OFA model checkpoint.
+- `ofa_classification_head` Use a classification on the model's output instead of recasting the MMSRL problem as VQA or SNLI.
+- `vqa_input` Input to the OFA encoder for the VQA task (either "question ocr", "ocr question", or "question").
+- `only_score_answer` The prediction is only made by looking at the likelihood of answer tokens, not tokens part of the question ocr, etc.
+
+#### OFA SNLI parameters
+The output of the SNLI model for an entity looks like the following row-normalized table:
+
+| | no | maybe | yes |
+| :--- | :---: | :---: | :---: |
+| hero | A | B | C |
+| victim | A | B | C |
+| villain | A | B | C |
+| *(other)* | *a* | *b* | *c* |
+
+The output must be of size 4: one score for each of hero, victim, villain and other.
+The conversion of the above table into this prediction on size 4 is parametrized by the following options:
+
+- `maybe_is` Whether to group `maybe` with `yes`, `no` (or ignore it).
+- `neither_for_other` Whether a prompt is generated for *other*, if this is false the "other" class is the sum of the `no` column(s), otherwise a prompt "entity is neither a hero, villain nor victim" is used.
+
+When `neither_for_other` is `True`, the prediction is simply the column C (or B+C if `maybe_is="yes"`).
+Otherwise, when the last row does not exist, the predictions for hero, victim and villain remain the same, but the prediction for other is set to the sum of unsused cells.
+
+### Baseline model parameters
+- `modelname` Which transformers model to use.
+- `hidden_size` Size of the hidden layer of the MLP for the baseline model.
+- `dropout` Dropout of the transformers' output, before the MLP.
+
+### Optimizer parameters
+- `learning_rate` Learning rate.
+- `weight_decay` Weight decay.
+- `max_grad_norm` Max gradient normalization.
+- `subsample_other` Subsampling of the others class to counter the large class imbalance.
+- `subsample_labels` Same but per-label.
+- `class_weights` Class weights for loss computation. Doesn't work because the class imbalance is too big. Because when we have a class with low frequency, the loss gets huge and the gradient too.
+
+### Trainer parameters
+- `batch_size` Model batch size.
+- `accumulation` Nb of accumulation steps.
+- `batch_per_epoch` Batch per epoch.
+- `no_initial_valid` Do not validate before training.
+- `max_epoch` Max epoch.
+- `patience` If the dev score worsen 3 epoch in a row, stop the training.
+- `amp` Automatic Mixed Precision training.
+- `workers` Number of workers used to generate data.
+
+### Dataset parameters (Numbers chosen from exploration of the datasets)
+- `max_text_len` Max text length (nb of tokens). (observed: 321)
+- `max_nb_ent` Max nb of entities per image, for padding. (observed: 29)
+- `max_ent_len` Max entity name length (nb of tokens).
+- `image_max_edge` Resize images such that the longest edge has this length.
diff --git a/OFA b/OFA
@@ -0,0 +1 @@
+Subproject commit 32e457c8a98b411c77ff0acf7ab4738291f7a5be
diff --git a/README.md b/README.md
@@ -0,0 +1,79 @@
+# Multimodal-semantic-role-labeling
+
+https://codalab.lisn.upsaclay.fr/competitions/906
+
+## How-to
+
+Export the `DATA_PATH` variable to the location where you want the dataset stored:
+```
+$ export DATA_PATH="$PWD/../data"
+```
+
+Run the dataset preparation script (the data will be downloaded if not already present):
+```
+$ ./prepare_data.sh
+```
+
+Run the baseline and create a `predictions.pkl` files with the probabilities on the validation set:
+```
+$ python -m mmsrl.train configs/baseline.py --output_val=predictions.pkl
+```
+
+Show the content `predictions.pkl`:
+```
+$ python -m mmsrl.show_predictions predictions.pkl
+```
+
+Create a new ensemble prediction in the current directory `.` from directories containing several prediction files (they should contain `val` and `test` in their name):
+```
+$ python -m mmsrl.ensembling . path/to/directory1 path/to/directory2 …
+```
+
+Create a `.zip` containing a `.jsonl` for submission on Codalab val dataset:
+```
+$ python -m mmsrl.submission submission.zip predictions.pkl
+```
+
+To run using ipython:
+```
+ipython --pdb -c "%run -m mmsrl.train -- configs/ofa_vqa.py --learning_rate=1e-5"
+```
+
+See `CONFIG.md` for more details on the handling of hyperparameters.
+
+## Preparing environment
+```
+pip install -r requirements
+```
+
+Install `CLIP`:
+```
+pip install git+https://github.com/openai/CLIP.git
+```
+
+Install `fairseq`:
+```
+git clone https://github.com/pytorch/fairseq.git
+cd fairseq
+pip install --use-feature=in-tree-build .
+```
+
+Use `OFA` (from the repository root) and get checkpoints:
+```
+git submodule update --init --recursive
+mkdir OFA/checkpoints
+cd OFA/checkpoints
+wget https://ofa-beijing.oss-cn-beijing.aliyuncs.com/checkpoints/ofa_base.pt
+wget https://ofa-beijing.oss-cn-beijing.aliyuncs.com/checkpoints/vqa_large_best.pt
+wget https://ofa-beijing.oss-cn-beijing.aliyuncs.com/checkpoints/snli_ve_large_best.pt
+```
+
+
+## Extracting features
+
+### Image features:
+python mmsrl/generate_image_features.py --output_folder /data/mmsrl/all/features --image_dir /data/mmsrl/all/images/ --modelname vgg
+python mmsrl/generate_image_features.py --output_folder/data/mmsrl/all/features --image_dir /data/mmsrl/all/images/ --modelname b7
+
+### Generating captions:
+python -m mmsrl.generate_captions.py
+\ No newline at end of file
diff --git a/annotations/annotations of A b/annotations/annotations of A
@@ -0,0 +1,100 @@
+covid_memes_3977.png extroverts 2
+memes_2046.png donald trump 1
+memes_988.png barack obama 0
+covid_memes_2476.png germany 2
+memes_1716.png donald trump 1
+memes_317.png chris wallace 1
+covid_memes_2973.png china 1
+memes_6576.png liberals 1
+memes_3731.png gary johnson 0
+covid_memes_934.png toilet paper 3
+memes_2764.png joe biden 2
+memes_1705.png donald trump 1
+covid_memes_2743.png work from home 1
+covid_memes_2356.png donald trump 1
+covid_memes_3081.png isarelis 3
+covid_memes_1768.png coronavirus memes 3
+memes_6880.png american flag 0
+covid_memes_3286.png corona beer 0
+memes_1224.png economy 2
+covid_memes_2310.png doanld trump 1
+memes_266.png college students 2
+memes_8245.png mexicans 1
+memes_954.png iron man 0
+memes_4336.png republicans 1
+memes_5071.png republican party 1
+covid_memes_3907.png centers for disease control and prevention 1
+memes_4828.png ? 3
+covid_memes_5678.png spooktober memes 3
+memes_4796.png donald trump 1
+memes_1381.png hiliary clinton 1
+covid_memes_2128.png people who invented the covid19 vaccine 2
+memes_5094.png republicans 1
+memes_4291.png donald trump 0
+covid_memes_265.png stores in china 2
+memes_4521.png terry 1
+memes_2713.png joe biden 2
+memes_1991.png sweden 0
+memes_6600.png electoral college 3
+memes_1047.png barack obama 3
+memes_5969.png bernie sanders 0
+memes_1382.png bernie sanders 0
+memes_3840.png 3
+covid_memes_1438.png homeschool teachers 3
+memes_3598.png democrats 0
+memes_1300.png time 1
+memes_1151.png democratic party 3
+memes_8046.png donald trump 1
+memes_843.png barack obama 0
+covid_memes_1014.png inventor of plastic 1
+memes_1702.png joe biden 0
+memes_5850.png american 1
+memes_4989.png democrats 0
+covid_memes_2528.png america 3
+memes_5882.png barack obama 1
+covid_memes_5664.png people 3
+memes_4402.png ted cruz 3
+memes_4768.png donald trump 1
+covid_memes_470.png coronavirus 1
+memes_8281.png donald trump 1
+covid_memes_5480.png america 2
+memes_4952.png democrats 0
+memes_8283.png illegals 1
+covid_memes_3094.png xi jinping 1
+memes_1979.png scott morrison 2
+covid_memes_4788.png corona 1
+covid_memes_3209.png china 1
+memes_1468.png ben shapiro 1
+memes_3497.png gary johnson 0
+covid_memes_3334.png chinese 1
+memes_5658.png hiliary clinton 3
+memes_4399.png presidential debates 3
+covid_memes_4293.png corona beer 0
+covid_memes_4193.png lysol 0
+memes_1858.png green party 0
+covid_memes_829.png donald trump 2
+memes_1525.png homicide 1
+covid_memes_252.png thais 3
+memes_5098.png grand old party (gop) 3
+memes_1629.png antifa 0
+covid_memes_2700.png work from home 3
+memes_3675.png libertarians 1
+covid_memes_1818.png introvert 0
+covid_memes_775.png democrats 1
+memes_912.png barack obama 3
+memes_3335.png donald trump 1
+memes_175.png prison 3
+memes_5693.png hillary clinton 1
+memes_6743.png minors 3
+covid_memes_4193.png purell 3
+memes_2623.png joe biden 2
+memes_8119.png illegal immigrants 1
+covid_memes_797.png introverts 3
+memes_53.png green party 2
+covid_memes_349.png spanish people 2
+covid_memes_4865.png pilots 2
+covid_memes_1678.png spirit 3
+memes_5922.png women 2
+memes_826.png economy 3
+memes_4662.png donald trump 1
+memes_1647.png america 3
diff --git a/annotations/annotations of D b/annotations/annotations of D
@@ -0,0 +1,100 @@
+covid_memes_3977.png extroverts 3
+memes_2046.png donald trump 3
+memes_988.png barack obama 0
+covid_memes_2476.png germany 1
+memes_1716.png donald trump 1
+memes_317.png chris wallace 2
+covid_memes_2973.png china 1
+memes_6576.png liberals 1
+memes_3731.png gary johnson 0
+covid_memes_934.png toilet paper 3
+memes_2764.png joe biden 2
+memes_1705.png donald trump 2
+covid_memes_2743.png work from home 3
+covid_memes_2356.png donald trump 1
+covid_memes_3081.png isarelis 2
+covid_memes_1768.png coronavirus memes 3
+memes_6880.png american flag 3
+covid_memes_3286.png corona beer 3
+memes_1224.png economy 2
+covid_memes_2310.png doanld trump 0
+memes_266.png college students 2
+memes_8245.png mexicans 1
+memes_954.png iron man 0
+memes_4336.png republicans 3
+memes_5071.png republican party 1
+covid_memes_3907.png centers for disease control and prevention 1
+memes_4828.png ? 3
+covid_memes_5678.png spooktober memes 3
+memes_4796.png donald trump 1
+memes_1381.png hiliary clinton 1
+covid_memes_2128.png people who invented the covid19 vaccine 2
+memes_5094.png republicans 1
+memes_4291.png donald trump 1
+covid_memes_265.png stores in china 2
+memes_4521.png terry 1
+memes_2713.png joe biden 0
+memes_1991.png sweden 0
+memes_6600.png electoral college 3
+memes_1047.png barack obama 0
+memes_5969.png bernie sanders 0
+memes_1382.png bernie sanders 0
+memes_3840.png 3
+covid_memes_1438.png homeschool teachers 0
+memes_3598.png democrats 1
+memes_1300.png time 3
+memes_1151.png democratic party 1
+memes_8046.png donald trump 1
+memes_843.png barack obama 0
+covid_memes_1014.png inventor of plastic 1
+memes_1702.png joe biden 0
+memes_5850.png american 1
+memes_4989.png democrats 0
+covid_memes_2528.png america 2
+memes_5882.png barack obama 3
+covid_memes_5664.png people 2
+memes_4402.png ted cruz 3
+memes_4768.png donald trump 1
+covid_memes_470.png coronavirus 1
+memes_8281.png donald trump 2
+covid_memes_5480.png america 2
+memes_4952.png democrats 0
+memes_8283.png illegals 2
+covid_memes_3094.png xi jinping 1
+memes_1979.png scott morrison 1
+covid_memes_4788.png corona 3
+covid_memes_3209.png china 2
+memes_1468.png ben shapiro 1
+memes_3497.png gary johnson 0
+covid_memes_3334.png chinese 1
+memes_5658.png hiliary clinton 0
+memes_4399.png presidential debates 3
+covid_memes_4293.png corona beer 3
+covid_memes_4193.png lysol 3
+memes_1858.png green party 0
+covid_memes_829.png donald trump 2
+memes_1525.png homicide 3
+covid_memes_252.png thais 2
+memes_5098.png grand old party (gop) 1
+memes_1629.png antifa 1
+covid_memes_2700.png work from home 3
+memes_3675.png libertarians 1
+covid_memes_1818.png introvert 1
+covid_memes_775.png democrats 1
+memes_912.png barack obama 0
+memes_3335.png donald trump 1
+memes_175.png prison 3
+memes_5693.png hillary clinton 1
+memes_6743.png minors 2
+covid_memes_4193.png purell 0
+memes_2623.png joe biden 2
+memes_8119.png illegal immigrants 1
+covid_memes_797.png introverts 0
+memes_53.png green party 1
+covid_memes_349.png spanish people 3
+covid_memes_4865.png pilots 3
+covid_memes_1678.png spirit 3
+memes_5922.png women 2
+memes_826.png economy 3
+memes_4662.png donald trump 1
+memes_1647.png america 2
diff --git a/annotations/annotations of S b/annotations/annotations of S
@@ -0,0 +1,100 @@
+covid_memes_3977.png extroverts 2
+memes_2046.png donald trump 1
+memes_988.png barack obama 0
+covid_memes_2476.png germany 2
+memes_1716.png donald trump 1
+memes_317.png chris wallace 2
+covid_memes_2973.png china 1
+memes_6576.png liberals 1
+memes_3731.png gary johnson 0
+covid_memes_934.png toilet paper 3
+memes_2764.png joe biden 2
+memes_1705.png donald trump 1
+covid_memes_2743.png work from home 3
+covid_memes_2356.png donald trump 1
+covid_memes_3081.png isarelis 2
+covid_memes_1768.png coronavirus memes 3
+memes_6880.png american flag 0
+covid_memes_3286.png corona beer 0
+memes_1224.png economy 2
+covid_memes_2310.png doanld trump 1
+memes_266.png college students 3
+memes_8245.png mexicans 2
+memes_954.png iron man 3
+memes_4336.png republicans 1
+memes_5071.png republican party 1
+covid_memes_3907.png centers for disease control and prevention 3
+memes_4828.png ? 3
+covid_memes_5678.png spooktober memes 3
+memes_4796.png donald trump 1
+memes_1381.png hiliary clinton 1
+covid_memes_2128.png people who invented the covid19 vaccine 2
+memes_5094.png republicans 1
+memes_4291.png donald trump 3
+covid_memes_265.png stores in china 2
+memes_4521.png terry 1
+memes_2713.png joe biden 2
+memes_1991.png sweden 0
+memes_6600.png electoral college 3
+memes_1047.png barack obama 3
+memes_5969.png bernie sanders 0
+memes_1382.png bernie sanders 0
+memes_3840.png 3
+covid_memes_1438.png homeschool teachers 1
+memes_3598.png democrats 1
+memes_1300.png time 3
+memes_1151.png democratic party 1
+memes_8046.png donald trump 1
+memes_843.png barack obama 3
+covid_memes_1014.png inventor of plastic 1
+memes_1702.png joe biden 0
+memes_5850.png american 3
+memes_4989.png democrats 0
+covid_memes_2528.png america 3
+memes_5882.png barack obama 3
+covid_memes_5664.png people 2
+memes_4402.png ted cruz 3
+memes_4768.png donald trump 1
+covid_memes_470.png coronavirus 3
+memes_8281.png donald trump 2
+covid_memes_5480.png america 3
+memes_4952.png democrats 0
+memes_8283.png illegals 2
+covid_memes_3094.png xi jinping 1
+memes_1979.png scott morrison 2
+covid_memes_4788.png corona 1
+covid_memes_3209.png china 1
+memes_1468.png ben shapiro 1
+memes_3497.png gary johnson 0
+covid_memes_3334.png chinese 3
+memes_5658.png hiliary clinton 3
+memes_4399.png presidential debates 3
+covid_memes_4293.png corona beer 0
+covid_memes_4193.png lysol 3
+memes_1858.png green party 3
+covid_memes_829.png donald trump 2
+memes_1525.png homicide 3
+covid_memes_252.png thais 3
+memes_5098.png grand old party (gop) 3
+memes_1629.png antifa 1
+covid_memes_2700.png work from home 3
+memes_3675.png libertarians 1
+covid_memes_1818.png introvert 3
+covid_memes_775.png democrats 3
+memes_912.png barack obama 3
+memes_3335.png donald trump 3
+memes_175.png prison 3
+memes_5693.png hillary clinton 3
+memes_6743.png minors 3
+covid_memes_4193.png purell 3
+memes_2623.png joe biden 3
+memes_8119.png illegal immigrants 1
+covid_memes_797.png introverts 0
+memes_53.png green party 2
+covid_memes_349.png spanish people 2
+covid_memes_4865.png pilots 2
+covid_memes_1678.png spirit 3
+memes_5922.png women 2
+memes_826.png economy 3
+memes_4662.png donald trump 1
+memes_1647.png america 2
diff --git a/annotations/annotations of U b/annotations/annotations of U
@@ -0,0 +1,100 @@
+covid_memes_3977.png extroverts 2
+memes_2046.png donald trump 1
+memes_988.png barack obama 0
+covid_memes_2476.png germany 1
+memes_1716.png donald trump 1
+memes_317.png chris wallace 2
+covid_memes_2973.png china 1
+memes_6576.png liberals 1
+memes_3731.png gary johnson 0
+covid_memes_934.png toilet paper 3
+memes_2764.png joe biden 2
+memes_1705.png donald trump 1
+covid_memes_2743.png work from home 2
+covid_memes_2356.png donald trump 1
+covid_memes_3081.png isarelis 2
+covid_memes_1768.png coronavirus memes 3
+memes_6880.png american flag 3
+covid_memes_3286.png corona beer 0
+memes_1224.png economy 2
+covid_memes_2310.png doanld trump 0
+memes_266.png college students 3
+memes_8245.png mexicans 2
+memes_954.png iron man 3
+memes_4336.png republicans 3
+memes_5071.png republican party 1
+covid_memes_3907.png centers for disease control and prevention 1
+memes_4828.png ? 3
+covid_memes_5678.png spooktober memes 3
+memes_4796.png donald trump 1
+memes_1381.png hiliary clinton 1
+covid_memes_2128.png people who invented the covid19 vaccine 2
+memes_5094.png republicans 1
+memes_4291.png donald trump 3
+covid_memes_265.png stores in china 2
+memes_4521.png terry 1
+memes_2713.png joe biden 3
+memes_1991.png sweden 3
+memes_6600.png electoral college 3
+memes_1047.png barack obama 0
+memes_5969.png bernie sanders 0
+memes_1382.png bernie sanders 0
+memes_3840.png 3
+covid_memes_1438.png homeschool teachers 0
+memes_3598.png democrats 1
+memes_1300.png time 3
+memes_1151.png democratic party 1
+memes_8046.png donald trump 1
+memes_843.png barack obama 2
+covid_memes_1014.png inventor of plastic 1
+memes_1702.png joe biden 0
+memes_5850.png american 1
+memes_4989.png democrats 2
+covid_memes_2528.png america 1
+memes_5882.png barack obama 0
+covid_memes_5664.png people 2
+memes_4402.png ted cruz 3
+memes_4768.png donald trump 1
+covid_memes_470.png coronavirus 3
+memes_8281.png donald trump 1
+covid_memes_5480.png america 3
+memes_4952.png democrats 2
+memes_8283.png illegals 2
+covid_memes_3094.png xi jinping 1
+memes_1979.png scott morrison 2
+covid_memes_4788.png corona 3
+covid_memes_3209.png china 1
+memes_1468.png ben shapiro 1
+memes_3497.png gary johnson 0
+covid_memes_3334.png chinese 2
+memes_5658.png hiliary clinton 3
+memes_4399.png presidential debates 3
+covid_memes_4293.png corona beer 0
+covid_memes_4193.png lysol 0
+memes_1858.png green party 0
+covid_memes_829.png donald trump 2
+memes_1525.png homicide 3
+covid_memes_252.png thais 0
+memes_5098.png grand old party (gop) 1
+memes_1629.png antifa 1
+covid_memes_2700.png work from home 3
+memes_3675.png libertarians 1
+covid_memes_1818.png introvert 2
+covid_memes_775.png democrats 1
+memes_912.png barack obama 0
+memes_3335.png donald trump 1
+memes_175.png prison 3
+memes_5693.png hillary clinton 0
+memes_6743.png minors 2
+covid_memes_4193.png purell 0
+memes_2623.png joe biden 1
+memes_8119.png illegal immigrants 3
+covid_memes_797.png introverts 0
+memes_53.png green party 1
+covid_memes_349.png spanish people 2
+covid_memes_4865.png pilots 0
+covid_memes_1678.png spirit 3
+memes_5922.png women 3
+memes_826.png economy 2
+memes_4662.png donald trump 1
+memes_1647.png america 2
diff --git a/annotations/annotations of É b/annotations/annotations of É
@@ -0,0 +1,100 @@
+covid_memes_3977.png extroverts 2
+memes_2046.png donald trump 1
+memes_988.png barack obama 0
+covid_memes_2476.png germany 2
+memes_1716.png donald trump 1
+memes_317.png chris wallace 3
+covid_memes_2973.png china 3
+memes_6576.png liberals 1
+memes_3731.png gary johnson 0
+covid_memes_934.png toilet paper 3
+memes_2764.png joe biden 1
+memes_1705.png donald trump 1
+covid_memes_2743.png work from home 3
+covid_memes_2356.png donald trump 1
+covid_memes_3081.png isarelis 2
+covid_memes_1768.png coronavirus memes 3
+memes_6880.png american flag 0
+covid_memes_3286.png corona beer 3
+memes_1224.png economy 2
+covid_memes_2310.png doanld trump 3
+memes_266.png college students 2
+memes_8245.png mexicans 1
+memes_954.png iron man 3
+memes_4336.png republicans 3
+memes_5071.png republican party 0
+covid_memes_3907.png centers for disease control and prevention 3
+memes_4828.png ? 3
+covid_memes_5678.png spooktober memes 3
+memes_4796.png donald trump 1
+memes_1381.png hiliary clinton 1
+covid_memes_2128.png people who invented the covid19 vaccine 2
+memes_5094.png republicans 1
+memes_4291.png donald trump 3
+covid_memes_265.png stores in china 2
+memes_4521.png terry 1
+memes_2713.png joe biden 0
+memes_1991.png sweden 0
+memes_6600.png electoral college 3
+memes_1047.png barack obama 3
+memes_5969.png bernie sanders 0
+memes_1382.png bernie sanders 0
+memes_3840.png 3
+covid_memes_1438.png homeschool teachers 0
+memes_3598.png democrats 1
+memes_1300.png time 3
+memes_1151.png democratic party 1
+memes_8046.png donald trump 1
+memes_843.png barack obama 0
+covid_memes_1014.png inventor of plastic 1
+memes_1702.png joe biden 0
+memes_5850.png american 3
+memes_4989.png democrats 0
+covid_memes_2528.png america 3
+memes_5882.png barack obama 0
+covid_memes_5664.png people 2
+memes_4402.png ted cruz 1
+memes_4768.png donald trump 1
+covid_memes_470.png coronavirus 3
+memes_8281.png donald trump 1
+covid_memes_5480.png america 2
+memes_4952.png democrats 0
+memes_8283.png illegals 1
+covid_memes_3094.png xi jinping 1
+memes_1979.png scott morrison 3
+covid_memes_4788.png corona 3
+covid_memes_3209.png china 1
+memes_1468.png ben shapiro 0
+memes_3497.png gary johnson 0
+covid_memes_3334.png chinese 2
+memes_5658.png hiliary clinton 3
+memes_4399.png presidential debates 2
+covid_memes_4293.png corona beer 0
+covid_memes_4193.png lysol 3
+memes_1858.png green party 3
+covid_memes_829.png donald trump 2
+memes_1525.png homicide 1
+covid_memes_252.png thais 1
+memes_5098.png grand old party (gop) 1
+memes_1629.png antifa 1
+covid_memes_2700.png work from home 3
+memes_3675.png libertarians 3
+covid_memes_1818.png introvert 3
+covid_memes_775.png democrats 1
+memes_912.png barack obama 0
+memes_3335.png donald trump 1
+memes_175.png prison 0
+memes_5693.png hillary clinton 1
+memes_6743.png minors 2
+covid_memes_4193.png purell 3
+memes_2623.png joe biden 0
+memes_8119.png illegal immigrants 2
+covid_memes_797.png introverts 3
+memes_53.png green party 2
+covid_memes_349.png spanish people 2
+covid_memes_4865.png pilots 3
+covid_memes_1678.png spirit 3
+memes_5922.png women 2
+memes_826.png economy 2
+memes_4662.png donald trump 1
+memes_1647.png america 2
diff --git a/annotations/annotator A.ipynb b/annotations/annotator A.ipynb
@@ -0,0 +1,103 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "laughing-being",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "d350896381334b159aa781dc8f2cc28a",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "HTML(value='\\n <p><strong>This notebook is for annotator A!</strong></p>\\n <p><strong>In…"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "import annotations.annotator\n",
+ "annotator = annotations.annotator.Annotator(\"A\", \"/images\")\n",
+ "annotator.instructions()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "proper-campus",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# No need to run this multiple times, it calls itself back once you select the answer\n",
+ "annotator.ask_next()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a99955f5",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "hide_input": false,
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.2"
+ },
+ "latex_envs": {
+ "LaTeX_envs_menu_present": true,
+ "autoclose": false,
+ "autocomplete": true,
+ "bibliofile": "biblio.bib",
+ "cite_by": "apalike",
+ "current_citInitial": 1,
+ "eqLabelWithNumbers": true,
+ "eqNumInitial": 1,
+ "hotkeys": {
+ "equation": "Ctrl-E",
+ "itemize": "Ctrl-I"
+ },
+ "labels_anchors": false,
+ "latex_user_defs": false,
+ "report_style_numbering": false,
+ "user_envs_cfg": false
+ },
+ "toc": {
+ "base_numbering": 1,
+ "nav_menu": {},
+ "number_sections": true,
+ "sideBar": true,
+ "skip_h1_title": false,
+ "title_cell": "Table of Contents",
+ "title_sidebar": "Contents",
+ "toc_cell": false,
+ "toc_position": {},
+ "toc_section_display": true,
+ "toc_window_display": false
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/annotations/annotator D.ipynb b/annotations/annotator D.ipynb
@@ -0,0 +1,118 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "laughing-being",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "3c936d0eb6e94c6d886efb748ef1ce18",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "HTML(value='\\n <p><strong>This notebook is for annotator D!</strong></p>\\n <p><strong>In…"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "import annotations.annotator\n",
+ "annotator = annotations.annotator.Annotator(\"D\", \"/images\")\n",
+ "annotator.instructions()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "proper-campus",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "4e34a804a0e7446a8f44732392e796e8",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "VBox(children=(HTML(value='\\n <img src=\"/images/covid_memes_2973.png\"/>\\n <pre>MEANWHILE\\nAT THE…"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "# No need to run this multiple times, it calls itself back once you select the answer\n",
+ "annotator.ask_next()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "982fb423",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "hide_input": false,
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.2"
+ },
+ "latex_envs": {
+ "LaTeX_envs_menu_present": true,
+ "autoclose": false,
+ "autocomplete": true,
+ "bibliofile": "biblio.bib",
+ "cite_by": "apalike",
+ "current_citInitial": 1,
+ "eqLabelWithNumbers": true,
+ "eqNumInitial": 1,
+ "hotkeys": {
+ "equation": "Ctrl-E",
+ "itemize": "Ctrl-I"
+ },
+ "labels_anchors": false,
+ "latex_user_defs": false,
+ "report_style_numbering": false,
+ "user_envs_cfg": false
+ },
+ "toc": {
+ "base_numbering": 1,
+ "nav_menu": {},
+ "number_sections": true,
+ "sideBar": true,
+ "skip_h1_title": false,
+ "title_cell": "Table of Contents",
+ "title_sidebar": "Contents",
+ "toc_cell": false,
+ "toc_position": {},
+ "toc_section_display": true,
+ "toc_window_display": false
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/annotations/annotator S.ipynb b/annotations/annotator S.ipynb
@@ -0,0 +1,117 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "id": "laughing-being",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "bc11c9139aea4ff98a1ee2151a18bc05",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "HTML(value='\\n <p><strong>This notebook is for annotator S!</strong></p>\\n <p><strong>In…"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "import annotations.annotator\n",
+ "annotator = annotations.annotator.Annotator(\"S\", \"/images\")\n",
+ "annotator.instructions()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "id": "proper-campus",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "VBox(children=(HTML(value='\\n <img src=\"/images/memes_2046.png\"/>\\n <pre>The e The\\nWHAT VOTING\\…"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "annotator.ask_next()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5ab3f97a",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "hide_input": false,
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.2"
+ },
+ "latex_envs": {
+ "LaTeX_envs_menu_present": true,
+ "autoclose": false,
+ "autocomplete": true,
+ "bibliofile": "biblio.bib",
+ "cite_by": "apalike",
+ "current_citInitial": 1,
+ "eqLabelWithNumbers": true,
+ "eqNumInitial": 1,
+ "hotkeys": {
+ "equation": "Ctrl-E",
+ "itemize": "Ctrl-I"
+ },
+ "labels_anchors": false,
+ "latex_user_defs": false,
+ "report_style_numbering": false,
+ "user_envs_cfg": false
+ },
+ "toc": {
+ "base_numbering": 1,
+ "nav_menu": {},
+ "number_sections": true,
+ "sideBar": true,
+ "skip_h1_title": false,
+ "title_cell": "Table of Contents",
+ "title_sidebar": "Contents",
+ "toc_cell": false,
+ "toc_position": {},
+ "toc_section_display": true,
+ "toc_window_display": false
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/annotations/annotator U.ipynb b/annotations/annotator U.ipynb
@@ -0,0 +1,110 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "laughing-being",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "1541fdfe7f5d4d70bbcd13573f936eea",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "HTML(value='\\n <p><strong>This notebook is for annotator U!</strong></p>\\n <p><strong>In…"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "import annotations.annotator\n",
+ "annotator = annotations.annotator.Annotator(\"U\", \"/images\")\n",
+ "annotator.instructions()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "proper-campus",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "798c56553624432fb01dee3a671f61bb",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "VBox(children=(HTML(value='\\n <img src=\"/images/memes_2046.png\"/>\\n <pre>The e The\\nWHAT VOTING\\…"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "# No need to run this multiple times, it calls itself back once you select the answer\n",
+ "annotator.ask_next()"
+ ]
+ }
+ ],
+ "metadata": {
+ "hide_input": false,
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.2"
+ },
+ "latex_envs": {
+ "LaTeX_envs_menu_present": true,
+ "autoclose": false,
+ "autocomplete": true,
+ "bibliofile": "biblio.bib",
+ "cite_by": "apalike",
+ "current_citInitial": 1,
+ "eqLabelWithNumbers": true,
+ "eqNumInitial": 1,
+ "hotkeys": {
+ "equation": "Ctrl-E",
+ "itemize": "Ctrl-I"
+ },
+ "labels_anchors": false,
+ "latex_user_defs": false,
+ "report_style_numbering": false,
+ "user_envs_cfg": false
+ },
+ "toc": {
+ "base_numbering": 1,
+ "nav_menu": {},
+ "number_sections": true,
+ "sideBar": true,
+ "skip_h1_title": false,
+ "title_cell": "Table of Contents",
+ "title_sidebar": "Contents",
+ "toc_cell": false,
+ "toc_position": {},
+ "toc_section_display": true,
+ "toc_window_display": false
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/annotations/annotator É.ipynb b/annotations/annotator É.ipynb
@@ -0,0 +1,109 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "laughing-being",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "c5641c631c5041f9b18905e30c4da904",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "HTML(value='\\n <p><strong>This notebook is for annotator É!</strong></p>\\n <p><strong>In…"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "import annotations.annotator\n",
+ "annotator = annotations.annotator.Annotator(\"É\", \"/images\")\n",
+ "annotator.instructions()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "proper-campus",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "39a03e9dbe6d4404b7e71c7d6cb18046",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "VBox(children=(HTML(value='\\n <img src=\"/images/memes_988.png\"/>\\n <pre>OBAMA: \".CAUSE I\\nWON BO…"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "annotator.ask_next()"
+ ]
+ }
+ ],
+ "metadata": {
+ "hide_input": false,
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.2"
+ },
+ "latex_envs": {
+ "LaTeX_envs_menu_present": true,
+ "autoclose": false,
+ "autocomplete": true,
+ "bibliofile": "biblio.bib",
+ "cite_by": "apalike",
+ "current_citInitial": 1,
+ "eqLabelWithNumbers": true,
+ "eqNumInitial": 1,
+ "hotkeys": {
+ "equation": "Ctrl-E",
+ "itemize": "Ctrl-I"
+ },
+ "labels_anchors": false,
+ "latex_user_defs": false,
+ "report_style_numbering": false,
+ "user_envs_cfg": false
+ },
+ "toc": {
+ "base_numbering": 1,
+ "nav_menu": {},
+ "number_sections": true,
+ "sideBar": true,
+ "skip_h1_title": false,
+ "title_cell": "Table of Contents",
+ "title_sidebar": "Contents",
+ "toc_cell": false,
+ "toc_position": {},
+ "toc_section_display": true,
+ "toc_window_display": false
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/annotations/annotator.py b/annotations/annotator.py
@@ -0,0 +1,178 @@
+from typing import Any, Dict, List, Set, Tuple, Union
+import itertools
+import os
+import pathlib
+import random
+
+import IPython.display
+import ipywidgets
+
+import mmsrl.dataset
+import mmsrl.utils
+
+
+SAMPLE_PER_LABEL = 25 # Total of 100 annotations
+SEED = 0 # We don't want to cherry pick what we annotate…
+#PREFIX = pathlib.Path(os.environ["DATA_PATH"]) / "mmsrl" # we can put annotations in data
+PREFIX = pathlib.Path(".") # The current directory (annotations in the git repository) seems better (we can commit the annotations)
+SELECTED_PATH = PREFIX / "selected for annotation"
+
+
+def get_data() -> Tuple[Dict[str, Any], List[Set[Tuple[str, str]]]]:
+ """
+ Get the sample needing annotations split into the 4 labels.
+
+ Returns:
+ - the dataset, a image→sample dictionary
+ - sets of (image, entity) pairs grouped by label
+ """
+ datasets = {}
+ for split in ["train", "val", "test"]:
+ config = mmsrl.utils.dotdict(dataname="all", model="none")
+ datasets[split] = mmsrl.dataset.MemeDataset(config, split, is_eval=True)
+ data = {sample["image_name"]: sample for sample in itertools.chain(*datasets.values())}
+ annotable = [set() for _ in range(4)]
+ for image, sample in data.items():
+ for entity, label in zip(sample["entities_name"], sample["labels"]):
+ annotable[label].add((image, entity))
+ return data, annotable
+
+
+def select_annotations(annotable: List[Set[Tuple[str, str]]]):
+ """ Write the keys to the selected file if it does not exist. """
+ if SELECTED_PATH.exists():
+ raise RuntimeError("The file containing selected samples already exists, some people might have started annotation, don't change it now!")
+ rng = random.Random(SEED)
+ selected: List[Tuple[str, str]] = []
+ for label in annotable:
+ domain: List[Tuple[str, str]] = sorted(label) # to ensure determinism
+ selected.extend(rng.sample(domain, SAMPLE_PER_LABEL))
+ rng.shuffle(selected) # we don't want to present all victim in a row, etc.
+ with SELECTED_PATH.open("w") as selected_file:
+ for image, entity in selected:
+ print(f"{image}\t{entity}", file=selected_file)
+
+
+class Annotator:
+ def __init__(self, identity: str, image_prefix: Union[pathlib.Path, str]):
+ self.identity = identity
+ self.image_prefix = pathlib.Path(image_prefix)
+ self.annotated_path = PREFIX / f"annotations of {self.identity}"
+ self.data, _ = get_data()
+
+ def instructions(self):
+ IPython.display.display(ipywidgets.HTML(f"""
+ <p><strong>This notebook is for annotator {self.identity}!</strong></p>
+ <p><strong>Instructions:</strong></p>
+ <ul>
+ <li>Given a meme and an entity, determine the role of the entity in the meme: hero vs. villain vs. victim vs. other.</li>
+ <li>The meme is to be analyzed from the perspective of the author of the meme.</li>
+ <li>Class definitions:<dl>
+ <dt>Hero</dt>
+ <dd>The entity is presented in a positive light. Glorified for their actions conveyed via the meme or gathered from background context.</dd>
+ <dt>Villain</td>
+ <dd>The entity is portrayed negatively, e.g., in an association with adverse traits like wickedness, cruelty, hypocrisy, etc.</dd>
+ <dt>Victim</dt>
+ <dd>The entity is portrayed as suffering the negative impact of someone else’s actions or conveyed implicitly within the meme.</dd>
+ <dt>Other</dt>
+ <dd>The entity is not a hero, a villain, or a victim.</dd>
+ </dl></li>
+ <li>The four classes are balanced, you should give each answer roughly 25% of the time.</li>
+ <li>Your answers are saved at each annotation, you can leave and come back at any time.</li>
+ </ul>
+ """))
+
+ def get_already_annotated(self) -> Set[Tuple[str, str]]:
+ """ Get list of samples already annotated. """
+ annotated = set()
+ try:
+ with self.annotated_path.open("r") as annotated_file:
+ for line in annotated_file:
+ image, entity, _ = line.rstrip('\n').split('\t')
+ annotated.add((image, entity))
+ except FileNotFoundError:
+ pass
+ return annotated
+
+ def get_next_annotation(self) -> Tuple[str, str]:
+ """ Get the next (image, entity) pair to annotate. """
+ # We re-read the file for each annotation to avoid an error-prone in-memory updating of the list
+ annotated = self.get_already_annotated()
+ # It would be hard to be less algorithmicaly efficient, but better be safe than sorry
+ with SELECTED_PATH.open("r") as selected_file:
+ for i, line in enumerate(selected_file):
+ image, entity = line.rstrip('\n').split('\t')
+ if (image, entity) not in annotated:
+ return i, image, entity
+ return -1, None, None
+
+ def write_answer(self, image: str, entity: str, answer: int):
+ with self.annotated_path.open("a") as annotated_file:
+ print(f"{image}\t{entity}\t{answer}", file=annotated_file)
+
+ def get_sample_html(self, image: str, entity: str):
+ sample = self.data[image]
+ ocr = sample["raw_text"]
+ html = f"""
+ <img src="{self.image_prefix / image}"/>
+ <pre>{ocr}</pre>
+ """
+ return ipywidgets.HTML(html)
+
+ def delete_last_annotation(self):
+ annotations = []
+ try:
+ with self.annotated_path.open("r") as annotated_file:
+ annotations = annotated_file.readlines()
+ except FileNotFoundError:
+ pass
+ if annotations:
+ annotations.pop()
+ with self.annotated_path.open("w") as annotated_file:
+ annotated_file.writelines(annotations)
+
+ def answer_clicked(self, image: str, entity: str, label: int):
+ def callback(button):
+ IPython.display.clear_output()
+ if label == -1:
+ # Undo last answer
+ self.delete_last_annotation()
+ else:
+ assert(button.description == mmsrl.utils.LABELS[label])
+ self.write_answer(image, entity, label)
+ self.ask_next()
+ return callback
+
+ def ask_annotation(self, current_id: int, image: str, entity: str):
+ sample = self.get_sample_html(image, entity)
+ buttons = []
+ for ilabel, label in enumerate(mmsrl.utils.LABELS):
+ button = ipywidgets.Button(description=label)
+ button.on_click(self.answer_clicked(image, entity, ilabel))
+ buttons.append(button)
+ button = ipywidgets.Button(description="Undo last answer")
+ button.on_click(self.answer_clicked(None, None, -1))
+ buttons.append(button)
+ hbox = ipywidgets.HBox(buttons)
+ instructions = ipywidgets.HTML(f'({current_id+1}/{4*SAMPLE_PER_LABEL}) <strong>What is the role of <span style="font-size:150%;">{entity}</span> in the above meme?</strong>')
+ vbox = ipywidgets.VBox([sample, instructions, hbox])
+ IPython.display.display(vbox)
+
+ def completed(self):
+ message = ipywidgets.HTML(f'You completed all annotations! Thank you!')
+ button = ipywidgets.Button(description="Undo last answer")
+ button.on_click(self.answer_clicked(None, None, -1))
+ vbox = ipywidgets.VBox([message, button])
+ IPython.display.display(vbox)
+
+ def ask_next(self):
+ current_id, image, entity = self.get_next_annotation()
+ if current_id < 0 and image is None and entity is None:
+ self.completed()
+ else:
+ self.ask_annotation(current_id, image, entity)
+
+
+# This only need to be done once for all annotators
+# And it was already done, DO NOT UNCOMMENT
+# select_annotations(get_data()[1])
diff --git a/annotations/selected for annotation b/annotations/selected for annotation
@@ -0,0 +1,100 @@
+covid_memes_3977.png extroverts
+memes_2046.png donald trump
+memes_988.png barack obama
+covid_memes_2476.png germany
+memes_1716.png donald trump
+memes_317.png chris wallace
+covid_memes_2973.png china
+memes_6576.png liberals
+memes_3731.png gary johnson
+covid_memes_934.png toilet paper
+memes_2764.png joe biden
+memes_1705.png donald trump
+covid_memes_2743.png work from home
+covid_memes_2356.png donald trump
+covid_memes_3081.png isarelis
+covid_memes_1768.png coronavirus memes
+memes_6880.png american flag
+covid_memes_3286.png corona beer
+memes_1224.png economy
+covid_memes_2310.png doanld trump
+memes_266.png college students
+memes_8245.png mexicans
+memes_954.png iron man
+memes_4336.png republicans
+memes_5071.png republican party
+covid_memes_3907.png centers for disease control and prevention
+memes_4828.png ?
+covid_memes_5678.png spooktober memes
+memes_4796.png donald trump
+memes_1381.png hiliary clinton
+covid_memes_2128.png people who invented the covid19 vaccine
+memes_5094.png republicans
+memes_4291.png donald trump
+covid_memes_265.png stores in china
+memes_4521.png terry
+memes_2713.png joe biden
+memes_1991.png sweden
+memes_6600.png electoral college
+memes_1047.png barack obama
+memes_5969.png bernie sanders
+memes_1382.png bernie sanders
+memes_3840.png
+covid_memes_1438.png homeschool teachers
+memes_3598.png democrats
+memes_1300.png time
+memes_1151.png democratic party
+memes_8046.png donald trump
+memes_843.png barack obama
+covid_memes_1014.png inventor of plastic
+memes_1702.png joe biden
+memes_5850.png american
+memes_4989.png democrats
+covid_memes_2528.png america
+memes_5882.png barack obama
+covid_memes_5664.png people
+memes_4402.png ted cruz
+memes_4768.png donald trump
+covid_memes_470.png coronavirus
+memes_8281.png donald trump
+covid_memes_5480.png america
+memes_4952.png democrats
+memes_8283.png illegals
+covid_memes_3094.png xi jinping
+memes_1979.png scott morrison
+covid_memes_4788.png corona
+covid_memes_3209.png china
+memes_1468.png ben shapiro
+memes_3497.png gary johnson
+covid_memes_3334.png chinese
+memes_5658.png hiliary clinton
+memes_4399.png presidential debates
+covid_memes_4293.png corona beer
+covid_memes_4193.png lysol
+memes_1858.png green party
+covid_memes_829.png donald trump
+memes_1525.png homicide
+covid_memes_252.png thais
+memes_5098.png grand old party (gop)
+memes_1629.png antifa
+covid_memes_2700.png work from home
+memes_3675.png libertarians
+covid_memes_1818.png introvert
+covid_memes_775.png democrats
+memes_912.png barack obama
+memes_3335.png donald trump
+memes_175.png prison
+memes_5693.png hillary clinton
+memes_6743.png minors
+covid_memes_4193.png purell
+memes_2623.png joe biden
+memes_8119.png illegal immigrants
+covid_memes_797.png introverts
+memes_53.png green party
+covid_memes_349.png spanish people
+covid_memes_4865.png pilots
+covid_memes_1678.png spirit
+memes_5922.png women
+memes_826.png economy
+memes_4662.png donald trump
+memes_1647.png america
diff --git a/annotations/stats.py b/annotations/stats.py
@@ -0,0 +1,75 @@
+from typing import Any, Dict, List, Tuple
+
+import numpy
+import sklearn.metrics
+import statsmodels.stats.inter_rater
+
+import annotations.annotator
+import mmsrl.utils
+
+
+def show_scores(annotator: str, data: Dict[str, Any], answers: List[Tuple[str, str, int]]):
+ truth = []
+ predictions = []
+ assert(len(answers) == 100)
+ for image, entity, answer in answers:
+ sample = data[image]
+ truth.append(sample["labels"][sample["entities_name"].index(entity)].item())
+ predictions.append(answer)
+
+ f1 = sklearn.metrics.f1_score(truth, predictions, average="macro")
+ class_precision, class_recall, class_f1, _ = sklearn.metrics.precision_recall_fscore_support(truth, predictions)
+ class_scores = ", ".join(f"{label}[F{class_f1[i]:4.2f}/P{class_precision[i]:4.2f}/R{class_recall[i]:4.2f}]" for i, label in enumerate(mmsrl.utils.LABELS))
+ print(f"Annotator {annotator}, Macro-F1: {f1:5.3f}, {class_scores}")
+ return f1
+
+
+def get_annotations(annotator: str) -> List[Tuple[str, str, int]]:
+ answers = []
+ with open(f"annotations/annotations of {annotator}", "r") as file:
+ for line in file:
+ image, entity, answer = line.rstrip('\n').split('\t')
+ answers.append((image, entity, int(answer)))
+ return answers
+
+
+def main(annotators: List[str]):
+ data, _ = annotations.annotator.get_data()
+ answers = {}
+ x = numpy.empty((100, len(annotators)), dtype=numpy.int32)
+ m = numpy.zeros((100, 4), dtype=numpy.int32)
+ scores = []
+ for i, annotator in enumerate(annotators):
+ answers[annotator] = get_annotations(annotator)
+ f1 = show_scores(annotator, data, answers[annotator])
+ scores.append(f1)
+ for j, (_, _, answer) in enumerate(answers[annotator]):
+ x[j, i] = answer
+ m[j, answer] += 1
+ print(f"Average Macro-F1: {numpy.mean(scores)*100:.1f}")
+ print(f"Std Macro-F1: {numpy.std(scores)*100:.1f}")
+ y = numpy.array([data[image]["labels"][data[image]["entities_name"].index(entity)].item() for image, entity, _ in answers[annotator]], dtype=numpy.int32)
+ print("")
+
+ print(f"Fleiss kappa: {statsmodels.stats.inter_rater.fleiss_kappa(m):.2f}")
+ kappas = []
+ for i, annotatorA in enumerate(annotators):
+ for jmi, annotatorB in enumerate(annotators[i+1:]):
+ j = i + 1 + jmi
+ kappa = sklearn.metrics.cohen_kappa_score(x[:, i], x[:, j])
+ kappas.append(kappa)
+ print(f"Cohen kappa between annotators {annotatorA} and {annotatorB}: {kappa:.2f}")
+ avg_kappa = numpy.mean(kappas)
+ print(f"Average Cohen kappa: {avg_kappa:.2f}")
+
+
+ print("")
+ print(f"Percent of samples where all annotators agree: {numpy.equal(x[:, [0]], x[:, 1:]).all(axis=1).sum()}%")
+ for i, annotatorA in enumerate(annotators):
+ for jmi, annotatorB in enumerate(annotators[i+1:]):
+ j = i + 1 + jmi
+ print(f"Percent agreement between {annotatorA} and {annotatorB}: {(x[:, i] == x[:, j]).sum()}%")
+
+
+if __name__ == "__main__":
+ main(["A", "É", "S", "D", "U"])
diff --git a/configs/baseline.py b/configs/baseline.py
@@ -0,0 +1,30 @@
+dataname = "all"
+model = "baseline"
+
+from configs.label_distribution import label_frequencies
+from configs.micro_to_macro import multilabel, batch_per_epoch, cyclic_subsample, subsample_labels
+
+# Baseline model parameters
+modelname = "bert-base-uncased"
+hidden_size = 256
+dropout = 0
+use_caption = False
+
+# Optimizer parameters
+learning_rate = 2e-5
+weight_decay = 0.0001
+max_grad_norm = 1
+
+# Trainer parameters
+batch_size = 8
+accumulation = 16
+max_epoch = 25
+patience = 3
+
+max_text_len = 128
+max_nb_ent = 10
+max_ent_len = 15
+image_max_edge = 384
+
+amp = True
+workers = 3
diff --git a/configs/cyclic.py b/configs/cyclic.py
@@ -0,0 +1,14 @@
+cyclic_subsample_shorter = [
+ {"subsample_labels": "macro"},
+ {"subsample_labels": "micro"},
+]
+
+cyclic_subsample = [
+ {"subsample_labels": "macro"},
+ {"subsample_labels": "affine", "frequency_factor": 0.1, "frequency_bias": 0.05},
+ {"subsample_labels": "micro"},
+ {"subsample_labels": "affine", "frequency_factor": 0.7, "frequency_bias": 0.05},
+]
+
+multilabel = False
+subsample_labels = None
diff --git a/configs/generic.py b/configs/generic.py
@@ -0,0 +1,41 @@
+dataname = "all"
+model = "generic"
+
+use_clip = True
+modelname = 'ViT-L/14' #"ViT-B/16"
+workers = 3
+amp = False
+unfreeze="classifier"
+pooling = "mlp"
+use_caption = False
+use_entities = False
+features_path = False #"features_b7"
+use_sbert = False
+sbert_model = '/gpfswork/rech/lco/url46ht/.cache/torch/sentence_transformers/sentence-transformers_paraphrase-distilroberta-base-v1/'
+
+use_visualbert = False
+visualbert_name = 'uclanlp/visualbert-vcr-coco-pre'#'uclanlp/visualbert-nlvr2-coco-pre'
+visualbert_text_model = "bert-base-uncased" #limjiayi/bert-hateful-memes-expanded facebook/bart-large
+
+from configs.label_distribution import label_frequencies
+from configs.micro_to_macro import multilabel, batch_per_epoch, cyclic_subsample, subsample_labels
+
+# Model parameters
+hidden_size = 768
+dropout = 0
+
+# Optimizer parameters
+learning_rate = 1e-4
+weight_decay = 0.001
+max_grad_norm = 1
+
+# Trainer parameters
+batch_size = 8
+accumulation = 16
+max_epoch = 25
+patience = 3
+
+max_text_len = 128
+max_nb_ent = 10
+max_ent_len = 15
+image_max_edge = 384
diff --git a/configs/label_distribution.py b/configs/label_distribution.py
@@ -0,0 +1,3 @@
+# Can be used with cofig.subsample_labels to ensure all labels are shown equally often
+# These frequencies come from the "all" train set.
+label_frequencies = [0.027121160214685396, 0.13857485440219253, 0.05195843325339728, 0.7823455521297248]
diff --git a/configs/micro_to_macro.py b/configs/micro_to_macro.py
@@ -0,0 +1,4 @@
+multilabel = False
+batch_per_epoch = 500
+cyclic_subsample = None
+subsample_labels = "interpolate_micro_to_macro"
diff --git a/configs/ofa.py b/configs/ofa.py
@@ -0,0 +1,27 @@
+dataname = "all"
+model = "ofa"
+
+from configs.label_distribution import label_frequencies
+from configs.micro_to_macro import multilabel, batch_per_epoch, cyclic_subsample, subsample_labels
+
+# OFA parameters
+unfreeze = "all"
+ofa_classification_head = True
+
+# Optimizer parameters
+learning_rate = 2e-5
+weight_decay = 0.0001
+max_grad_norm = 1
+
+# Trainer parameters
+batch_size = 8
+accumulation = 16
+max_epoch = 25
+patience = 3
+
+max_text_len = 128
+max_nb_ent = 10
+image_max_edge = 384
+
+amp = True
+workers = 3
diff --git a/configs/ofa_new_old_best.py b/configs/ofa_new_old_best.py
@@ -0,0 +1,18 @@
+from configs.ofa_old_best import *
+
+
+# Observed distribution of the ofa_old_best settings
+#subsample_labels = "target"
+#frequency_target = [0.08273819892004877, 0.16164431283748476, 0.09946002438599548, 0.6561574638564711]
+
+subsample_labels = "affine"
+frequency_factor = 0.762
+frequency_bias = 0.06
+
+subsample_other = 1
+
+keep_all_batches = False
+group_image_entities = False
+
+# The mean batch size of the old code was 1.068093023255814
+batch_size = 1
diff --git a/configs/ofa_old_best.py b/configs/ofa_old_best.py
@@ -0,0 +1,37 @@
+dataname = "all"
+model = "ofa"
+
+multilabel = False
+from configs.label_distribution import label_frequencies
+subsample_labels = "macro"
+subsample_other = 0.5
+
+# OFA parameters
+ofa_task = "vqa_gen"
+ofa_dir = "OFA/checkpoints/"
+ofa_path = f"{ofa_dir}/ofa_base.pt"
+vqa_input = "question ocr"
+decoder_input = ""
+unfreeze = "all"
+ofa_classification_head = True
+
+keep_all_batches = True
+group_image_entities = True
+
+# Optimizer parameters
+learning_rate = 1e-5
+weight_decay = 0.0001
+max_grad_norm = 1
+
+# Trainer parameters
+batch_size = 1
+accumulation = 16
+max_epoch = 35
+patience = 25
+
+max_text_len = 128
+max_nb_ent = 10
+image_max_edge = 384
+
+amp = True
+workers = 3
diff --git a/configs/ofa_seq2seq.py b/configs/ofa_seq2seq.py
@@ -0,0 +1,3 @@
+constrain_output = True
+decoder_input = "{label}"
+ofa_classification_head = False
diff --git a/configs/ofa_snli.py b/configs/ofa_snli.py
@@ -0,0 +1,5 @@
+ofa_task = "snli_ve"
+ofa_path = "OFA/checkpoints/snli_ve_large_best.pt"
+ofa_classification_head = False
+neither_for_other = True
+maybe_is = "ignored"
diff --git a/configs/ofa_vqa.py b/configs/ofa_vqa.py
@@ -0,0 +1,3 @@
+ofa_task = "vqa_gen"
+ofa_path = "OFA/checkpoints/vqa_large_best.pt"
+vqa_input = "question ocr"
diff --git a/extract_results_table.py b/extract_results_table.py
@@ -0,0 +1,473 @@
+from typing import Any, Dict, List, Optional, Tuple
+import argparse
+import collections
+import functools
+import math
+import os
+import pathlib
+import re
+
+import numpy
+import scipy.stats
+import statsmodels.stats.contingency_tables
+
+import mmsrl.ensembling
+import mmsrl.utils
+
+RESULT_REGEX = re.compile(r"^Epoch\s+(\d+) (val|test), Macro-F1: (\d\.\d{3}), hero\[F(\d\.\d{2})/P(\d\.\d{2})/R(\d\.\d{2})\], villain\[F(\d\.\d{2})/P(\d\.\d{2})/R(\d\.\d{2})\], victim\[F(\d\.\d{2})/P(\d\.\d{2})/R(\d\.\d{2})\], other\[F(\d\.\d{2})/P(\d\.\d{2})/R(\d\.\d{2})\]$")
+RESULT_KEYS = ["epoch", "split", "macro-F1", "hero F1", "hero P", "hero R", "villain F1", "villain P", "villain R", "victim F1", "victim P", "victim R", "other F1", "other P", "other R"]
+CONFIG_REGEX = re.compile(r"^([^:]*): (.*)$")
+CONFIG_UNIQUE_KEYS = ["output_test", "output_val"]
+
+
+def all_equals(l: List[str]) -> bool:
+ return all(x == l[0] for x in l[1:])
+
+
+def config_to_run(config: Dict[str, str]) -> Any:
+ key = config.copy()
+ for unique in CONFIG_UNIQUE_KEYS:
+ key.pop(unique, None)
+ return tuple(sorted(key.items()))
+
+
+def run_to_config(run: Any) -> Dict[str, str]:
+ return dict(list(run))
+
+
+@functools.cache
+def levenshtein(lhs: str, rhs: str) -> int:
+ if not lhs or not rhs:
+ return len(lhs) + len(rhs)
+ if lhs[0] == rhs[0]:
+ return levenshtein(lhs[1:], rhs[1:])
+ return min(2+levenshtein(lhs[1:], rhs), 2+levenshtein(lhs, rhs[1:]), 1+levenshtein(lhs[1:], rhs[1:]))
+
+
+def get_experiments(args: argparse.Namespace, path: Optional[pathlib.Path] = None) -> Dict[str, List[pathlib.Path]]:
+ if path is None:
+ path = args.path
+ experiments: Dict[str, List[pathlib.Path]] = {}
+ sub_experiments: Dict[str, List[pathlib.Path]] = {}
+ for filename in sorted(os.listdir(path)):
+ if (path / filename).is_dir():
+ if filename not in args.exclude:
+ sub_experiments.update(get_experiments(args, path / filename))
+ continue
+ if not (filename.startswith(args.prefix) and filename.endswith(args.suffix)):
+ continue
+
+ matched: bool = False
+ for candidate, filenames in experiments.items():
+ if levenshtein(filename, candidate) < args.group_threshold:
+ filenames.append(path / filename)
+ matched = True
+ break
+
+ if not matched:
+ experiments[filename] = [path / filename]
+
+ return experiments | sub_experiments
+
+
+def get_result(args: argparse.Namespace, data: List[str]) -> Dict[str, float]:
+ last = {}
+ for line in data:
+ match = RESULT_REGEX.fullmatch(line)
+ if match:
+ last[match[2]] = match
+
+ if args.split not in last or int(last[args.split][1]) < args.min_epoch:
+ return {}
+
+ result = dict(zip(RESULT_KEYS, last[args.split].groups()))
+ result.pop("split")
+ return {key: float(value) for key, value in result.items()}
+
+
+def get_config(data: List[str]) -> Dict[str, str]:
+ if not data or data[0] != "\033[33mConfiguration:\033[0m":
+ return {}
+
+ config: Dict[str, str] = {}
+ for line in data[1:]:
+ if "\033" in line or line.startswith("Epoch "):
+ break
+
+ match = CONFIG_REGEX.fullmatch(line)
+ if match:
+ config[match[1]] = match[2]
+ else:
+ break
+ return config
+
+
+def merge_configs(configs: Dict[pathlib.Path, Dict[str, str]]) -> Tuple[Dict[str, str], Dict[pathlib.Path, Dict[str, str]]]:
+ transpose: Dict[str, List[str]] = collections.defaultdict(list)
+ for config in configs.values():
+ for key, value in config.items():
+ transpose[key].append(value)
+
+ base_config: Dict[str, str] = {}
+ for key, values in transpose.items():
+ if all_equals(values) and len(values) == len(configs):
+ base_config[key] = values[0]
+
+ delta_configs: Dict[pathlib.Path, Dict[str, str]] = {}
+ for path, config in configs.items():
+ delta_configs[path] = {key: value for key, value in config.items() if key not in base_config}
+
+ return base_config, delta_configs
+
+
+def merge_runs(configs: Dict[pathlib.Path, Dict[str, str]]) -> Dict[Any, List[pathlib.Path]]:
+ runs: Dict[List[Tuple[str, str]], List[pathlib.Path]] = collections.defaultdict(list)
+ for path, config in configs.items():
+ runs[config_to_run(config)].append(path)
+ return runs
+
+
+def aggregate_results(results: List[Dict[str, float]]):
+ aggregated: Dict[str, Dict[str, Any]] = {}
+ for key in results[0].keys():
+ values = numpy.array([result[key] for result in results])
+ aggregated[key] = {
+ "mean": values.mean(),
+ "std": values.std(),
+ "min": values.min(),
+ "max": values.max(),
+ "values": values
+ }
+ return aggregated
+
+
+def merge_results(runs: Dict[Any, List[pathlib.Path]], results: Dict[pathlib.Path, Dict[str, float]]) -> Dict[Any, Dict[str, Any]]:
+ merged: Dict[Any, Dict[str, Any]] = {}
+ for run, paths in runs.items():
+ merged[run] = {}
+ merged[run]["runs"] = [results[path] for path in paths],
+ merged[run].update(aggregate_results([results[path] for path in paths]))
+ return merged
+
+
+def mark_best(results: Dict[Any, Dict[str, Any]]) -> None:
+ best = collections.defaultdict(lambda: 0)
+ for run in results.values():
+ for key, values in run.items():
+ if isinstance(values, dict) and isinstance(values.get("mean"), float):
+ best[key] = max(best[key], values["mean"])
+
+ for run in results.values():
+ for key, values in run.items():
+ if isinstance(values, dict) and isinstance(values.get("mean"), float):
+ values["best"] = (values["mean"] >= best[key])
+
+
+def get_predictions(args: argparse.Namespace, runs: Dict[Any, List[pathlib.Path]], delta_configs: Dict[pathlib.Path, Dict[str, str]]) -> Tuple[Dict[Any, numpy.ndarray], numpy.ndarray]:
+ predictions: Dict[Tuple[str, str], Dict[Any, List[int]]] = collections.defaultdict(lambda: collections.defaultdict(list))
+ gold = {sample["image"]: sample for sample in mmsrl.ensembling.read_data(args.split)}
+
+ for run, paths in runs.items():
+ for stdout_path in paths:
+ base_path = stdout_path.parent
+ prediction_path = pathlib.Path(delta_configs[stdout_path][f"output_{args.split}"])
+ prediction = mmsrl.ensembling.read_prediction_file(base_path / prediction_path.parts[-1])
+ for image, entities in prediction.items():
+ for entity, probabilities in entities.items():
+ predictions[(image, entity)][run].append(probabilities.argmax())
+
+ dataset_size: int = len(predictions)
+ num_seeds: int = len(paths)
+ prediction_array = {run: numpy.empty((dataset_size, num_seeds), dtype=numpy.int32) for run in runs}
+ gold_array = numpy.empty((dataset_size,), dtype=numpy.int32)
+
+ for i, ((image, entity), run_predictions) in enumerate(predictions.items()):
+ assigned = False
+ for ilabel, label in enumerate(mmsrl.utils.LABELS):
+ if entity in gold[image][label]:
+ gold_array[i] = ilabel
+ assigned = True
+ assert(assigned)
+
+ for run, seed_predictions in run_predictions.items():
+ prediction_array[run][i] = seed_predictions
+ return prediction_array, gold_array
+
+
+def compute_contingency_matrix(aggregate: str, lhs: numpy.ndarray, rhs: numpy.ndarray) -> numpy.ndarray:
+ contingency = numpy.zeros((4, 4), dtype=numpy.int32)
+ if aggregate == "first":
+ for y, x in zip(lhs[:, 0], rhs[:, 0]):
+ contingency[y, x] += 1
+ elif aggregate == "flatten":
+ for y, x in zip(lhs.flatten(), rhs.flatten()):
+ contingency[y, x] += 1
+ elif aggregate == "cartesian mean":
+ for ys, xs in zip(lhs, rhs):
+ for y in ys:
+ for x in xs:
+ contingency[y, x] += 1
+ contingency = contingency // (len(ys) * len(xs))
+ return contingency
+
+
+def compute_contingency_matrices(aggregate: str, predictions: Dict[Any, numpy.ndarray]) -> Dict[Tuple[Any, Any], numpy.ndarray]:
+ contingencies = {}
+ for run_lhs, lhs in predictions.items():
+ for run_rhs, rhs in predictions.items():
+ contingencies[(run_lhs, run_rhs)] = compute_contingency_matrix(aggregate, lhs, rhs)
+ return contingencies
+
+
+def compute_mcnemar_bowker(contingency: numpy.ndarray) -> float:
+ return statsmodels.stats.contingency_tables.mcnemar(contingency, exact=False, correction=True).pvalue
+
+
+def compute_macro_f1_t_test(lhs: Dict[str, Any], rhs: Dict[str, Any], equal_var: bool) -> float:
+ return scipy.stats.ttest_ind(lhs["macro-F1"]["values"], rhs["macro-F1"]["values"], equal_var=equal_var, alternative="greater").pvalue
+
+
+def compute_accuracies_t_test(lhs: numpy.ndarray, rhs: numpy.ndarray, gold: numpy.ndarray, equal_var: bool) -> float:
+ lhs = (lhs == gold[:, None]).flatten()
+ rhs = (rhs == gold[:, None]).flatten()
+ return scipy.stats.ttest_ind(lhs, rhs, equal_var=equal_var, alternative="greater").pvalue
+
+
+def compute_pvalues(args: argparse.Namespace, runs: Dict[Any, List[pathlib.Path]], delta_configs: Dict[pathlib.Path, Dict[str, str]], run_results: Dict[Any, Dict[str, Any]]) -> Dict[str, Dict[Tuple[Any, Any], float]]:
+ if args.all_test:
+ predictions, gold = get_predictions(args, runs, delta_configs)
+ contingencies: Dict[str, Dict[Tuple[Any, Any], numpy.ndarray]] = {
+ aggregation: compute_contingency_matrices(aggregation, predictions)
+ for aggregation in ["first", "flatten", "cartesian mean"]
+ }
+
+ pvalues: Dict[str, Dict[Tuple[Any, Any], float]] = collections.defaultdict(dict)
+ for run_lhs in runs:
+ for run_rhs in runs:
+ nseed = len(runs[run_lhs])
+ pvalues[f'one-sided t-test of macro-F1'][(run_lhs, run_rhs)] = compute_macro_f1_t_test(run_results[run_lhs], run_results[run_rhs], equal_var=True)
+ pvalues[f'one-sided Welch\'s unequal variances t-test of macro-F1'][(run_lhs, run_rhs)] = compute_macro_f1_t_test(run_results[run_lhs], run_results[run_rhs], equal_var=False)
+
+ if args.all_test:
+ pvalues[f'one-sided t-test of accuracies ({nseed} seed{"s" if nseed>1 else ""})'][(run_lhs, run_rhs)] = compute_accuracies_t_test(predictions[run_lhs], predictions[run_rhs], gold, equal_var=True)
+ pvalues[f'one-sided Welch\'s unequal variances t-test of accuracies ({nseed} seed{"s" if nseed>1 else ""})'][(run_lhs, run_rhs)] = compute_accuracies_t_test(predictions[run_lhs], predictions[run_rhs], gold, equal_var=False)
+
+ for aggregation, contingency in contingencies.items():
+ pvalues[f"McNemar-Bowker ({aggregation})"][(run_lhs, run_rhs)] = compute_mcnemar_bowker(contingency[(run_lhs, run_rhs)])
+ return pvalues
+
+
+def pre_html() -> None:
+ print("""<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
+<head>
+ <title>MMSRL results</title>
+ <meta charset="UTF-8" />
+ <style type="text/css">
+ /*<![CDATA[*/
+ .pvalue {
+ paint-order: stroke fill;
+ -webkit-text-stroke: 10px rgba(255, 255, 255, .3);
+ }
+ table {
+ border-collapse: collapse;
+ }
+ table, td, th {
+ border: 1px solid;
+ }
+ td, th {
+ max-width: 600px;
+ padding: 4px;
+ }
+ dl.base_config > dt:before {
+ content: "";
+ display: block;
+ }
+ .best_true {
+ font-weight: bold;
+ color: #800;
+ }
+ dt { font-weight: bold; }
+ dt:after { content: "="; }
+ dt, dd {
+ display: inline;
+ margin: 0;
+ }
+ /*]]>*/
+ </style>
+</head>
+<body>""")
+
+
+def output_html_config(config: Dict[str, str], style: str) -> None:
+ print(f'<dl class="{style}">')
+ for key, value in config.items():
+ print(f'<dt>{key}</dt><dd>{value}</dd>')
+ print('</dl>')
+
+
+def output_html_pvalue_color(pvalue: float, better: bool) -> None:
+ if not numpy.isfinite(pvalue):
+ hue = 200
+ else:
+ hue: int = round(min(-math.log(max(pvalue, 1e-10), 10), 2.5)/2.5*120)
+ saturation: int = 100 if better else 25
+ print(f'<td class="pvalue" style="background-color: hsl({hue}, {saturation}%, 50%);">{pvalue:.2e}</td>', end='')
+
+
+def output_html(experiment: str, base_config: Dict[str, str], runs: Dict[Any, List[pathlib.Path]], run_results: Dict[Any, Dict[str, Any]], pvalues: Dict[str, Dict[Tuple[Any, Any], float]]) -> None:
+ run_order = sorted(list(run_results.keys()))
+
+ print('<section>')
+ print(f'<h1>{experiment}</h1>')
+ print('<details>')
+ print('<summary>Base config</summary>')
+ output_html_config(base_config, "base_config")
+ print('</details>')
+ print('<table>')
+ print(' <thead>')
+ print(' <tr><th rowspan="2">ID</th><th rowspan="2">Config</th><th rowspan="2">Macro-F1</th><th colspan="3">hero</th><th colspan="3">villain</th><th colspan="3">victim</th><th colspan="3">other</th></tr>')
+ print(' <tr><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th></tr>')
+ print(' </thead>')
+ print(' <tbody>')
+ for irun, run in enumerate(run_order):
+ result = run_results[run]
+ print(' <tr>')
+ print(f' <td>{chr(ord("α")+irun)}</td>')
+ print(' <td>', end='')
+ print('<details><summary><code>', end='')
+ config = run_to_config(run)
+ output_html_config(config, "inline_config")
+ print('</code></summary>')
+ print(' <ul>')
+ for path in runs[run]:
+ print(f' <li>{path}</li>')
+ print(' </ul></details>')
+ print('</td>')
+ for key in ["macro-F1", "hero F1", "hero P", "hero R", "villain F1", "villain P", "villain R", "victim F1", "victim P", "victim R", "other F1", "other P", "other R"]:
+ data = result[key]
+ details = "\n".join(f"{key}: {value}" for key, value in data.items())
+ print(' <td>', end='')
+ print(f'<abbr title="{details}" class="best_{data["best"]}">{data["mean"]:.2f}</abbr>', end='')
+ print('</td>')
+ print(' </tr>')
+ print(' </tbody>')
+ print('</table>')
+
+ for test, pvalue in pvalues.items():
+ print('<section>')
+ print(f'<details><summary><h3>{test}</h3></summary>')
+ print('<table>')
+ print(' <tr>')
+ print(' <th></th>')
+ for irun in range(len(run_order)):
+ print(f' <th>{chr(ord("α")+irun)}</th>')
+ print(' </tr>')
+ for irun, lhs in enumerate(run_order):
+ print(' <tr>')
+ print(f' <th>{chr(ord("α")+irun)}</th>')
+ for rhs in run_order:
+ output_html_pvalue_color(pvalue[(lhs, rhs)], run_results[lhs]["macro-F1"]["mean"] > run_results[rhs]["macro-F1"]["mean"])
+ print(' </tr>')
+ print('</table></details>')
+ print('</section>')
+
+ print('</section>')
+
+
+def post_html() -> None:
+ print('</body>')
+ print('</html>')
+
+
+def pre_latex() -> None:
+ print(r"""\documentclass[a4paper, landscape]{article}
+\usepackage[a4paper,margin=1in,landscape]{geometry}
+\usepackage{fontspec}
+\usepackage{booktabs}
+\usepackage{enumitem}
+\usepackage{multirow}
+\begin{document}""")
+
+
+def output_latex_escape(x: str):
+ return x.replace('_', r'\_').replace('[', r'\([\)').replace(']', r'\(]\)').replace('{', r'\({\)').replace('}', r'\(}\)')
+
+
+def output_latex(experiment: str, base_config: Dict[str, str], runs: Dict[Any, List[pathlib.Path]], run_results: Dict[Any, Dict[str, Any]], pvalues: Dict[str, Dict[Tuple[Any, Any], float]]) -> None:
+ print(f'\\section{{{output_latex_escape(experiment)}}}')
+ print('\paragraph{Base config:}')
+ print(r'\begin{description}[nosep]')
+ for key, value in base_config.items():
+ key = output_latex_escape(key)
+ print(f'\\item[{key}=] \\verb|{value}|')
+ print(r'\end{description}')
+ print(r'\begin{tabular}{p{5cm} r r r r r r r r r r r r r}')
+ print(r' \toprule')
+ print(r' \multirow{2}{*}{Config} & \multirow{2}{*}{Macro-\(F_1\)} & \multicolumn{3}{c}{hero} & \multicolumn{3}{c}{villain} & \multicolumn{3}{c}{victim} & \multicolumn{3}{c}{other} \\')
+ print(r' & & \(F_1\) & \(P\) & \(R\) & \(F_1\) & \(P\) & \(R\) & \(F_1\) & \(P\) & \(R\) & \(F_1\) & \(P\) & \(R\) \\')
+ print(r' \midrule')
+ for run, result in run_results.items():
+ config = run_to_config(run)
+ for key, value in config.items():
+ key = output_latex_escape(key)
+ print(f' \\strong{{{key}=}} \\verb|{value}|', end='')
+ for key in ["macro-F1", "hero F1", "hero P", "hero R", "villain F1", "villain P", "villain R", "victim F1", "victim P", "victim R", "other F1", "other P", "other R"]:
+ data = result[key]
+ if data["best"]:
+ print(f' & \strong{{{data["mean"]:.2f}}}', end='')
+ else:
+ print(f' & {data["mean"]:.2f}', end='')
+ print(r'\\')
+ print(r' \bottomrule')
+ print(r'\end{tabular}')
+
+
+def post_latex() -> None:
+ print(r"\end{document}")
+
+
+def main(args: argparse.Namespace) -> None:
+ if args.html:
+ pre, output, post = pre_html, output_html, post_html
+ elif args.latex:
+ pre, output, post = pre_latex, output_latex, post_latex
+ pre()
+ experiments = get_experiments(args)
+ for experiment, paths in experiments.items():
+ configs = {}
+ results = {}
+ for path in paths:
+ with path.open("r") as file:
+ data = list(map(lambda line: line.rstrip("\n"), file.readlines()))
+ result = get_result(args, data)
+ config = get_config(data)
+ if result and config:
+ results[path] = result
+ configs[path] = config
+ if not configs:
+ continue
+ base_config, delta_configs = merge_configs(configs)
+ runs = merge_runs(delta_configs)
+ run_results = merge_results(runs, results)
+ mark_best(run_results)
+ pvalues = compute_pvalues(args, runs, delta_configs, run_results)
+ output(experiment, base_config, runs, run_results, pvalues)
+ post()
+
+
+if __name__=="__main__":
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--all-test", action="store_true", help="Compute all significance test methods.")
+ parser.add_argument("--exclude", type=str, nargs='+', default=["old"], help="Ignore these directories")
+ parser.add_argument("--group-threshold", type=int, default=10, help="Maximum levenshtein (add=del=2, mod=1) between experiences grouped in the same experiment.")
+ parser.add_argument("--html", action="store_true", help="Output detailed table in HTML format.")
+ parser.add_argument("--latex", action="store_true", help="Output table in LaTeX format.")
+ parser.add_argument("--min-epoch", type=int, default=25, help="Ignore runs that did not reached this epoch number.")
+ parser.add_argument("--prefix", type=str, default="", help="Prefix of the stdout files to consider, can be used to select a specific experiment.")
+ parser.add_argument("--split", type=str, default="test", help="Output results on the given split (val or test).")
+ parser.add_argument("--suffix", type=str, default=".out", help="Suffix of the stdout files to consider.")
+ parser.add_argument("path", type=pathlib.Path, help="Path to the directory containing stdout files.")
+ args = parser.parse_args()
+ if args.html == args.latex:
+ raise RuntimeError("You must use exactly one of --html or --latex.")
+ main(args)
diff --git a/interpret_entity_embeddings.py b/interpret_entity_embeddings.py
@@ -0,0 +1,103 @@
+import numpy as np
+import os
+import json
+import collections
+import ipdb
+import clip
+import torch
+from sklearn.manifold import TSNE
+from sklearn.decomposition import PCA
+import tqdm
+import pandas as pd
+import matplotlib.pyplot as plt
+import seaborn as sns
+import pickle
+
+labels_list = ["hero", "villain", "victim", "other"]
+label_frequencies = [0.027121160214685396, 0.13857485440219253, 0.05195843325339728, 0.7823455521297248]
+
+
+device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
+min_occurences=30 #min nb of occurrences of entities
+
+entities_dict = collections.defaultdict(list)
+annot_file = os.path.join('../mmsrl', 'all', 'annotations', 'train' + '.jsonl')
+with open(annot_file, 'r') as json_file:
+ dataset = list(map(json.loads, json_file))
+for sample in dataset:
+ for i, label in enumerate(labels_list):
+ for entity in sample[label]:
+ entities_dict[entity].append(label)
+
+def normalise_label_freq(counter):
+ counts = np.array([counter.get(label, 0) for label in labels_list])
+ counts = np.divide(counts, label_frequencies)
+ counts = counts / sum(counts)
+ return counts
+
+
+entities = []
+main_label = []
+label_distrib = []
+frequency = []
+for entity, labels in entities_dict.items():
+ entities.append(entity)
+ counter = collections.Counter(labels)
+ distrib = normalise_label_freq(counter)
+ main_label.append(distrib.argmax())
+ label_distrib.append(distrib)
+ frequency.append(len(labels))
+
+print(f"Extracted {len(entities_dict)} entities.")
+entities_dict_reduc = {k:v for k,v in entities_dict.items() if len(v)>min_occurences}
+print(f"keeping {len(entities_dict_reduc)} entities appearing more than {min_occurences} times in the dataset.")
+labels_reduc = [l for l, e in zip(main_label, entities) if e in entities_dict_reduc]
+distrib_reduc = [l for l, e in zip(label_distrib, entities) if e in entities_dict_reduc]
+
+print(collections.Counter(labels_reduc))
+# Get list of entities and encode them
+entities_list_final = entities_dict_reduc.keys()
+entities_input_ids = [clip.tokenize(entity, truncate=True).squeeze(0).to(device) for entity in entities_list_final]
+embeddings = []
+
+# load model and get entities embeddings
+
+model_path = '/data/almanach/user/smontari/scratch/constraint_challenge/Multimodal-semantic-role-labeling/models/model_clip_0.bin'
+model = torch.load(model_path, map_location=torch.device(device))
+
+print("loading model...")
+clip_model, _ = clip.load('ViT-L/14', device=device)
+for entity_encoding in tqdm.tqdm(entities_input_ids):
+ entities_features = clip_model.encode_text(entity_encoding.unsqueeze(0)).to(dtype=model['entities_linear.weight'].dtype)
+ entities_embedding = torch.nn.functional.linear(entities_features, weight = model['entities_linear.weight'], bias = model['entities_linear.bias'])
+ embeddings.append(np.array(entities_embedding.squeeze(0).cpu().detach(), dtype='float32'))
+
+
+# tsne = TSNE(n_components=2, init='random').fit_transform(embeddings)
+
+
+pca = PCA(n_components=2)
+pca.fit(embeddings)
+print(pca.explained_variance_ratio_) #array([0.3405144 , 0.18068498])
+
+pca_features = pca.transform(embeddings)
+
+df = pd.DataFrame()
+df["comp-1"] = pca_features[:,0]
+df["comp-2"] = pca_features[:,1]
+
+final_dict = {}
+for i, entity in enumerate(entities_list_final):
+ final_dict[entity] = {}
+ counter = collections.Counter(entities_dict[entity])
+ final_dict[entity]['distrib'] = np.array([counter.get(label, 0) for label in labels_list])
+ final_dict[entity]['feature'] = pca_features[i]
+
+pickle.dump(final_dict, open('pca_coordinates.pkl', 'wb'))
+
+plt.figure(figsize=(8,5))
+
+sns.scatterplot(x="comp-1", y="comp-2", hue=labels_reduc,
+ palette=sns.color_palette("hls", 4),
+ data=df, legend="full").set(title="T-SNE projection")
+plt.savefig('embeddings_pca_correct.png')
diff --git a/mmsrl/baseline.py b/mmsrl/baseline.py
@@ -0,0 +1,43 @@
+import torch
+import transformers
+
+import mmsrl.loss
+
+
+class Model(torch.nn.Module):
+ def __init__(self, config, label_weights):
+ super().__init__()
+ self.config = config
+ self.transformer = transformers.AutoModel.from_pretrained(self.config.modelname)
+ # TODO tune the smaller output size for the linears to compute attention?
+ self.text_linear= torch.nn.Linear(self.transformer.config.hidden_size, self.config.hidden_size)
+ self.entities_linear= torch.nn.Linear(self.transformer.config.hidden_size, self.config.hidden_size)
+ self.classifier = torch.nn.Linear(self.transformer.config.hidden_size, 4) # output * number of classes
+ #self.relu = torch.nn.ReLU()
+ self.loss = mmsrl.loss.Loss(self.config, label_weights)
+
+ def forward(self, input_ids, attention_mask, entities_mask, entities_input_ids, entities_attention_mask, labels=None, **kwargs):
+ # entities are tri-dimensionnal (batch * nb of entities * emb dim) while text is bi-dimensional
+ nb_entities = entities_input_ids.shape[1]
+ batch_size = entities_input_ids.shape[0]
+ fake_batch_size = batch_size*nb_entities
+
+ text_output = self.transformer(input_ids, attention_mask)
+
+ text = self.text_linear(text_output.last_hidden_state)
+ entities = self.entities_linear(self.transformer(entities_input_ids.view(fake_batch_size, -1), entities_attention_mask.view(fake_batch_size, -1)).pooler_output).view(batch_size, nb_entities, -1)
+ attention = torch.einsum("blf,bef->bel", text, entities)
+
+ # attention = dot product on each element of the batch, on the entities
+ attention_entities_mask = entities_mask.unsqueeze(2) & attention_mask.unsqueeze(1) # make the softmax only on entities and words --> be->be1 & bl->b1l --> bel
+
+ attention_max = torch.max(attention,dim=2,keepdim=True)[0] # max along sentence length
+ attention_exp = torch.exp(attention-attention_max)
+ attention_exp = attention_exp * attention_entities_mask.float()
+ attention_softmax = attention_exp /( torch.sum(attention_exp,dim=2,keepdim=True) + (1-attention_entities_mask.float()) )
+
+ # use this as query for image transformer output
+ entity_attention_output = torch.sum(attention_softmax.unsqueeze(3) * text_output.last_hidden_state.unsqueeze(1), dim=2) # sum along sentence length: torch.Size([8, 10, 64, 1]) + torch.Size([8, 1, 64, 768] --> torch.Size([8, 10, 768])
+
+ logits = self.classifier(entity_attention_output) # batch * entity * nb_labels (4)
+ return self.loss(logits, labels)
diff --git a/mmsrl/batcher.py b/mmsrl/batcher.py
@@ -0,0 +1,164 @@
+from typing import Any, Dict, Iterable, Iterator, List, Optional, Set, Tuple
+import collections
+
+import torch
+
+import mmsrl.utils
+
+
+class Batcher:
+ """
+ Batch a group of sample together.
+
+ Two new features are derived from the "text": its length and a mask.
+ """
+
+ def __init__(self, config: mmsrl.utils.dotdict, text_pad: int, label_pad: int, image_pad: int) -> None:
+ """ Initialize a Batcher, using the provided value to pad text. """
+ self.config: mmsrl.utils.dotdict = config
+ self.text_pad: int = text_pad
+ self.label_pad: int = label_pad
+ self.image_pad: int = image_pad
+
+ def process_text(self, batch: Dict[str, Any], target: str, prefix: str, pad: int) -> None:
+ """ Build mask and text batch by padding sentences. """
+ if target not in batch:
+ return
+ batch_size: int = len(batch[target])
+ if isinstance(batch[target][0], torch.Tensor):
+ length: torch.Tensor = torch.tensor([len(text) for text in batch[target]], dtype=torch.int64)
+ max_seq_len: int = max(length.max().item(), self.config.get("min_text_len", 0))
+
+ text: torch.Tensor = torch.empty((batch_size, max_seq_len), dtype=torch.int64)
+ mask: torch.Tensor = torch.empty((batch_size, max_seq_len), dtype=torch.bool)
+ for b, sentence in enumerate(batch[target]):
+ text[b, :sentence.shape[0]] = sentence
+ text[b, sentence.shape[0]:] = pad
+ mask[b, :sentence.shape[0]] = True
+ mask[b, sentence.shape[0]:] = False
+
+ batch[target] = text
+ batch[f"{prefix}length"] = length
+ batch[f"{prefix}attention_mask"] = mask
+ else:
+ assert(isinstance(batch[target][0], list) and isinstance(batch[target][0][0], torch.Tensor))
+ generate_mask2: bool = "decoder_answer_mask" in batch and target == "decoder_target"
+ generate_mask3: bool = "decoder_constraint" in batch and target == "decoder_target"
+
+ sup_length: int = len(batch[target][0])
+ length: torch.Tensor = torch.tensor([[len(text) for text in sup] for sup in batch[target]], dtype=torch.int64)
+ max_seq_len: int = max(length.max().item(), self.config.get("min_text_len", 0))
+
+ text: torch.Tensor = torch.empty((batch_size, sup_length, max_seq_len), dtype=torch.int64)
+ mask: torch.Tensor = torch.empty((batch_size, sup_length, max_seq_len), dtype=torch.bool)
+ if generate_mask2:
+ mask2: torch.Tensor = torch.empty((batch_size, sup_length, max_seq_len), dtype=torch.bool)
+ if generate_mask3:
+ vocabulary_size: int = batch["decoder_constraint"][0][0].shape[1]
+ mask3: torch.Tensor = torch.empty((batch_size, sup_length, max_seq_len, vocabulary_size), dtype=torch.bool)
+ for b, sup in enumerate(batch[target]):
+ for i, sentence in enumerate(sup):
+ text[b, i, :sentence.shape[0]] = sentence
+ text[b, i, sentence.shape[0]:] = pad
+ mask[b, i, :sentence.shape[0]] = True
+ mask[b, i, sentence.shape[0]:] = False
+ if generate_mask2:
+ mask2[b, i, :sentence.shape[0]] = batch["decoder_answer_mask"][b][i]
+ mask2[b, i, sentence.shape[0]:] = False
+ if generate_mask3:
+ mask3[b, i, :sentence.shape[0]] = batch["decoder_constraint"][b][i]
+ mask3[b, i, sentence.shape[0]:] = False
+
+ batch[target] = text
+ batch[f"{prefix}length"] = length
+ batch[f"{prefix}attention_mask"] = mask
+ if generate_mask2:
+ batch["decoder_answer_mask"] = mask2
+ if generate_mask3:
+ batch["decoder_constraint"] = mask3
+
+ def process_entities(self, batch: Dict[str, Any]) -> None:
+ """ Build mask and entities text batch by padding list of entities and their input ids. """
+ batch_size: int = len(batch["entities_name"])
+ max_num_entities: int = max(len(entities) for entities in batch["entities_name"])
+ max_num_entities = max(max_num_entities, self.config.get("min_nb_ent", 0))
+
+ if "labels" in batch:
+ if batch["labels"][0].ndim == 2:
+ assert(batch["labels"][0].shape[1] == len(mmsrl.utils.LABELS))
+ labels: torch.Tensor = torch.empty((batch_size, max_num_entities, len(mmsrl.utils.LABELS)), dtype=torch.bool)
+ else:
+ labels: torch.Tensor = torch.empty((batch_size, max_num_entities), dtype=torch.int64)
+
+ for b, entities in enumerate(batch["entities_name"]):
+ labels[b, :len(entities)] = batch["labels"][b]
+ labels[b, len(entities):] = self.label_pad
+ batch["labels"] = labels
+
+ mask: torch.Tensor = torch.empty((batch_size, max_num_entities), dtype=torch.bool)
+ for b, entities in enumerate(batch["entities_name"]):
+ mask[b, :len(entities)] = True
+ mask[b, len(entities):] = False
+ batch["entities_mask"] = mask
+
+ if "entities_input_ids" in batch:
+ lengths: torch.Tensor = torch.zeros((batch_size, max_num_entities), dtype=torch.int64)
+ for b, entities in enumerate(batch["entities_input_ids"]):
+ for i, entity in enumerate(entities):
+ lengths[b, i] = len(entity)
+ max_seq_len: int = max(lengths.max().item(), self.config.get("min_ent_len", 0))
+
+ text: torch.Tensor = torch.empty((batch_size, max_num_entities, max_seq_len), dtype=torch.int64)
+ text_mask: torch.Tensor = torch.empty((batch_size, max_num_entities, max_seq_len), dtype=torch.bool)
+ for b, entities in enumerate(batch["entities_input_ids"]):
+ text[b, len(entities):] = self.text_pad
+ text_mask[b, len(entities):] = False
+ for i, entity in enumerate(entities):
+ text[b, i, :len(entity)] = entity
+ text[b, i, len(entity):] = self.text_pad
+ text_mask[b, i, :len(entity)] = True
+ text_mask[b, i, len(entity):] = False
+
+ batch["entities_input_ids"] = text
+ batch["entities_length"] = lengths
+ batch["entities_attention_mask"] = text_mask
+
+ def process_image(self, batch: Dict[str, Any]) -> None:
+ """ Build width, height and mask by padding images. """
+ batch_size: int = len(batch["image"])
+ heights: torch.Tensor = torch.tensor([image.shape[1] for image in batch["image"]], dtype=torch.int64)
+ widths: torch.Tensor = torch.tensor([image.shape[2] for image in batch["image"]], dtype=torch.int64)
+ height: int = max(heights.max().item(), self.config.get("height_min", 0))
+ width: int = max(widths.max().item(), self.config.get("width_min", 0))
+ images: torch.Tensor = torch.full((batch_size, 3, height, width), self.image_pad, dtype=torch.float32)
+ mask: torch.Tensor = torch.zeros((batch_size, height, width), dtype=torch.bool)
+
+ for b, image in enumerate(batch["image"]):
+ mask[b, :image.shape[1], :image.shape[2]] = True
+ images[b, :, :image.shape[1], :image.shape[2]] = image
+
+ batch["image"] = images
+ batch["image_mask"] = mask
+ batch["height"] = heights
+ batch["width"] = widths
+
+ def __call__(self, samples: List[Dict[str, Any]]) -> Dict[str, Any]:
+ """ Batch the provided samples """
+ batch = collections.defaultdict(list)
+ for sample in samples:
+ for key, value in sample.items():
+ batch[key].append(value)
+
+ self.process_text(batch, "input_ids", "", self.text_pad)
+ self.process_text(batch, "vbert_input_ids", "vbert_", self.text_pad)
+ self.process_text(batch, "caption_input_ids", "caption_", self.text_pad)
+ self.process_text(batch, "decoder_input_ids", "decoder_", self.text_pad)
+ self.process_text(batch, "decoder_target", "decoder_target_", self.label_pad)
+ self.process_entities(batch)
+ if batch.get("image"):
+ self.process_image(batch)
+ for feature in list(batch.keys()):
+ if isinstance(batch[feature], list) and isinstance(batch[feature][0], torch.Tensor):
+ batch[feature] = torch.stack(batch[feature])
+
+ return batch
diff --git a/mmsrl/dataset.py b/mmsrl/dataset.py
@@ -0,0 +1,498 @@
+from typing import List
+
+import json
+import os
+import pathlib
+import pandas
+import random
+import collections
+
+import torch
+import PIL
+import torchvision
+import mmsrl.utils
+
+
+class MemeDataset(torch.utils.data.IterableDataset):
+ """
+ Creating a custom dataset for reading the dataset and
+ loading it into the dataloader to pass it to the
+ neural network for finetuning the model
+
+ """
+
+ def __init__(self, config, split, is_eval, tokenizer=None, ofa_task=None, image_processor=None, image_features_loader=None, vbert_tokenizer = None):
+ """
+ Initializes a Dataset class
+
+ Args:
+ dataname in ['uspolitics','covid19', 'all']
+ split in ['train', 'val', 'test', 'unseen_test']
+ tokenizer (transformers.tokenizer): Transformers tokenizer
+ """
+ super().__init__()
+ self.config = config
+ self.tokenizer = tokenizer
+ self.split = split
+ self.is_eval = is_eval
+ self.ofa_task = ofa_task
+ self.has_labels = ('unseen' not in split)
+
+ if os.environ.get("DATA_PATH", "") == "":
+ raise RuntimeError("Environment variable DATA_PATH is not set. Run the following command: export DATA_PATH=\"$PWD/../data\"")
+ data_path = pathlib.Path(os.environ['DATA_PATH'])
+ annot_file = data_path / "mmsrl" / self.config.dataname / 'annotations' / f"{split}.jsonl"
+ with annot_file.open('r') as json_file:
+ self.dataset = list(map(json.loads, json_file))
+
+ self.images_path = data_path / "mmsrl" / self.config.dataname / 'images'
+ if self.config.get('use_caption') and self.config.dataname == "all":
+ caption_path = data_path / "mmsrl" / self.config.dataname / "captions.csv"
+ self.captions_df = pandas.read_csv(caption_path, sep='\t', header=None)
+
+ mean = [0.5, 0.5, 0.5]
+ std = [0.5, 0.5, 0.5]
+ self.image_transform = image_processor
+ if image_features_loader is not None:
+ self.image_features_loader = image_features_loader
+ if vbert_tokenizer is not None:
+ self.vbert_tokenizer = vbert_tokenizer
+ elif self.config.model in ["ofa", "mmf"]:
+ self.image_transform = torchvision.transforms.Compose([
+ lambda image: image.convert("RGB"),
+ self.resize_image,
+ torchvision.transforms.ToTensor(),
+ torchvision.transforms.Normalize(mean=mean, std=std),
+ ])
+
+ if self.config.model == "ofa" and not self.config.get("group_image_entities"):
+ self.dataset_len = 0
+ for sample in self.dataset:
+ if "entity_list" in sample:
+ self.dataset_len += len(set(sample["entity_list"]))
+ else:
+ self.dataset_len += sum(len(sample[label]) for label in mmsrl.utils.LABELS)
+ # If we ever remove entities appearing twice in samples:
+ #entities = set()
+ #for label in mmsrl.utils.LABELS:
+ # entities.update(sample[label])
+ #self.dataset_len += len(entities)
+
+ if self.ofa_task:
+ self.ofa_bos = torch.LongTensor([self.ofa_task.src_dict.bos()])
+ self.ofa_eos = torch.LongTensor([self.ofa_task.src_dict.eos()])
+
+ if self.config.get("maybe_is", "") in ["yes", "no"]:
+ self.snli_answers = ["no", "maybe", "yes"]
+ else:
+ self.snli_answers = ["no", "yes"]
+
+ if self.config.get("constrain_output"):
+ assert(self.config.model == "ofa")
+ assert(not self.config.get("only_score_answer"))
+ assert(not self.config.get("only_train_answer"))
+ assert(not self.config.get("ofa_classification_head"))
+ assert(self.config.get("decoder_input") is not None and self.config.decoder_input == "{label}")
+ if self.config.ofa_task == "vqa_gen":
+ self.build_trie(mmsrl.utils.LABELS)
+ else:
+ self.build_trie(self.snli_answers)
+
+ self.seed(0)
+ self.compute_subsample(epoch=0)
+
+ def seed(self, id: int):
+ self.common_rng = random.Random(0)
+ self.rng = random.Random(id)
+
+ def __len__(self):
+ """returns the length of dataframe"""
+ if not self.is_eval and self.config.get("batch_per_epoch"):
+ return self.config.batch_per_epoch * self.config.batch_size
+ elif self.config.model == "ofa" and not self.config.get("group_image_entities"):
+ return self.dataset_len
+ else:
+ # Approximation due to sub-sampling entities deleting empty samples.
+ return len(self.dataset)
+
+ def get_label_weights(self):
+ if self.config.get("class_weights"):
+ label_weights = torch.zeros(4, dtype=torch.float32, requires_grad=False)
+ for sample in self.dataset:
+ for i, label in enumerate(mmsrl.utils.LABELS):
+ label_weights[i] += len(sample[label])
+ label_weights = 1/label_weights
+ label_weights /= label_weights.mean() # normalise
+ else:
+ label_weights = torch.ones(4, dtype=torch.float32, requires_grad=False)
+ return label_weights
+
+ def compute_subsample(self, epoch):
+ if self.is_eval:
+ return
+ if self.config.get("cyclic_subsample") is not None:
+ config = mmsrl.utils.dotdict(self.config.cyclic_subsample[epoch % len(self.config.cyclic_subsample)])
+ elif epoch>0 and not self.config.get("subsample_labels", "").startswith("interpolate"):
+ return
+ else:
+ config = self.config
+
+ frequencies = self.config.label_frequencies
+ if config.get("subsample_labels", "micro") == "micro":
+ target_frequencies = frequencies
+ elif config.subsample_labels == "macro":
+ target_frequencies = [0.25]*4
+ elif config.subsample_labels == "interpolate_micro_to_macro":
+ initial_frequencies = frequencies
+ final_frequencies = [0.25]*4
+ factor = epoch / (self.config.max_epoch - 1)
+ target_frequencies = [
+ factor * final + (1 - factor) * initial
+ for initial, final in zip(initial_frequencies, final_frequencies)]
+ elif config.subsample_labels == "affine":
+ target_frequencies = [frequency * config.frequency_factor + config.frequency_bias for frequency in frequencies]
+ elif config.subsample_labels == "target":
+ target_frequencies = config.frequency_target
+ else:
+ raise RuntimeError(f"Unknown value for config.subsample_labels: {config.subsample_labels}")
+ target_frequencies[3] *= config.get("subsample_other", 1)
+ target_frequencies = [frequency / sum(target_frequencies) for frequency in target_frequencies]
+ sampling_frequencies = [frequency / target for frequency, target in zip(frequencies, target_frequencies)]
+ sampling_frequencies = [frequency / sum(sampling_frequencies) for frequency in sampling_frequencies]
+ self.subsample_probs = [min(sampling_frequencies) / frequency for frequency in sampling_frequencies]
+
+ repr_frequencies = ", ".join(f"{f:5.3f}" for f in target_frequencies)
+ repr_subsample = ", ".join(f"{p:5.3f}" for p in self.subsample_probs)
+ if self.split == "train":
+ print(f"\033[33mSubsampling factors are [{repr_subsample}]\033[0m (target frequencies: [{repr_frequencies}])")
+
+ def resize_image(self, image):
+ if max(image.size) > self.config.image_max_edge:
+ factor: float = self.config.image_max_edge / max(image.size)
+ height: int = round(image.size[0] * factor)
+ width: int = round(image.size[1] * factor)
+ image = torchvision.transforms.functional.resize(image, (height, width), interpolation=torchvision.transforms.InterpolationMode.BICUBIC)
+ return image
+
+ def process_image(self, image_path: pathlib.Path):
+ if self.config.model in ["baseline", "none"]:
+ return {}
+ image = PIL.Image.open(image_path)
+ if self.config.model in ['clip_momenta', 'clip', 'generic']:
+ image = self.image_transform(image)
+ if self.config.features_path:
+ feature = self.image_features_loader(image_path)
+ return {'image': image, 'image_feature': feature}
+ else:
+ return {'image': image}
+ elif self.config.model == 'visualbert':
+ image = self.image_transform(image_path)
+ elif self.image_transform is not None:
+ image = self.image_transform(image)
+ return {'image': image}
+
+ def process_text(self, text):
+ if self.config.model == "ofa":
+ if isinstance(text, str):
+ text = text.lower().replace('-', ' ').replace('/', ' ')
+ if not text:
+ return torch.tensor([], dtype=torch.int64)
+ encoded = self.ofa_task.tgt_dict.encode_line(
+ line=self.ofa_task.bpe.encode(f" {text}"),
+ add_if_not_exist=False,
+ append_eos=False
+ ).long()
+ return encoded[:self.config.max_text_len]
+ elif isinstance(text, tuple):
+ prefix, infix, suffix = tuple(map(self.process_text, text))
+ total_length = prefix.shape[0] + infix.shape[0] + suffix.shape[0] + 2
+ if total_length > self.config.max_text_len:
+ infix = infix[:max(total_length - self.config.max_text_len, 0)]
+ encoded = torch.cat([self.ofa_bos, prefix, infix, suffix, self.ofa_eos])
+ return encoded[:self.config.max_text_len]
+ else:
+ raise RuntimeError("text is of wrong type {type(text)}")
+ elif self.config.model in ["clip", "clip_momenta", "generic"]:
+ return self.tokenizer(text, truncate=True).squeeze(0)[:self.config.max_text_len]
+ elif self.config.model in ["baseline", 'visualbert']:
+ return self.tokenizer.encode_plus(
+ text,
+ max_length=self.config.max_text_len,
+ truncation=True,
+ return_tensors="pt",
+ )["input_ids"].squeeze(0).to(dtype=torch.long)
+ else:
+ return None
+
+ def create_affirmation(self, entity: str, label: str) -> str:
+ if label != "other":
+ return f"{entity} is a {label}."
+ else:
+ assert(self.config.get("neither_for_other")) # For another commit
+ classes = mmsrl.utils.LABELS.copy()
+ classes.remove("other")
+ self.rng.shuffle(classes)
+ class_string = ", ".join(classes[:-1]) + f" nor {classes[-1]}"
+ return f"{entity} is neither a {class_string}."
+
+ def create_question(self, entity, ocr):
+ classes = mmsrl.utils.LABELS.copy()
+ self.rng.shuffle(classes)
+ class_string = ", ".join(classes[:-1]) + f" or {classes[-1]}"
+ #question = f"{entity} belongs to one of the following categorie: {class_string}. Which one is it ?"
+ question = f"What is the category of {entity} between {class_string} ?"
+
+ if self.config.vqa_input == "question ocr":
+ return (question, ocr, "")
+ elif self.config.vqa_input == "ocr question":
+ return ("", ocr, question)
+ elif self.config.vqa_input == "question":
+ return ("", question, "")
+ else:
+ raise RuntimeError(f"Unknown value for config.vqa_input: {self.config.vqa_input}")
+
+ def create_premises(self, entity, ocr):
+ if self.config.get("ofa_classification_head"):
+ labels = ["other"]
+ else:
+ labels = mmsrl.utils.LABELS
+
+ premises = []
+ for label in labels:
+ affirmation = self.create_affirmation(entity, label)
+ premises.append((('can image and text1 "', ocr, '" imply text2 " {affirmation} "?'), label))
+ return premises
+
+ def prepare_ofa_io(self, ocr, entity, true_label):
+ test_answers: bool = self.is_eval and not self.config.get("ofa_classification_head")
+ if self.config.get("ofa_classification_head"):
+ true_label = "other"
+
+ encoder_input = []
+ decoder_input = []
+ if self.config.ofa_task == "vqa_gen":
+ source = self.create_question(entity, ocr)
+ encoder_input.append(source)
+ if test_answers:
+ for label in mmsrl.utils.LABELS:
+ if self.config.get("decoder_input") is not None:
+ decoder_input.append((self.config.decoder_input.format(label=label), "", ""))
+ else:
+ decoder_input.append((source[0], source[1], f"{source[2]} {label}"))
+ else:
+ if self.config.get("decoder_input") is not None:
+ decoder_input.append((self.config.decoder_input.format(label=true_label), "", ""))
+ else:
+ decoder_input.append((source[0], source[1], f"{source[2]} {true_label}"))
+ elif self.config.ofa_task == "snli_ve":
+ for source, label in self.create_premises(entity, ocr):
+ encoder_input.append(source)
+ if test_answers:
+ for answer in self.snli_answers:
+ if self.config.get("decoder_input") is not None:
+ decoder_input.append((self.config.decoder_input.format(label=answer), "", ""))
+ else:
+ decoder_input.append((source[0], source[1], f"{source[2]} {answer}"))
+ else:
+ answer = "yes" if label == true_label else "no"
+ if self.config.get("decoder_input") is not None:
+ decoder_input.append((self.config.decoder_input.format(label=answer), "", ""))
+ else:
+ decoder_input.append((source[0], source[1], f"{source[2]} {answer}"))
+ else:
+ raise RuntimeError(f"Unknown value for self.config.ofa_task: {self.config.ofa_task}")
+ return encoder_input, decoder_input
+
+ def subsample_labels(self, entities, labels):
+ if self.subsample_probs == [1, 1, 1, 1]:
+ return entities, labels
+
+ delete_me: List[int] = []
+ for i, label in enumerate(labels):
+ if self.config.get("multilabel"):
+ for j, val in enumerate(label):
+ if val and torch.rand(1) > self.subsample_probs[j]:
+ label[j] = 0
+ if label[j] == [0,0,0,0]:
+ delete_me.append(i)
+ else:
+ if torch.rand(1) > self.subsample_probs[label]:
+ delete_me.append(i)
+
+ # We good a good score while doing this is old code…
+ if self.config.get("keep_all_batches"):
+ if len(delete_me) == len(entities):
+ self.rng.shuffle(delete_me)
+ delete_me = delete_me[1:]
+
+ delete_me.sort(reverse=True)
+ for index in delete_me:
+ labels.pop(index)
+ entities.pop(index)
+
+ return entities, labels
+
+ def generate_mask(self, input_ids, text):
+ mask = torch.zeros(input_ids.shape[0]-1, dtype=torch.bool)
+ mask[-2] = True
+ if text[-1].endswith("victim") or text[-1].endswith("villain"):
+ mask[-3] = True
+ return mask
+
+ def build_trie(self, answers: List[str]):
+ self.trie = collections.defaultdict(lambda: torch.zeros(len(self.ofa_task.tgt_dict), dtype=torch.bool))
+ for answer in answers:
+ tokenized = torch.cat([self.process_text(answer), self.ofa_eos])
+ for i in range(len(tokenized)):
+ self.trie[tuple(tokenized[:i].tolist())][tokenized[i]] = True
+
+ def get_constraint(self, target: torch.Tensor) -> torch.Tensor:
+ constraint = torch.zeros((target.shape[0], len(self.ofa_task.tgt_dict)), dtype=torch.bool)
+ for i in range(target.shape[0]):
+ constraint[i] = self.trie[tuple(target[:i].tolist())]
+ return constraint
+
+ def index_to_sample(self, index):
+ """return the input ids, images and target ids"""
+ sample = {}
+
+ sample["image_name"] = self.dataset[index]["image"]
+ image_data = self.process_image(self.images_path / sample["image_name"])
+ if image_data is None:
+ return
+ if self.config.model == "visualbert":
+ sample['image_feature'] = image_data['image']
+ else:
+ sample.update(image_data)
+
+ sample["raw_text"] = self.dataset[index]["OCR"]
+ if self.config.get('use_caption'):
+ caption = self.captions_df[self.captions_df[0]==sample["image_name"]][1].item() + '. '
+ sample["caption"] = caption
+ sample["caption_input_ids"] = self.process_text(sample["caption"])
+
+ if self.has_labels:
+ targets = {l: self.dataset[index][l] for l in mmsrl.utils.LABELS}
+ entities = [t for l in targets for t in targets[l]]
+
+ if self.config.get("multilabel"):
+ labels = []
+ for ent in entities:
+ ent_lab = [mmsrl.utils.LABELS.index(l) for l in targets for t in targets[l] if t==ent]
+ ent_lab_distrib = [0]*len(mmsrl.utils.LABELS)
+ for l in ent_lab: ent_lab_distrib[l] = 1
+ labels.append(ent_lab_distrib)
+ else:
+ labels = [mmsrl.utils.LABELS.index(l) for l in targets for t in targets[l]]
+
+ if not self.is_eval:
+ entities, labels = self.subsample_labels(entities, labels)
+ if not entities:
+ return
+
+ ids = list(range(len(entities)))
+ self.rng.shuffle(ids)
+ ids = ids[:self.config.get("max_nb_ent", len(ids))]
+ entities = [entities[i] for i in ids]
+ labels = [labels[i] for i in ids]
+
+ sample["labels"] = torch.tensor(labels, dtype=torch.int64)
+ else:
+ entities = self.dataset[index]["entity_list"]
+
+ sample["entities_name"] = entities
+
+ if self.config.model == "ofa":
+ if self.config.get("group_image_entities"):
+ assert(self.config.get("ofa_classification_head"))
+ if self.has_labels:
+ sample["labels"] = torch.tensor(labels, dtype=torch.int64)
+ else:
+ None
+ sample["text"] = []
+ sample["decoder_text"] = []
+ for i, entity in enumerate(entities):
+ label = labels[i] if self.has_labels else None
+ label_text = mmsrl.utils.LABELS[label] if label is not None else None
+ encoder_input, decoder_input = self.prepare_ofa_io(sample["raw_text"], entity, label_text)
+ assert(len(encoder_input) == 1)
+ assert(len(decoder_input) == 1)
+ sample["text"].append(encoder_input[0])
+ sample["decoder_text"].append(decoder_input[0])
+ sample["input_ids"] = list(map(self.process_text, sample["text"]))
+ decoder_text = list(map(self.process_text, sample["decoder_text"]))
+ if not self.config.get("ofa_classification_head") and not self.config.get("keep_eof"):
+ decoder_text = [text[:-1] for text in decoder_text]
+ sample["decoder_input_ids"] = [text[:-1] for text in decoder_text]
+ sample["decoder_target"] = [text[1:] for text in decoder_text]
+ if self.config.get("constrain_output"):
+ sample["decoder_constraint"] = [self.get_constraint(target) for target in sample["decoder_target"]]
+ yield sample
+ else:
+ for i, entity in enumerate(entities):
+ current_sample = {}
+ current_sample["raw_text"] = sample["raw_text"]
+ current_sample["image_name"] = sample["image_name"]
+ current_sample["image"] = sample["image"]
+
+ if self.has_labels:
+ label = labels[i]
+ current_sample["labels"] = torch.tensor([label], dtype=torch.int64)
+ else:
+ label = None
+
+ label_text = mmsrl.utils.LABELS[label] if label is not None else None
+ encoder_input, decoder_input = self.prepare_ofa_io(sample["raw_text"], entity, label_text)
+ current_sample["text"] = encoder_input
+ current_sample["decoder_text"] = decoder_input
+
+ current_sample["input_ids"] = list(map(self.process_text, encoder_input))
+ decoder_text = list(map(self.process_text, decoder_input))
+ if not self.config.get("ofa_classification_head") and not self.config.get("keep_eof"):
+ decoder_text = [text[:-1] for text in decoder_text]
+ current_sample["decoder_input_ids"] = [text[:-1] for text in decoder_text]
+ current_sample["decoder_target"] = [text[1:] for text in decoder_text]
+ if self.config.get("only_score_answer") or self.config.get("only_train_answer"):
+ current_sample["decoder_answer_mask"] = [self.generate_mask(input_ids, text) for input_ids, text in zip(decoder_text, decoder_input)]
+ if self.config.get("constrain_output"):
+ current_sample["decoder_constraint"] = [self.get_constraint(target) for target in current_sample["decoder_target"]]
+
+ current_sample["entities_name"] = [entity]
+ yield current_sample
+ else:
+ sample["text"] = sample["raw_text"]
+ sample["input_ids"] = self.process_text(sample["text"])
+ if self.config.get('use_visualbert'):
+ sample["vbert_input_ids"] = self.vbert_tokenizer.encode_plus(
+ sample["text"],
+ max_length=self.config.max_text_len,
+ truncation=True,
+ return_tensors="pt",
+ )["input_ids"].squeeze(0).to(dtype=torch.long)
+ sample["entities_input_ids"] = [self.process_text(entity) for entity in entities]
+ yield sample
+
+ def __iter__(self):
+ worker_info = torch.utils.data.get_worker_info()
+ if worker_info is None:
+ worker_modulo: int = 1
+ worker_residue: int = 0
+ else:
+ worker_modulo: int = worker_info.num_workers
+ worker_residue: int = worker_info.id
+
+ if not self.is_eval and self.config.get("batch_per_epoch"):
+ sample_per_worker = len(self) // worker_modulo + (1 if worker_residue < len(self) % worker_modulo else 0)
+ sample_generated = 0
+ while sample_generated < sample_per_worker:
+ samples = list(self.index_to_sample(self.rng.randint(0, len(self.dataset)-1)))
+ yield from samples[:sample_per_worker - sample_generated]
+ sample_generated += len(samples)
+ else:
+ self.order = list(range(len(self.dataset)))
+ if not self.is_eval:
+ self.common_rng.shuffle(self.order)
+
+ for i, index in enumerate(self.order):
+ if i % worker_modulo == worker_residue:
+ yield from self.index_to_sample(index)
diff --git a/mmsrl/ensembling.py b/mmsrl/ensembling.py
@@ -0,0 +1,285 @@
+from typing import Any, Dict, List, Tuple
+import argparse
+import collections
+import json
+import os
+import re
+import sys
+import pathlib
+import pickle
+
+import tqdm
+import numpy
+import sklearn.metrics
+import sklearn.linear_model
+import xgboost
+
+import mmsrl.utils
+
+
+def build_model_paths(paths):
+ model_paths = set()
+ for path in paths:
+ for filename in os.listdir(path):
+ modelname = filename.replace("val", "{}").replace("test", "{}")
+ if modelname.count("{}") == 1:
+ model_paths.add(path / modelname)
+ elif modelname.count("{}") > 1:
+ print(f"\033[31;1mWARNING: ignoring file with several \"val\" and \"test\" in filename: {filename}.\033[0m")
+ return model_paths
+
+
+def read_prediction_file(path):
+ try:
+ data = collections.defaultdict(dict)
+ with open(path, "rb") as file:
+ try:
+ while True:
+ image, entities = pickle.load(file)
+ for entity, prediction in entities.items():
+ data[image][entity] = prediction.numpy()
+ except EOFError:
+ return data
+ except FileNotFoundError:
+ return {}
+
+
+def read_predictions(paths):
+ data = {}
+ for path in tqdm.tqdm(paths, desc="Loading predictions", leave=False):
+ prediction = str(path)
+ data[prediction] = {}
+ for split in ["val", "test"]:
+ path = prediction.replace("{}", split)
+ data[prediction][split] = read_prediction_file(path)
+ return data
+
+
+def read_data(split):
+ data_path = pathlib.Path(os.environ["DATA_PATH"])
+ path = data_path / "mmsrl" / "all" / "annotations" / f"{split}.jsonl"
+ with path.open("r") as file:
+ return list(map(json.loads, file))
+
+
+def write_prediction_file(path, data):
+ with open(path, "wb") as file:
+ for image, predictions in data.items():
+ pickle.dump((image, predictions), file)
+
+
+def get_sample_entities(sample):
+ if "entity_list" in sample:
+ return sample["entity_list"]
+ else:
+ return sum((sample[label] for label in mmsrl.utils.LABELS), [])
+
+
+def models_to_tensors(data, models):
+ tensors = {}
+ for sample in data:
+ image = sample["image"]
+ tensors[image] = {}
+ for entity in set(get_sample_entities(sample)):
+ if not entity:
+ continue
+ values = []
+ for model in models:
+ values.append(model[image][entity])
+ tensors[image][entity] = numpy.stack(values)
+ return tensors
+
+
+def model_is_valid(model, dataset):
+ try:
+ for sample in dataset:
+ image = sample["image"]
+ for entity in get_sample_entities(sample):
+ if entity:
+ pred = model[image][entity]
+ if not numpy.isfinite(pred).all():
+ return False
+ except KeyError:
+ return False
+ return True
+
+
+def compute_score(predictions, truth, final, split):
+ y_pred = []
+ y_truth = []
+ for sample in truth:
+ image = sample["image"]
+ for ilabel, label in enumerate(mmsrl.utils.LABELS):
+ for entity in sample[label]:
+ if entity:
+ y_pred.append(predictions[image][entity].argmax().item())
+ else:
+ y_pred.append(3)
+ y_truth.append(ilabel)
+
+ f1 = sklearn.metrics.f1_score(y_truth, y_pred, average="macro", zero_division=0)
+ class_precision, class_recall, class_f1, _ = sklearn.metrics.precision_recall_fscore_support(y_truth, y_pred, zero_division=0)
+ class_scores = ", ".join(f"{label}[F{class_f1[i]:4.2f}/P{class_precision[i]:4.2f}/R{class_recall[i]:4.2f}]" for i, label in enumerate(mmsrl.utils.LABELS))
+ color = "32" if final else "33"
+ return f1, f"{split} Macro-F1: \033[{color};1m{f1:5.3f}\033[0m, {class_scores}"
+
+
+def models_to_xy(models, valid):
+ nmodels = len(models)
+ x = []
+ y = []
+ order = []
+ for sample in valid:
+ image = sample["image"]
+ for ilabel, label in enumerate(mmsrl.utils.LABELS):
+ for entity in sample[label]:
+ if not entity:
+ continue
+ row = numpy.empty((nmodels,4), dtype=numpy.float32)
+ for im, model in enumerate(models):
+ row[im] = model[image][entity]
+ x.append(row)
+ y.append(ilabel)
+ order.append((image, entity))
+ return numpy.stack(x), numpy.array(y, dtype=numpy.int64), order
+
+
+def build_ensemble(config, valid, models):
+ tmodels = models_to_tensors(valid, models)
+ if config.aggregator in ["linear", "xgboost"]:
+ x, y, order = models_to_xy(models, valid)
+
+ if config.aggregator in "linear":
+ if config.full_matrix:
+ linear = sklearn.linear_model.LogisticRegression(penalty="none", class_weight="balanced", max_iter=1000)
+ linear.fit(x.reshape(x.shape[0], -1), y)
+ else:
+ nsample = y.shape[0]
+ label_weight = nsample / 4 / numpy.bincount(y)
+ nmodels = len(models)
+ x = x.transpose(0, 2, 1)
+ x = x.reshape(-1, nmodels)
+ y = y.repeat(4)
+ sample_weight = label_weight[y]
+ y = (y.reshape(nsample, 4) == numpy.arange(4)).flatten()
+
+ linear = sklearn.linear_model.LogisticRegression(penalty="none", max_iter=1000)
+ linear.fit(x, y, sample_weight=sample_weight)
+ ensemble = linear.coef_
+ elif config.aggregator == "xgboost":
+ nsample = y.shape[0]
+ label_weight = nsample / 4 / numpy.bincount(y)
+ param = {'objective': 'multi:softmax', 'num_class': 4}
+ param.update(eval(config.xgboost_params))
+ print(f"\033[33mXGBoost parameters: {param}\033[0m")
+ ensemble = xgboost.XGBRegressor(**param)
+ ensemble.fit(x.reshape(2067, -1), y, sample_weight=label_weight[y])
+ if config.save_xgboost:
+ ensemble.save_model(str(config.output / get_output_name(config)) + ".xgboost.json")
+ with open(str(config.output / get_output_name(config)) + ".xgboost.map", "wb") as outfile:
+ pickle.dump(order, outfile)
+ else:
+ ensemble = None
+
+ prediction = ensemble_predict(config, valid, ensemble, models)
+ print(compute_score(prediction, valid, True, "VALID")[1])
+ return ensemble
+
+
+def ensemble_predict_sample(config, ensemble, models):
+ if config.aggregator == "mean":
+ return models.mean(0)
+ elif config.aggregator == "max":
+ return models.max(0)[0]
+ elif config.aggregator == "min":
+ return models.min(0)[0]
+ elif config.aggregator == "median":
+ models.sort(0)
+ return models[models.shape[0]//2, :]
+ elif config.aggregator == "linear":
+ return ensemble.dot(models.flatten() if config.full_matrix else models)
+ elif config.aggregator == "xgboost":
+ probabilities = numpy.zeros(4, dtype=numpy.float32)
+ prediction = ensemble.predict(models.reshape(1, -1))
+ probabilities[int(prediction)] = 1
+ return probabilities
+ else:
+ raise RuntimeError(f"Unknown aggregator {config.aggregator}")
+
+
+def ensemble_predict(config, test, ensemble, models):
+ tmodels = models_to_tensors(test, models)
+ output = {}
+ for sample in test:
+ image = sample["image"]
+ output[image] = {}
+ for entity in set(get_sample_entities(sample)):
+ if entity:
+ output[image][entity] = ensemble_predict_sample(config, ensemble, tmodels[image][entity])
+ else:
+ output[image][entity] = numpy.array([0, 0, 0, 1], dtype=numpy.float32)
+ return output
+
+
+def filter_models(config: mmsrl.utils.dotdict, models, val_data, test_data):
+ useme = {}
+ for model_name, model in models.items():
+ if not (model_is_valid(model["val"], val_data) and model_is_valid(model["test"], test_data)):
+ continue
+ f1, msg = compute_score(model["val"], val_data, False, "VALID")
+ if f1 > config.cut_off:
+ useme[model_name] = model
+ if config.show_all:
+ print(f"\033[33m{model_name}\033[0m")
+ print(msg)
+ return useme
+
+
+def get_output_name(config: mmsrl.utils.dotdict) -> str:
+ name = f"ensemble_{config.aggregator}_"
+ name += str(config.cut_off).split('.')[-1]
+ if config.aggregator == "linear" and config.full_matrix:
+ name += "_fullmatrix"
+ if config.aggregator == "xgboost":
+ name += re.sub("[^a-zA-Z0-9.]+", "_", config.xgboost_params).rstrip('_')
+ return f"{name}.pkl"
+
+
+def perform_ensembling(config: mmsrl.utils.dotdict, models, val_data, test_data):
+ models_filtered = filter_models(config, models, val_data, test_data)
+ models_order = list(models_filtered.keys())
+ models = [models[key] for key in models_order]
+ ensemble = build_ensemble(config, val_data, [model["val"] for model in models])
+ prediction = ensemble_predict(config, test_data, ensemble, [model["test"] for model in models])
+ outfile = config.output / get_output_name(config)
+ print(f"Writing test prediction to \033[1m{outfile}\033[0m")
+ write_prediction_file(outfile, prediction)
+ print(compute_score(prediction, test_data, True, "TEST ")[1])
+
+
+def main(config: mmsrl.utils.dotdict) -> None:
+ val_data = read_data("val")
+ test_data = read_data("test")
+ models_path = build_model_paths(config.prediction)
+ models = read_predictions(models_path)
+ if config.cut_off==0:
+ for cut in range(31, 45, 2):
+ print(cut/100)
+ config.cut_off = cut /100
+ perform_ensembling(config, models, val_data, test_data)
+ else:
+ perform_ensembling(config, models, val_data, test_data)
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser()
+ parser.add_argument("output", type=pathlib.Path, help="Path were we write the test prediction output.")
+ parser.add_argument("prediction", type=pathlib.Path, nargs='+', help="Paths to directories containing prediction files, the filenames should contain \"val\" and \"test\" such that substituting one for other gives the corresponding predictions on the other split.")
+ parser.add_argument("--aggregator", type=str, default="mean", help="How to aggregate results.")
+ parser.add_argument("--full-matrix", action="store_true", help="When using logistic regression, learn a full 4*nmodel→4 matrix instead of a 4→1 one.")
+ parser.add_argument("--cut-off", type=float, default=0.3, help="Ignore models with a lower F1 on the validation set.")
+ parser.add_argument("--show-all", action="store_true", help="Show validation score of all (not cut-off) models.")
+ parser.add_argument("--save-xgboost", action="store_true", help="Save the xgboost model and mapping.")
+ parser.add_argument("--xgboost-params", type=str, default="{}", help="Should be eval()uable to a dict which is then added to xgboost parameters.")
+ args = mmsrl.utils.dotdict(vars(parser.parse_args()))
+ main(args)
diff --git a/mmsrl/exploration.py b/mmsrl/exploration.py
@@ -0,0 +1,119 @@
+import numpy as np
+import pandas as pd
+import os
+import json
+from scipy.stats import entropy
+import collections
+from transformers import AutoTokenizer
+
+import mmsrl.utils
+
+
+tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
+
+dataset = {}
+
+#datasetnames = ['uspolitics','covid19']
+datasetnames = ['all']
+for name in datasetnames:
+ dataset[name] = {}
+ for split in ['train', 'val', 'test']:
+ annot_file = os.path.join('../mmsrl', name, 'annotations', split + '.jsonl')
+ with open(annot_file, 'r') as json_file:
+ dataset[name][split] = list(map(json.loads, json_file))
+
+labels_list = mmsrl.utils.LABELS
+stats_table = pd.DataFrame()
+metrics_names = ['size', 'avg_nb_tokens', 'max_nb_tokens', 'nb_token_quantile95','class_distrib', 'avg_nb_ents', 'max_nb_ents', 'avg_nb_token_ents', 'max_nb_token_ents', 'total_nb_ents', 'percent_ents_in_text', 'total_nb_unique_ents', 'nb_duplicated_ent', 'samples_with_duplicate', 'entropy']
+stats_table['metric_name'] = metrics_names
+
+def fill_dict(sample, ent_label_dict):
+ for i, label in enumerate(labels_list):
+ for ent in sample[label]:
+ if ent not in ent_label_dict:
+ ent_label_dict[ent] = [0]*len(labels_list)
+ ent_label_dict[ent][i] +=1
+
+for name in datasetnames:
+ for split in ['train', 'val', 'test']:
+ colname = name + '_' + split
+ metrics = []
+ data = dataset[name][split]
+ metrics.append(len(data))
+ text = [data[i]['OCR'] for i in range(len(data))]
+ tokenized = [tokenizer.tokenize(t) for t in text]
+ nbtok = [len(t) for t in tokenized]
+ metrics.append(np.mean(nbtok))
+ metrics.append(np.max(nbtok))
+ metrics.append(round(np.percentile(nbtok, 95), 2))
+ #print(pd.Series(nbtok).describe())
+ class_distrib = [0]*len(labels_list)
+ nbents = []
+ entities = []
+ duplicated_ents = 0
+ sample_with_duplicates = 0
+ ent_label_dict = {}
+ nb_ent_in_text = []
+ for sample in data:
+ sample_ents = []
+ # add entities and labels to the global dictionary
+ fill_dict(sample, ent_label_dict)
+ for i, label in enumerate(labels_list):
+ # compute class distribution
+ class_distrib[i] += len(sample[label])
+ sample_ents.extend(sample[label])
+ # check if some entities are in several classes
+ if len(sample_ents) != len(set(sample_ents)):
+ #print("Duplicated entity!")
+ #print(sample)
+ duplicated_ents += (len(sample_ents)- len(set(sample_ents)))
+ sample_with_duplicates +=1
+ entities.extend(sample_ents)
+ nbents.append(len(sample_ents))
+ # check if the entities are in the text
+ ents_in_text = [(ent in sample['OCR']) for ent in sample_ents]
+ nb_ent_in_text.append(np.sum(ents_in_text))
+
+ class_distrib /= np.sum(class_distrib)
+ class_distrib = [round(num, 2) for num in class_distrib]
+ metrics.append(class_distrib)
+ metrics.append(np.mean(nbents))
+ metrics.append(np.max(nbents))
+
+ tokenized = [tokenizer.tokenize(t) for t in entities]
+ nbtokent = [len(t) for t in tokenized]
+ metrics.append(np.mean(nbtokent))
+ metrics.append(np.max(nbtokent))
+
+ metrics.append(len(entities))
+
+ percent_ents_in_text = np.sum(nb_ent_in_text) / len(entities) * 100
+ metrics.append(round(percent_ents_in_text, 2))
+ metrics.append(len(set(entities)))
+
+ metrics.append(duplicated_ents)
+ metrics.append(sample_with_duplicates)
+
+ # entropy: how are the classes distributed fow each entity
+
+ dataset_avg_var = 0
+ dataset_avg_entropy = 0
+ for ent, distrib in ent_label_dict.items():
+ mean_abs_dev_ent_ditrib = np.mean(distrib - np.mean(distrib)**2)
+ dataset_avg_var += mean_abs_dev_ent_ditrib
+ dataset_avg_entropy += entropy(distrib)
+ #mean_abs_dev_ent_ditrib = np.mean(np.absolute(distrib - np.mean(distrib)))
+ dataset_avg_var /= len(ent_label_dict)
+ dataset_avg_entropy /= len(ent_label_dict)
+
+ metrics.append((round(dataset_avg_var,2), round(dataset_avg_entropy, 2)))
+
+ stats_table[colname] = metrics
+
+
+ stats_table[colname] = metrics
+
+
+print(stats_table)
+
+stats_table.to_csv('stats_table.csv', sep='\t')
diff --git a/mmsrl/extract_image_features.py b/mmsrl/extract_image_features.py
@@ -0,0 +1,36 @@
+import numpy as np
+import os
+import torch
+
+class ImageFeatureExtractor():
+ def __init__(self, config):
+ self.config = config
+
+ def process_image(self, images):
+ """
+ return pooled or full vecteur, for mlp or attention pooling respectively.
+ If all features are used, return pooled features concatenated (only for mlp model)
+ """
+
+ if self.config.features_path == "all":
+ if self.config.pooling == 'mlp':
+ b7_pooled = np.load(str(images).replace('.png', '_pooled.npy').replace('images', "features_b7")).squeeze(0)
+ vgg_pooled = np.load(str(images).replace('.png', '_pooled.npy').replace('images', "features_vgg")).squeeze(0)
+ frcnn = np.load(str(images).replace('.png', '.npy').replace('images', "features_frcnn")).mean(0)
+ all_features = np.concatenate((vgg_pooled, b7_pooled, frcnn), axis=0)[None,:]
+ else:
+ b7 = np.load(str(images).replace('.png', '.npy').replace('images', "features_b7")).squeeze(0).reshape((2560,196)).transpose()
+ vgg = np.load(str(images).replace('.png', '.npy').replace('images', "features_vgg")).squeeze(0).reshape((512,196)).transpose()
+ frcnn = np.load(str(images).replace('.png', '.npy').replace('images', "features_frcnn")) # 2048, 36
+ all_features = [vgg, b7, frcnn]
+ raise NotImplementedError()
+ return all_features
+ elif self.config.features_path in "features_frcnn":
+ feature = np.load(str(images).replace('.png', '.npy').replace('images', self.config.features_path))
+ return feature.mean(0)[None,:] if self.config.pooling == 'mlp' else feature
+ elif self.config.features_path in ["features_b7", "features_vgg"]:
+ extension = '_pooled.npy' if self.config.pooling == 'mlp' else '.npy'
+ vec = np.load(str(images).replace('.png', extension).replace('images', self.config.features_path))
+ return vec if self.config.pooling == 'mlp' else vec.reshape((vec.shape[1],196)).transpose()
+ else:
+ raise RuntimeError(f"Unknown config value for feature path: {self.config.features_path}")
diff --git a/mmsrl/generate_captions.py b/mmsrl/generate_captions.py
@@ -0,0 +1,129 @@
+import numpy as np
+import os, sys
+import torch
+from fairseq import utils, tasks
+from fairseq import checkpoint_utils
+import PIL
+import torchvision
+import json
+import pathlib
+import pandas as pd
+import tqdm
+
+import mmsrl
+sys.path.append(mmsrl.__path__[0] + "/../OFA")
+
+from utils.eval_utils import eval_step
+from tasks.mm_tasks.caption import CaptionTask
+from models.ofa import OFAModel
+
+
+class ImageCaption():
+ def __init__(self, images_path):
+ # Register caption task
+ tasks.register_task('caption',CaptionTask)
+ self.images_path = images_path
+
+ # turn on cuda if GPU is available
+ self.use_cuda = torch.cuda.is_available()
+ # use fp16 only when GPU is available
+ self.use_fp16 = False
+
+ # Load pretrained ckpt & config
+ overrides={"bpe_dir":"OFA/utils/BPE", "eval_cider":False, "beam":5, "max_len_b":16, "no_repeat_ngram_size":3, "seed":7}
+ self.models, self.cfg, self.task = checkpoint_utils.load_model_ensemble_and_task(
+ utils.split_paths('OFA/checkpoints/caption_large_best_clean.pt'),
+ arg_overrides=overrides
+ )
+
+ # Move models to GPU
+ for model in self.models:
+ model.eval()
+ if self.use_fp16:
+ model.half()
+ if self.use_cuda and not self.cfg.distributed_training.pipeline_model_parallel:
+ model.cuda()
+ model.prepare_for_inference_(self.cfg)
+
+ # Initialize generator
+ self.generator = self.task.build_generator(self.models, self.cfg.generation)
+
+ # Text preprocess
+ self.bos_item = torch.LongTensor([self.task.src_dict.bos()])
+ self.eos_item = torch.LongTensor([self.task.src_dict.eos()])
+ self.pad_idx = self.task.src_dict.pad()
+
+ mean = [0.5, 0.5, 0.5]
+ std = [0.5, 0.5, 0.5]
+
+ self.patch_resize_transform = torchvision.transforms.Compose([
+ lambda image: image.convert("RGB"),
+ torchvision.transforms.Resize((self.cfg.task.patch_image_size, self.cfg.task.patch_image_size), interpolation=PIL.Image.BICUBIC),
+ torchvision.transforms.ToTensor(),
+ torchvision.transforms.Normalize(mean=mean, std=std),
+ ])
+
+ def encode_text(self, text, length=None, append_bos=False, append_eos=False):
+ s = self.task.tgt_dict.encode_line(
+ line=self.task.bpe.encode(text),
+ add_if_not_exist=False,
+ append_eos=False
+ ).long()
+ if length is not None:
+ s = s[:length]
+ if append_bos:
+ s = torch.cat([self.bos_item, s])
+ if append_eos:
+ s = torch.cat([s, self.eos_item])
+ return s
+
+ def construct_sample(self, data, length=None, ocr=False):
+ image = PIL.Image.open(self.images_path / data['image'])
+ patch_image = self.patch_resize_transform(image).unsqueeze(0)
+ patch_mask = torch.tensor([True])
+ src_text = self.encode_text(" what does the image describe?", append_bos=True, append_eos=True)
+ if ocr:
+ ocr_text = data['OCR'].replace('\n', ' ')
+ ocr_encoded = self.encode_text(ocr_text, length=length, append_bos=True, append_eos=True)
+ src_text = torch.cat((ocr_encoded, src_text))
+
+ src_length = torch.LongTensor([src_text.ne(self.pad_idx).long().sum()])
+ sample = {
+ "id":np.array(['42']),
+ "net_input": {
+ "src_tokens": src_text.unsqueeze(0),
+ "src_lengths": src_length,
+ "patch_images": patch_image,
+ "patch_masks": patch_mask
+ }
+ }
+ return sample
+
+ def generate(self, data, length=None, ocr = False):
+
+ # Construct input sample & preprocess for GPU if cuda available
+ sample = self.construct_sample(data, length, ocr)
+ sample = utils.move_to_cuda(sample) if self.use_cuda else sample
+ sample = utils.apply_to_sample(apply_half, sample) if self.use_fp16 else sample
+
+ with torch.no_grad():
+ result, scores = eval_step(self.task, self.generator, self.models, sample)
+
+ return result[0]['caption']
+
+
+if __name__=='__main__':
+
+ data_path = pathlib.Path(os.environ['DATA_PATH'])
+ images_path = data_path / "mmsrl" / "all" / 'images'
+ caption_generator = ImageCaption(images_path)
+ captions_path = data_path / "mmsrl" / "all" / "captions.csv"
+ with open(captions_path, 'w', buffering=1) as f:
+ for split in ['val', 'unseen_test', 'train']:
+ annot_file = data_path / "mmsrl" / "all" / 'annotations' / f"{split}.jsonl"
+ with annot_file.open('r') as json_file:
+ dataset = list(map(json.loads, json_file))
+ #print("split: ", len(dataset), "samples")
+ for data in tqdm.tqdm(dataset, desc=f"{split}", leave=False):
+ caption = caption_generator.generate(data)
+ f.write(data['image'] + '\t' + caption + '\n')
+\ No newline at end of file
diff --git a/mmsrl/generate_image_features.py b/mmsrl/generate_image_features.py
@@ -0,0 +1,280 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+
+"""
+ Run with for example:
+ python3 mmf/tools/scripts/features/frcnn/extract_features_vgg.py \
+ --image_dir ./example_images --output_folder ./output_features
+"""
+
+import argparse
+import copy
+import logging
+import os
+import torchvision
+import numpy as np
+import torch
+import glob
+import math
+import cv2
+import os
+
+
+def get_image_files(
+ image_dir,
+ exclude_list=None,
+ partition=None,
+ max_partition=None,
+ start_index=0,
+ end_index=None,
+ output_folder=None,
+):
+ files = glob.glob(os.path.join(image_dir, "*.png"))
+ files.extend(glob.glob(os.path.join(image_dir, "*.jpg")))
+ files.extend(glob.glob(os.path.join(image_dir, "*.jpeg")))
+
+ files = set(files)
+ exclude = set()
+
+ if os.path.exists(exclude_list):
+ with open(exclude_list) as f:
+ lines = f.readlines()
+ for line in lines:
+ exclude.add(line.strip("\n").split(os.path.sep)[-1].split(".")[0])
+ output_ignore = set()
+ if output_folder is not None:
+ output_files = glob.glob(os.path.join(output_folder, "*.npy"))
+ for f in output_files:
+ file_name = f.split(os.path.sep)[-1].split(".")[0]
+ output_ignore.add(file_name)
+
+ for f in list(files):
+ file_name = f.split(os.path.sep)[-1].split(".")[0]
+ if file_name in exclude or file_name in output_ignore:
+ files.remove(f)
+
+ files = list(files)
+ files = sorted(files)
+
+ if partition is not None and max_partition is not None:
+ interval = math.floor(len(files) / max_partition)
+ if partition == max_partition:
+ files = files[partition * interval :]
+ else:
+ files = files[partition * interval : (partition + 1) * interval]
+
+ if end_index is None:
+ end_index = len(files)
+
+ files = files[start_index:end_index]
+
+ return files
+
+
+def chunks(array, chunk_size):
+ for i in range(0, len(array), chunk_size):
+ yield array[i : i + chunk_size], i
+
+
+class ModelFeatureExtractor(torch.nn.Module):
+ def __init__(self, model):
+ super(ModelFeatureExtractor, self).__init__()
+ # Extract VGG-16 Feature Layers
+ self.features = list(model.features)
+ self.features = torch.nn.Sequential(*self.features)
+
+ # Extract VGG-16 Average Pooling Layer
+ self.pooling = model.avgpool
+ # Convert the image into one-dimensional vector
+ self.flatten = torch.nn.Flatten()
+ # Extract the first part of fully-connected layer from VGG16
+ #self.fc = model.classifier[0]
+
+ def forward(self, x):
+ # It will take the input 'x' until it returns the feature vector called 'out'
+ feature = self.features(x)
+ pooled = self.pooling(feature)
+ pooled = self.flatten(pooled)
+ #out = self.fc(out)
+ return feature, pooled
+
+
+class FeatureExtractor:
+ def __init__(self):
+ self.args = self.get_parser().parse_args()
+ self.args.output_folder = self.args.output_folder + '_' + self.args.modelname + '/'
+ if not os.path.exists(self.args.output_folder):
+ os.mkdir(self.args.output_folder)
+ if self.args.modelname == 'vgg':
+ model_vgg_pretrained = torchvision.models.vgg16(pretrained=True)
+ self.model = ModelFeatureExtractor(model_vgg_pretrained)
+ elif self.args.modelname == 'b7':
+ model_b7_pretrained = torchvision.models.efficientnet_b7(pretrained=True)
+ self.model = ModelFeatureExtractor(model_b7_pretrained)
+ else:
+ print("Model unknown")
+
+ # Change the device to GPU
+ self.device = torch.device('cuda:0' if torch.cuda.is_available() else "cpu")
+ self.model = self.model.to(self.device)
+
+ # Transform the image, so it becomes readable with the model
+ self.transform_center = torchvision.transforms.Compose([
+ torchvision.transforms.ToPILImage(),
+ torchvision.transforms.CenterCrop(512),
+ torchvision.transforms.Resize(448),
+ torchvision.transforms.ToTensor(),
+ torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406],
+ std=[0.229, 0.224, 0.225])
+ ])
+
+ def get_parser(self):
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ "--start_index", default=0, type=int, help="Index to start from "
+ )
+ parser.add_argument("--end_index", default=None, type=int, help="")
+ parser.add_argument("--batch_size", type=int, default=1, help="Batch size")
+ parser.add_argument(
+ "--num_features",
+ type=int,
+ default=100,
+ help="Number of features to extract.",
+ )
+ parser.add_argument(
+ "--output_folder", type=str, default="/data/almanach/user/smontari/scratch/constraint_challenge/mmsrl/all/features", help="Output folder"
+ )
+ parser.add_argument("--image_dir", type=str, help="Image directory or file", default='/data/almanach/user/smontari/scratch/constraint_challenge/mmsrl/all/images/')
+ parser.add_argument(
+ "--modelname",
+ type=str,
+ help="The name of the feature to extract",
+ default="vgg",
+ )
+ parser.add_argument(
+ "--exclude_list",
+ type=str,
+ help="List of images to be excluded from feature conversion. "
+ + "Each image on a new line",
+ default="./list",
+ )
+ parser.add_argument(
+ "--confidence_threshold",
+ type=float,
+ default=0,
+ help="Threshold of detection confidence above which boxes will be selected",
+ )
+ # TODO finish background flag
+ parser.add_argument(
+ "--background",
+ action="store_true",
+ help="The model will output predictions for the background class when set",
+ )
+ parser.add_argument(
+ "--padding",
+ type=str,
+ default=None,
+ help="You can set your padding, i.e. 'max_detections'",
+ )
+ parser.add_argument(
+ "--visualize",
+ type=bool,
+ default=False,
+ help="Add this flag to save the extra file used for visualization",
+ )
+ parser.add_argument(
+ "--partition",
+ type=int,
+ default=None,
+ help="Add this flag to save the extra file used for visualization",
+ )
+ parser.add_argument(
+ "--max_partition",
+ type=int,
+ default=None,
+ help="Add this flag to save the extra file used for visualization",
+ )
+ return parser
+
+ def _save_feature(self, file_name, feature, pooled):
+ file_base_name = os.path.basename(file_name)
+ file_base_name = file_base_name.split(".")[0]
+ feat_list_base_name = file_base_name + ".npy"
+ np.save(
+ os.path.join(self.args.output_folder, file_base_name + ".npy"),
+ feature.cpu().numpy(),
+ )
+ np.save(
+ os.path.join(self.args.output_folder, file_base_name + "_pooled.npy"),
+ pooled.cpu().numpy(),
+ )
+
+ def extract_features(self):
+ image_dir = self.args.image_dir
+ if os.path.isfile(image_dir):
+ feature, pooled = self.get_image_center([image_dir])
+ self._save_feature(image_dir, feature, pooled)
+ else:
+ files = get_image_files(
+ self.args.image_dir,
+ exclude_list=self.args.exclude_list,
+ partition=self.args.partition,
+ max_partition=self.args.max_partition,
+ start_index=self.args.start_index,
+ end_index=self.args.end_index,
+ )
+
+ finished = 0
+ total = len(files)
+ failed = 0
+ failedNames = []
+
+ file_names = copy.deepcopy(files)
+
+ for chunk, begin_idx in chunks(files, self.args.batch_size):
+ try:
+ feature, pooled = self.get_image_center(chunk)
+ for idx, file_name in enumerate(chunk):
+ self._save_feature(
+ file_names[begin_idx + idx],
+ feature, pooled,
+ )
+ finished += len(chunk)
+
+ if finished % 200 == 0:
+ print(f"Processed {finished}/{total}")
+ except Exception:
+ failed += len(chunk)
+ for idx, file_name in enumerate(chunk):
+ failedNames.append(file_names[begin_idx + idx])
+ logging.exception("message")
+ if self.args.partition is not None:
+ print("Partition " + str(self.args.partition) + " done.")
+ print("Failed: " + str(failed))
+ print("Failed Names: " + str(failedNames))
+
+ def get_image_center(self, chunk):
+ # Set the image path
+ imgs: torch.Tensor = torch.empty((len(chunk), 3, 448, 448), dtype=torch.float32)
+ for i, path in enumerate(chunk):
+ img = cv2.imread(path)
+ img = self.transform_center(img)
+ # Reshape the image. PyTorch model reads 4-dimensional tensor
+ # [batch_size, channels, width, height]
+ img = img.reshape(3, 448, 448)
+ imgs[i] = img
+
+ imgs = imgs.to(self.device)
+ # We only extract features, so we don't need gradient
+ with torch.no_grad():
+ # Extract the feature from the image
+ #feature = model_vgg(imgs).squeeze()
+ feature, pooled = self.model(imgs)
+
+ return feature, pooled
+
+
+
+if __name__ == "__main__":
+
+ feature_extractor = FeatureExtractor()
+ feature_extractor.extract_features()
diff --git a/mmsrl/generic_classifier.py b/mmsrl/generic_classifier.py
@@ -0,0 +1,225 @@
+import numpy as np
+import torch
+import transformers
+from sentence_transformers import SentenceTransformer
+
+import mmsrl.loss
+import mmsrl.classification
+
+class Model(torch.nn.Module):
+ def __init__(self, config, label_weights, model=None):
+ super().__init__()
+ self.config = config
+ assert self.config.use_clip == True or self.config.use_visualbert == True
+
+ if self.config.features_path=="features_b7":
+ self.features_dim = 2560
+ elif self.config.features_path == "features_vgg":
+ self.features_dim = 25088 if self.config.pooling=='mlp' else 512
+ elif self.config.features_path=="all":
+ self.features_dim = 29696
+ elif self.config.features_path=="features_frcnn":
+ self.features_dim = 2048
+
+ self.model = model
+ if self.config.use_clip: output_emb_dim = self.model.ln_final.bias.shape[0]
+ if self.config.unfreeze == "classifier":
+ self.model.requires_grad_(False)
+ if self.config.use_visualbert:
+ self.image_linear = torch.nn.Linear(self.features_dim, 512)
+ self.transformer = transformers.VisualBertModel.from_pretrained(self.config.visualbert_name, add_pooling_layer = True)
+ if not self.config.use_clip:
+ output_emb_dim = self.transformer.config.hidden_size
+ if self.config.unfreeze == "classifier":
+ self.transformer.requires_grad_(False)
+
+ if self.config.features_path:
+ self.dense_vgg_1024 = torch.nn.Linear(self.features_dim, 1024)
+ self.dense_img_feat = torch.nn.Linear(1024, output_emb_dim)
+ self.drop20 = torch.nn.Dropout(p=0.2)
+ if self.config.use_sbert:
+ self.model_sent_trans = SentenceTransformer(self.config.sbert_model)
+
+ if self.config.pooling == "attention":
+ self.linear_key = torch.nn.Linear(output_emb_dim, self.config.hidden_size)
+ self.linear_value = torch.nn.Linear(output_emb_dim, self.config.hidden_size)
+
+ self.entities_linear = torch.nn.Linear(output_emb_dim, self.config.hidden_size)
+ self.classifier = torch.nn.Linear(self.config.hidden_size, 4) # output * number of classes
+ self.loss = mmsrl.loss.Loss(self.config, label_weights)
+ self.relu = torch.nn.ReLU()
+
+ self.num_features = 0
+ if self.config.use_clip: self.num_features +=2
+ if self.config.use_visualbert: self.num_features +=1
+ if self.config.features_path and self.config.use_clip: self.num_features +=1
+ if self.config.use_caption: self.num_features +=1
+ if self.config.use_entities: self.num_features +=1
+ print("Number of features in total: ", self.num_features)
+
+ if self.config.pooling == "attention":
+ self.segment_embeddings = torch.nn.Embedding(self.num_features, self.config.hidden_size)
+ elif self.config.pooling != "mlp":
+ raise RuntimeError(f"Unknown config value for config.pooling in clip: {self.config.pooling}")
+
+ #self.model.transformer.resblocks[-1].requires_grad_(True)
+ #self.model.visual.transformer.resblocks[-1].requires_grad_(True)
+
+ def train(self, value=True):
+ super().train(value)
+ self.model.eval()
+ if self.config.use_visualbert:
+ self.transformer.eval()
+
+ def clip_encode_text_without_pooling(self, text):
+ """ This code comes from clip, except that we removed the selection along text length. """
+ x = self.model.token_embedding(text).type(self.model.dtype) # [batch_size, n_ctx, d_model]
+
+ x = x + self.model.positional_embedding.type(self.model.dtype)
+ x = x.permute(1, 0, 2) # NLD -> LND
+ x = self.model.transformer(x)
+ x = x.permute(1, 0, 2) # LND -> NLD
+ x = self.model.ln_final(x).type(self.model.dtype)
+
+ # x.shape = [batch_size, n_ctx, transformer.width]
+ x = x @ self.model.text_projection
+ return x
+
+ def clip_encode_image_without_pooling(self, image):
+ """ This code comes from clip, except that we removed the selection along image length. """
+ x = image.to(dtype=self.model.dtype)
+ x = self.model.visual.conv1(x) # shape = [*, width, grid, grid]
+ x = x.reshape(x.shape[0], x.shape[1], -1) # shape = [*, width, grid ** 2]
+ x = x.permute(0, 2, 1) # shape = [*, grid ** 2, width]
+ x = torch.cat([self.model.visual.class_embedding.to(x.dtype) + torch.zeros(x.shape[0], 1, x.shape[-1], dtype=x.dtype, device=x.device), x], dim=1) # shape = [*, grid ** 2 + 1, width]
+ x = x + self.model.visual.positional_embedding.to(x.dtype)
+ x = self.model.visual.ln_pre(x)
+
+ x = x.permute(1, 0, 2) # NLD -> LND
+ x = self.model.visual.transformer(x)
+ x = x.permute(1, 0, 2) # LND -> NLD
+
+ x = self.model.visual.ln_post(x)
+
+ if self.model.visual.proj is not None:
+ x = x @ self.model.visual.proj
+
+ return x
+
+ def forward(self, input_ids, image, attention_mask, entities_mask, entities_input_ids, entities_attention_mask, entities_name, image_feature = None, vbert_attention_mask = None, vbert_input_ids = None, caption_input_ids = None, caption_attention_mask=None, labels=None, **kwargs):
+ # entities are tri-dimensionnal (batch * nb of entities * emb dim) while text is bi-dimensional
+ nb_entities = entities_input_ids.shape[1]
+ batch_size = entities_input_ids.shape[0]
+ fake_batch_size = batch_size*nb_entities
+
+ features = []
+ masks =[]
+ segments = []
+
+ # 1) Encoding from CLIP
+ if self.config.use_clip:
+ num_feat = len(features)
+ if self.config.pooling == "mlp":
+ clip_text_features = self.model.encode_text(input_ids)
+ clip_image_features = self.model.encode_image(image)
+ elif self.config.pooling == "attention":
+ clip_text_features = self.clip_encode_text_without_pooling(input_ids)
+ clip_image_features = self.clip_encode_image_without_pooling(image)
+
+ features.append(clip_text_features)
+ features.append(clip_image_features)
+ attention_mask = input_ids!=0
+ segments.append(attention_mask.new_ones(batch_size, clip_text_features.shape[1], dtype=torch.int64)*num_feat)
+ segments.append(attention_mask.new_ones(batch_size, clip_image_features.shape[1], dtype=torch.int64)*num_feat)
+ masks.append(attention_mask)
+ masks.append(attention_mask.new_ones(batch_size, clip_image_features.shape[1]))
+
+ # 2) Image features pre-extracted using various models
+ if self.config.features_path and self.config.use_clip: # visualbert already use them
+ num_feat = len(features)
+ image_feature_tensor = torch.tensor(np.array(image_feature), device=input_ids.device)
+ image_feature_tensor = self.drop20(torch.nn.functional.relu(self.dense_img_feat(self.drop20(torch.nn.functional.relu(self.dense_vgg_1024(image_feature_tensor))))))
+ if self.config.pooling == "mlp":
+ image_feature_tensor = image_feature_tensor.squeeze(1)
+ features.append(image_feature_tensor)
+ # All image features have the same shape
+ segments.append(attention_mask.new_ones(batch_size, image_feature_tensor.shape[1], dtype=torch.int64)*num_feat)
+ masks.append(attention_mask.new_ones(batch_size, image_feature_tensor.shape[1]))
+
+ # 3) List of all entities of the meme, with one embedding per entity, concatenated.
+ if self.config.use_entities:
+ num_feat = len(features)
+ if self.config.get("use_sbert"):
+ ent = torch.vstack([torch.mean(torch.tensor(self.model_sent_trans.encode(entities)), dim=0) for entities in entities_name])
+ entities_list_embedding = torch.stack(tuple([torch.tensor(self.model_sent_trans.encode(entities + ['']*(entities_mask.shape[1] - len(entities)))) for entities in entities_name])).to(device = input_ids.device)
+ else:
+ entities_list_embedding = self.model.encode_text(entities_input_ids.view(fake_batch_size, -1)).view(batch_size, nb_entities, -1)
+ #entities_feat = self.drop5(torch.nn.functional.relu(entities_embedding)) # batch * nb_ent * emb_dim
+ if self.config.pooling == "mlp":
+ entities_list_embedding = (entities_list_embedding*entities_mask.unsqueeze(2)).sum(1)/entities_mask.sum(1,keepdim=True)
+ features.append(entities_list_embedding)
+ segments.append(entities_mask.new_ones(batch_size, entities_list_embedding.shape[1], dtype=torch.int64)*num_feat)
+ masks.append(entities_mask)
+
+ # 4) Caption of the meme
+ if self.config.use_caption:
+ num_feat = len(features)
+ # should we use sentenceBERT and just concat it with the clip_text_emb ?
+ #cap = torch.tensor(self.model_sent_trans.encode(caption)).to(device = input_ids.device)
+ #text_embedding_pooled = self.caption_entiy_linear(torch.cat((entities_embedding_pooled, self.caption_linear(cap)), dim=1))
+ if self.config.pooling == "mlp":
+ caption_embedding = self.model.encode_text(caption_input_ids)
+ elif self.config.pooling == "attention":
+ caption_embedding = self.clip_encode_text_without_pooling(caption_input_ids)
+ features.append(caption_embedding)
+ caption_attention_mask = caption_input_ids!=0
+ segments.append(caption_attention_mask.new_ones(batch_size, caption_embedding.shape[1], dtype=torch.int64)*num_feat)
+ masks.append(caption_attention_mask)
+
+ # 5) Features from visualBERT:
+ if self.config.use_visualbert:
+ num_feat = len(features)
+ visual_embeds = self.image_linear(torch.tensor(np.array(image_feature), device=input_ids.device).to(device=input_ids.device))
+ visual_attention_mask = torch.ones(visual_embeds.shape[:-1], dtype=torch.long).to(device=input_ids.device)
+ visual_token_type_ids = torch.ones(visual_embeds.shape[:-1], dtype=torch.long).to(device=input_ids.device)
+ token_type_ids = torch.ones(vbert_attention_mask.shape, dtype=torch.long).to(device=input_ids.device)
+ outputs = self.transformer(input_ids=vbert_input_ids, attention_mask=vbert_attention_mask, token_type_ids=token_type_ids,
+ visual_embeds=visual_embeds, visual_attention_mask=visual_attention_mask, visual_token_type_ids=visual_token_type_ids, output_hidden_states=True, return_dict=True)
+ if self.config.pooling == "mlp":
+ features.append(outputs.pooler_output)
+ elif self.config.pooling == "attention":
+ features.append(outputs.last_hidden_state)
+ segments.append(outputs.last_hidden_state.new_ones(batch_size, outputs.last_hidden_state.shape[1], dtype=torch.int64)*num_feat)
+ masks.append(entities_mask.new_ones(batch_size, outputs.last_hidden_state.shape[1]))
+
+ # 6) Entity feature to be used as query
+ entities_features = self.model.encode_text(entities_input_ids.view(fake_batch_size, -1)).view(batch_size, nb_entities, -1)
+ entities_embedding = self.entities_linear(entities_features.to(dtype=self.entities_linear.weight.dtype))
+
+ assert self.num_features == len(features)
+ features_value = list(map(self.linear_value, [f.to(dtype=self.linear_value.weight.dtype) for f in features]))
+
+ # 7) MLP and attention
+ if self.config.pooling == "mlp":
+ features_value = sum(features_value)
+ # We want the joint representation of the features and entities.
+ hidden = self.relu(features_value.unsqueeze(1) + entities_embedding / len(features) +1 )
+ elif self.config.pooling == "attention":
+ features_key = list(map(self.linear_key, [f.to(dtype=self.linear_key.weight.dtype) for f in features]))
+
+ memory_key = torch.cat(features_key, dim=1) + self.segment_embeddings(torch.cat(segments, dim=1))
+ memory_value = torch.cat(features_value, dim=1) + self.segment_embeddings(torch.cat(segments, dim=1))
+ mask = torch.cat(masks, dim=1)
+
+ attention = torch.einsum("blf,bef->bel", memory_key, entities_embedding)
+ attention_entities_mask = entities_mask.unsqueeze(2) & mask.unsqueeze(1) # make the softmax only on entities and words --> be->be1 & bl->b1l --> bel
+ attention_max = torch.max(attention, dim=2, keepdim=True)[0] # max along sentence length
+ attention_exp = torch.exp(attention-attention_max)
+ attention_exp = attention_exp * attention_entities_mask.float()
+ attention_softmax = attention_exp / (attention_exp.sum(dim=2,keepdim=True) + (1-attention_entities_mask.float()))
+
+ hidden = torch.sum(attention_softmax.unsqueeze(3) * memory_value.unsqueeze(1), dim=2) # sum along sentence length: torch.Size([8, nbent, seqlen, 1]) + torch.Size([8, 1, seqlen, f] --> torch.Size([8, nbent, f])
+ hidden = self.relu(hidden)
+
+ logits = self.classifier(hidden) # batch * entity * nb_labels (4)
+ return self.loss(logits, labels)
diff --git a/mmsrl/loss.py b/mmsrl/loss.py
@@ -0,0 +1,34 @@
+from typing import Optional
+
+import torch
+
+
+class Loss(torch.nn.Module):
+ def __init__(self, config, label_weights):
+ super().__init__()
+ self.config = config
+ self.label_weights = label_weights
+ # reduction none because we weight it with class weights and then we do the average ourself.
+ if self.config.multilabel:
+ self.loss = torch.nn.BCEWithLogitsLoss(reduction="none")
+ else:
+ self.loss = torch.nn.CrossEntropyLoss(reduction="none", label_smoothing=self.config.get("label_smoothing", 0))
+
+ def forward(self, logits: torch.Tensor, labels: Optional[torch.Tensor] = None):
+ if labels is None:
+ loss = None
+ elif self.config.multilabel:
+ loss = self.loss(logits, labels)
+ loss = loss*self.label_weights[labels%4] # %4: artisanal way to avoid the -100 when indexing
+ loss = loss.mean()
+ else:
+ loss = self.loss(logits.view(labels.numel(), 4), labels.view(-1)) # numel = nb of elements in labels in total (product of dimensions)
+ loss = loss*self.label_weights[labels.view(-1)%4] # %4: artisanal way to avoid the -100 when indexing
+ loss = loss.mean()
+
+ if self.config.multilabel:
+ prediction = torch.nn.functional.sigmoid(logits, dim=2)
+ else:
+ prediction = torch.nn.functional.softmax(logits, dim=2)
+
+ return loss, prediction
diff --git a/mmsrl/ofa.py b/mmsrl/ofa.py
@@ -0,0 +1,180 @@
+import math
+
+import torch
+
+import mmsrl.loss
+import mmsrl.utils
+
+
+class Model(torch.nn.Module):
+ def __init__(self, config, label_weights, ofa_task, ofa_config, ofa_models):
+ super().__init__()
+ self.config = config
+ self.ofa_task = ofa_task
+ self.ofa_config = ofa_config
+ assert(len(ofa_models) == 1) # Check if it's really an ensemble or a single model
+ self.model = ofa_models[0]
+
+ if self.config.get("ofa_classification_head"):
+ self.model.register_classification_head("mmsrl", 4)
+ self.loss = mmsrl.loss.Loss(self.config, label_weights)
+ else:
+ self.loss = torch.nn.NLLLoss(reduction="none")
+
+ if self.config.unfreeze != "all":
+ self.model.requires_grad_(False)
+
+ if self.config.unfreeze == "classifier":
+ if self.config.get("ofa_classification_head"):
+ self.model.classification_heads["mmsrl"].requires_grad_(True)
+ else:
+ self.model.decoder.output_projection.requires_grad_(True)
+
+ if self.config.get("freeze_embeddings"):
+ self.model.encoder.embed_tokens.requires_grad_(False)
+ self.model.decoder.embed_tokens.requires_grad_(False)
+
+ def train(self, mode=True):
+ super().train(mode)
+ if self.config.get("freeze_resnet"):
+ for module in self.model.encoder.embed_images.modules():
+ if isinstance(module, torch.nn.BatchNorm2d):
+ module.eval()
+ module.requires_grad_(False)
+
+ def masked_log_softmax(self, logits, mask):
+ if mask is None:
+ return torch.nn.functional.log_softmax(logits, dim=2)
+
+ fake_batch_size: int = logits.shape[0]
+ sequence_length: int = logits.shape[1]
+ vocabulary_size: int = logits.shape[2]
+ mask = mask.view(fake_batch_size, sequence_length, vocabulary_size)
+
+ logits_max = torch.max(logits, dim=2, keepdim=True)[0] # max along vocabulary
+ logits_exp = torch.exp(logits - logits_max)
+ logits_exp = mask.float() * logits_exp
+ logits_sum = torch.sum(logits_exp, dim=2, keepdim=True)
+ logits_sum += (logits_sum==0).float() # avoid log(0) when everything is masked
+ log_softmax = logits - logits_max - torch.log(logits_sum)
+ return mask.float() * log_softmax
+
+ def forward(self, input_ids, length, image, image_mask, decoder_input_ids, decoder_target, decoder_target_attention_mask, labels=None, **kwargs):
+ if self.config.get("ofa_classification_head"):
+ if self.config.get("group_image_entities"):
+ batch_size: int = input_ids.shape[0]
+ num_entities: int = input_ids.shape[1]
+ fake_batch_size: int = batch_size * num_entities
+ input_ids = input_ids.view(fake_batch_size, -1)
+ length = length.view(fake_batch_size)
+ decoder_input_ids = decoder_input_ids.view(fake_batch_size, -1)
+ image = image.repeat_interleave(repeats=num_entities, dim=0)
+ image_mask = image_mask.repeat_interleave(repeats=num_entities, dim=0)
+ else:
+ input_ids = input_ids.squeeze(1)
+ length = length.squeeze(1)
+ decoder_input_ids = decoder_input_ids.squeeze(1)
+
+ outputs = self.model(
+ input_ids,
+ length,
+ decoder_input_ids,
+ patch_images=image,
+ patch_masks=image_mask[:, 0, 0],
+ classification_head_name="mmsrl")[0]
+ if input_ids.shape[0] == 1:
+ # fix for stupid wild squeeze from OFA/fairseq.
+ outputs = outputs.unsqueeze(0)
+ if self.config.get("group_image_entities"):
+ logits = outputs.view(batch_size, num_entities, 4)
+ else:
+ # When using OFA without group_image_entities, we always have 1 entity per sample
+ logits = outputs.unsqueeze(1)
+ return self.loss(logits, labels)
+
+ batch_size: int = input_ids.shape[0]
+ inner_input_size: int = input_ids.shape[1]
+ fake_input_batch_size: int = batch_size * inner_input_size
+
+ encoded = self.model.encoder(
+ input_ids.view(fake_input_batch_size, -1),
+ src_lengths=length.view(fake_input_batch_size),
+ patch_images=image.repeat_interleave(inner_input_size, 0),
+ # FIXME build patch_masks the same way patches are built from image.
+ patch_masks=image_mask.repeat_interleave(inner_input_size, 0)[:, 0, 0])
+
+ inner_output_size: int = decoder_input_ids.shape[1]
+ fake_output_batch_size: int = batch_size * inner_output_size
+ decode_per_encode: int = inner_output_size // inner_input_size
+ decoder_sequence_length: int = decoder_input_ids.shape[2]
+
+ encoded["encoder_out"] = [encoded["encoder_out"][0].repeat_interleave(decode_per_encode, 1)]
+ encoded["encoder_padding_mask"] = [encoded["encoder_padding_mask"][0].repeat_interleave(decode_per_encode, 0)]
+ encoded["position_embeddings"] = [encoded["position_embeddings"][0].repeat_interleave(decode_per_encode, 0)]
+
+ decoded = self.model.decoder(
+ decoder_input_ids.view(fake_output_batch_size, decoder_sequence_length),
+ encoder_out=encoded)
+ lm_logits = self.masked_log_softmax(decoded[0], kwargs.get("decoder_constraint"))
+
+ fake_loss_batch_size: int = fake_output_batch_size * decoder_sequence_length
+ lm_loss = self.loss(lm_logits.view(fake_loss_batch_size, -1), decoder_target.view(fake_loss_batch_size))
+ lm_loss = lm_loss.view(fake_output_batch_size, decoder_sequence_length)
+ loss = lm_loss
+
+ if self.config.get("label_smoothing"):
+ loss *= (1 - self.config.label_smoothing)
+ mask = kwargs["decoder_constraint"].view(fake_output_batch_size, decoder_sequence_length, -1)
+ num_classes = mask.float().sum(2, keepdim=True)
+ num_classes += (num_classes == 0)
+ uniform = mask.float() / num_classes
+ loss += self.config.label_smoothing * - torch.sum(uniform * lm_logits, 2)
+
+ if self.config.get("encouraging_loss"):
+ bonus = torch.log(-torch.expm1(lm_logits))
+ log_end = self.config.encouraging_loss
+ if log_end != 1:
+ probs = torch.exp(lm_logits)
+ y_log_end = math.log1p(-log_end)
+ bonus_after_log_end = 1 / (log_end - 1) * (probs - log_end) + y_log_end
+ bonus = torch.where(probs > log_end, bonus_after_log_end, bonus)
+ ec_loss = self.loss(bonus.view(fake_loss_batch_size, -1), decoder_target.view(fake_loss_batch_size)).view(fake_output_batch_size, decoder_sequence_length)
+
+ if self.config.get("label_smoothing"):
+ ec_loss *= (1 - self.config.label_smoothing)
+ mask = kwargs["decoder_constraint"].view(fake_output_batch_size, decoder_sequence_length, -1)
+ num_classes = mask.float().sum(2, keepdim=True)
+ num_classes += (num_classes == 0)
+ uniform = mask.float() / num_classes
+ ec_loss += self.config.label_smoothing * - torch.sum(uniform * bonus, 2)
+
+ loss += ec_loss
+
+ if self.config.get("only_train_answer"):
+ loss *= kwargs["decoder_answer_mask"].view(fake_output_batch_size, decoder_sequence_length)
+ loss = loss.sum(1)
+ if self.config.get("loss_mean_sentence_length"):
+ loss /= decoder_target_attention_mask.view(fake_output_batch_size, decoder_sequence_length).sum(1)
+ loss = loss.mean()
+
+ if self.config.get("only_score_answer"):
+ lm_loss *= kwargs["decoder_answer_mask"].view(fake_output_batch_size, decoder_sequence_length)
+ lm_loss = lm_loss.sum(1)
+
+ if self.config.ofa_task == "vqa_gen" and decode_per_encode == 4:
+ logits = -lm_loss.view(batch_size, decode_per_encode)
+ prediction = torch.nn.functional.softmax(logits, dim=1).unsqueeze(1)
+ elif self.config.ofa_task == "snli_ve" and decode_per_encode in [2, 3]:
+ assert(self.config.get("neither_for_other")) # For another commit
+ assert(self.config.maybe_is not in ["yes", "no"]) # For another commit too =(
+ # logits is of size: batch × label × answer:
+ # - label comes from the encoding of "entity is a hero", "entity is a villain", etc.
+ # - answer comes from the SNLI question answer: "no", "maybe" or "yes"
+ logits = -lm_loss.view(batch_size, inner_input_size, decode_per_encode)
+
+ # FIXME just compare the probabilities of "yes" for now
+ prediction = torch.nn.functional.softmax(logits[:, :, -1], dim=1).unsqueeze(1)
+ else:
+ prediction = None
+
+ return loss, prediction
diff --git a/mmsrl/output.py b/mmsrl/output.py
@@ -0,0 +1,57 @@
+from typing import Any, Dict, Optional, Type
+import pathlib
+import pickle
+import types
+
+import torch
+
+import mmsrl.utils
+
+
+class Output:
+ """
+ Class for outputing data about the learned model.
+ """
+ # TODO parametrize this class
+
+ def __init__(self, config: mmsrl.utils.dotdict, split: str, epoch: int, final: bool) -> None:
+ """ Initialize the outputs variables, but do not acquire necessary resources yet. """
+ self.config: mmsrl.utils.dotdict = config
+ self.split = {"unseen_test": "test"}.get(split, split)
+ self.epoch = epoch
+ self.final = final
+ self.path = None
+ self.file = None
+
+ def __enter__(self):
+ """ Acquire resources needed for outputing the data. """
+ if self.final and self.config.get(f"output_{self.split}"):
+ self.path = pathlib.Path(self.config[f"output_{self.split}"])
+ if '{' in str(self.path):
+ self.path = pathlib.Path(str(path).format(self.epoch))
+ self.file = self.path.open("wb")
+ return self
+
+ def __exit__(self,
+ exc_type: Optional[Type[BaseException]],
+ exc_inst: Optional[BaseException],
+ exc_tb: Optional[types.TracebackType]) -> None:
+ """ Free used resources. """
+ if self.file is not None:
+ self.file.close()
+
+ def __call__(self, batch: Dict[str, Any], predictions: Optional[torch.Tensor]) -> None:
+ """
+ Update model output data files with the given batch and the outputs of the model on this batch.
+
+ Args:
+ batch: the input values used for evaluation
+ predictions: the output of the model
+ """
+ if self.file and predictions is not None:
+ for ib in range(len(batch["image_name"])):
+ entities = dict(zip(
+ filter(None, batch["entities_name"][ib]),
+ predictions[ib].detach().cpu()))
+ pickle.dump((batch["image_name"][ib], entities), self.file)
+ self.file.flush()
diff --git a/mmsrl/show_predictions.py b/mmsrl/show_predictions.py
@@ -0,0 +1,23 @@
+import pathlib
+import pickle
+import sys
+
+
+def show_predictions(path: pathlib.Path) -> None:
+ with path.open("rb") as input_file:
+ try:
+ while True:
+ image, entities = pickle.load(input_file)
+ for entity, prediction in entities.items():
+ prediction_str = " ".join(map(str, prediction.tolist()))
+ print(f"{image}\t{entity}\t{prediction_str}")
+ print("")
+ except EOFError:
+ pass
+
+
+if __name__ == "__main__":
+ if len(sys.argv) != 2:
+ print(f"Usage: {sys.argv[0]} prediction_file", file=sys.stderr)
+ sys.exit(1)
+ show_predictions(pathlib.Path(sys.argv[1]))
diff --git a/mmsrl/submission.py b/mmsrl/submission.py
@@ -0,0 +1,59 @@
+from typing import Any, Dict
+import argparse
+import collections
+import json
+import os
+import pathlib
+import pickle
+import zipfile
+
+import mmsrl.utils
+
+
+def make_submission_file(config: mmsrl.utils.dotdict) -> None:
+ inputs: Dict[str, Dict[str, Any]] = collections.defaultdict(dict)
+ with config.prediction.open("rb") as input_file:
+ try:
+ while True:
+ image, entities = pickle.load(input_file)
+ for entity, prediction in entities.items():
+ inputs[image][entity] = prediction
+ except EOFError:
+ pass
+
+ data_path = pathlib.Path(os.environ['DATA_PATH'])
+ annot_file = data_path / "mmsrl" / "all" / "annotations" / f"unseen_test.jsonl"
+ with annot_file.open('r') as json_file:
+ dataset = list(map(json.loads, json_file))
+
+ outputfile_path = config.output / pathlib.Path(config.prediction.stem).with_suffix('.json')
+ with outputfile_path.open("w") as output_file:
+ for sample in dataset:
+ image = sample["image"]
+ output: Dict[str, Any] = {"image": image}
+ for label in mmsrl.utils.LABELS:
+ output[label] = []
+
+ for i, entity in enumerate(sample["entity_list"]):
+ if entity == "":
+ prediction = mmsrl.utils.LABELS.index("other")
+ else:
+ predictions = [(p, i) for i, p in enumerate(inputs[image][entity].tolist())]
+ predictions.sort(reverse=True)
+ if len(set(entities)) == len(entities) or entities.count(entity) == 1:
+ prediction = predictions[0][1]
+ else:
+ prediction = predictions[entities[:i].count(entity)][1]
+ output[mmsrl.utils.LABELS[prediction]].append(entity)
+
+ print(json.dumps(output), file=output_file)
+
+ with zipfile.ZipFile(str(outputfile_path.with_suffix('.zip')), mode="w") as archive:
+ archive.write(str(outputfile_path), arcname="answer.jsonl")
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser()
+ parser.add_argument("output", type=pathlib.Path, help="Path were we write the submission json file.")
+ parser.add_argument("prediction", type=pathlib.Path, help="Prediction file to use.")
+ args = mmsrl.utils.dotdict(vars(parser.parse_args()))
+ make_submission_file(args)
diff --git a/mmsrl/train.py b/mmsrl/train.py
@@ -0,0 +1,305 @@
+import argparse
+import contextlib
+import copy
+import logging
+import math
+import multiprocessing
+import pathlib
+import signal
+import sys
+import tqdm
+import types
+
+# Do this do deal with the badly packaged OFA
+import mmsrl
+sys.path.append(mmsrl.__path__[0] + "/../OFA")
+
+import clip
+import fairseq.checkpoint_utils
+import fairseq.dataclass.utils
+import fairseq.options
+import fairseq.tasks
+import fairseq.utils
+import sklearn.metrics
+import tasks # OFA
+import torch
+import transformers
+
+from mmsrl.dataset import MemeDataset
+import mmsrl.batcher
+import mmsrl.extract_image_features
+import mmsrl.output
+import mmsrl.utils
+
+
+logging.basicConfig(level=logging.INFO)
+transformers_logger = logging.getLogger("transformers")
+transformers_logger.setLevel(logging.WARNING)
+
+
+class Trainer:
+ def __init__(self, config):
+ self.config = config
+ self.device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
+ self.prepare_framework()
+ self.prepare_data()
+ self.prepare_model()
+
+ def hook_signals(self) -> None:
+ """ Change the behavior of SIGINT (^C) to change a variable `self.interrupted' before killing the process. """
+ self.interrupted = False
+
+ def handler(sig: int, frame: types.FrameType) -> None:
+ if multiprocessing.current_process().name != "MainProcess":
+ return
+
+ print("\n\033[31mInterrupted, execution will stop at the next end of validation.\n\033[1mNEXT ^C WILL KILL THE PROCESS!\033[0m\n", file=sys.stderr)
+ self.interrupted = True
+ signal.signal(signal.SIGINT, signal.SIG_DFL)
+
+ signal.signal(signal.SIGINT, handler)
+
+ def get_ofa_config(self):
+ parser = fairseq.options.get_generation_parser()
+ input_args = ["",
+ f"--task={self.config.ofa_task}",
+ "--unnormalized",
+ f"--path={self.config.ofa_path}",
+ "--bpe-dir=OFA/utils/BPE"]
+ args = fairseq.options.parse_args_and_arch(parser, input_args)
+ ofa_config = fairseq.dataclass.utils.convert_namespace_to_omegaconf(args)
+ return ofa_config
+
+ def prepare_feature(self, feature):
+ """ Adapt image and text feature to the model being in float16 or float32 setting. """
+ if isinstance(feature, torch.Tensor):
+ if self.config.get("half"):
+ feature = feature.to(dtype=torch.float16)
+ feature = feature.to(device=self.device)
+ return feature
+
+ def prepare_framework(self):
+ if self.config.model == "ofa":
+ if self.config.ofa_task == 'vqa_gen':
+ fairseq.tasks.register_task('vqa_gen', tasks.mm_tasks.vqa_gen.VqaGenTask)
+ elif self.config.ofa_task == 'snli_ve':
+ fairseq.tasks.register_task('snli_ve', tasks.mm_tasks.snli_ve.SnliVeTask)
+ else:
+ raise RuntimeError(f"Unknown OFA task {self.config.ofa_task}")
+ ofa_pre_config = self.get_ofa_config()
+ self.ofa_task = fairseq.tasks.setup_task(ofa_pre_config.task)
+ self.ofa_models, self.ofa_config = fairseq.checkpoint_utils.load_model_ensemble(
+ fairseq.utils.split_paths(ofa_pre_config.common_eval.path),
+ task=self.ofa_task
+ )
+ elif self.config.model in ["clip", "clip_momenta", "generic"]:
+ self.clip_model, self.clip_preprocess = clip.load(self.config.modelname, device=self.device)
+
+ def prepare_data(self):
+ self.epoch = 0
+ self.dataset = {}
+ self.iterator = {}
+ if self.config.model == "ofa":
+ extra_dataset_kwargs = {"ofa_task": self.ofa_task}
+ text_pad: int = self.ofa_task.src_dict.pad()
+ elif self.config.model == "clip":
+ text_pad: int = 0
+ if self.config.features_path:
+ self.image_feature_extractor = mmsrl.extract_image_features.ImageFeatureExtractor(self.config)
+ extra_dataset_kwargs = {"image_processor": self.clip_preprocess, "tokenizer": clip.tokenize, "image_features_loader": self.image_feature_extractor.process_image}
+ else:
+ extra_dataset_kwargs = {"image_processor": self.clip_preprocess, "tokenizer": clip.tokenize}
+ elif self.config.model == "clip_momenta":
+ text_pad: int = 0
+ self.image_feature_extractor = mmsrl.extract_image_features.ImageFeatureExtractor(self.config)
+ extra_dataset_kwargs = {"image_processor": self.clip_preprocess, "tokenizer": clip.tokenize, "image_features_loader": self.image_feature_extractor.process_image}
+ elif self.config.model == "visualbert":
+ self.image_feature_extractor = mmsrl.extract_image_features.ImageFeatureExtractor(self.config)
+ self.tokenizer = transformers.AutoTokenizer.from_pretrained(self.config.modelname)
+ text_pad: int = self.tokenizer.pad_token_id
+ extra_dataset_kwargs = {"image_processor": self.image_feature_extractor.process_image, "tokenizer": self.tokenizer}
+ elif self.config.model == "generic":
+ # tokenizer:
+ extra_dataset_kwargs = {"tokenizer": clip.tokenize, "image_processor": self.clip_preprocess}
+ text_pad: int = 0
+ if self.config.use_visualbert or self.config.features_path:
+ self.tokenizer = transformers.AutoTokenizer.from_pretrained(self.config.visualbert_text_model)
+ self.image_feature_extractor = mmsrl.extract_image_features.ImageFeatureExtractor(self.config)
+ extra_dataset_kwargs.update({"image_features_loader": self.image_feature_extractor.process_image, "vbert_tokenizer": self.tokenizer})
+ else:
+ self.tokenizer = transformers.AutoTokenizer.from_pretrained(self.config.modelname)
+ text_pad: int = self.tokenizer.pad_token_id
+ extra_dataset_kwargs = {"tokenizer": self.tokenizer}
+
+ self.batcher = mmsrl.batcher.Batcher(self.config, text_pad=text_pad, label_pad=-100, image_pad=0)
+
+ for split in ["train", "val", "test"]:
+ self.dataset[split] = MemeDataset(self.config, split, split!="train", **extra_dataset_kwargs)
+ self.iterator[split] = lambda split=split, trainer=self: tqdm.tqdm(
+ torch.utils.data.DataLoader(dataset=trainer.dataset[split],
+ batch_size=trainer.config["batch_size"],
+ collate_fn=trainer.batcher,
+ num_workers=trainer.config.get("workers", 0),
+ worker_init_fn=trainer.dataset[split].seed,
+ ), desc=f"{split}, epoch: {self.epoch}", leave=False)
+
+ def prepare_model(self) -> None:
+ label_weights = self.dataset['train'].get_label_weights()
+ if self.config.model=="ofa":
+ from mmsrl.ofa import Model
+ model_kwargs = {"ofa_models": self.ofa_models, "ofa_config": self.ofa_config, "ofa_task": self.ofa_task}
+ elif self.config.model=="perceiver":
+ from mmsrl.perceiver import Model
+ model_kwargs = {}
+ elif self.config.model=="clip":
+ from mmsrl.clip import Model
+ model_kwargs = {"model": self.clip_model}
+ elif self.config.model=="clip_momenta":
+ from mmsrl.clip_momenta import Model
+ model_kwargs = {"model": self.clip_model}
+ elif self.config.model == "visualbert":
+ from mmsrl.visualbert import Model
+ model_kwargs = {}
+ elif self.config.model == "generic":
+ from mmsrl.generic_classifier import Model
+ model_kwargs = {"model": self.clip_model}
+ else:
+ from mmsrl.baseline import Model
+ model_kwargs = {}
+ self.model = Model(self.config, label_weights.to(device=self.device), **model_kwargs)
+ if self.config.get("half"):
+ self.model.half()
+ self.model.to(self.device)
+ self.optimizer = torch.optim.AdamW(self.model.parameters(),
+ lr=self.config["learning_rate"],
+ weight_decay=self.config.get("weight_decay", 0))
+ step_per_epoch = math.ceil(len(self.dataset["train"]) / self.config.batch_size / self.config.accumulation)
+ max_iterations: int = self.config["max_epoch"] * step_per_epoch
+
+ warmup_step = self.config.get("warmup_step", 0)
+ warmup_step = warmup_step * max_iterations if 0 < warmup_step < 1 else warmup_step
+ self.scheduler = transformers.get_scheduler(self.config.get("lr_scheduler", "linear"), self.optimizer, warmup_step, max_iterations)
+ self.scaler = torch.cuda.amp.GradScaler(1) # Used to scale gradients to avoid rounding to 0 when using float16
+ self.amp = torch.cuda.amp.autocast if self.config.amp else contextlib.nullcontext
+
+ def eval(self, split: str, final: bool = False) -> float:
+ with torch.no_grad(), mmsrl.output.Output(self.config, split, self.epoch, final) as output:
+ predictions = []
+ truth = []
+ self.model.eval()
+ accuracy_accumulator = 0
+ num_samples = 0
+ loop = self.iterator[split]()
+ for batch in loop:
+ batch = {key: self.prepare_feature(value) for key, value in batch.items()}
+ with self.amp(): # Use mixed-precision float16
+ loss, prediction = self.model(**batch)
+ output(batch, prediction)
+ if prediction is not None and "labels" in batch:
+ predictions.extend(prediction.argmax(2)[batch['entities_mask']].tolist())
+ truth.extend(batch["labels"][batch['entities_mask']].tolist())
+ accuracy_accumulator += ((prediction.argmax(2) == batch["labels"])[batch['entities_mask']]).sum().item()
+ num_samples += batch["entities_mask"].sum().item()
+ loop.set_postfix(accuracy=f"{accuracy_accumulator/num_samples:5.3f}") # for the tqdm
+ if predictions and truth:
+ f1 = sklearn.metrics.f1_score(truth, predictions, average="macro")
+ class_precision, class_recall, class_f1, _ = sklearn.metrics.precision_recall_fscore_support(truth, predictions)
+ class_scores = ", ".join(f"{label}[F{class_f1[i]:4.2f}/P{class_precision[i]:4.2f}/R{class_recall[i]:4.2f}]" for i, label in enumerate(mmsrl.utils.LABELS))
+ print(f"Epoch {self.epoch:3} {split}, Macro-F1: {f1:5.3f}, {class_scores}")
+ return f1
+ return None
+
+ def train_step(self) -> None:
+ """ Apply the gradients to the parameters. """
+ if self.config.amp:
+ self.scaler.unscale_(self.optimizer) # Gradients are scaled in order to avoid rounding to 0 when casting to float16
+ torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.config["max_grad_norm"]) # Clip to gradients to a maximum value (to be robust to outliers)
+ if self.config.amp:
+ self.scaler.step(self.optimizer) # Add the gradient to the parameters
+ else:
+ self.optimizer.step()
+ self.scheduler.step() # Update the learning rate
+ if self.config.amp:
+ self.scaler.update() # Update by how much the gradients should be scaled to avoid rounding to 0
+ self.optimizer.zero_grad() # Set the gradients to 0 to prepare for the next iteration
+
+ def train(self) -> None:
+ best_dev = -math.inf
+ best_dev_epoch = 0
+ best_state_dict = None # Used to save the best model
+ metric_key = None
+
+ self.hook_signals()
+ if not self.config.get("no_initial_valid"):
+ best_dev = self.eval("val", final=True)
+
+ for self.epoch in range(1, 1 + self.config["max_epoch"]):
+ # Accumulate gradient to better estimate it even with small batches
+ if self.interrupted == True:
+ break
+ metric_accumulator = 0
+ num_samples = 0
+ self.model.train()
+ epoch_loop = self.iterator["train"]()
+ for batch_id, batch in enumerate(epoch_loop):
+ if self.interrupted == True:
+ break
+ if batch_id >= self.config.get("batch_per_epoch", math.inf):
+ break
+ batch = {key: self.prepare_feature(value) for key, value in batch.items()}
+ with self.amp(): # Use mixed-precision float16
+ loss, prediction = self.model(**batch)
+ if loss is None:
+ break
+
+ if prediction is not None:
+ metric_accumulator += ((prediction.argmax(2) == batch["labels"])[batch['entities_mask']]).sum().item()
+ metric_key = "accuracy"
+ elif loss is not None:
+ metric_accumulator += loss * batch["entities_mask"].sum().item()
+ metric_key = "loss"
+ num_samples += batch["entities_mask"].sum().item()
+ if metric_key is not None:
+ epoch_loop.set_postfix(**{metric_key: f"{metric_accumulator/num_samples:5.3f}"}) # for the tqdm
+
+ self.scaler.scale(loss).backward() # Accumulate the (scaled) gradients
+ if (1+batch_id) % self.config.get("accumulation", 1) == 0:
+ self.train_step()
+
+ if metric_key is not None:
+ print(f"Epoch {self.epoch:3} train, {metric_key}: {metric_accumulator/num_samples:.4f}")
+ else:
+ print(f"Epoch {self.epoch:3} train, done")
+ if loss is None:
+ break
+
+ if (batch_id+1) % self.config.get("accumulation", 1) != 0:
+ self.train_step()
+
+ dev = self.eval("val")
+ if dev > best_dev: # If dev score improved
+ best_dev = dev
+ best_dev_epoch = self.epoch
+ if self.config.get("save_model"):
+ torch.save(self.model.state_dict(), self.config.save_model)
+ best_state_dict = True
+ else:
+ best_state_dict = copy.deepcopy(self.model.state_dict()) # Save the weights of the model
+ elif self.epoch - best_dev_epoch > self.config.get("patience", 0): # If dev score worsened for several steps (or 1 if patience is 0)
+ break # Early stopping
+
+ self.dataset["train"].compute_subsample(self.epoch)
+ if best_state_dict is not None: # If we improved over random initialization
+ if self.config.get("save_model"):
+ self.model.load_state_dict(torch.load(self.config.save_model))
+ else:
+ self.model.load_state_dict(best_state_dict) # Load the model with the best dev score
+ for split in ["val", "test"]:
+ self.eval(split, final=True)
+
+
+if __name__=="__main__":
+ config: mmsrl.utils.dotdict = mmsrl.utils.parse_args()
+ print("\033[33mConfiguration:\033[0m")
+ mmsrl.utils.print_dict(config)
+ Trainer(config).train()
diff --git a/mmsrl/utils.py b/mmsrl/utils.py
@@ -0,0 +1,128 @@
+
+from typing import Any, Callable, Dict, List, Union
+import importlib
+import logging
+import sys
+import types
+
+logger = logging.getLogger(__name__)
+
+
+LABELS = ["hero", "villain", "victim", "other"]
+
+class dotdict(dict):
+ """ Dictionary which can be access through var.key instead of var["key"]. """
+ def __getattr__(self, name: str):
+ if name not in self:
+ raise AttributeError(f"Config key {name} not found")
+ return dotdict(self[name]) if type(self[name]) is dict else self[name]
+ __setattr__ = dict.__setitem__
+ __delattr__ = dict.__delitem__
+
+
+def eval_arg(config: Dict[str, Any], arg: str) -> None:
+ """
+ Evaluate arg in the context config, and update it.
+
+ The argument is expected to be of the form:
+ (parent.)*key(=value)?
+ If no value is provided, the key is assumed to be a boolean and True is assigned to it.
+ When passing a string argument through the shell, it must be enclosed in quote (like all python string), which usually need to be escaped.
+ """
+ key: str
+ value: Any
+ if '=' in arg:
+ key, value = arg.split('=', maxsplit=1)
+ try:
+ value = eval(value, config)
+ except NameError:
+ pass
+ config.pop("__builtins__", None)
+ else:
+ key, value = arg, True
+ path: List[str] = key.split('.')
+ for component in path[:-1]:
+ config = config[component]
+ key = path[-1]
+ if isinstance(config, list):
+ key = int(key)
+ config[key] = value
+
+
+def import_arg(config: Dict[str, Any], arg: str) -> None:
+ """
+ Load file arg, and update config with its content.
+
+ The file is loaded in an independent context, all the variable defined in the file (even through import) are added to config, with the exception of builtins and whole modules.
+ """
+ if arg.endswith(".py"):
+ arg = arg[:-3].replace('/', '.')
+ module: types.ModuleType = importlib.import_module(arg)
+ for key, value in vars(module).items():
+ if key not in module.__builtins__ and not key.startswith("__") and not isinstance(value, types.ModuleType): # pytype: disable=attribute-error
+ config[key] = value
+
+
+def parse_args() -> dotdict:
+ """
+ Parse command line arguments and return config dictionary.
+
+ Two kind of argument are supported:
+ - When the argument starts with -- it is evaluated by the eval_arg function
+ - Otherwise the argument is assumed to be a file which is loaded by the import_arg function
+ """
+ config: Dict[str, Any] = {}
+ for arg in sys.argv[1:]:
+ if arg.startswith("--"):
+ eval_arg(config, arg[2:])
+ else:
+ import_arg(config, arg)
+ return dotdict(config)
+
+
+def display_dict(output: Callable[[str], None], input: Dict[str, Any], depth: int = 0) -> None:
+ """ Display nested dictionaries in input using the provided output function. """
+ for key, value in input.items():
+ indent = '\t'*depth
+ output(f"{indent}{key}:")
+ if isinstance(value, dict):
+ output('\n')
+ display_dict(output, value, depth+1)
+ else:
+ output(f" {value}\n")
+
+
+def print_dict(input: Dict[str, Any]) -> None:
+ """ Print dictionary to standard output. """
+ display_dict(lambda x: print(x, end=""), input)
+
+
+def log_dict(logger: logging.Logger, input: Dict[str, Any]) -> None:
+ """ Log dictionary to the provided logger. """
+ class log:
+ buf: str = ""
+
+ def __call__(self, x: str) -> None:
+ self.buf += x
+ if self.buf.endswith('\n'):
+ logger.info(self.buf[:-1])
+ self.buf = ""
+ display_dict(log(), input)
+
+
+def flatten_dict(input: Dict[str, Any]) -> Dict[str, Union[bool, int, float, str]]:
+ """
+ Replace nested dict by dot-separated keys, and cast keys to simple types.
+
+ repr() is used to cast non-base-type to str.
+ """
+ def impl(result: Dict[str, Union[bool, int, float, str]], input: Dict[str, Any], prefix: str):
+ for key, value in input.items():
+ if isinstance(value, dict):
+ impl(result, value, f"{key}.")
+ else:
+ result[f"{prefix}{key}"] = value if type(value) in [bool, int, float, str] else repr(value)
+
+ result: Dict[str, Union[bool, int, float, str]] = {}
+ impl(result, input, "")
+ return result
diff --git a/oarscripts/extract_image_features.sh b/oarscripts/extract_image_features.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+#
+#
+# Comments starting with #OAR are used by the resource manager if using "oarsub -S"
+#
+# The job is submitted to the default queue
+#OAR -q default
+#OAR -p gpu='YES'
+#OAR -l /gpunum=1,walltime=20:00:00
+#OAR -t besteffort
+
+
+module purge
+module load conda
+source activate pytorch
+module load cuda/10.2
+module load gcc/7.3.0
+module load cudnn/7.6-cuda-10.2
+
+#python -m mmsrl.generate_captions
+python mmsrl/extract_features_vgg.py
+
+# PYTHONPATH=/data/almanach/user/smontari/scratch/constraint_challenge/mmf: python3 -u \
+# /data/almanach/user/smontari/scratch/constraint_challenge/mmf/tools/scripts/features/frcnn/extract_features_frcnn.py \
+# --model_file /data/almanach/user/smontari/scratch/constraint_challenge/Multimodal-semantic-role-labeling/mmfsrl/fcrnn/pytorch_model.bin \
+# --config_file /data/almanach/user/smontari/scratch/constraint_challenge/Multimodal-semantic-role-labeling/mmfsrl/fcrnn/config.yaml \
+# --output_folder "/data/almanach/user/smontari/scratch/constraint_challenge/mmsrl/all/features_frcnn/" \
+# --image_dir "/data/almanach/user/smontari/scratch/constraint_challenge/mmsrl/all/images/"
+
+# oarsub -p "gpu='YES' and host='nefgpu39.inria.fr'" -l /gpunum=1,walltime=20:00:00 -t besteffort ./oarscripts/extract_image_features.sh
+# oarsub -p "gpu='YES' and host='nefgpu47.inria.fr'" -l /gpunum=1,walltime=20:00:00 -t besteffort -S "./oarscripts/train_clip.sh mlp"
+# oarsub -S "./oarscripts/train_clip.sh attention"
diff --git a/oarscripts/train_snli.sh b/oarscripts/train_snli.sh
@@ -0,0 +1,34 @@
+#!/bin/bash
+#
+#
+# Comments starting with #OAR are used by the resource manager if using "oarsub -S"
+#
+# Note : quoting style of parameters matters, follow the example
+#
+# The job is submitted to the default queue
+#OAR -q default
+# Path to the binary to run
+
+module purge
+module load conda
+source activate pytorch
+module load cuda/10.2
+module load gcc/7.3.0
+module load cudnn/7.6-cuda-10.2
+
+
+echo python -m mmsrl.train configs/ofa_vqa.py --ofa_path=OFA/checkpoints/snli_ve_large_best.pt \
+ --output_val=outputs/ofa_snli_val.pkl \
+ --output_test=outputs/ofa_snli_test.pkl \
+ --save_model=models/ofa_snli.bin
+
+python -m mmsrl.train configs/ofa_vqa.py --ofa_path=OFA/checkpoints/snli_ve_large_best.pt \
+ --output_val=outputs/ofa_snli_val.pkl \
+ --output_test=outputs/ofa_snli_test.pkl \
+ --save_model=models/ofa_snli.bin
+
+# oarsub -p "gpu='YES'" -l /gpunum=1,walltime=20:00:00 -t besteffort ./oarscripts/train_vqa.sh
+# oarsub -p "gpu='YES' and host='nefgpu47.inria.fr'" -l /gpunum=1,walltime=20:00:00 -t besteffort ./oarscripts/train_snli.sh
+
+#OAR_JOB_ID=12239573
+# "--ofa_input='ocr question'" \
+\ No newline at end of file
diff --git a/oarscripts/train_vqa.sh b/oarscripts/train_vqa.sh
@@ -0,0 +1,42 @@
+#!/bin/bash
+#
+#
+# Comments starting with #OAR are used by the resource manager if using "oarsub -S"
+#
+# Note : quoting style of parameters matters, follow the example
+#
+# The job is submitted to the default queue
+#OAR -q default
+# Path to the binary to run
+
+module purge
+module load conda
+source activate pytorch
+module load cuda/10.2
+module load gcc/7.3.0
+module load cudnn/7.6-cuda-10.2
+
+model=$1
+unfreeze=$2
+learningrate=$3
+subsamble_other=$4
+
+echo python -m mmsrl.train \
+ --ofa_filename=$model \
+ configs/ofa_vqa.py --unfreeze=$unfreeze --learning_rate=$learningrate --subsamble_other=$subsamble_other\
+ --output_val=\"outputs/ofa_vqa_${model}-${unfreeze}-${learningrate}-${subsamble_other}_val.pkl\"\
+ --output_test=\"outputs/ofa_vqa_${model}-${unfreeze}-${learningrate}-${subsamble_other}_test.pkl\"\
+ --save_model=\"models/ofa_vqa_${model}-${unfreeze}-${learningrate}-${subsamble_other}.bin\"
+
+python -m mmsrl.train --ofa_filename=$model configs/ofa_vqa.py --ofa_path=ofa_dir+ofa_filename --unfreeze=$unfreeze --learning_rate=$learningrate --subsamble_other=$subsamble_other\
+ --output_val=\"outputs/ofa_vqa_${model}-${unfreeze}-${learningrate}-${subsamble_other}_val.pkl\"\
+ --output_test=\"outputs/ofa_vqa_${model}-${unfreeze}-${learningrate}-${subsamble_other}_test.pkl\"\
+ --save_model=\"models/ofa_vqa_${model}-${unfreeze}-${learningrate}-${subsamble_other}.bin\"
+
+# oarsub -p "gpu='YES'" -l /gpunum=1,walltime=20:00:00 -t besteffort ./oarscripts/train_vqa.sh
+# oarsub -p "gpu='YES' and host='nefgpu39.inria.fr'" -l /gpunum=1,walltime=20:00:00 -t besteffort ./oarscripts/train_vqa.sh
+# oarsub -p "gpu='YES' and host='nefgpu50.inria.fr'" -l /nodes=1/gpunum=1,walltime=20:00:00 --array-param-file oarscripts/ofa_params.txt -t besteffort ./oarscripts/train_vqa.sh
+
+
+#OAR_JOB_ID=12236582
+# "--ofa_input='ocr question'" \
+\ No newline at end of file
diff --git a/oarscripts/train_vqa_old.sh b/oarscripts/train_vqa_old.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+#
+#
+# Comments starting with #OAR are used by the resource manager if using "oarsub -S"
+#
+# Note : quoting style of parameters matters, follow the example
+#
+# The job is submitted to the default queue
+#OAR -q default
+# Path to the binary to run
+
+module purge
+module load conda
+source activate pytorch
+module load cuda/10.2
+module load gcc/7.3.0
+module load cudnn/7.6-cuda-10.2
+
+echo python -m mmsrl.train configs/ofa_old_best.py --output_val=outputs/ofa_vqa_old_max35_lr5_val.pkl \
+ --output_test=outputs/ofa_vqa_old_max35_lr5_test.pkl \
+ --save_model=models/ofa_vqa_old.bin
+
+python -m mmsrl.train configs/ofa_old_best.py --output_val=outputs/ofa_vqa_old_max35_lr5_val.pkl \
+ --output_test=outputs/ofa_vqa_old_max35_lr5_test.pkl \
+ --save_model=models/ofa_vqa_old.bin
+
+# oarsub -p "gpu='YES'" -l /gpunum=1,walltime=20:00:00 -t besteffort ./oarscripts/train_vqa_old.sh
+# oarsub -p "gpu='YES' and mem>80000 and host='nefgpu39.inria.fr'" -l /gpunum=1,walltime=20:00:00 -t besteffort ./oarscripts/train_vqa_old.sh
+#OAR_JOB_ID=12236582
+# --unfreeze=\"classifier\" \
+
+#ipython --pdb -c "%run -m mmsrl.train configs/ofa_vqa.py \"--ofa_path=ofa_path.replace('vqa_large_best','ofa_large')\" --ofa_unfreeze="all" --learning_rate=1e-4 --output_val=outputs/ofa_vqa_large_val_lr4.pkl --output_test=outputs/ofa_vqa_large_test_lr4.pkl --save_model=models/ofa_vqa_large_lr4.bin"
+\ No newline at end of file
diff --git a/prepare_data.sh b/prepare_data.sh
@@ -0,0 +1,48 @@
+#!/bin/sh
+
+if [ -z "$DATA_PATH" ]; then
+ echo "Error: Variable DATA_PATH is not set."
+ echo "Please set it to the path where you want to store the data."
+ echo "For example, from the root of the git repository you can type:"
+ echo ' export DATA_PATH="$PWD/../data"'
+ exit 1
+fi
+
+if [ ! -f "$DATA_PATH/mmsrl/constraint-data.zip" ]; then
+ echo 'Data not found at $DATA_PATH/mmsrl/constraint-data.zip.'
+ echo "For information DATA_PATH is set to: $(readlink -m $DATA_PATH)"
+ echo "Downloading dataset..."
+ mkdir -p "$DATA_PATH/mmsrl"
+ curl "https://esimon.eu/tmp/constraint-data.zip" --output "$DATA_PATH/mmsrl/constraint-data.zip"
+fi
+
+if [ ! -f "$DATA_PATH/mmsrl/constraint22_dataset_unseen_test.zip" ]; then
+ echo 'Test data not found at $DATA_PATH/mmsrl/constraint22_dataset_unseen_test.zip.'
+ echo "For information DATA_PATH is set to: $(readlink -m $DATA_PATH)"
+ echo "Downloading test dataset..."
+ curl "https://esimon.eu/tmp/constraint22_dataset_unseen_test.zip" --output "$DATA_PATH/mmsrl/constraint22_dataset_unseen_test.zip"
+fi
+
+echo "Preparing data in $DATA_PATH"
+echo "Unzipping"
+unzip -q "$DATA_PATH/mmsrl/constraint-data.zip" -d "$DATA_PATH/mmsrl"
+unzip -q "$DATA_PATH/mmsrl/constraint22_dataset_covid19.zip" -d "$DATA_PATH/mmsrl/covid19"
+unzip -q "$DATA_PATH/mmsrl/constraint22_dataset_uspolitics.zip" -d "$DATA_PATH/mmsrl/uspolitics"
+unzip -q "$DATA_PATH/mmsrl/constraint22_dataset_unseen_test.zip" -d "$DATA_PATH/mmsrl/all"
+
+if [ ! -f "$DATA_PATH/mmsrl/all/annotations/test.jsonl" ]; then
+ echo 'Labeled test data not found at $DATA_PATH/mmsrl/all/annotations/test.jsonl.'
+ echo "For information DATA_PATH is set to: $(readlink -m $DATA_PATH)"
+ echo "Downloading test dataset..."
+ curl "https://esimon.eu/tmp/constraint22_dataset_unseen_test_gold_labels.jsonl" --output "$DATA_PATH/mmsrl/all/annotations/test.jsonl"
+fi
+
+echo "Merging datasets"
+cat "$DATA_PATH/mmsrl/covid19/annotations/train.jsonl" "$DATA_PATH/mmsrl/uspolitics/annotations/train.jsonl" | grep -v '"hero": \[\], "villain": \[\], "victim": \[\], "other": \[\]' > "$DATA_PATH/mmsrl/all/annotations/train.jsonl"
+cat "$DATA_PATH/mmsrl/covid19/annotations/val.jsonl" "$DATA_PATH/mmsrl/uspolitics/annotations/val.jsonl" | grep -v '"hero": \[\], "villain": \[\], "victim": \[\], "other": \[\]' > "$DATA_PATH/mmsrl/all/annotations/val.jsonl"
+cp -rl "$DATA_PATH/mmsrl/covid19/images" "$DATA_PATH/mmsrl/all"
+cp -rl "$DATA_PATH/mmsrl/uspolitics/images" "$DATA_PATH/mmsrl/all"
+rm -r "$DATA_PATH/mmsrl/covid19/images"
+rm -r "$DATA_PATH/mmsrl/uspolitics/images"
+ln -s "$DATA_PATH/mmsrl/all/images" "$DATA_PATH/mmsrl/covid19/images"
+ln -s "$DATA_PATH/mmsrl/all/images" "$DATA_PATH/mmsrl/uspolitics/images"
diff --git a/requirements b/requirements
@@ -0,0 +1,13 @@
+matplotlib 3.5.1
+numpy 1.22.3
+pandas 1.4.1
+scikit-learn 1.0.2
+scipy 1.8.0
+sentence-transformers 2.2.0
+statsmodels 0.13.2
+torch 1.11.0
+torchvision 0.12.0
+tqdm 4.63.0
+transformers 4.17.0
+typing_extensions 4.1.1
+xgboost 1.5.2
+\ No newline at end of file
diff --git a/results_table.csv b/results_table.csv
@@ -0,0 +1,161 @@
+12240051 - clip_attention-14-1e-4-1_val.pkl : ['Epoch 21 val', ' Macro-F1: 0.377']
+12240050 - clip_attention-14-1e-4-2_val.pkl : ['Epoch 8 val', ' Macro-F1: 0.339']
+12240049 - clip_attention-14-1e-5-1_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.275']
+12240152 - clip_attention-epoch30-1e-4-1_val.pkl : ['Epoch 9 val', ' Macro-F1: 0.361']
+12240154 - clip_attention-epoch30-1e-4-2_val.pkl : ['Epoch 15 val', ' Macro-F1: 0.350']
+12240151 - clip_attention-epoch30-1e-5-1_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.270']
+12240153 - clip_attention-epoch30-1e-5-2_val.pkl : ['Epoch 18 val', ' Macro-F1: 0.207']
+12237153 - clip_attention_val.pkl : ['Epoch 14 val', ' Macro-F1: 0.376']
+12240048 - clip_mlp-14-1e-5-0.5_val.pkl : ['Epoch 21 val', ' Macro-F1: 0.322']
+12240046 - clip_mlp-14-1e-5-2_val.pkl : ['Epoch 21 val', ' Macro-F1: 0.317']
+12240047 - clip_mlp-14-1e-5-3_val.pkl : ['Epoch 21 val', ' Macro-F1: 0.316']
+12240045 - clip_mlp-14-1e-6-1_val.pkl : ['Epoch 10 val', ' Macro-F1: 0.076']
+12240150 - clip_mlp-epoch30-1e-4-2_val.pkl : ['Epoch 21 val', ' Macro-F1: 0.401']
+12237161 - clip_mlp_val.pkl : ['Epoch 14 val', ' Macro-F1: 0.422']
+12241360 - clip_momenta_all-lr1e-4-mt128--sberFalse-caption+text_val.pkl : ['Epoch 30 val', ' Macro-F1: 0.403']
+12250780 - clip_momenta_all-lr1e-4-mt128-affine-sberFalse-caption+text_cyclic5_val.pkl : ['Epoch 30 val', ' Macro-F1: 0.440']
+12250812 - clip_momenta_all-lr1e-4-mt128-affine-sberFalse-caption+text_cyclic5shorter_val.pkl : ['Epoch 3 train', ' accuracy: 0.2847\n']
+12241363 - clip_momenta_all-lr1e-4-mt300--sbermacro-caption+text_val.pkl : ['Epoch 30 val', ' Macro-F1: 0.422']
+12250783 - clip_momenta_all-lr1e-4-mt300-affine-sberFalse-caption+text_cyclic5_val.pkl : ['Epoch 30 val', ' Macro-F1: 0.441']
+12250815 - clip_momenta_all-lr1e-4-mt300-affine-sberFalse-caption+text_cyclic5shorter_val.pkl : ['Epoch 23 val', ' Macro-F1: 0.439']
+12241345 - clip_momenta_all-lr1e-5-mt128--sberFalse-caption+text_val.pkl : ['Epoch 12 val', ' Macro-F1: 0.331']
+12241357 - clip_momenta_all-lr1e-5-mt128--sberFalse-caption+text_val.pkl : ['Epoch 8 val', ' Macro-F1: 0.331']
+12241348 - clip_momenta_all-lr1e-5-mt128--sberFalse-separated_val.pkl : ['Epoch 28 val', ' Macro-F1: 0.364']
+12241354 - clip_momenta_all-lr1e-5-mt128--sberTrue-caption+text_val.pkl : ['Epoch 22 val', ' Macro-F1: 0.345']
+12241351 - clip_momenta_all-lr1e-5-mt128--sberTrue-separated_val.pkl : ['Epoch 23 val', ' Macro-F1: 0.335']
+12250777 - clip_momenta_all-lr1e-5-mt128-affine-sberFalse-caption+text_cyclic5_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.249']
+12250809 - clip_momenta_all-lr1e-5-mt128-affine-sberFalse-caption+text_cyclic5shorter_val.pkl : ['Epoch 12 val', ' Macro-F1: 0.242']
+12250765 - clip_momenta_all-lr1e-5-mt128-macro-sberFalse-caption+text_cyclic5_val.pkl : ['Epoch 12 val', ' Macro-F1: 0.282']
+12250797 - clip_momenta_all-lr1e-5-mt128-macro-sberFalse-caption+text_cyclic5shorter_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.217']
+12250768 - clip_momenta_all-lr1e-5-mt128-macro-sberFalse-separated_cyclic5_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.300']
+12250800 - clip_momenta_all-lr1e-5-mt128-macro-sberFalse-separated_cyclic5shorter_val.pkl : ['Epoch 15 val', ' Macro-F1: 0.217']
+12250774 - clip_momenta_all-lr1e-5-mt128-macro-sberTrue-caption+text_cyclic5_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.278']
+12250806 - clip_momenta_all-lr1e-5-mt128-macro-sberTrue-caption+text_cyclic5shorter_val.pkl : ['Epoch 12 val', ' Macro-F1: 0.226']
+12250771 - clip_momenta_all-lr1e-5-mt128-macro-sberTrue-separated_cyclic5_val.pkl : ['Epoch 14 val', ' Macro-F1: 0.217']
+12250803 - clip_momenta_all-lr1e-5-mt128-macro-sberTrue-separated_cyclic5shorter_val.pkl : ['Epoch 15 val', ' Macro-F1: 0.217']
+12250786 - clip_momenta_all-lr2e-4-mt128-affine-sberTrue-separated_cyclic5_val.pkl : ['Epoch 30 val', ' Macro-F1: 0.435']
+12250818 - clip_momenta_all-lr2e-4-mt128-affine-sberTrue-separated_cyclic5shorter_val.pkl : ['Epoch 27 val', ' Macro-F1: 0.422']
+12250789 - clip_momenta_all-lr2e-4-mt300-affine-sberTrue-separated_cyclic5_val.pkl : ['Epoch 25 val', ' Macro-F1: 0.416']
+12250821 - clip_momenta_all-lr2e-4-mt300-affine-sberTrue-separated_cyclic5shorter_val.pkl : ['Epoch 27 val', ' Macro-F1: 0.433']
+12241359 - clip_momenta_features_b7-lr1e-4-mt128--sberFalse-caption+text_val.pkl : ['Epoch 18 val', ' Macro-F1: 0.445']
+12250779 - clip_momenta_features_b7-lr1e-4-mt128-affine-sberFalse-caption+text_cyclic5_val.pkl : ['Epoch 30 val', ' Macro-F1: 0.436']
+12250811 - clip_momenta_features_b7-lr1e-4-mt128-affine-sberFalse-caption+text_cyclic5shorter_val.pkl : ['Epoch 4 val', ' Macro-F1: 0.278']
+12241362 - clip_momenta_features_b7-lr1e-4-mt300--sberFalse-caption+text_val.pkl : ['Epoch 8 val', ' Macro-F1: 0.361']
+12250782 - clip_momenta_features_b7-lr1e-4-mt300-macro-sberFalse-caption+text_cyclic5_val.pkl : ['Epoch 30 val', ' Macro-F1: 0.434']
+12250814 - clip_momenta_features_b7-lr1e-4-mt300-macro-sberFalse-caption+text_cyclic5shorter_val.pkl : ['Epoch 5 val', ' Macro-F1: 0.328']
+12241344 - clip_momenta_features_b7-lr1e-5-mt128--sberFalse-caption+text_val.pkl : ['Epoch 30 val', ' Macro-F1: 0.359']
+12241356 - clip_momenta_features_b7-lr1e-5-mt128--sberFalse-caption+text_val.pkl : ['Epoch 7 val', ' Macro-F1: 0.299']
+12241347 - clip_momenta_features_b7-lr1e-5-mt128--sberFalse-separated_val.pkl : ['Epoch 18 val', ' Macro-F1: 0.363']
+12241353 - clip_momenta_features_b7-lr1e-5-mt128--sberTrue-caption+text_val.pkl : ['Epoch 30 val', ' Macro-F1: 0.344']
+12241350 - clip_momenta_features_b7-lr1e-5-mt128--sberTrue-separated_val.pkl : ['Epoch 15 train', ' accuracy: 0.4647\n']
+12250776 - clip_momenta_features_b7-lr1e-5-mt128-affine-sberFalse-caption+text_cyclic5_val.pkl : ['Epoch 30 val', ' Macro-F1: 0.281']
+12250808 - clip_momenta_features_b7-lr1e-5-mt128-affine-sberFalse-caption+text_cyclic5shorter_val.pkl : ['Epoch 11 val', ' Macro-F1: 0.217']
+12250764 - clip_momenta_features_b7-lr1e-5-mt128-macro-sberFalse-caption+text_cyclic5_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.234']
+12250796 - clip_momenta_features_b7-lr1e-5-mt128-macro-sberFalse-caption+text_cyclic5shorter_val.pkl : ['Epoch 15 val', ' Macro-F1: 0.217']
+12250767 - clip_momenta_features_b7-lr1e-5-mt128-macro-sberFalse-separated_cyclic5_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.263']
+12250799 - clip_momenta_features_b7-lr1e-5-mt128-macro-sberFalse-separated_cyclic5shorter_val.pkl : ['Epoch 15 val', ' Macro-F1: 0.217']
+12250773 - clip_momenta_features_b7-lr1e-5-mt128-macro-sberTrue-caption+text_cyclic5_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.234']
+12250805 - clip_momenta_features_b7-lr1e-5-mt128-macro-sberTrue-caption+text_cyclic5shorter_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.229']
+12250770 - clip_momenta_features_b7-lr1e-5-mt128-macro-sberTrue-separated_cyclic5_val.pkl : ['Epoch 15 val', ' Macro-F1: 0.217']
+12250802 - clip_momenta_features_b7-lr1e-5-mt128-macro-sberTrue-separated_cyclic5shorter_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.217']
+12250785 - clip_momenta_features_b7-lr2e-4-mt128-affine-sberFalse-caption+text_cyclic5_val.pkl : ['Epoch 27 val', ' Macro-F1: 0.421']
+12250817 - clip_momenta_features_b7-lr2e-4-mt128-affine-sberFalse-caption+text_cyclic5shorter_val.pkl : ['Epoch 21 val', ' Macro-F1: 0.439']
+12250788 - clip_momenta_features_b7-lr2e-4-mt300-macro-sberTrue-separated_cyclic5_val.pkl : ['Epoch 30 val', ' Macro-F1: 0.427']
+12250820 - clip_momenta_features_b7-lr2e-4-mt300-macro-sberTrue-separated_cyclic5shorter_val.pkl : ['Epoch 28 val', ' Macro-F1: 0.354']
+12241358 - clip_momenta_features_frcnn-lr1e-4-mt128--sberFalse-caption+text_val.pkl : ['Epoch 25 val', ' Macro-F1: 0.391']
+12250778 - clip_momenta_features_frcnn-lr1e-4-mt128-affine-sberFalse-caption+text_cyclic5_val.pkl : ['Epoch 3 train', ' accuracy: 0.7837\n']
+12250810 - clip_momenta_features_frcnn-lr1e-4-mt128-affine-sberFalse-caption+text_cyclic5shorter_val.pkl : ['Epoch 5 train', ' accuracy: 0.3127\n']
+12241361 - clip_momenta_features_frcnn-lr1e-4-mt300--sberFalse-caption+text_val.pkl : ['Epoch 9 val', ' Macro-F1: 0.367']
+12250781 - clip_momenta_features_frcnn-lr1e-4-mt300-macro-sberFalse-caption+text_cyclic5_val.pkl : ['Epoch 30 val', ' Macro-F1: 0.449']
+12250813 - clip_momenta_features_frcnn-lr1e-4-mt300-macro-sberFalse-caption+text_cyclic5shorter_val.pkl : ['Epoch 3 val', ' Macro-F1: 0.306']
+12241343 - clip_momenta_features_frcnn-lr1e-5-mt128--sberFalse-caption+text_val.pkl : ['Epoch 21 val', ' Macro-F1: 0.367']
+12241355 - clip_momenta_features_frcnn-lr1e-5-mt128--sberFalse-caption+text_val.pkl : ['Epoch 7 val', ' Macro-F1: 0.356']
+12241346 - clip_momenta_features_frcnn-lr1e-5-mt128--sberFalse-separated_val.pkl : ['Epoch 25 val', ' Macro-F1: 0.241']
+12241352 - clip_momenta_features_frcnn-lr1e-5-mt128--sberTrue-caption+text_val.pkl : ['Epoch 30 val', ' Macro-F1: 0.343']
+12241349 - clip_momenta_features_frcnn-lr1e-5-mt128--sberTrue-separated_val.pkl : ['Epoch 15 val', ' Macro-F1: 0.309']
+12250775 - clip_momenta_features_frcnn-lr1e-5-mt128-affine-sberFalse-caption+text_cyclic5_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.300']
+12250807 - clip_momenta_features_frcnn-lr1e-5-mt128-affine-sberFalse-caption+text_cyclic5shorter_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.217']
+12250763 - clip_momenta_features_frcnn-lr1e-5-mt128-macro-sberFalse-caption+text_cyclic5_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.284']
+12250795 - clip_momenta_features_frcnn-lr1e-5-mt128-macro-sberFalse-caption+text_cyclic5shorter_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.217']
+12250766 - clip_momenta_features_frcnn-lr1e-5-mt128-macro-sberFalse-separated_cyclic5_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.288']
+12250798 - clip_momenta_features_frcnn-lr1e-5-mt128-macro-sberFalse-separated_cyclic5shorter_val.pkl : ['Epoch 15 val', ' Macro-F1: 0.217']
+12250772 - clip_momenta_features_frcnn-lr1e-5-mt128-macro-sberTrue-caption+text_cyclic5_val.pkl : ['Epoch 11 val', ' Macro-F1: 0.217']
+12250804 - clip_momenta_features_frcnn-lr1e-5-mt128-macro-sberTrue-caption+text_cyclic5shorter_val.pkl : ['Epoch 13 val', ' Macro-F1: 0.260']
+12250769 - clip_momenta_features_frcnn-lr1e-5-mt128-macro-sberTrue-separated_cyclic5_val.pkl : ['Epoch 14 val', ' Macro-F1: 0.217']
+12250801 - clip_momenta_features_frcnn-lr1e-5-mt128-macro-sberTrue-separated_cyclic5shorter_val.pkl : ['Epoch 14 val', ' Macro-F1: 0.222']
+12250784 - clip_momenta_features_frcnn-lr2e-4-mt128-affine-sberFalse-caption+text_cyclic5_val.pkl : ['Epoch 25 val', ' Macro-F1: 0.447']
+12250816 - clip_momenta_features_frcnn-lr2e-4-mt128-affine-sberFalse-caption+text_cyclic5shorter_val.pkl : ['Epoch 25 val', ' Macro-F1: 0.436']
+12250787 - clip_momenta_features_frcnn-lr2e-4-mt300-macro-sberTrue-separated_cyclic5_val.pkl : ['Epoch 7 val', ' Macro-F1: 0.265']
+12250819 - clip_momenta_features_frcnn-lr2e-4-mt300-macro-sberTrue-separated_cyclic5shorter_val.pkl : ['Epoch 28 val', ' Macro-F1: 0.335']
+12235573 - not saved : ['Epoch 26 val', ' Macro-F1: 0.462']
+12236743 - not saved : ['Epoch 15 val', ' Macro-F1: 0.361']
+12239573 - ofa_snli_val.pkl : ['Epoch 4 val', ' Macro-F1: 0.245']
+12239965 - ofa_snli_val.pkl : ['Epoch 1 val', ' Macro-F1: 0.091']
+12236582 - ofa_vqa_base_val.pkl : ['Epoch 12 val', ' Macro-F1: 0.303']
+12237268 - ofa_vqa_base_val_ocr+q.pkl : ['Epoch 4 val', ' Macro-F1: 0.144']
+12237723 - ofa_vqa_base_val_ocr+q.pkl : ['Epoch 11 val', ' Macro-F1: 0.288']
+12237742 - ofa_vqa_base_val_unfreeze.pkl : ['Epoch 14 val', ' Macro-F1: 0.384']
+12237743 - ofa_vqa_large_val.pkl : ['Epoch 9 val', ' Macro-F1: 0.329']
+12240112 - ofa_vqa_ofa_base.pt-all-1e-4-1_val.pkl : ['Epoch 1 val', ' Macro-F1: 0.217']
+12240126 - ofa_vqa_ofa_base.pt-all-1e-4-1_val.pkl : ['Epoch 1 val', ' Macro-F1: 0.214']
+12240115 - ofa_vqa_ofa_base.pt-all-1e-5-2_val.pkl : ['Epoch 1 val', ' Macro-F1: 0.128']
+12240129 - ofa_vqa_ofa_base.pt-all-1e-5-2_val.pkl : ['Epoch 1 val', ' Macro-F1: 0.229']
+12240114 - ofa_vqa_ofa_base.pt-all-1e-6-1_val.pkl : ['Epoch 1 val', ' Macro-F1: 0.069']
+12240128 - ofa_vqa_ofa_base.pt-all-1e-6-1_val.pkl : ['Epoch 1 val', ' Macro-F1: 0.077']
+12240130 - ofa_vqa_ofa_base.pt-all-5e-5-2_val.pkl : ['Epoch 1 val', ' Macro-F1: 0.167']
+12240113 - ofa_vqa_ofa_base.pt-classifier-1e-4-1_val.pkl : ['Epoch 2 val', ' Macro-F1: 0.103']
+12240127 - ofa_vqa_ofa_base.pt-classifier-1e-4-1_val.pkl : ['Epoch 11 val', ' Macro-F1: 0.287']
+12241299 - ofa_vqa_old_max35_lr5_val.pkl : ['Epoch 29 val', ' Macro-F1: 0.379']
+12241228 - ofa_vqa_old_new_val.pkl : ['Epoch 26 val', ' Macro-F1: 0.395']
+12241201 - ofa_vqa_old_val.pkl : ['Epoch 26 val', ' Macro-F1: 0.460']
+12236083 - ofa_vqa_val.pkl : ['Epoch 20 val', ' Macro-F1: 0.409']
+12240108 - ofa_vqa_vqa_large_best.pt-all-1e-5-2_val.pkl : ['Epoch 1 val', ' Macro-F1: 0.072']
+12240106 - ofa_vqa_vqa_large_best.pt-all-1e-6-1_val.pkl : ['Epoch 1 val', ' Macro-F1: 0.137']
+12240110 - ofa_vqa_vqa_large_best.pt-all-5e-5-2_val.pkl : ['Epoch 1 val', ' Macro-F1: 0.109']
+12240009 - visualbert_lbert-nlvr2-coco-pre_1e-4_val.pkl : ['Epoch 20 val', ' Macro-F1: 0.387']
+12240120 - visualbert_lbert-nlvr2-coco-pre_1e-5_sb0.5-a_val.pkl : ['Epoch 31 val', ' Macro-F1: 0.366']
+12240086 - visualbert_lbert-nlvr2-coco-pre_1e-5_sb0.5_val.pkl : ['Epoch 9 val', ' Macro-F1: 0.286']
+12240083 - visualbert_lbert-nlvr2-coco-pre_1e-5_sb1_val.pkl : ['Epoch 15 val', ' Macro-F1: 0.303']
+12240123 - visualbert_lbert-nlvr2-coco-pre_1e-5_sb2-a_val.pkl : ['Epoch 25 val', ' Macro-F1: 0.369']
+12240006 - visualbert_lbert-nlvr2-coco-pre_1e-6_val.pkl : ['Epoch 5 val', ' Macro-F1: 0.213']
+12250862 - visualbert_ualbert-vcr-coco-pre_1e-4_capFalse_affine_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.462']
+12250836 - visualbert_ualbert-vcr-coco-pre_1e-4_capFalse_cyclic5_val.pkl : ['Epoch 30 val', ' Macro-F1: 0.442']
+12250854 - visualbert_ualbert-vcr-coco-pre_1e-4_capFalse_macro_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.440']
+12250860 - visualbert_ualbert-vcr-coco-pre_1e-4_capTrue_affine_val.pkl : ['Epoch 25 val', ' Macro-F1: 0.465']
+12250834 - visualbert_ualbert-vcr-coco-pre_1e-4_capTrue_cyclic5_val.pkl : ['Epoch 31 val', ' Macro-F1: 0.444']
+12250852 - visualbert_ualbert-vcr-coco-pre_1e-4_capTrue_macro_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.452']
+12240132 - visualbert_ualbert-vcr-coco-pre_1e-4_sb1-c_val.pkl : ['Epoch 17 val', ' Macro-F1: 0.406']
+12240138 - visualbert_ualbert-vcr-coco-pre_1e-4_sb2-b_val.pkl : ['Epoch 19 val', ' Macro-F1: 0.367']
+12240134 - visualbert_ualbert-vcr-coco-pre_1e-4_sb2-c_val.pkl : ['Epoch 31 val', ' Macro-F1: 0.450']
+12240010 - visualbert_ualbert-vcr-coco-pre_1e-4_val.pkl : ['Epoch 19 val', ' Macro-F1: 0.390']
+12250866 - visualbert_ualbert-vcr-coco-pre_1e-5_capFalse_affine_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.378']
+12250840 - visualbert_ualbert-vcr-coco-pre_1e-5_capFalse_cyclic5_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.378']
+12250858 - visualbert_ualbert-vcr-coco-pre_1e-5_capFalse_macro_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.399']
+12250864 - visualbert_ualbert-vcr-coco-pre_1e-5_capTrue_affine_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.395']
+12250838 - visualbert_ualbert-vcr-coco-pre_1e-5_capTrue_cyclic5_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.417']
+12250856 - visualbert_ualbert-vcr-coco-pre_1e-5_capTrue_macro_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.411']
+12240007 - visualbert_ualbert-vcr-coco-pre_1e-6_val.pkl : ['Epoch 11 val', ' Macro-F1: 0.220']
+12240121 - visualbert_ualbert-vcr-coco-pre_5e-4_sb0.5-a_val.pkl : ['Epoch 7 val', ' Macro-F1: 0.217']
+12240084 - visualbert_ualbert-vcr-coco-pre_5e-4_sb1_val.pkl : ['Epoch 5 val', ' Macro-F1: 0.013']
+12240124 - visualbert_ualbert-vcr-coco-pre_5e-4_sb2-a_val.pkl : ['Epoch 8 val', ' Macro-F1: 0.217']
+12240090 - visualbert_ualbert-vcr-coco-pre_5e-4_sb2_val.pkl : ['Epoch 5 val', ' Macro-F1: 0.089']
+12250861 - visualbert_ualbert-vqa-coco-pre_1e-4_capFalse_affine_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.450']
+12250835 - visualbert_ualbert-vqa-coco-pre_1e-4_capFalse_cyclic5_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.461']
+12250853 - visualbert_ualbert-vqa-coco-pre_1e-4_capFalse_macro_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.474']
+12250859 - visualbert_ualbert-vqa-coco-pre_1e-4_capTrue_affine_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.473']
+12250833 - visualbert_ualbert-vqa-coco-pre_1e-4_capTrue_cyclic5_val.pkl : ['Epoch 15 val', ' Macro-F1: 0.432']
+12250851 - visualbert_ualbert-vqa-coco-pre_1e-4_capTrue_macro_val.pkl : ['Epoch 22 val', ' Macro-F1: 0.439']
+12240131 - visualbert_ualbert-vqa-coco-pre_1e-4_sb1-c_val.pkl : ['Epoch 18 val', ' Macro-F1: 0.421']
+12240133 - visualbert_ualbert-vqa-coco-pre_1e-4_sb2-c_val.pkl : ['Epoch 16 val', ' Macro-F1: 0.418']
+12240011 - visualbert_ualbert-vqa-coco-pre_1e-4_val.pkl : ['Epoch 22 val', ' Macro-F1: 0.413']
+12250865 - visualbert_ualbert-vqa-coco-pre_1e-5_capFalse_affine_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.401']
+12250839 - visualbert_ualbert-vqa-coco-pre_1e-5_capFalse_cyclic5_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.397']
+12250857 - visualbert_ualbert-vqa-coco-pre_1e-5_capFalse_macro_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.391']
+12250863 - visualbert_ualbert-vqa-coco-pre_1e-5_capTrue_affine_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.419']
+12250837 - visualbert_ualbert-vqa-coco-pre_1e-5_capTrue_cyclic5_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.394']
+12250855 - visualbert_ualbert-vqa-coco-pre_1e-5_capTrue_macro_val.pkl : ['Epoch 35 val', ' Macro-F1: 0.391']
+12240008 - visualbert_ualbert-vqa-coco-pre_1e-6_val.pkl : ['Epoch 10 val', ' Macro-F1: 0.225']
+12240122 - visualbert_ualbert-vqa-coco-pre_5e-4_sb0.5-a_val.pkl : ['Epoch 7 val', ' Macro-F1: 0.228']
+12240085 - visualbert_ualbert-vqa-coco-pre_5e-4_sb1_val.pkl : ['Epoch 5 val', ' Macro-F1: 0.013']
+12240125 - visualbert_ualbert-vqa-coco-pre_5e-4_sb2-a_val.pkl : ['Epoch 5 val', ' Macro-F1: 0.028']
+12240091 - visualbert_ualbert-vqa-coco-pre_5e-4_sb2_val.pkl : ['Epoch 5 val', ' Macro-F1: 0.072']
diff --git a/slurmscripts/distrib_search.sh b/slurmscripts/distrib_search.sh
@@ -0,0 +1,27 @@
+#!/bin/bash
+#SBATCH -A lco@gpu
+#SBATCH --job-name=mmsrl # job name
+#SBATCH --ntasks=1 # number of MP tasks
+#SBATCH --ntasks-per-node=1 # number of MPI tasks per node
+#SBATCH --gres=gpu:1 # number of GPUs per node
+#SBATCH --cpus-per-task=10 # number of cores per tasks
+#SBATCH --hint=nomultithread # we get physical cores not logical
+#SBATCH --distribution=block:block # we pin the tasks on contiguous cores
+#SBATCH --time=5:00:00 # maximum execution time (HH:MM:SS)
+#SBATCH --output=new_old_freq_%j_%x_%A_%a.out # output file name
+#SBATCH --error=new_old_freq_%j_%x_%A_%a.err # error file name
+#SBATCH --array=0-19
+
+
+cd $WORK/mmsrl
+
+module purge
+source ~/.bashrc
+conda activate pytorch
+export DATA_PATH=$WORK/data
+
+FACTOR=('0.1' '0.3' '0.5' '0.7' '0.9')
+BIAS=('0.02' '0.05' '0.1' '0.2')
+ifactor=$(($SLURM_ARRAY_TASK_ID%5))
+ibias=$(($SLURM_ARRAY_TASK_ID/5))
+python -m mmsrl.train configs/ofa_new_old_best.py --frequency_factor=${FACTOR[$ifactor]} --frequency_bias=${BIAS[$ibias]} --output_val=\"val_${FACTOR[$ifactor]}_${BIAS[$ibias]}.pkl\" --output_test=\"test_${FACTOR[$ifactor]}_${BIAS[$ibias]}.pkl\"
diff --git a/slurmscripts/generic_features.sh b/slurmscripts/generic_features.sh
@@ -0,0 +1,52 @@
+#!/bin/bash
+#SBATCH -A lco@gpu
+#SBATCH --job-name=mmsrl # job name
+#SBATCH --ntasks=1 # number of MP tasks
+#SBATCH --ntasks-per-node=1 # number of MPI tasks per node
+#SBATCH --gres=gpu:1 # number of GPUs per node
+#SBATCH --cpus-per-task=10 # number of cores per tasks
+#SBATCH --hint=nomultithread # we get physical cores not logical
+#SBATCH --distribution=block:block # we pin the tasks on contiguous cores
+#SBATCH --time=10:00:00 # maximum execution time (HH:MM:SS)
+#SBATCH --output=/gpfswork/rech/lco/url46ht/outputs/generic_features/generic_feature_%j_%x_%A_%a.out # output file name
+#SBATCH --error=/gpfswork/rech/lco/url46ht/outputs/generic_features/generic_feature_%j_%x_%A_%a.err # error file name
+#SBATCH --array=0-17
+
+
+cd $WORK/mmsrl
+
+module purge
+source ~/.bashrc
+conda activate pytorch
+export DATA_PATH=$WORK/data
+export HF_DATASETS_OFFLINE=1
+export TRANSFORMERS_OFFLINE=1
+
+
+base_config=("--features_path='all' --use_sbert=True --use_caption=True --use_entities=True --use_visualbert=True --use_clip=True"
+ "--features_path=features_frcnn --use_sbert=True --use_caption=True --use_entities=True --use_visualbert=True --use_clip=True"
+ "--features_path=features_b7 --use_sbert=True --use_caption=True --use_entities=True --use_visualbert=True --use_clip=True"
+ "--features_path=features_vgg --use_sbert=True --use_caption=True --use_entities=True --use_visualbert=True --use_clip=True"
+ "--features_path=features_frcnn --use_sbert=True --use_entities=True --use_visualbert=True --use_clip=True"
+ "--features_path=features_frcnn --use_sbert=True --use_caption=True --use_visualbert=True --use_clip=True"
+ "--features_path=features_frcnn --use_sbert=True --use_caption=True --use_entities=True --use_clip=True"
+ "--features_path=features_frcnn --use_sbert=True --use_caption=True --use_entities=True --use_visualbert=True"
+ "--features_path=features_frcnn --use_caption=True --use_entities=True --use_visualbert=True --use_clip=True")
+
+base_name=("all"
+ "frcnn"
+ "b7"
+ "vgg"
+ "nocaption"
+ "noentities"
+ "novisualbert"
+ "noclip"
+ "nosbert")
+base_pooling=(attention mlp)
+
+iconfig=$(($SLURM_ARRAY_TASK_ID%9))
+ipooling=$(($SLURM_ARRAY_TASK_ID/9%2))
+config=${base_config[$iconfig]}
+pooling=${base_pooling[$ipooling]}
+name=${base_name[$iconfig]}_${pooling}
+python -m mmsrl.train configs/generic.py --pooling=$pooling $config --patience=26 --output_val=\"$WORK/outputs/generic_features/val_${name}_${SLURM_ARRAY_TASK_ID}.pkl\" --output_test=\"$WORK/outputs/generic_features/test_${name}_${SLURM_ARRAY_TASK_ID}.pkl\"
diff --git a/slurmscripts/generic_features_attention.sh b/slurmscripts/generic_features_attention.sh
@@ -0,0 +1,54 @@
+#!/bin/bash
+#SBATCH -A lco@gpu
+#SBATCH --job-name=mmsrl # job name
+#SBATCH --ntasks=1 # number of MP tasks
+#SBATCH --ntasks-per-node=1 # number of MPI tasks per node
+#SBATCH --gres=gpu:1 # number of GPUs per node
+#SBATCH --cpus-per-task=10 # number of cores per tasks
+#SBATCH --hint=nomultithread # we get physical cores not logical
+#SBATCH --distribution=block:block # we pin the tasks on contiguous cores
+#SBATCH --time=10:00:00 # maximum execution time (HH:MM:SS)
+#SBATCH --output=/gpfswork/rech/lco/url46ht/outputs/generic_features_attention/generic_features_attention_%j_%x_%A_%a.out # output file name
+#SBATCH --error=/gpfswork/rech/lco/url46ht/outputs/generic_features_attention/generic_features_attention_%j_%x_%A_%a.err # error file name
+#SBATCH --array=0-39
+
+
+cd $WORK/mmsrl
+
+module purge
+source ~/.bashrc
+conda activate pytorch
+export DATA_PATH=$WORK/data
+export TRANSFORMERS_CACHE=$HOME/.cache/huggingface/transformers
+export TRANSFORMERS_OFFLINE=1
+
+base_config=("--features_path=features_b7 --use_sbert=True --use_caption=True --use_entities=True --use_visualbert=True --use_clip=True"
+ "--use_clip=True"
+ "--features_path=features_b7 --use_clip=True"
+ "--features_path=features_vgg --use_clip=True"
+ "--features_path=features_frcnn --use_clip=True"
+ "--features_path='all' --use_caption=True --use_clip=True"
+ "--features_path='all' --use_sbert=True --use_entities=True --use_clip=True"
+ "--features_path='all' --use_visualbert=True --use_clip=True")
+
+base_name=("all"
+ "clip"
+ "b7"
+ "vgg"
+ "frcnn"
+ "caption"
+ "entities"
+ "visualbert")
+
+DIRNAME=generic_features_attention
+mkdir $WORK/outputs/$DIRNAME
+
+iconfig=$(($SLURM_ARRAY_TASK_ID%8))
+config=${base_config[$iconfig]}
+name=${base_name[$iconfig]}}
+
+python -m mmsrl.train configs/generic.py --pooling=attention $config \
+ --patience=26 --batch_per_epoch=1000 \
+ --cyclic_subsample=None --subsample_labels='interpolate_micro_to_macro' \
+ --output_val=\"$WORK/outputs/$DIRNAME/val_${name}_${SLURM_ARRAY_TASK_ID}.pkl\" \
+ --output_test=\"$WORK/outputs/$DIRNAME/test_${name}_${SLURM_ARRAY_TASK_ID}.pkl\"
diff --git a/slurmscripts/generic_features_attention2.sh b/slurmscripts/generic_features_attention2.sh
@@ -0,0 +1,44 @@
+#!/bin/bash
+#SBATCH -A lco@gpu
+#SBATCH --job-name=mmsrl # job name
+#SBATCH --ntasks=1 # number of MP tasks
+#SBATCH --ntasks-per-node=1 # number of MPI tasks per node
+#SBATCH --gres=gpu:1 # number of GPUs per node
+#SBATCH --cpus-per-task=10 # number of cores per tasks
+#SBATCH --hint=nomultithread # we get physical cores not logical
+#SBATCH --distribution=block:block # we pin the tasks on contiguous cores
+#SBATCH --time=10:00:00 # maximum execution time (HH:MM:SS)
+#SBATCH --output=/gpfswork/rech/lco/url46ht/outputs/generic_features_attention/generic_features_attention_%j_%x_%A_%a.out # output file name
+#SBATCH --error=/gpfswork/rech/lco/url46ht/outputs/generic_features_attention/generic_features_attention_%j_%x_%A_%a.err # error file name
+#SBATCH --array=0-39
+
+
+cd $WORK/mmsrl
+
+module purge
+source ~/.bashrc
+conda activate pytorch
+export DATA_PATH=$WORK/data
+export TRANSFORMERS_CACHE=$HOME/.cache/huggingface/transformers
+export TRANSFORMERS_OFFLINE=1
+
+base_config=("--use_caption=True --use_clip=True"
+ "--use_sbert=True --use_entities=True --use_clip=True"
+ "--use_visualbert=True --use_clip=True")
+
+base_name=("caption"
+ "entities"
+ "visualbert")
+
+DIRNAME=generic_features_attention
+mkdir $WORK/outputs/$DIRNAME
+
+iconfig=$(($SLURM_ARRAY_TASK_ID%8))
+config=${base_config[$iconfig]}
+name=${base_name[$iconfig]}}
+
+python -m mmsrl.train configs/generic.py --pooling=attention $config \
+ --patience=26 --batch_per_epoch=1000 \
+ --cyclic_subsample=None --subsample_labels='interpolate_micro_to_macro' \
+ --output_val=\"$WORK/outputs/$DIRNAME/val_${name}_${SLURM_ARRAY_TASK_ID}.pkl\" \
+ --output_test=\"$WORK/outputs/$DIRNAME/test_${name}_${SLURM_ARRAY_TASK_ID}.pkl\"
diff --git a/slurmscripts/generic_features_mlp.sh b/slurmscripts/generic_features_mlp.sh
@@ -0,0 +1,53 @@
+#!/bin/bash
+#SBATCH -A lco@gpu
+#SBATCH --job-name=mmsrl # job name
+#SBATCH --ntasks=1 # number of MP tasks
+#SBATCH --ntasks-per-node=1 # number of MPI tasks per node
+#SBATCH --gres=gpu:1 # number of GPUs per node
+#SBATCH --cpus-per-task=10 # number of cores per tasks
+#SBATCH --hint=nomultithread # we get physical cores not logical
+#SBATCH --distribution=block:block # we pin the tasks on contiguous cores
+#SBATCH --time=10:00:00 # maximum execution time (HH:MM:SS)
+#SBATCH --output=/gpfswork/rech/lco/url46ht/outputs/generic_features_mlp/generic_features_mlp%j_%x_%A_%a.out # output file name
+#SBATCH --error=/gpfswork/rech/lco/url46ht/outputs/generic_features_mlp/generic_features_mlp%j_%x_%A_%a.err # error file name
+#SBATCH --array=0-39
+
+
+cd $WORK/mmsrl
+
+module purge
+source ~/.bashrc
+conda activate pytorch
+export DATA_PATH=$WORK/data
+export TRANSFORMERS_CACHE=$HOME/.cache/huggingface/transformers
+export TRANSFORMERS_OFFLINE=1
+
+base_config=("--features_path='all' --use_sbert=True --use_caption=True --use_entities=True --use_visualbert=True --use_clip=True"
+ "--use_clip=True"
+ "--features_path=features_b7 --use_clip=True"
+ "--features_path=features_vgg --use_clip=True"
+ "--features_path=features_frcnn --use_clip=True"
+ "--features_path='all' --use_caption=True --use_clip=True"
+ "--features_path='all' --use_sbert=True --use_entities=True --use_clip=True"
+ "--features_path='all' --use_visualbert=True --use_clip=True")
+
+base_name=("all"
+ "clip"
+ "b7"
+ "vgg"
+ "frcnn"
+ "caption"
+ "entities"
+ "novisualbert")
+
+DIRNAME=generic_features_mlp
+
+iconfig=$(($SLURM_ARRAY_TASK_ID%8))
+config=${base_config[$iconfig]}
+name=${base_name[$iconfig]}}
+
+python -m mmsrl.train configs/generic.py --pooling=mlp $config \
+ --patience=26 --batch_per_epoch=1000 \
+ --cyclic_subsample=None --subsample_labels='interpolate_micro_to_macro' \
+ --output_val=\"$WORK/outputs/$DIRNAME/val_${name}_${SLURM_ARRAY_TASK_ID}.pkl\" \
+ --output_test=\"$WORK/outputs/$DIRNAME/test_${name}_${SLURM_ARRAY_TASK_ID}.pkl\"
diff --git a/slurmscripts/generic_features_visualbert.sh b/slurmscripts/generic_features_visualbert.sh
@@ -0,0 +1,55 @@
+#!/bin/bash
+#SBATCH -A lco@gpu
+#SBATCH --job-name=mmsrl # job name
+#SBATCH --ntasks=1 # number of MP tasks
+#SBATCH --ntasks-per-node=1 # number of MPI tasks per node
+#SBATCH --gres=gpu:1 # number of GPUs per node
+#SBATCH --cpus-per-task=10 # number of cores per tasks
+#SBATCH --hint=nomultithread # we get physical cores not logical
+#SBATCH --distribution=block:block # we pin the tasks on contiguous cores
+#SBATCH --time=10:00:00 # maximum execution time (HH:MM:SS)
+#SBATCH --output=/gpfswork/rech/lco/url46ht/outputs/generic_features_visualbert/generic_features_visualbert_%j_%x_%A_%a.out # output file name
+#SBATCH --error=/gpfswork/rech/lco/url46ht/outputs/generic_features_visualbert/generic_features_visualbert_%j_%x_%A_%a.err # error file name
+#SBATCH --array=0-95
+
+
+cd $WORK/mmsrl
+
+module purge
+source ~/.bashrc
+conda activate pytorch
+export DATA_PATH=$WORK/data
+export TRANSFORMERS_CACHE=$HOME/.cache/huggingface/transformers
+export TRANSFORMERS_OFFLINE=1
+
+base_config=("--features_path='all' --use_sbert=True --use_caption=True --use_entities=True"
+ "--features_path=features_frcnn"
+ "--features_path=features_b7"
+ "--features_path=features_vgg"
+ "--features_path=features_b7 --use_sbert=True --use_entities=True"
+ "--features_path=features_b7 --use_caption=True")
+
+base_name=("all"
+ "frcnn"
+ "b7"
+ "vgg"
+ "entities"
+ "caption")
+base_pooling=(attention mlp)
+base_lr=(5e-3 1e-4 5e-4 1e-5)
+
+iconfig=$(($SLURM_ARRAY_TASK_ID%6))
+ipooling=$(($SLURM_ARRAY_TASK_ID/6%2))
+ilr=$(($SLURM_ARRAY_TASK_ID/12%4))
+
+config=${base_config[$iconfig]}
+pooling=${base_pooling[$ipooling]}
+lr=${base_lr[$ilr]}
+name=${base_name[$iconfig]}_${pooling}_lr${lr}
+
+python -m mmsrl.train configs/generic.py --use_clip=False --pooling=$pooling $config --patience=26 \
+ --batch_per_epoch=1000 --use_visualbert=True\
+ --learning_rate=$lr \
+ --cyclic_subsample=None --subsample_labels='interpolate_micro_to_macro' \
+ --output_val=\"$WORK/outputs/generic_features_visualbert/val_${name}_${SLURM_ARRAY_TASK_ID}.pkl\" \
+ --output_test=\"$WORK/outputs/generic_features_visualbert/test_${name}_${SLURM_ARRAY_TASK_ID}.pkl\"
diff --git a/slurmscripts/generic_subsample.sh b/slurmscripts/generic_subsample.sh
@@ -0,0 +1,50 @@
+#!/bin/bash
+#SBATCH -A lco@gpu
+#SBATCH --job-name=mmsrl # job name
+#SBATCH --ntasks=1 # number of MP tasks
+#SBATCH --ntasks-per-node=1 # number of MPI tasks per node
+#SBATCH --gres=gpu:1 # number of GPUs per node
+#SBATCH --cpus-per-task=10 # number of cores per tasks
+#SBATCH --hint=nomultithread # we get physical cores not logical
+#SBATCH --distribution=block:block # we pin the tasks on contiguous cores
+#SBATCH --time=10:00:00 # maximum execution time (HH:MM:SS)
+#SBATCH --output=/gpfswork/rech/lco/url46ht/outputs/generic_subsample/generic_subsample_%j_%x_%A_%a.out # output file name
+#SBATCH --error=/gpfswork/rech/lco/url46ht/outputs/generic_subsample/generic_subsample_%j_%x_%A_%a.err # error file name
+#SBATCH --array=0-104
+
+
+cd $WORK/mmsrl
+
+module purge
+source ~/.bashrc
+conda activate pytorch
+export DATA_PATH=$WORK/data
+export TRANSFORMERS_CACHE=$HOME/.cache/huggingface/transformers
+export TRANSFORMERS_OFFLINE=1
+
+base_config=("--cyclic_subsample=None --subsample_labels=micro"
+ "--cyclic_subsample=None --subsample_labels=macro"
+ "--cyclic_subsample=None --subsample_labels=target --frequency_target=[2,1.5,1.75,1]"
+ "--cyclic_subsample=None --subsample_labels='interpolate_micro_to_macro'"
+ "--cyclic_subsample=None --subsample_labels=affine --frequency_factor=0.762 --frequency_bias=0.06"
+ "configs/cyclic.py"
+ "configs/cyclic.py --cyclic_subsample=cyclic_subsample_shorter")
+
+base_name=("micro"
+ "macro"
+ "supramacro"
+ "interpolate"
+ "affine"
+ "long_cycle"
+ "short_cycle")
+base_bpe=(500 1000 2000)
+
+mkdir $WORK/outputs/generic_subsample
+
+
+iconfig=$(($SLURM_ARRAY_TASK_ID%7))
+ibpe=$(($SLURM_ARRAY_TASK_ID/7%3))
+config=${base_config[$iconfig]}
+bpe=${base_bpe[$ibpe]}
+name=${base_name[$iconfig]}_bpe${bpe}
+python -m mmsrl.train configs/generic.py --pooling=mlp $config --patience=26 --batch_per_epoch=$bpe --output_val=\"$WORK/outputs/generic_subsample/val_${name}_${SLURM_ARRAY_TASK_ID}.pkl\" --output_test=\"$WORK/outputs/generic_subsample/test_${name}_${SLURM_ARRAY_TASK_ID}.pkl\"
diff --git a/slurmscripts/ofa_search.sh b/slurmscripts/ofa_search.sh
@@ -0,0 +1,37 @@
+#!/bin/bash
+#SBATCH -A lco@gpu
+#SBATCH -C v100-32g
+#SBATCH --job-name=mmsrl # job name
+#SBATCH --ntasks=1 # number of MP tasks
+#SBATCH --ntasks-per-node=1 # number of MPI tasks per node
+#SBATCH --gres=gpu:1 # number of GPUs per node
+#SBATCH --cpus-per-task=10 # number of cores per tasks
+#SBATCH --hint=nomultithread # we get physical cores not logical
+#SBATCH --distribution=block:block # we pin the tasks on contiguous cores
+#SBATCH --time=5:00:00 # maximum execution time (HH:MM:SS)
+#SBATCH --output=ofa_%j_%x_%A_%a.out # output file name
+#SBATCH --error=ofa_%j_%x_%A_%a.err # error file name
+#SBATCH --array=0-35
+
+
+cd $WORK/mmsrl
+
+module purge
+source ~/.bashrc
+conda activate pytorch
+export DATA_PATH=$WORK/data
+
+TASK=('configs/ofa_vqa.py' 'configs/ofa_snli.py')
+PARAMS=('configs/ofa_repo.py' 'configs/ofa_ours.py' 'configs/ofa_ours.py --freeze_resnet --freeze_embeddings')
+CYCLE=('--subsample_labels=affine --frequency_factor=0.1 --frequency_bias=0.05' 'configs/alternate_macro_micro.py' 'configs/alternate_macro_micro.py --cyclic_subsample=cyclic_subsample_longer')
+HEAD=('--ofa_classification_head=False' '--ofa_classification_head=True')
+itask=$(($SLURM_ARRAY_TASK_ID%2))
+iparams=$(($SLURM_ARRAY_TASK_ID%6/2))
+icycle=$(($SLURM_ARRAY_TASK_ID%18/6))
+ihead=$(($SLURM_ARRAY_TASK_ID/18))
+task=${TASK[$itask]}
+params=${PARAMS[$iparams]}
+cycle=${CYCLE[$icycle]}
+head=${HEAD[$ihead]}
+
+python -m mmsrl.train $task $params $cycle $head --patience=10 --output_val=\"val_${itask}_${iparams}_${icycle}_${ihead}.pkl\" --output_test=\"test_${itask}_${iparams}_${icycle}_${ihead}.pkl\"
diff --git a/slurmscripts/ofa_seq2seq.sh b/slurmscripts/ofa_seq2seq.sh
@@ -0,0 +1,36 @@
+#!/bin/bash
+#SBATCH -A lco@gpu
+#SBATCH -C v100-32g
+#SBATCH --job-name=mmsrl # job name
+#SBATCH --ntasks=1 # number of MP tasks
+#SBATCH --ntasks-per-node=1 # number of MPI tasks per node
+#SBATCH --gres=gpu:1 # number of GPUs per node
+#SBATCH --cpus-per-task=10 # number of cores per tasks
+#SBATCH --hint=nomultithread # we get physical cores not logical
+#SBATCH --distribution=block:block # we pin the tasks on contiguous cores
+#SBATCH --time=10:00:00 # maximum execution time (HH:MM:SS)
+#SBATCH --output=/gpfswork/rech/lco/url46ht/outputs/ofa_seq2seq_%j_%x_%A_%a.out # output file name
+#SBATCH --error=/gpfswork/rech/lco/url46ht/outputs/ofa_seq2seq_%j_%x_%A_%a.err # error file name
+#SBATCH --array=0-11
+
+set -x
+
+cd $WORK/mmsrl
+
+module purge
+source ~/.bashrc
+conda activate pytorch
+export DATA_PATH=$WORK/data
+
+base_config=(""
+ "--ofa_path=OFA/checkpoints/ofa_base.pt"
+ "--weight_decay=0.01"
+ "--ofa_path=OFA/checkpoints/ofa_base.pt --weight_decay=0.01")
+base_name=("s2s_vqa_large"
+ "s2s_base"
+ "s2s_vqa_large_ddecay"
+ "s2s_base_ddecay")
+iconfig=$(($SLURM_ARRAY_TASK_ID%4))
+config=${base_config[$iconfig]}
+name=${base_name[$iconfig]}
+python -m mmsrl.train configs/ofa_ours.py configs/ofa_vqa.py configs/ofa_seq2seq.py --patience=26 $config --output_val=\"$WORK/outputs/val_${name}_${SLURM_ARRAY_TASK_ID}.pkl\" --output_test=\"$WORK/outputs/test_${name}_${SLURM_ARRAY_TASK_ID}.pkl\"
diff --git a/slurmscripts/ofa_seq2seq_freeze.sh b/slurmscripts/ofa_seq2seq_freeze.sh
@@ -0,0 +1,58 @@
+#!/bin/bash
+#SBATCH -A lco@gpu
+#SBATCH -C v100-32g
+#SBATCH --job-name=mmsrl # job name
+#SBATCH --ntasks=1 # number of MP tasks
+#SBATCH --ntasks-per-node=1 # number of MPI tasks per node
+#SBATCH --gres=gpu:1 # number of GPUs per node
+#SBATCH --cpus-per-task=10 # number of cores per tasks
+#SBATCH --hint=nomultithread # we get physical cores not logical
+#SBATCH --distribution=block:block # we pin the tasks on contiguous cores
+#SBATCH --time=10:00:00 # maximum execution time (HH:MM:SS)
+#SBATCH --output=/gpfswork/rech/lco/url46ht/outputs/fixed_seq2seq_%j_%x_%A_%a.out # output file name
+#SBATCH --error=/gpfswork/rech/lco/url46ht/outputs/fixed_seq2seq_%j_%x_%A_%a.err # error file name
+#SBATCH --array=0-321
+
+set -x
+
+cd $WORK/mmsrl
+
+module purge
+source ~/.bashrc
+conda activate pytorch
+export DATA_PATH=$WORK/data
+
+base_config=(""
+ "--ofa_path=OFA/checkpoints/ofa_base.pt"
+ "--weight_decay=0.01"
+ "--ofa_path=OFA/checkpoints/ofa_base.pt --weight_decay=0.01")
+base_config_name=("vqa_large"
+ "base"
+ "vqa_large_ddecay"
+ "base_ddecay")
+base_freeze=("--unfreeze='all'"
+ "--unfreeze='all' --freeze_resnet"
+ "--unfreeze='all' --freeze_embeddings --freeze_resnet"
+ "--unfreeze=classifier --freeze_resnet")
+base_freeze_name=("none"
+ "resnet"
+ "resnet_emb"
+ "all")
+base_loss=("--subsample_labels=macro"
+ "--subsample_labels=macro --label_smoothing=0.1"
+ "--subsample_labels=interpolate_micro_to_macro"
+ "--subsample_labels=interpolate_micro_to_macro --label_smoothing=0.1")
+base_loss_name=("macro_onehot"
+ "macro_smoothed"
+ "interpolate_onehot"
+ "interpolate_smooth")
+
+iconfig=$(($SLURM_ARRAY_TASK_ID%4))
+ifreeze=$(($SLURM_ARRAY_TASK_ID/4%4))
+iloss=$(($SLURM_ARRAY_TASK_ID/16%4))
+config=${base_config[$iconfig]}
+freeze=${base_freeze[$ifreeze]}
+loss=${base_loss[$iloss]}
+name=${base_config_name[$iconfig]}_${base_freeze_name[$ifreeze]}_${base_loss_name[$iloss]}
+
+python -m mmsrl.train configs/ofa_ours.py configs/ofa_vqa.py configs/ofa_seq2seq.py --patience=26 $config $freeze $loss --output_val=\"$WORK/outputs/val_${name}_${SLURM_ARRAY_TASK_ID}.pkl\" --output_test=\"$WORK/outputs/test_${name}_${SLURM_ARRAY_TASK_ID}.pkl\"
diff --git a/slurmscripts/ofa_subsample.sh b/slurmscripts/ofa_subsample.sh
@@ -0,0 +1,43 @@
+#!/bin/bash
+#SBATCH -A lco@gpu
+#SBATCH -C v100-32g
+#SBATCH --job-name=mmsrl # job name
+#SBATCH --ntasks=1 # number of MP tasks
+#SBATCH --ntasks-per-node=1 # number of MPI tasks per node
+#SBATCH --gres=gpu:1 # number of GPUs per node
+#SBATCH --cpus-per-task=10 # number of cores per tasks
+#SBATCH --hint=nomultithread # we get physical cores not logical
+#SBATCH --distribution=block:block # we pin the tasks on contiguous cores
+#SBATCH --time=10:00:00 # maximum execution time (HH:MM:SS)
+#SBATCH --output=/gpfswork/rech/lco/url46ht/outputs/ofa_subsample_bpe_%j_%x_%A_%a.out # output file name
+#SBATCH --error=/gpfswork/rech/lco/url46ht/outputs/ofa_subsample_bpe_%j_%x_%A_%a.err # error file name
+#SBATCH --array=0-53
+
+
+cd $WORK/mmsrl
+
+module purge
+source ~/.bashrc
+conda activate pytorch
+export DATA_PATH=$WORK/data
+
+base_config=("--cyclic_subsample=None --subsample_labels=micro"
+ "--cyclic_subsample=None --subsample_labels=macro"
+ "--cyclic_subsample=None --subsample_labels=target --frequency_target=[2,1.5,1.75,1]"
+ "--cyclic_subsample=None --subsample_labels='interpolate_micro_to_macro'"
+ "configs/alternate_macro_micro.py"
+ "configs/alternate_macro_micro.py --cyclic_subsample=cyclic_subsample_shorter")
+base_name=("micro"
+ "macro"
+ "supramacro"
+ "interpolate"
+ "long_cycle"
+ "short_cycle")
+base_bpe=(500 1000 2000)
+
+iconfig=$(($SLURM_ARRAY_TASK_ID%6))
+ibpe=$(($SLURM_ARRAY_TASK_ID/6%3))
+config=${base_config[$iconfig]}
+bpe=${base_bpe[$ibpe]}
+name=${base_name[$iconfig]}_bpe${bpe}
+python -m mmsrl.train configs/ofa_ours.py configs/ofa_vqa.py --ofa_classification_head --patience=26 --batch_per_epoch=$bpe $config --output_val=\"$WORK/outputs/val_${name}_${SLURM_ARRAY_TASK_ID}.pkl\" --output_test=\"$WORK/outputs/test_${name}_${SLURM_ARRAY_TASK_ID}.pkl\"
diff --git a/stats_table.csv b/stats_table.csv
@@ -0,0 +1,16 @@
+ metric_name all_train all_val all_test
+0 size 5375 623 718
+1 avg_nb_tokens 27.685395348837208 30.32744783306581 29.43732590529248
+2 max_nb_tokens 321 221 256
+3 nb_token_quantile95 68.0 72.0 76.0
+4 class_distrib [0.03, 0.14, 0.05, 0.78] [0.03, 0.15, 0.06, 0.77] [0.02, 0.14, 0.05, 0.79]
+5 avg_nb_ents 3.2584186046511627 3.321027287319422 3.3885793871866294
+6 max_nb_ents 29 15 22
+7 avg_nb_token_ents 2.1221308667351835 2.038182696955051 2.207973695026716
+8 max_nb_token_ents 15 9 12
+9 total_nb_ents 17514 2069 2433
+10 percent_ents_in_text 14.1 15.42 13.4
+11 total_nb_unique_ents 4395 906 1104
+12 nb_duplicated_ent 37 6 2
+13 samples_with_duplicate 32 6 2
+14 entropy (-51.31, 0.07) (-2.72, 0.06) (-5.16, 0.05)
