README.md (2235B)
1 # Multimodal-semantic-role-labeling 2 3 https://codalab.lisn.upsaclay.fr/competitions/906 4 5 ## How-to 6 7 Export the `DATA_PATH` variable to the location where you want the dataset stored: 8 ``` 9 $ export DATA_PATH="$PWD/../data" 10 ``` 11 12 Run the dataset preparation script (the data will be downloaded if not already present): 13 ``` 14 $ ./prepare_data.sh 15 ``` 16 17 Run the baseline and create a `predictions.pkl` files with the probabilities on the validation set: 18 ``` 19 $ python -m mmsrl.train configs/baseline.py --output_val=predictions.pkl 20 ``` 21 22 Show the content `predictions.pkl`: 23 ``` 24 $ python -m mmsrl.show_predictions predictions.pkl 25 ``` 26 27 Create a new ensemble prediction in the current directory `.` from directories containing several prediction files (they should contain `val` and `test` in their name): 28 ``` 29 $ python -m mmsrl.ensembling . path/to/directory1 path/to/directory2 … 30 ``` 31 32 Create a `.zip` containing a `.jsonl` for submission on Codalab val dataset: 33 ``` 34 $ python -m mmsrl.submission submission.zip predictions.pkl 35 ``` 36 37 To run using ipython: 38 ``` 39 ipython --pdb -c "%run -m mmsrl.train -- configs/ofa_vqa.py --learning_rate=1e-5" 40 ``` 41 42 See `CONFIG.md` for more details on the handling of hyperparameters. 43 44 ## Preparing environment 45 ``` 46 pip install -r requirements 47 ``` 48 49 Install `CLIP`: 50 ``` 51 pip install git+https://github.com/openai/CLIP.git 52 ``` 53 54 Install `fairseq`: 55 ``` 56 git clone https://github.com/pytorch/fairseq.git 57 cd fairseq 58 pip install --use-feature=in-tree-build . 59 ``` 60 61 Use `OFA` (from the repository root) and get checkpoints: 62 ``` 63 git submodule update --init --recursive 64 mkdir OFA/checkpoints 65 cd OFA/checkpoints 66 wget https://ofa-beijing.oss-cn-beijing.aliyuncs.com/checkpoints/ofa_base.pt 67 wget https://ofa-beijing.oss-cn-beijing.aliyuncs.com/checkpoints/vqa_large_best.pt 68 wget https://ofa-beijing.oss-cn-beijing.aliyuncs.com/checkpoints/snli_ve_large_best.pt 69 ``` 70 71 72 ## Extracting features 73 74 ### Image features: 75 python mmsrl/generate_image_features.py --output_folder /data/mmsrl/all/features --image_dir /data/mmsrl/all/images/ --modelname vgg 76 python mmsrl/generate_image_features.py --output_folder/data/mmsrl/all/features --image_dir /data/mmsrl/all/images/ --modelname b7 77 78 ### Generating captions: 79 python -m mmsrl.generate_captions.py