README.md (1621B)
1 # minhash-retrieval 2 Finds your documents in the HPLT datasets. 3 4 **Inputs**: your documents should be stored in the jsonl format with fields ```article```, ```headline```. 5 Each document will be added to the index twice: with and without the headline. Then each HPLT document will be searched in the index. 6 7 **Outputs**: jsonl with the following informations for each near-match: 8 * qid: file path and line number (starting from 1) identifying an HPLT document 9 * tid: line number (starting from 1) identifying your document in the input json, or negated line number 10 * sim: Jaccard simliarity 11 * text: text of the retrieved HPLT document 12 13 # Dependencies 14 See [requirements_nird.lock](./requirements_nird.lock). 15 16 # Preprocessing 17 Assume your documents are stored in fixed.jsonl. Preprocessing removes all non-letters to make retrieval independent from whitespaces, newlines and other special characters. 18 ```python retriever.py preprocess_content fixed.jsonl``` 19 fixed.jsonl.preproc.pkl is dumped to disk, your docuemnts will be read from this file during retrieval 20 21 # Retrieval 22 For search in the deduplicate/cleaned versions of data release 2: 23 24 ```./run_retrieval1.sh fixed.jsonl.preproc.pkl deduplicated/ ./fixed-in-deduplicated-run2 text '*.zst'``` 25 26 This creates a MinHash index and puts docuemnts form fixed.jsonl.preproc.pkl there. Then retreives near-duplicates for HPLT docuemnts stored in deduplicated/. The outputs are dumped to the folder ./fixed-in-deduplicated-run2 27 28 For search in the outputs of stage2 of data release 2: 29 30 ```./run_retrieval1.sh fixed.jsonl.preproc.pkl _stage2out/ ./fixed-in-stage2out t text.zst``` 31