A tool for Discourse Annotation. Inheritor of ToNy and DisCut, segmentors for DISRPT 2019 and 2021. The goal of this version is to be easy to use with or without IT knowledge.
-**Discourse Segmentation:** Take a raw text as input, use a loaded model to make predictions. Output the same text but with EDU segmentation. --> config_1
-**Segmentation Evaluation:** Take an EDU gold segmented text as input, use a loaded model to make predictions. Output scores of model predictions against gold, and output discrepancies. --> config_2
-**Custom Model Creation:** Fine-tuning (over one or two level) a pretrained Language Model with a specific dataset or combination of datasets. Then make predictions and evaluation. --> config_3
## Usecases
-**Discourse Segmentation: "annotation"** Take a raw text as input, use a loaded model to make predictions. Output the same text but with EDU segmentation.
→`config_global_1.1.json` and `config_global_1.2.json`.
-**Segmentation Evaluation: "test"** Take an EDU gold segmented text as input, use a loaded model to make predictions. Output scores of model predictions against gold, and output discrepancies.
→`config_global_2.json`.
-**Custom Model Creation: "train"** Train a new model using a pretrained Language Model (BERT, etc) and a specific dataset or combination of datasets. Then make predictions and evaluation.
→`config_global_3.json`.
-**Custom Model fine-tuning: "fine_tune"** Fine-tune an existing model using a pretrained Language Model (BERT, etc) and a specific dataset or combination of datasets. Then make predictions and evaluation.
→`config_global_4.json`.
## Content description
-`README.md` Description of project.
-`global_config_file_guideline.md` Contains detailed documentation to build well formed config_global file.
-`data/my_cool_dataset/` Contains raw data that need the same name of the directory.
-`code/` Contains main scripts.
-`config_global_XX.json` A file to be completed for your specific project.
-`utils/` Contains useful scripts to be called.
-`discut22_1.py` One python script to run them all.
-`model/` Contains model to be loaded.
-`config_training.jsonnet` A file to be completed for usecases 3 and 4.
-`projects/` This directory will be created automatically.
-`my_cool_exp_v1/` Name of your run. This directory will be created automatically. (see Usage)
-`logs_global.json` Logs of all processes, data and results. This file will be created automatically.
-`data_converted/` Contains pre-processed data if needed. This directory will be created automatically.
-`results/` Contains output files, logs and metrics, if any. This directory will be created automatically.
-`train/` Contains specific output related to train (like the model created), if any.
-`fine_tune/` Contains specific output related to train (like the model created), if any.
## Content description
[TBD : xplain directories automatically created during scripts run]
-`data/my.cool.dataset/` Contains input data, raw and/or pre-processed format(s).
-`results.{stamp}/` Contains output data, scores and post-processed data. (Also logs of allennlp)
-`code/` Contains main scripts.
-`discut22_1.py` One python script to run them all.
-`config_XX.json` A file to be completed for your specific project (or a dir with choise between simple use_case configs and a template for a custom config). See **
-`utils/` Contains useful scripts to be called.
-`model/` Contains model to be loaded or created.
-`config_training.jsonnet` A file to be completed. (TBD automatically saved with model when done)
-`global_config_file_guideline.md` Contains detailed documentation to build well formed config file.
## Set up environnement
-Conda stuff pour python 3.7 (TBD ?)
-DICUT22 run on Python 3.7. Advise : create a specific virtual envireonment (Miniconda...).
- Install all librairies required with the following command:
