publications

2023

1. ACL

   Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations

   Lucy Lu Wang, Yulia Otmakhova, Jay DeYoung, and 4 more authors

   In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jul 2023

   Abs

   Evaluating multi-document summarization (MDS) quality is difficult. This is especially true in the case of MDS for biomedical literature reviews, where models must synthesize contradicting evidence reported across different documents. Prior work has shown that rather than performing the task, models may exploit shortcuts that are difficult to detect using standard n-gram similarity metrics such as ROUGE. Better automated evaluation metrics are needed, but few resources exist to assess metrics when they are proposed. Therefore, we introduce a dataset of human-assessed summary quality facets and pairwise preferences to encourage and support the development of better automated evaluation methods for literature review MDS. We take advantage of community submissions to the Multi-document Summarization for Literature Review (MSLR) shared task to compile a diverse and representative sample of generated summaries. We analyze how automated summarization evaluation metrics correlate with lexical features of generated summaries, to other automated metrics including several we propose in this work, and to aspects of human-assessed summary quality. We find that not only do automated metrics fail to capture aspects of quality as assessed by humans, in many cases the system rankings produced by these metrics are anti-correlated with rankings according to human annotators.

2. Automating Quality Assessment of Medical Evidence in Systematic Reviews: Model Development and Validation Study

   Simon Šuster, Timothy Baldwin, Jey Han Lau, and 4 more authors

   J Med Internet Res, Mar 2023

2022

1. EMNLP

   M3: Multi-level dataset for Multi-document summarisation of Medical studies

   Yulia Otmakhova, Karin Verspoor, Timothy Baldwin, and 2 more authors

   In Findings of the Association for Computational Linguistics: EMNLP 2022, Dec 2022

   Abs

   We present M3 (Multi-level dataset for Multi-document summarisation of Medical studies), a benchmark dataset for evaluating the quality of summarisation systems in the biomedical domain. The dataset contains sets of multiple input documents and target summaries of three levels of complexity: documents, sentences, and propositions. The dataset also includes several levels of annotation, including biomedical entities, direction, and strength of relations between them, and the discourse relationships between the input documents (“contradiction” or “agreement”). We showcase usage scenarios of the dataset by testing 10 generic and domain-specific summarisation models in a zero-shot setting, and introduce a probing task based on counterfactuals to test if models are aware of the direction and strength of the conclusions generated from input studies.

2. AACL

   Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation

   Thinh Hung Truong, Yulia Otmakhova, Timothy Baldwin, and 3 more authors

   In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Nov 2022

   Abs

   Negation is poorly captured by current language models, although the extent of this problem is not widely understood. We introduce a natural language inference (NLI) test suite to enable probing the capabilities of NLP methods, with the aim of understanding sub-clausal negation. The test suite contains premise–hypothesis pairs where the premise contains sub-clausal negation and the hypothesis is constructed by making minimal modifications to the premise in order to reflect different possible interpretations. Aside from adopting standard NLI labels, our test suite is systematically constructed under a rigorous linguistic framework. It includes annotation of negation types and constructions grounded in linguistic theory, as well as the operations used to construct hypotheses. This facilitates fine-grained analysis of model performance. We conduct experiments using pre-trained language models to demonstrate that our test suite is more challenging than existing benchmarks focused on negation, and show how our annotation supports a deeper understanding of the current NLI capabilities in terms of negation and quantification.

3. COLING

   LED down the rabbit hole: exploring the potential of global attention for biomedical multi-document summarisation

   Yulia Otmakhova, Thinh Hung Truong, Timothy Baldwin, and 3 more authors

   In Proceedings of the Third Workshop on Scholarly Document Processing, Oct 2022

   Abs

   In this paper we report the experiments performed for the submission to the Multidocument summarisation for Literature Review (MSLR) Shared Task. In particular, we adopt Primera model to the biomedical domain by placing global attention on important biomedical entities in several ways. We analyse the outputs of 23 resulting models and report some patterns related to the presence of additional global attention, number of training steps and the inputs configuration.

4. Multi-label classification for biomedical literature: an overview of the BioCreative VII LitCovid Track for COVID-19 literature topic annotations

   Qingyu Chen, Alexis Allot, Robert Leaman, and 36 more authors

   Database, Aug 2022

   Abs

   The coronavirus disease 2019 (COVID-19) pandemic has been severely impacting global society since December 2019. The related findings such as vaccine and drug development have been reported in biomedical literature—at a rate of about 10 000 articles on COVID-19 per month. Such rapid growth significantly challenges manual curation and interpretation. For instance, LitCovid is a literature database of COVID-19-related articles in PubMed, which has accumulated more than 200 000 articles with millions of accesses each month by users worldwide. One primary curation task is to assign up to eight topics (e.g. Diagnosis and Treatment) to the articles in LitCovid. The annotated topics have been widely used for navigating the COVID literature, rapidly locating articles of interest and other downstream studies. However, annotating the topics has been the bottleneck of manual curation. Despite the continuing advances in biomedical text-mining methods, few have been dedicated to topic annotations in COVID-19 literature. To close the gap, we organized the BioCreative LitCovid track to call for a community effort to tackle automated topic annotation for COVID-19 literature. The BioCreative LitCovid dataset—consisting of over 30 000 articles with manually reviewed topics—was created for training and testing. It is one of the largest multi-label classification datasets in biomedical scientific literature. Nineteen teams worldwide participated and made 80 submissions in total. Most teams used hybrid systems based on transformers. The highest performing submissions achieved 0.8875, 0.9181 and 0.9394 for macro-F1-score, micro-F1-score and instance-based F1-score, respectively. Notably, these scores are substantially higher (e.g. 12\%, higher for macro F1-score) than the corresponding scores of the state-of-art multi-label classification method. The level of participation and results demonstrate a successful track and help close the gap between dataset curation and method development. The dataset is publicly available via https://ftp.ncbi.nlm.nih.gov/pub/lu/LitCovid/biocreative/ for benchmarking and further development.Database URLhttps://ftp.ncbi.nlm.nih.gov/pub/lu/LitCovid/biocreative/

5. Cross-linguistic Comparison of Linguistic Feature Encoding in BERT Models for Typologically Different Languages

   Yulia Otmakhova, Karin Verspoor, and Jey Han Lau

   In Proceedings of the 4th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Jul 2022

   Abs

   Though recently there have been an increased interest in how pre-trained language models encode different linguistic features, there is still a lack of systematic comparison between languages with different morphology and syntax. In this paper, using BERT as an example of a pre-trained model, we compare how three typologically different languages (English, Korean, and Russian) encode morphology and syntax features across different layers. In particular, we contrast languages which differ in a particular aspect, such as flexibility of word order, head directionality, morphological type, presence of grammatical gender, and morphological richness, across four different tasks.

6. ACL

   The patient is more dead than alive: exploring the current state of the multi-document summarisation of the biomedical literature

   Yulia Otmakhova, Karin Verspoor, Timothy Baldwin, and 1 more author

   In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), May 2022

   Abs

   Although multi-document summarisation (MDS) of the biomedical literature is a highly valuable task that has recently attracted substantial interest, evaluation of the quality of biomedical summaries lacks consistency and transparency. In this paper, we examine the summaries generated by two current models in order to understand the deficiencies of existing evaluation approaches in the context of the challenges that arise in the MDS task. Based on this analysis, we propose a new approach to human evaluation and identify several challenges that must be overcome to develop effective biomedical MDS systems.

7. ITTC @ TREC 2021 Clinical Trials Track

   Thinh Hung Truong, Yulia Otmakhova, Rahmad Mahendra, and 6 more authors

   May 2022

2021

1. Impact of detecting clinical trial elements in exploration of COVID-19 literature

   Simon Šuster, Karin Verspoor, Timothy Baldwin, and 4 more authors

   May 2021
2. Team ITTC at BioCreative VII LitCovid Track 5: combining pretrained and bag-of-words models

   Yulia Otmakhova, and Antonio Jimeno Yepes

   In Proceedings of the seventh BioCreative challenge evaluation workshop, May 2021
3. Brief Description of COVID-SEE: The Scientific Evidence Explorer for COVID-19 Related Research

   Karin Verspoor, Simon Šuster, Yulia Otmakhova, and 7 more authors

   In Advances in Information Retrieval, May 2021

   Abs

   We present COVID-SEE, a system for medical literature discovery based on the concept of information exploration, which builds on several distinct text analysis and natural language processing methods to structure and organise information in publications, and augments search through a visual overview of a collection enabling exploration to identify key articles of interest. We developed this system over COVID-19 literature to help medical professionals and researchers explore the literature evidence, and improve findability of relevant information. COVID-SEE is available at http://covid-see.com.

2020

1. Improved Topic Representations of Medical Documents to Assist COVID-19 Literature Exploration

   Yulia Otmakhova, Karin Verspoor, Timothy Baldwin, and 1 more author

   In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Dec 2020

   Abs

   Efficient discovery and exploration of biomedical literature has grown in importance in the context of the COVID-19 pandemic, and topic-based methods such as latent Dirichlet allocation (LDA) are a useful tool for this purpose. In this study we compare traditional topic models based on word tokens with topic models based on medical concepts, and propose several ways to improve topic coherence and specificity.

2015

1. NAACL

   Do We Really Need Lexical Information? Towards a Top-down Approach to Sentiment Analysis of Product Reviews

   Yulia Otmakhova, and Hyopil Shin

   In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, May 2015

2013

1. Applying Graph-based Keyword Extraction to Document Retrieval

   Youngsam Kim, Munhyong Kim, Andrew Cattle, and 3 more authors

   In Proceedings of the Sixth International Joint Conference on Natural Language Processing, Oct 2013