PEER2024 Program

The 3rd Workshop on Processing and Evaluation Event Representations (PEER2024) was held at Cornell University on March 29, 2024. Both talks and breaks were on the top floor of the Johnson Art Museum.

  • 08:30–09:30
    Breakfast

  • 09:30–9:40
    Opening Remarks

  • 09:40–10:40
    Keynote: Expectation-based Semantics in Language Comprehension

    Noortje Venhuizen (Tilburg University)

    The processing difficulty of each word we encounter in a sentence is affected by both our prior linguistic experience and our general knowledge about the world. Computational models of language processing have, however, been limited in accounting for the influence of world knowledge. I present an incremental model of language comprehension that integrates linguistic experience and world knowledge at the level of utterance interpretation. On a word-by-word basis, our model constructs rich, distributed representations that capture utterance meaning in terms of propositional co-occurrence across observations in the world. These representations are inherently compositional and probabilistic, capturing entailment as well as probabilistic inference. To quantify linguistic processing effort in the model, we adopt Surprisal Theory; in contrast with typical language model implementations, our model instantiates surprisal as a comprehension-centric metric that reflects the likelihood of the unfolding utterance meaning as established after processing each word. I will present simulations that illustrate how linguistic experience and world knowledge combine in determining online expectations in the model and discuss the implications for neurocognitive theories and models of language comprehension. Finally, I will discuss directions for future research within the framework of Distributional Formal Semantics.

  • 10:40–11:00
    Coffee Break

  • 11:00–11:20
    Generating event descriptions with syntactic and semantic constraints

    Angela Cao (University of Rochester)

    This presentation examines manual, corpus-based, and LLM-based methods for generating large-scale samples of event descriptions while constraining for syntactic and semantic requirements. We assess the effectiveness of each approach by collecting annotator ratings of naturalness, typicality, and contrasting uses of different verb-senses. We find that while all methods reliably generate natural and typical items given a verb-sense, constraining items on pre-existing sense inventories such as Propbank’s generates a more varied representation of sense distributions, compared to when we use LLM-generated sense inventories. This work provides insights into each procedure’s suitability for linguists’ various requirements when generating linguistic stimuli, and enables the possibility of studying linguistic phenomena at the scale of the lexicon.

  • 11:20–11:40
    Situating phonological phenomena within events

    John Starr (Cornell University)

    It is well-established that syntactic and semantic information generally governs the cognitive representation of events: _Tessa sees Kru_ is distinct from both _Kru sees Tessa_ and _Tessa observes Kru_. However, in the psycholinguistic wild, we receive other kinds of linguistic input beyond the syntactic and semantic, input which can interact with other structures. In this project, we investigate how judgments of two phonologically-associated phenomena -- phonotactic acceptability judgments of nonce words, binomial ordering preferences -- arise when they are situated in different kinds of events. Our findings demonstrate that layers of event embedding can both manipulate if and when judgments of these phenomena occur, suggesting that judgments of phonologically-oriented structures may variably compute when situated in different kinds of events.

  • 11:40–12:00
    Reasoning about propositional attitude reports in Bayesian dynamic semantics

    Julian Grove (University of Rochester)

    Recent work by Grove and White (2023) has shown how Bayesian models of experimentally collected inference-judgment data can be seamlessly derived from the compositional semantics of the expressions triggering inferences, allowing semantic theories and theories of experimental behavior to be rigorously formulated and quantitatively compared. In this talk, I show how Grove and White's approach can be stated in general terms as a Bayesian theory of dynamic semantics that crucially makes use of a state. In the end, the state of a discourse corresponds to the model parameters in a Bayesian model. As a case study, I show how the framework may be used to encode a sophisticated semantics of propositional attitude reports and the inferences derived from them, following recent work by Djärv (2019). I show how, in this case, a model formulated in a Bayesian dynamic semantics may be used to learn the distinct semantic profiles of doxastic and emotive factive predicates from experimental data.

  • 12:00–13:00
    Lunch

  • 13:00–13:20
    Are Triggers All You Need for Document-Level Event Extraction?

    Shaden Shaar (Cornell University)

    Majority of the work done for event extraction, has been focused on sentence-level; as sentence-level dataset creation is easier than that of document-level. In sentence-level event extraction and document-level argument extraction, many works have relied on the use of _Triggers_; with some claiming the necessity of having such annotation. In this paper, we investigate if it is necessary to have _Triggers_ when working on document-level event extraction. This question is important as the few available datasets don't have _Triggers_ and obtaining such annotation is expensive. We analyze the usefulness of learning _Triggers_ obtained from different annotation sources (human, LLM, keyword-based, and random), by comparing the performance of 5 different models on 2 datasets. We also release the human, LLM, and keyword annotations we collected and annotated for the datasets.

  • 13:20–13:40
    Causality in Hate Speech

    Kaelyn Lamp (Cornell University)

    Limited work quantitatively analyzes how linguistic phenomena are used to manipulate the perception of causality and blame when expressing hate speech. Using annotated hate speech datasets, we conducted a corpus analysis of select causal constructions and strategies of indirect speech to see what syntactic choices were made by writers of hate speech when referencing minority (targeted) groups and majority groups. We found that hate speech tends to use causal constructions to minimize the causal responsibility of majority groups and indirect speech strategies such as word order and passivization to focus on the minority group’s involvement in events.

  • 13:40–14:00
    Event-Keyed Summarization

    William Gantt (University of Rochester)

    Traditional approaches to summarization in natural language processing focus on summarizing whole documents. As readers, however, we often seek information about a particular situation. This talk will discuss some recent work on a novel task we call event-keyed summarization, in which the aim is to generate summaries targeted to a specific situation described in a document, given the document and an event representation extracted from it. We describe a dataset for this task, a suite of computational experiments on that dataset, as well as the results of a user study, showing that event-keyed summarization represents an interesting new synthesis of both traditional summarization and event extraction.

  • 14:00–14:20
    Coffee Break

  • 14:20–14:40
    FAMuS: Frames Across Multiple Sources

    Alexander Martin (University of Rochester)

    Understanding event descriptions is a central aspect of language processing, but current approaches focus overwhelmingly on single sentences or documents. Aggregating information about an event across documents can offer a much richer understanding. To this end, we present FAMuS, a new corpus of Wikipedia passages that report on some event, paired with underlying, genre-diverse (non-Wikipedia) source articles for the same event. Events and (cross-sentence) arguments in both report and source are annotated against FrameNet, providing broad coverage of different event types. We present results on two key event understanding tasks enabled by FAMuS: source validation -- determining whether a document is a valid source for a target report event -- and cross-document argument extraction -- full-document argument extraction for a target event from both its report and the correct source article. We release both FAMuS and our models to support further research.

  • 14:40–15:00
    Probing Representations for Document-level Event Extraction

    Barry Wang (Cornell University)

    The probing classifiers framework has been employed for interpreting deep neural network models for a variety of natural language processing (NLP) applications. Studies, however, have largely focused on sentencelevel NLP tasks. This work is the first to apply the probing paradigm to representations learned for document-level information extraction (IE). We designed eight embedding probes to analyze surface, semantic, and event-understanding capabilities relevant to document-level event extraction. We apply them to the representations acquired by learning models from three different LLM-based document-level IE approaches on a standard dataset. We found that trained encoders from these models yield embeddings that can modestly improve argument detections and labeling but only slightly enhance event-level tasks, albeit trade-offs in information helpful for coherence and event-type prediction. We further found that encoder models struggle with document length and cross-sentence discourse.

  • 15:00–15:20
    Enhancing Language Model Representations with Attributed Network Embeddings

    Jacob Matthews (Cornell University)

    Networks like citation graphs are rich sources of metatextual information, which may not be easily learnable from unstructured text data alone. Citation and publication events in particular can give valuable metatextual insight into a document's social and temporal contexts. We will demonstrate a fast, simple, and interpretable method for generating embeddings from attributed networks. We then explore use cases for these attributed network embeddings on downstream tasks involving pretrained language models, such as authorship attribution and diachronic word embedding generation.

  • 15:20–15:40
    Coffee Break

  • 15:40–16:40
    Keynote: Events and objects in mind and language

    Anna Papafragou (University of Pennsylvania)

    Understanding how humans represent, recognize, remember and talk about events is important for several disciplines that focus on the human mind and brain. In this talk I bring together an interdisciplinary set of tools to address the nature of event representation. I present experimental evidence showing that abstract properties of event structure underlie both the conceptual and the linguistic encoding of dynamic events; furthermore, in several crucial respects, the representation of events resembles that of objects. I argue that notions of boundaries and structure underlie mental units in both the event and the object domain, and that such notions explain otherwise mysterious similarities in cross-domain correspondences. These findings have implications for both cognitive and linguistic event theories, as well as the relation between language and thought.

  • 16:40–17:00
    Closing Remarks