Unsupervised Semantic Role Labeling

The data (extracted argument signatures, system and baseline predictions) and evaluation scripts are downloadable from this page. The goal of this data is to facilitate the comparison between methods used for induction of semantic roles.

Relevant publication:


This package contains the data which can be used to reproduce the experimental set-up and evaluation metrics used of Titov and Klementiev (EACL 2012). The set-up is a very close reimplementation of the one used in Lang and Lapata (ACL 2010, ACL 2011, EMNLP 2011).

The data includes:

All this is available both for gold and predicted syntax (MALT parsers from CoNLL 08 distribution), and for gold and predicted arguments (a reimplementation of argument selection heuristic from Lang and Lapata (ACL 2011)).

The data is represented in CoNLL-08 shared task format but only columns 11+ are present (i.e., only predicate and predicted role fields are fields). In order to use this data, one needs to obtain the CoNLL-08 dataset from LDC and add 1-11 columns to the dataset (e.g., using the paste shell command).

See README for details.



The evaluator computes the purity, collocation and their harmonic mean for the induced semantic roles (on correctly identified arguments). The input files can be either in the CoNLL-08 or CoNLL 09 shared task formats (all columns should be present).

Example usage:

java  -Xms10000m -Xmx10000m -cp ~/usrl-evaluator.jar unisaar.bayes.synsem.learning.srl.SrlEvalExt --model predictions/model-predictions.conll --gold  gold-data/gold-data.conll