SpanBERT: Improving Pre-training by Representing and Predicting Spans

Mandar JoshiDanqi ChenYinhan LiuDaniel S. WeldLuke ZettlemoyerOmer Levy

   Papers with code   Abstract  PDF

We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it... (read more)

Benchmarked Models

RANK
MODEL
REPO
CODE RESULT
PAPER RESULT
ε-REPRODUCED
BUILD
RANK
MODEL
REPO
CODE RESULT
PAPER RESULT
ε-REPRODUCED
BUILD