1. Home
  2. Benchmarks
  3. Question Answering on SQuAD1.1 dev
Dataset Question Answering on SQuAD1.1 dev

Question Answering on SQuAD1.1 dev

#
MODEL
REPOSITORY
EM F1 SPEED
PAPER
ε-REPRODUCES PAPER
1
SpanBERT
facebookresearch / SpanBERT
88.1% 94.0% 28.8
--
2
BERT large
(whole word masking, uncased)
huggingface / transformers
86.9% 93.2%
--
3
BERT large
(whole word masking, cased)
huggingface / transformers
86.3% 92.8%
--
4
DistilBERT
huggingface / transformers
79.1% 86.9%
5
BIDAF
(single)
allenai / allennlp
68.4% 77.8% 205.6
Models on Papers with Code for which code has not been tried out yet.
MODEL
PAPER
EM F1
PWC RANK
T5-11B
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
90.1% 95.6%
1
XLNet
(no data aug)
XLNet: Generalized Autoregressive Pretraining for Language Understanding
88.9% 94.5%
2
T5-3B
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
88.5% 95.0%
3
T5-Large
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
86.7% 93.8%
4
T5-Base
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
85.4% 92.1%
5
BERT large
(+TriviaQA)
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
84.2% 91.1%
6
BERT large
(single)
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
84.1% 90.9%
7
BERT base
(single)
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
80.8% 88.5%
8
T5-Small
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
79.1% 87.2%
9
R.M-Reader
(single)
Reinforced Mnemonic Reader for Machine Reading Comprehension
78.9% 86.3%
10
KAR
Explicit Utilization of General Knowledge in Machine Reading Comprehension
76.7% 84.9%
12
SAN
(single)
Stochastic Answer Networks for Machine Reading Comprehension
76.2% 84.1%
13
FusionNet
FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension
75.3% 83.6%
14
QANet
(data aug x3)
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
75.1% 83.8%
15
DCN+
(single)
DCN+: Mixed Objective and Deep Residual Coattention for Question Answering
74.5% 83.1%
17
QANet
(data aug x2)
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
74.5% 83.2%
16
QANet
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
73.6% 82.7%
18
PhaseCond
(single)
Phase Conductor on Multi-layered Attentions for Machine Comprehension
72.1% 81.4%
19
SRU
Simple Recurrent Units for Highly Parallelizable Recurrence
71.4% 80.2%
20
Smarnet
Smarnet: Teaching Machines to Read and Comprehend Like Human
71.4% 80.2%
21
DCN
(Char + CoVe)
Learned in Translation: Contextualized Word Vectors
71.3% 79.9%
22
R-NET
(single)
Gated Self-Matching Networks for Reading Comprehension and Question Answering
71.1% 79.5%
23
Ruminating Reader
Ruminating Reader: Reasoning with Gated Multi-Hop Attention
70.6% 79.5%
24
FastQAExt
(beam-size 5)
Making Neural QA as Simple as Possible but not Simpler
70.3% 78.5%
25
DrQA
(Document Reader only)
Reading Wikipedia to Answer Open-Domain Questions
69.5% 78.8%
26
jNet
(TreeLSTM adaptation, QTLa, K=…)
Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering
69.1% 78.4%
27
SEDT-LSTM
Structural Embedding of Syntactic Trees for Machine Comprehension
67.9% 77.4%
28
SECT-LSTM
Structural Embedding of Syntactic Trees for Machine Comprehension
67.7% 77.2%
30
RASOR
Learning Recurrent Span Representations for Extractive Question Answering
66.4% 74.9%
31
MPCM
Multi-Perspective Context Matching for Machine Comprehension
66.1% 75.8%
32
DCN
Dynamic Coattention Networks For Question Answering
65.4% 75.6%
33
FABIR
A Fully Attention-Based Information Retriever
65.1% 75.6%
34
Match-LSTM with Bi-Ans-Ptr
(Boundary+Search+b)
Machine Comprehension Using Match-LSTM and Answer Pointer
64.1% 64.7%
35
OTF dict+spelling
(single)
Learning to Compute Word Embeddings On the Fly
63.1%
36
DCR
End-to-End Answer Chunk Extraction and Ranking for Reading Comprehension
62.5% 71.2%
37
FG fine-grained gate
Words or Characters? Fine-grained Gating for Reading Comprehension
60.0% 71.2%
38

This benchmark is evaluating models on the dev set of the SQuAD1.1 dataset.

Step 1: Evaluate models locally

First, use one of the public benchmarks libraries to evaluate your model.

Once you can run the benchmark locally, you are ready to connect it to our automatic service.

Step 2: Login and connect your GitHub Repository

Connect your GitHub repository to automatically start benchmarking your repository. Once connected we'll re-benchmark your master branch on every commit, giving your users confidence in using models in your repository and helping you spot any bugs.