1. Home
  2. Benchmarks
  3. Language Modelling on WikiText-103
Dataset Language Modelling on WikiText-103

Language Modelling on WikiText-103

#
MODEL
REPOSITORY
TEST PERPLEXITY SPEED
PAPER
ε-REPRODUCES PAPER
1
Transformer-XL
(RMS dynamic eval)
benkrause / dynamiceval-transformer
16.44 483.6
2
Transformer-XL Large
huggingface / transformers
18.19 2970.8
3
Transformer
(Adaptive inputs)
pytorch / fairseq
18.70 1685.4
4
GPT-2 Large
huggingface / transformers
22.61 4482.9
5
GPT-2 Medium
huggingface / transformers
26.51 9316.7
6
GPT-2 Small
huggingface / transformers
36.53 25200.1
7
DistilGPT2
huggingface / transformers
49.91 38425.0
--
8
GPT1
huggingface / transformers
90.61 29444.9
--
Models on Papers with Code for which code has not been tried out yet.
MODEL
PAPER
TEST PERPLEXITY VALIDATION PERPLEXITY
PWC RANK
Megatron-LM
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
10.80
4
Transformer-XL
(SGD dynamic eval)
Dynamic Evaluation of Transformer Language Models
17.00 16.30
6
Compressive Transformer
(18L, M=1024)
Compressive Transformers for Long-Range Sequence Modelling
17.10 16.00
7
GPT-2 Full
Language Models are Unsupervised Multitask Learners
17.48
8
All-attention network
(36 layers)
Augmenting Self-attention with Persistent Memory
20.60 19.70
11
DEQ-Transformer
(medium, adaptive embed)
Deep Equilibrium Models
23.70
13
Transformer-XL Standard
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
24.00 23.10
14
DEQ-TrellisNet
Deep Equilibrium Models
29.00
16
Trellis Network
Trellis Networks for Sequence Modeling
29.19
17
LSTM
(Hebbian, Cache, MbPA)
Fast Parametric Learning with Activation Memorization
29.20 29.00
18
LSTM
(Hebbian, Cache)
Fast Parametric Learning with Activation Memorization
29.70 29.90
19
LSTM
(RMC)
Relational recurrent neural networks
31.60 30.80
20
DEQ-Transformer
(small)
Deep Equilibrium Models
32.40
21
4 layer QRNN
An Analysis of Neural Language Modeling at Multiple Scales
33.00 32.00
22
LSTM
(Hebbian)
Fast Parametric Learning with Activation Memorization
34.30 34.10
23
LSTM
Fast Parametric Learning with Activation Memorization
36.40 36.00
24
GCNN-14
Language Modeling with Gated Convolutional Networks
37.20
25
Neural cache model
(size = 2,000)
Improving Neural Language Models with a Continuous Cache
40.80
27
Neural cache model
(size = 100)
Improving Neural Language Models with a Continuous Cache
44.80
28
GCNN-8
Language Modeling with Gated Convolutional Networks
44.90
29
TCN
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
45.19
30
Temporal CNN
Convolutional Sequence Modeling Revisited
45.20
31
LSTM
Improving Neural Language Models with a Continuous Cache
48.70
32

This benchmark is evaluating language models on the WikiText-103 dataset.

Step 1: Evaluate models locally

First, use our public benchmark library to evaluate your model. sotabench-eval is a framework-agnostic library that implements the WikiText-103 Benchmark. See sotabench-eval docs here.

Once you can run the benchmark locally, you are ready to connect it to our automatic service.

Step 2: Login and connect your GitHub Repository

Connect your GitHub repository to automatically start benchmarking your repository. Once connected we'll re-benchmark your master branch on every commit, giving your users confidence in using models in your repository and helping you spot any bugs.