Getting Started

Sotabench is a free service. Anyone can add models from their repositories and create public benchmarks.

For background information see the About page.

Adding Models from Your Repository

Sotabench provides a continous integration (CI) - like service, but instead of running unit tests, it benchmarks models by running the sotabench.py from your GitHub repository. This happens automatically on every commit of the master branch.

How to add your model.

Creating a New Public Benchmark

You can either create a new public benchmark or add a new implementation for an existing benchmark (e.g. to make it easier to benchmark models in a particular deep learning framework).

You can submit your benchmark either a pull request to sotabench-eval or torchbench. Or, you can create an entirely new Python package.

Once you are ready open a new topic on forum and we'll add your benchmark.