|REPOSITORY||MODEL||BENCHMARK||CODE RESULT||PAPER RESULT||ε-REPRODUCED|
How it Works
INTEGRATED WITH GITHUBConnect your repo and benchmark models for free
MODELS RUN ON BENCHMARKSFree GPUs to run your code on public benchmarks
COMPARED TO PAPERSResults compared to papers to test reproducibility
How to Add Your Repository
Step One : Evaluate Locally
Each public benchmark has its own instructions on how to use.
For example, to use the Image Classification on ImageNet benchmark
on your model in a framework-independent way, create a
sotabench.py file like this:
from sotabencheval.image_classification import ImageNetEvaluator evaluator = ImageNetEvaluator( # automatically compare to this paper model_name='ResNeXt-101-32x8d', paper_arxiv_id='1611.05431' ) predictions = ... # use your model to make predictions evaluator.add(predictions) evaluator.save()
Step Two : Connect Git and Showcase Results
Sotabench is like Continuous Integration, but instead of running
unit tests, it benchmarks models in
sotabench.py on every commit.
For developers, this is an easier way to continuously test their ML models, allowing for direct comparison with other repositories and papers.
For the community, this is a free and always up-to-date reference of ML model implementations.