Benchmarking

System for

Vitess

Historical results on the main branch

See more historical results >

How it works

Workloads

Seven workloads are executed against every commit we decide to benchmark. These workloads define the SQL schema, the queries we execute, the configuration of the Vitess cluster, and the configuration of sysbench. The settings of our workload can be found on the arewefastyet repository.

We have two categories of workloads: OLTP and TPCC, both are a modified version of the official workloads.

Frequency

There are three cron schedules that enable us to periodically benchmark Vitess, the definition of these schedules is available in our yaml configuration.

Generally, we benchmark the main and release branches of Vitess every night at midnight UTC. We also detect new PRs that need to be benchmarked every five minutes, and new tags/releases every minute.

Methodology

Each commit is benchmarked ten times by each workload. Under the hood, we use sysbench with a custom configuration to perform the benchmark. We then perform statistical analysis on the ten results of a given workload to get a more accurate and reliable result.

Every execution is done on the same hardware: 192Gb of RAM and 2x Intel Xeon Silver 4214 Processor 24-Core @ 2.20GHz.

Results

We collect the same results as sysbench (QPS, TPS, Error rate, latency, etc), along with several Golang metrics such as the CPU used per query, and the total memory used per query.

Architecture

execution pipelineexecution pipeline