automata benchmarks

History · Versus Spot and Oink

Versus Spot and Oink

Wall-clock medians of the same operation on the same input in automata (through its Python bindings, like Spot) and in the reference, from main · c432f068. The ratio is ours / reference: above 1 we are slower. They depend on the machine and are never compared automatically.

Settings: {"automata":"0.1.0","budget_s":1.0,"oink":{"binary":"/nix/store/lpniqhrsws0fsk208p8djv3rdim4bqyl-oink-parity-unstable-2026-06-03/bin/oink","solvers":["zlk","ppp"],"workers":1},"python":"3.13.17","spot":"2.16.2"}

Versus Spot and Oink

dpa/equivalence

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot122.58 µs117.70 µs1.04×– / –yes
hard/l-nba-2spot56.20 µs37.44 µs1.50×– / –yes
hard/m-nba-3spot238.82 µs140.78 µs1.70×– / –yes
ltl/and-gf-3spot9.16 µs12.98 µs0.71×– / –yes
ltl/eh-patterns-0spot21.78 µs20.94 µs1.04×– / –yes
ltl/gf-equiv-3spot20.04 µs34.12 µs0.59×– / –yes
ltl/gf-implies-3spot10.68 µs19.90 µs0.54×– / –yes
ltl/gh-q-3spot50.76 µs30.14 µs1.68×– / –yes
ltl/gh-r-3spot20.76 µs68.50 µs0.30×– / –yes
ltl/go-theta-3spot11.20 µs22.00 µs0.51×– / –yes
ltl/lily-patterns-1spot12.34 µs16.58 µs0.74×– / –yes
ltl/ms-phi-r-1spot10.22 µs38.62 µs0.26×– / –yes
ltl/ms-phi-s-1spot10.48 µs41.02 µs0.26×– / –yes
ltl/rv-counter-3spot28.78 µs14.28 µs2.02×– / –yes
ltl/sb-patterns-0spot12.46 µs14.68 µs0.85×– / –yes
ltl/sejk-k-2spot13.48 µs58.24 µs0.23×– / –yes
ltl/tv-uu-3spot41.32 µs39.36 µs1.05×– / –yes
ltl/u-left-3spot10.56 µs9.36 µs1.13×– / –yes
many_aps/and-gf-12spot168.96 µs76.56 µs2.21×– / –yes
many_aps/gf-equiv-12spot131.12 µs91.96 µs1.43×– / –yes
many_aps/r-right-10spot66.88 µs26.88 µs2.49×– / –yes
many_aps/random-n16-a8-p4-s1spot269.88 µs136.08 µs1.98×– / –yes
many_aps/random-n24-a12-p4-s1spot1.18 ms448.70 µs2.62×– / –yes
many_aps/random-n32-a16-p4-s1spot3.86 ms1.23 ms3.13×– / –yes
random/n128-a2-p6-s1spot338.82 µs215.86 µs1.57×– / –yes
random/n16-a1-p3-s1spot22.88 µs16.76 µs1.37×– / –yes
random/n256-a2-p8-s1spot891.45 µs489.03 µs1.82×– / –yes
random/n64-a2-p4-s1spot141.46 µs101.86 µs1.39×– / –yes

dpa/inclusion

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot71.66 µs62.14 µs1.15×– / –yes
hard/l-nba-2spot35.82 µs24.40 µs1.47×– / –yes
hard/m-nba-3spot148.08 µs91.18 µs1.62×– / –yes
ltl/and-gf-3spot5.90 µs8.66 µs0.68×– / –yes
ltl/eh-patterns-0spot14.28 µs17.90 µs0.80×– / –yes
ltl/gf-equiv-3spot11.42 µs19.08 µs0.60×– / –yes
ltl/gf-implies-3spot6.32 µs11.58 µs0.55×– / –yes
ltl/gh-q-3spot33.12 µs29.06 µs1.14×– / –yes
ltl/gh-r-3spot12.32 µs34.86 µs0.35×– / –yes
ltl/go-theta-3spot6.86 µs16.04 µs0.43×– / –yes
ltl/lily-patterns-1spot8.04 µs13.98 µs0.58×– / –yes
ltl/ms-phi-r-1spot6.04 µs19.24 µs0.31×– / –yes
ltl/ms-phi-s-1spot6.12 µs25.32 µs0.24×– / –yes
ltl/rv-counter-3spot18.08 µs14.52 µs1.25×– / –yes
ltl/sb-patterns-0spot7.64 µs12.12 µs0.63×– / –yes
ltl/sejk-k-2spot7.84 µs28.00 µs0.28×– / –yes
ltl/tv-uu-3spot28.46 µs30.08 µs0.95×– / –yes
ltl/u-left-3spot6.72 µs8.68 µs0.77×– / –yes
many_aps/and-gf-12spot125.80 µs54.50 µs2.31×– / –yes
many_aps/gf-equiv-12spot74.80 µs53.08 µs1.41×– / –yes
many_aps/r-right-10spot49.00 µs25.78 µs1.90×– / –yes
many_aps/random-n16-a8-p4-s1spot223.70 µs100.84 µs2.22×– / –yes
many_aps/random-n24-a12-p4-s1spot1.12 ms396.64 µs2.83×– / –yes
many_aps/random-n32-a16-p4-s1spot3.80 ms1.12 ms3.39×– / –yes
random/n128-a2-p6-s1spot198.46 µs128.34 µs1.55×– / –yes
random/n16-a1-p3-s1spot13.46 µs10.80 µs1.25×– / –yes
random/n256-a2-p8-s1spot512.25 µs317.64 µs1.61×– / –yes
random/n64-a2-p4-s1spot87.22 µs65.72 µs1.33×– / –yes

game/solve

inputreferenceoursreferenceours / referencesize (ours / reference)same result
ladder_10000oink-ppp2.75 ms328.00 µs8.39×10000 / 10000yes
ladder_10000oink-zlk2.75 ms335.00 µs8.21×10000 / 10000yes
ladder_10000spot2.75 ms10.17 ms0.27×10000 / 10000yes
ladder_50000oink-ppp18.93 ms2.12 ms8.92×50000 / 50000yes
ladder_50000oink-zlk18.93 ms2.47 ms7.66×50000 / 50000yes
ladder_50000spot18.93 ms77.59 ms0.24×50000 / 50000yes
random_few_10000oink-ppp6.98 ms1.46 ms4.80×10000 / 10000yes
random_few_10000oink-zlk6.98 ms2.76 ms2.53×10000 / 10000yes
random_few_10000spot6.98 ms21.11 ms0.33×10000 / 10000yes
random_few_50000oink-ppp77.03 ms16.00 ms4.81×50000 / 50000yes
random_few_50000oink-zlk77.03 ms20.71 ms3.72×50000 / 50000yes
random_few_50000spot77.03 ms679.94 ms0.11×50000 / 50000yes
random_many_10000oink-ppp6.31 ms1.64 ms3.84×10000 / 10000yes
random_many_10000oink-zlk6.31 ms3.04 ms2.07×10000 / 10000yes
random_many_10000spot6.31 ms29.88 ms0.21×10000 / 10000yes
random_many_50000oink-ppp82.90 ms35.20 ms2.35×50000 / 50000yes
random_many_50000oink-zlk82.90 ms51.28 ms1.62×50000 / 50000yes
random_many_50000spot82.90 ms779.68 ms0.11×50000 / 50000yes

History of the ratio

Geometric mean of ours / reference over the inputs of an operation, one point per commit of main; the values in the tooltips are per mille (1.5 is 1500).

dpa/equivalence vs spot
1.03×fc170083 · 2026-11-09 · 1.96 k067f8c28 · 2026-11-09 · 949 (-51.68 %)1bf383b2 · 2026-11-10 · 1.37 k (+44.36 %)8ce15b69 · 2026-11-10 · 1.26 k (-8.32 %)f83ff30f · 2026-11-10 · 1.31 k (+4.38 %)38e9f0d4 · 2026-11-11 · 1.23 k (-6.18 %)f002431b · 2026-11-11 · 1.09 k (-11.14 %)2b5473e7 · 2026-11-11 · 1.06 k (-3.20 %)91ab2741 · 2026-11-11 · 1.07 k (+0.85 %)c432f068 · 2026-11-11 · 1.03 k (-3.09 %)
dpa/inclusion vs spot
0.96×fc170083 · 2026-11-09 · 1.50 k067f8c28 · 2026-11-09 · 839 (-44.25 %)1bf383b2 · 2026-11-10 · 1.26 k (+50.66 %)8ce15b69 · 2026-11-10 · 1.16 k (-8.39 %)f83ff30f · 2026-11-10 · 1.20 k (+3.63 %)38e9f0d4 · 2026-11-11 · 1.12 k (-6.50 %)f002431b · 2026-11-11 · 1.00 k (-10.52 %)2b5473e7 · 2026-11-11 · 964 (-3.98 %)91ab2741 · 2026-11-11 · 978 (+1.45 %)c432f068 · 2026-11-11 · 959 (-1.94 %)
game/solve vs oink-ppp
5.00×fc170083 · 2026-11-09 · 5.47 k067f8c28 · 2026-11-09 · 5.38 k (-1.65 %)1bf383b2 · 2026-11-10 · 5.44 k (+1.17 %)8ce15b69 · 2026-11-10 · 5.61 k (+3.03 %)f83ff30f · 2026-11-10 · 5.63 k (+0.43 %)38e9f0d4 · 2026-11-11 · 5.42 k (-3.64 %)f002431b · 2026-11-11 · 5.10 k (-5.97 %)2b5473e7 · 2026-11-11 · 5.18 k (+1.59 %)91ab2741 · 2026-11-11 · 5.15 k (-0.62 %)c432f068 · 2026-11-11 · 5.00 k (-2.93 %)
game/solve vs oink-zlk
3.54×fc170083 · 2026-11-09 · 4.05 k067f8c28 · 2026-11-09 · 4.06 k (+0.27 %)1bf383b2 · 2026-11-10 · 4.06 k (+0.05 %)8ce15b69 · 2026-11-10 · 4.04 k (-0.69 %)f83ff30f · 2026-11-10 · 4.27 k (+5.82 %)38e9f0d4 · 2026-11-11 · 4.21 k (-1.52 %)f002431b · 2026-11-11 · 3.84 k (-8.65 %)2b5473e7 · 2026-11-11 · 3.91 k (+1.69 %)91ab2741 · 2026-11-11 · 3.87 k (-1.07 %)c432f068 · 2026-11-11 · 3.54 k (-8.31 %)
game/solve vs spot
0.19×fc170083 · 2026-11-09 · 204067f8c28 · 2026-11-09 · 208 (+1.96 %)1bf383b2 · 2026-11-10 · 215 (+3.37 %)8ce15b69 · 2026-11-10 · 213 (-0.93 %)f83ff30f · 2026-11-10 · 223 (+4.69 %)38e9f0d4 · 2026-11-11 · 212 (-4.93 %)f002431b · 2026-11-11 · 202 (-4.72 %)2b5473e7 · 2026-11-11 · 200 (-0.99 %)91ab2741 · 2026-11-11 · 196 (-2.00 %)c432f068 · 2026-11-11 · 194 (-1.02 %)