automata benchmarks

History · Versus Spot and Oink

PR #166 · f30631e1

CsrTs: real compressed sparse rows, nondeterministic, any alphabet

status done · queued 2026-11-11 17:40 · started 2026-11-11 17:40 · finished 2026-11-11 17:50

· · rustc 1.95.0 (59807616e 2026-04-14) (built from a source tarball) · valgrind-3.26.0 · pinned to cpu 1

Against merge base 38e9f0d4 — 8 regressions, 12 improvements, 246 unchanged, 0 new

benchmarkbaseheadchangeheap bytes
storage::storage::minimize_dense::redundant_20004 676 9104 834 920+3.38 %+0.00 %
storage::storage::minimize_dense::redundant_800018 364 89518 996 871+3.44 %+0.00 %
storage::storage::predecessors_of_all_states_csr::k2_2000102 046638 345+525.55 %
storage::storage::reachable_states_csr::k2_80001 101 1151 148 908+4.34 %+0.00 %
storage::storage::reachable_states_dense::k2_80001 133 6991 365 712+20.47 %+0.00 %
storage::storage::reachable_states_linked::k2_8000924 2791 148 290+24.24 %+0.00 %
storage::storage::run_many_words_csr::count_1000551 036726 035+31.76 %+0.00 %
storage::storage::run_one_word_csr::len_1000002 600 0543 500 048+34.61 %+0.00 %
storage::storage::clone_csr::k2_1000030 59025 138-17.82 %-8.70 %
storage::storage::complete_dense::partial_5000155 628150 672-3.18 %-58.82 %
storage::storage::edges_from_all_states_csr::k2_200072 04748 041-33.32 %+0.00 %
storage::storage::edges_from_all_states_csr::k8_2000108 04748 041-55.54 %+0.00 %
storage::storage::successor_all_pairs_csr::k2_2000122 242108 241-11.45 %+0.00 %
storage::storage::successor_all_pairs_csr::k8_2000702 302552 301-21.36 %+0.00 %
ts::ts::add_edges_in_order::k2_50001 402 6711 362 671-2.85 %+0.00 %
ts::ts::add_edges_in_order::k8_50006 669 2856 509 285-2.40 %+0.00 %
ts::ts::add_edges_in_reverse::k2_50001 402 6711 362 671-2.85 %+0.00 %
ts::ts::erase_state_colors_collect::scrambled_100002 377 1722 317 172-2.52 %+0.00 %
ts::ts::map_edge_colors_collect::scrambled_100002 368 7842 308 784-2.53 %+0.00 %
ts::ts::map_edge_colors_collect::scrambled_400009 415 4099 175 409-2.55 %+0.00 %
dfa::dfa::accepts_many_words::scrambled_1000343 072343 072+0.00 %+0.00 %
dfa::dfa::complement::scrambled_1000366 068366 068+0.00 %+0.00 %
dfa::dfa::complement::scrambled_40001 435 8731 435 873+0.00 %+0.00 %
dfa::dfa::equivalent::deep_difference_10061 66761 667+0.00 %+0.00 %
dfa::dfa::equivalent::deep_difference_400212 158212 158+0.00 %+0.00 %
dfa::dfa::equivalent::different_1009 0479 047+0.00 %+0.00 %
dfa::dfa::equivalent::different_4009 0889 088+0.00 %+0.00 %
dfa::dfa::equivalent::minimized_10035 84935 849+0.00 %+0.00 %
dfa::dfa::equivalent::minimized_400120 587120 587+0.00 %+0.00 %
dfa::dfa::give_word::empty_10001 872 6131 872 613+0.00 %+0.00 %
dfa::dfa::give_word::empty_40006 930 8236 930 823+0.00 %+0.00 %
dfa::dfa::give_word::far_1000164 342164 342+0.00 %+0.00 %
dfa::dfa::give_word::far_4000946 982946 982+0.00 %+0.00 %
dfa::dfa::intersection::scrambled_1003 319 7093 319 713+0.00 %+0.00 %
dfa::dfa::intersection::scrambled_20013 224 87313 224 877+0.00 %+0.00 %
dfa::dfa::intersection_is_empty::empty_10059 26659 266+0.00 %+0.00 %
dfa::dfa::intersection_is_empty::empty_200109 269109 269+0.00 %+0.00 %
dfa::dfa::is_empty_language::empty_10001 872 6151 872 615+0.00 %+0.00 %
dfa::dfa::is_empty_language::empty_40006 930 8256 930 825+0.00 %+0.00 %
dfa::dfa::minimize::minimal_10003 450 9243 461 926+0.32 %+0.00 %
dfa::dfa::minimize::minimal_400013 472 60213 516 602+0.33 %+0.00 %
dfa::dfa::minimize::redundant_10002 292 8072 297 812+0.22 %+0.00 %
dfa::dfa::minimize::redundant_40008 869 1068 889 109+0.23 %+0.00 %
dfa::dfa::minimize::scrambled_10003 670 2563 674 125+0.11 %+0.00 %
dfa::dfa::minimize::scrambled_400014 250 46914 265 475+0.11 %+0.00 %
dfa::dfa::minimize_moore_with_edge_colors::scrambled_10003 669 2523 673 121+0.11 %+0.00 %
dfa::dfa::minimize_moore_with_edge_colors::scrambled_400014 239 07314 254 079+0.11 %+0.00 %
dfa::dfa::union::scrambled_1003 319 7093 319 713+0.00 %+0.00 %
dfa::dfa::union::scrambled_20013 224 87313 224 877+0.00 %+0.00 %
dpa::dpa::accepted_word::chain_100113 482113 484+0.00 %
dpa::dpa::accepted_word::chain_400383 293383 295+0.00 %
dpa::dpa::accepted_word::scrambled_100145 688145 690+0.00 %
dpa::dpa::accepted_word::scrambled_400496 798496 800+0.00 %
dpa::dpa::accepted_word_fixtures::ltl546 210546 252+0.01 %
dpa::dpa::accepted_word_fixtures::synthetic1 402 2961 402 310+0.00 %
dpa::dpa::accepted_word_fixtures_by_atoms::ltl1 734 6751 734 717+0.00 %
dpa::dpa::accepted_word_fixtures_by_atoms::synthetic2 873 2562 873 270+0.00 %
dpa::dpa::collect_dpa::scrambled_1000299 640299 640+0.00 %
dpa::dpa::collect_dpa::scrambled_40001 169 0311 169 031+0.00 %
dpa::dpa::complement::chain_10004 096 9404 096 944+0.00 %
dpa::dpa::complement::chain_400016 653 21116 653 215+0.00 %
dpa::dpa::complement::scrambled_10004 958 4944 958 534+0.00 %
dpa::dpa::complement::scrambled_400020 037 45820 037 498+0.00 %
dpa::dpa::complement_fixtures::ltl1 480 5031 480 693+0.01 %
dpa::dpa::complement_fixtures::many_aps13 919 76413 919 870+0.00 %
dpa::dpa::complement_fixtures::synthetic9 652 5839 652 742+0.00 %
dpa::dpa::complement_without_normalizing::scrambled_1000574 441574 429-0.00 %
dpa::dpa::complement_without_normalizing::scrambled_40002 259 8232 259 822-0.00 %
dpa::dpa::equivalent_fixtures::many_aps21 649 39021 649 756+0.00 %
dpa::dpa::hoa_parse::ltl5 509 1615 507 276-0.03 %
dpa::dpa::hoa_parse::many_aps88 889 28188 878 935-0.01 %
dpa::dpa::hoa_parse::synthetic25 096 68125 084 967-0.05 %
dpa::dpa::hoa_print::ltl19 039 65119 039 652+0.00 %
dpa::dpa::hoa_print::many_aps19 675 34519 675 345+0.00 %
dpa::dpa::hoa_print::synthetic27 397 17627 397 177+0.00 %
dpa::dpa::included_fixtures::many_aps18 597 82318 598 029+0.00 %
dpa::dpa::included_in::complement_100270 299270 313+0.01 %
dpa::dpa::included_in::complement_2588 07588 085+0.01 %
dpa::dpa::included_in::complement_50151 571151 583+0.01 %
dpa::dpa::included_in::normalized_100543 003543 031+0.01 %
dpa::dpa::included_in::normalized_25184 848184 868+0.01 %
dpa::dpa::included_in::normalized_50313 519313 539+0.01 %
dpa::dpa::language_equivalent::complement_100279 874279 888+0.01 %
dpa::dpa::language_equivalent::complement_2590 60090 610+0.01 %
dpa::dpa::language_equivalent::complement_50156 446156 458+0.01 %
dpa::dpa::language_equivalent::normalized_100993 356993 400+0.00 %
dpa::dpa::language_equivalent::normalized_25326 815326 845+0.01 %
dpa::dpa::language_equivalent::normalized_50581 323581 353+0.01 %
dpa::dpa::map_edge_colors_collect::scrambled_1000297 297297 297+0.00 %
dpa::dpa::map_edge_colors_collect::scrambled_40001 157 6921 157 692+0.00 %
dpa::dpa::normalized::chain_10003 824 3863 824 390+0.00 %
dpa::dpa::normalized::chain_400015 564 65715 564 661+0.00 %
dpa::dpa::normalized::scrambled_10004 685 7424 685 782+0.00 %
dpa::dpa::normalized::scrambled_400018 948 70618 948 746+0.00 %
dpa::dpa::normalized::scrambled_k4_100026 000 72526 000 787+0.00 %
dpa::dpa::normalized_fixtures::ltl1 415 8891 416 101+0.01 %
dpa::dpa::normalized_fixtures::many_aps13 659 71513 659 799+0.00 %
dpa::dpa::normalized_fixtures::synthetic9 337 5919 337 761+0.00 %
dpa::dpa::prefix_partition::chain_163 339 0693 339 205+0.00 %
dpa::dpa::prefix_partition::chain_8861 644861 716+0.01 %
dpa::dpa::prefix_partition::scrambled_163 707 9093 708 189+0.01 %
dpa::dpa::prefix_partition::scrambled_81 022 8581 023 002+0.01 %
dpa::dpa::rejected_word::chain_100938 248938 255+0.00 %
dpa::dpa::rejected_word::chain_4003 643 9433 643 961+0.00 %
dpa::dpa::rejected_word::scrambled_100247 893247 895+0.00 %
dpa::dpa::rejected_word::scrambled_400935 362935 364+0.00 %
dpa::dpa::sccs::chain_1000500 888500 890+0.00 %
dpa::dpa::sccs::chain_40001 928 6151 928 617+0.00 %
dpa::dpa::sccs::scrambled_1000567 845567 847+0.00 %
dpa::dpa::sccs::scrambled_40002 257 4852 257 487+0.00 %
dpa::dpa::witness_colors::chain_1004 741 5404 741 555+0.00 %
dpa::dpa::witness_colors::chain_40046 520 65346 520 668+0.00 %
dpa::dpa::witness_colors::scrambled_1004 126 3724 126 387+0.00 %
dpa::dpa::witness_colors::scrambled_40031 846 70131 846 738+0.00 %
dpa::dpa::witness_inequivalence::complement_100279 873279 887+0.01 %
dpa::dpa::witness_inequivalence::complement_2590 59990 609+0.01 %
dpa::dpa::witness_inequivalence::complement_50156 445156 457+0.01 %
dpa::dpa::witness_not_subset_of::complement_100270 298270 312+0.01 %
dpa::dpa::witness_not_subset_of::complement_2588 07488 084+0.01 %
dpa::dpa::witness_not_subset_of::complement_50151 570151 582+0.01 %
dpa::dpa::witness_not_subset_of::normalized_100543 002543 030+0.01 %
dpa::dpa::witness_not_subset_of::normalized_25184 847184 867+0.01 %
dpa::dpa::witness_not_subset_of::normalized_50313 518313 538+0.01 %
dpa::dpa::witnesses_fixtures::ltl2 229 2152 229 287+0.00 %
dpa::dpa::witnesses_fixtures::synthetic19 359 35619 359 384+0.00 %
game::game::tangle_learning::few_priorities_10002 697 0482 687 985-0.34 %+0.00 %
game::game::tangle_learning::few_priorities_40008 120 6768 092 832-0.34 %+0.00 %
game::game::tangle_learning::ladder_10001 179 3141 179 088-0.02 %+0.00 %
game::game::tangle_learning::ladder_40004 408 1344 407 443-0.02 %+0.00 %
game::game::tangle_learning::many_priorities_10002 018 0282 013 656-0.22 %+0.00 %
game::game::tangle_learning::many_priorities_40006 752 8356 739 293-0.20 %+0.00 %
game::game::tangle_learning_min_odd::few_priorities_10001 521 6151 520 824-0.05 %+0.00 %
game::game::tangle_learning_min_odd::few_priorities_40008 017 7008 008 714-0.11 %+0.00 %
game::game::zielonka::few_priorities_10001 126 1991 126 202+0.00 %+0.00 %
game::game::zielonka::few_priorities_40003 470 2543 470 257+0.00 %+0.00 %
game::game::zielonka::ladder_10005 134 2815 134 284+0.00 %+0.00 %
game::game::zielonka::ladder_400019 816 37319 816 376+0.00 %+0.00 %
game::game::zielonka::many_priorities_10004 103 5184 103 521+0.00 %+0.00 %
game::game::zielonka::many_priorities_400012 909 83712 909 840+0.00 %+0.00 %
game::game::zielonka_min_odd::few_priorities_10002 139 0772 139 080+0.00 %+0.00 %
game::game::zielonka_min_odd::few_priorities_40006 302 0756 302 078+0.00 %+0.00 %
learning::learning::lstar_mealy::counting_161 816 8471 796 618-1.11 %-0.21 %
learning::learning::lstar_mealy::counting_8481 469473 174-1.72 %-0.47 %
learning::learning::sprout_parity::inf_ab_1001 270 7571 269 787-0.08 %-0.05 %
learning::learning::sprout_parity::inf_ab_4002 610 8202 611 163+0.01 %-0.02 %
nondeterministic::nondeterministic::fixture_accepts::ltl762 545762 661+0.02 %
nondeterministic::nondeterministic::nba_accepts::nba_400125 269 894125 269 894+0.00 %
nondeterministic::nondeterministic::nba_witness::nba_400014 689 96114 689 963+0.00 %
nondeterministic::nondeterministic::nca_accepts::nca_400266 229 440266 229 440+0.00 %
nondeterministic::nondeterministic::nca_witness::nca_400019 458 26119 458 265+0.00 %
nondeterministic::nondeterministic::nma_witness::nma_100025 744 46925 744 501+0.00 %
nondeterministic::nondeterministic::npa_accepts::empty_npa_400493 167 601493 167 513-0.00 %
nondeterministic::nondeterministic::npa_accepts::npa_10030 411 62830 411 608-0.00 %
nondeterministic::nondeterministic::npa_accepts::npa_400125 279 542125 279 522-0.00 %
nondeterministic::nondeterministic::npa_witness::empty_npa_10009 121 9529 121 954+0.00 %
nondeterministic::nondeterministic::npa_witness::empty_npa_400037 023 30337 023 305+0.00 %
nondeterministic::nondeterministic::npa_witness::npa_10002 479 5312 479 531+0.00 %
nondeterministic::nondeterministic::npa_witness::npa_400020 354 25520 354 255+0.00 %
nondeterministic::nondeterministic::nra_witness::nra_400020 639 56720 639 571+0.00 %
runs::runs::dfa_accepts_many_words::count_10004 303 2134 303 213+0.00 %+0.00 %
runs::runs::dfa_accepts_one_word::len_1003 4923 492+0.00 %+0.00 %
runs::runs::dfa_accepts_one_word::len_10000266 070266 070+0.00 %+0.00 %
runs::runs::dfa_accepts_one_word::len_1000002 650 7312 650 731+0.00 %+0.00 %
runs::runs::dfa_reached_state::count_10003 570 2113 570 211+0.00 %+0.00 %
runs::runs::dpa_accepts_lassos::long9 150 7459 145 303-0.06 %+0.00 %
runs::runs::dpa_accepts_lassos::short406 490400 803-1.40 %+0.00 %
runs::runs::dpa_recurrent_edge_colors::long15 116 26615 110 824-0.04 %+0.00 %
runs::runs::dpa_recurrent_edge_colors::short1 189 7901 184 103-0.48 %+0.00 %
runs::runs::dpa_recurrent_states::long10 502 98810 497 546-0.05 %+0.00 %
runs::runs::dpa_recurrent_states::short572 037566 350-0.99 %+0.00 %
runs::runs::dpa_visited_sequences::long8 738 2638 738 263+0.00 %+0.00 %
runs::runs::dpa_visited_sequences::short370 678370 678+0.00 %+0.00 %
runs::runs::fixture_accepts_lassos::ltl8 766 2718 769 271+0.03 %
runs::runs::fixture_accepts_lassos::synthetic4 340 1894 341 589+0.03 %
scc::scc::scc_queries::small_cycles_1000906 676906 678+0.00 %+0.00 %
scc::scc::scc_queries::small_cycles_40003 548 9193 548 921+0.00 %+0.00 %
scc::scc::tarjan::back_edges_1000512 099512 101+0.00 %+0.00 %
scc::scc::tarjan::back_edges_40001 987 6271 987 629+0.00 %+0.00 %
scc::scc::tarjan::binary_tree_1000571 217571 219+0.00 %+0.00 %
scc::scc::tarjan::binary_tree_40002 290 4592 290 461+0.00 %+0.00 %
scc::scc::tarjan::fragmented_1000658 522658 524+0.00 %+0.00 %
scc::scc::tarjan::fragmented_40002 534 1082 534 110+0.00 %+0.00 %
scc::scc::tarjan::grid_1024596 080596 082+0.00 %+0.00 %
scc::scc::tarjan::grid_40962 378 4842 378 486+0.00 %+0.00 %
scc::scc::tarjan::path_10001 929 1191 929 121+0.00 %+0.00 %
scc::scc::tarjan::path_40007 632 9087 632 910+0.00 %+0.00 %
scc::scc::tarjan::ring_1000512 099512 101+0.00 %+0.00 %
scc::scc::tarjan::ring_40001 987 6271 987 629+0.00 %+0.00 %
scc::scc::tarjan::scrambled_1000579 056579 058+0.00 %+0.00 %
scc::scc::tarjan::scrambled_40002 316 4972 316 499+0.00 %+0.00 %
scc::scc::tarjan::small_cycles_1000905 658905 660+0.00 %+0.00 %
scc::scc::tarjan::small_cycles_40003 544 9013 544 903+0.00 %+0.00 %
scc::scc::tarjan_from::fragmented_1000242 530242 532+0.00 %+0.00 %
scc::scc::tarjan_from::fragmented_4000907 762907 764+0.00 %+0.00 %
scc::scc::tarjan_from::path_10001 921 1581 921 160+0.00 %+0.00 %
scc::scc::tarjan_from::path_40007 545 5847 545 586+0.00 %+0.00 %
scc::scc::tarjan_from::ring_1000503 645503 647+0.00 %+0.00 %
scc::scc::tarjan_from::scrambled_1000570 602570 604+0.00 %+0.00 %
storage::storage::add_edges_dense::k2_50001 356 4011 356 401+0.00 %+0.00 %
storage::storage::add_edges_dense::k8_50004 247 0594 247 059+0.00 %+0.00 %
storage::storage::add_edges_linked::k2_50001 402 6701 402 670+0.00 %+0.00 %
storage::storage::add_edges_linked::k8_50006 669 2846 669 284+0.00 %+0.00 %
storage::storage::clone_dense::k2_1000015 11315 139+0.17 %+0.00 %
storage::storage::clone_linked::k2_10000370 824370 822-0.00 %+0.00 %
storage::storage::complete_linked::partial_50001 210 8541 225 141+1.18 %-0.12 %
storage::storage::edges_from_all_states_dense::k2_2000102 043102 043+0.00 %+0.00 %
storage::storage::edges_from_all_states_dense::k8_2000140 043140 043+0.00 %+0.00 %
storage::storage::edges_from_all_states_linked::k2_200068 03368 033+0.00 %+0.00 %
storage::storage::edges_from_all_states_linked::k8_2000152 033152 033+0.00 %+0.00 %
storage::storage::minimize_csr::redundant_20004 633 7564 584 603-1.06 %+0.00 %
storage::storage::minimize_csr::redundant_800018 159 86117 948 590-1.16 %+0.00 %
storage::storage::minimize_linked::redundant_20004 505 0384 585 040+1.78 %+0.00 %
storage::storage::minimize_linked::redundant_800017 714 72118 034 689+1.81 %+0.00 %
storage::storage::overwrite_edges_dense::k2_50002 126 4312 126 433+0.00 %+0.00 %
storage::storage::overwrite_edges_linked::k2_50003 790 7753 790 776+0.00 %+0.00 %
storage::storage::predecessors_of_all_states_dense::k2_2000290 369294 366+1.38 %+0.00 %
storage::storage::predecessors_of_all_states_linked::k2_200068 03368 033+0.00 %+0.00 %
storage::storage::product_successor_csr::n20064 09664 587+0.77 %+0.00 %
storage::storage::product_successor_dense::n20073 08773 087+0.00 %+0.00 %
storage::storage::product_successor_linked::n20046 54746 547+0.00 %+0.00 %
storage::storage::remove_state_dense::k2_100001 072 2121 092 207+1.86 %+0.00 %
storage::storage::remove_state_linked::k2_100001 094 0121 094 012+0.00 %+0.00 %
storage::storage::run_many_words_dense::count_1000558 044558 044+0.00 %+0.00 %
storage::storage::run_many_words_linked::count_1000481 044481 044+0.00 %+0.00 %
storage::storage::run_one_word_dense::len_1000002 800 0502 800 050+0.00 %+0.00 %
storage::storage::run_one_word_linked::len_1000002 350 0392 350 039+0.00 %+0.00 %
storage::storage::successor_all_pairs_dense::k2_2000162 237162 237+0.00 %+0.00 %
storage::storage::successor_all_pairs_dense::k8_2000814 297814 297+0.00 %+0.00 %
storage::storage::successor_all_pairs_linked::k2_2000108 236108 236+0.00 %+0.00 %
storage::storage::successor_all_pairs_linked::k8_2000750 296750 296+0.00 %+0.00 %
ts::ts::builder_with_edges::k2_500081 365 96781 441 560+0.09 %+0.00 %
ts::ts::canonical_naming::scrambled_15003 144 0903 144 066-0.00 %+0.00 %
ts::ts::canonical_naming::scrambled_500972 576972 552-0.00 %+0.00 %
ts::ts::clone::scrambled_10000371 163371 163+0.00 %+0.00 %
ts::ts::clone::scrambled_400001 481 4001 481 400+0.00 %+0.00 %
ts::ts::collect_dts::scrambled_100003 455 1283 455 128+0.00 %+0.00 %
ts::ts::collect_dts::scrambled_4000013 759 60713 759 607+0.00 %+0.00 %
ts::ts::determinize::nth_from_end_104 414 3204 414 320+0.00 %+0.00 %
ts::ts::determinize::nth_from_end_1219 628 00619 628 006+0.00 %+0.00 %
ts::ts::determinize::nth_from_end_81 007 2681 007 268+0.00 %+0.00 %
ts::ts::determinize_propositional::aps_6_n_64 333 8444 333 842-0.00 %+0.00 %
ts::ts::determinize_propositional::aps_7_n_64 525 3874 525 385-0.00 %+0.00 %
ts::ts::determinize_propositional::aps_8_n_626 425 54226 425 540-0.00 %+0.00 %
ts::ts::edges_from_all_states::scrambled_200072 66972 669+0.00 %+0.00 %
ts::ts::edges_from_all_states::scrambled_k8_2000156 601156 601+0.00 %+0.00 %
ts::ts::into_dfa::scrambled_100003 635 1663 635 166+0.00 %+0.00 %
ts::ts::matching_product_successor_index::n20075 66275 662+0.00 %+0.00 %
ts::ts::minimal_representative_of_all_states::scrambled_1002 102 8112 102 811+0.00 %+0.00 %
ts::ts::minimal_representative_of_all_states::scrambled_30016 917 26116 917 261+0.00 %+0.00 %
ts::ts::minimal_representatives::scrambled_20003 386 0403 386 040+0.00 %+0.00 %
ts::ts::minimal_representatives::scrambled_800014 908 24914 908 249+0.00 %+0.00 %
ts::ts::predecessors_of_all_states::scrambled_200072 66972 669+0.00 %+0.00 %
ts::ts::predecessors_of_all_states::scrambled_8000288 601288 601+0.00 %+0.00 %
ts::ts::reachable_states::scrambled_2000290 091290 091+0.00 %+0.00 %
ts::ts::reachable_states::scrambled_80001 148 6961 148 696+0.00 %+0.00 %
ts::ts::remove_edges_from_matching::scrambled_10000566 234566 234+0.00 %+0.00 %
ts::ts::remove_edges_from_matching::scrambled_40000896 234896 234+0.00 %+0.00 %
ts::ts::remove_state::scrambled_100001 091 0121 091 012+0.00 %+0.00 %
ts::ts::remove_state::scrambled_400001 421 0121 421 012+0.00 %+0.00 %
ts::ts::scc_minimal_representatives::small_cycles_100219 042219 044+0.00 %+0.00 %
ts::ts::scc_minimal_representatives::small_cycles_4001 242 4751 242 477+0.00 %+0.00 %
ts::ts::subset_construction::nth_from_end_1010 228 20610 228 206+0.00 %+0.00 %
ts::ts::subset_construction::nth_from_end_1245 253 02545 253 025+0.00 %+0.00 %
ts::ts::subset_construction::nth_from_end_82 233 5062 233 506+0.00 %+0.00 %
ts::ts::successor_index_all_pairs::scrambled_2000108 863108 863+0.00 %+0.00 %
ts::ts::successor_index_all_pairs::scrambled_k8_2000750 855750 855+0.00 %+0.00 %

Wall-clock times

Criterion means of a single run. They depend on the machine and are not compared; use the instruction counts above for that.

benchmarkbasehead
collect/clone/1000003.93 ms (3.84 ms … 4.03 ms)4.32 ms (4.31 ms … 4.32 ms)
collect/clone/100000044.28 ms (43.85 ms … 44.72 ms)38.03 ms (37.59 ms … 38.47 ms)
collect/collect_owned/1000007.23 ms (7.19 ms … 7.27 ms)8.32 ms (8.17 ms … 8.46 ms)
collect/collect_owned/100000081.41 ms (79.65 ms … 83.17 ms)88.83 ms (87.37 ms … 90.30 ms)
collect/collect_ref/1000005.21 ms (5.19 ms … 5.22 ms)5.27 ms (5.25 ms … 5.29 ms)
collect/collect_ref/100000089.22 ms (88.40 ms … 90.04 ms)89.59 ms (87.77 ms … 91.41 ms)
collect/map_edge_colors_collect_owned/1000008.02 ms (7.96 ms … 8.07 ms)8.79 ms (8.66 ms … 8.92 ms)
collect/map_edge_colors_collect_owned/100000089.84 ms (87.48 ms … 92.21 ms)76.33 ms (75.22 ms … 77.43 ms)
wallclock/dfa/accepts/word_100000927.76 µs (909.78 µs … 945.75 µs)924.26 µs (907.21 µs … 941.30 µs)
wallclock/dfa/minimize/redundant_40001.40 ms (1.39 ms … 1.42 ms)1.44 ms (1.44 ms … 1.44 ms)
wallclock/dpa/complement/scrambled_40003.59 ms (3.59 ms … 3.60 ms)3.65 ms (3.64 ms … 3.65 ms)
wallclock/dpa/normalized/scrambled_40003.43 ms (3.36 ms … 3.50 ms)3.38 ms (3.37 ms … 3.39 ms)
wallclock/hoa/parse/ltl944.29 µs (940.39 µs … 948.18 µs)965.67 µs (948.50 µs … 982.84 µs)
wallclock/nfa/determinize/nth_from_end_123.45 ms (3.43 ms … 3.48 ms)3.43 ms (3.42 ms … 3.45 ms)
wallclock/nfa/subset_construction/nth_from_end_128.96 ms (8.78 ms … 9.15 ms)9.07 ms (9.03 ms … 9.12 ms)
wallclock/scc/tarjan/scrambled_100002.21 ms (2.19 ms … 2.24 ms)2.25 ms (2.21 ms … 2.29 ms)

Versus Spot and Oink

dpa/equivalence

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot155.36 µs116.62 µs1.33×– / –yes
hard/l-nba-2spot67.62 µs35.88 µs1.88×– / –yes
hard/m-nba-3spot277.88 µs136.18 µs2.04×– / –yes
ltl/and-gf-3spot11.66 µs12.56 µs0.93×– / –yes
ltl/eh-patterns-0spot29.46 µs20.82 µs1.41×– / –yes
ltl/gf-equiv-3spot27.60 µs34.32 µs0.80×– / –yes
ltl/gf-implies-3spot14.44 µs19.84 µs0.73×– / –yes
ltl/gh-q-3spot64.16 µs29.86 µs2.15×– / –yes
ltl/gh-r-3spot30.80 µs74.16 µs0.42×– / –yes
ltl/go-theta-3spot14.46 µs21.56 µs0.67×– / –yes
ltl/lily-patterns-1spot14.92 µs15.92 µs0.94×– / –yes
ltl/ms-phi-r-1spot13.58 µs38.44 µs0.35×– / –yes
ltl/ms-phi-s-1spot13.68 µs39.36 µs0.35×– / –yes
ltl/rv-counter-3spot31.40 µs13.04 µs2.41×– / –yes
ltl/sb-patterns-0spot15.96 µs13.94 µs1.14×– / –yes
ltl/sejk-k-2spot17.78 µs57.18 µs0.31×– / –yes
ltl/tv-uu-3spot48.78 µs37.14 µs1.31×– / –yes
ltl/u-left-3spot13.08 µs8.32 µs1.57×– / –yes
many_aps/and-gf-12spot179.46 µs65.04 µs2.76×– / –yes
many_aps/gf-equiv-12spot163.42 µs90.14 µs1.81×– / –yes
many_aps/r-right-10spot77.32 µs26.12 µs2.96×– / –yes
many_aps/random-n16-a8-p4-s1spot305.98 µs127.82 µs2.39×– / –yes
many_aps/random-n24-a12-p4-s1spot1.29 ms431.12 µs2.99×– / –yes
many_aps/random-n32-a16-p4-s1spot3.97 ms1.16 ms3.42×– / –yes
random/n128-a2-p6-s1spot359.30 µs206.70 µs1.74×– / –yes
random/n16-a1-p3-s1spot24.96 µs15.46 µs1.61×– / –yes
random/n256-a2-p8-s1spot964.19 µs473.65 µs2.04×– / –yes
random/n64-a2-p4-s1spot155.50 µs96.04 µs1.62×– / –yes

dpa/inclusion

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot79.18 µs60.80 µs1.30×– / –yes
hard/l-nba-2spot40.28 µs22.80 µs1.77×– / –yes
hard/m-nba-3spot167.94 µs83.94 µs2.00×– / –yes
ltl/and-gf-3spot7.50 µs9.40 µs0.80×– / –yes
ltl/eh-patterns-0spot18.58 µs17.74 µs1.05×– / –yes
ltl/gf-equiv-3spot15.06 µs18.78 µs0.80×– / –yes
ltl/gf-implies-3spot8.20 µs11.52 µs0.71×– / –yes
ltl/gh-q-3spot39.72 µs27.20 µs1.46×– / –yes
ltl/gh-r-3spot16.00 µs34.54 µs0.46×– / –yes
ltl/go-theta-3spot9.06 µs16.18 µs0.56×– / –yes
ltl/lily-patterns-1spot10.20 µs13.84 µs0.74×– / –yes
ltl/ms-phi-r-1spot8.16 µs19.48 µs0.42×– / –yes
ltl/ms-phi-s-1spot8.06 µs24.98 µs0.32×– / –yes
ltl/rv-counter-3spot20.26 µs14.24 µs1.42×– / –yes
ltl/sb-patterns-0spot10.08 µs12.18 µs0.83×– / –yes
ltl/sejk-k-2spot10.32 µs29.12 µs0.35×– / –yes
ltl/tv-uu-3spot34.30 µs30.36 µs1.13×– / –yes
ltl/u-left-3spot8.78 µs9.44 µs0.93×– / –yes
many_aps/and-gf-12spot151.92 µs55.44 µs2.74×– / –yes
many_aps/gf-equiv-12spot92.20 µs51.28 µs1.80×– / –yes
many_aps/r-right-10spot55.22 µs23.36 µs2.36×– / –yes
many_aps/random-n16-a8-p4-s1spot242.46 µs96.88 µs2.50×– / –yes
many_aps/random-n24-a12-p4-s1spot1.23 ms382.54 µs3.22×– / –yes
many_aps/random-n32-a16-p4-s1spot3.85 ms1.09 ms3.53×– / –yes
random/n128-a2-p6-s1spot213.22 µs122.16 µs1.75×– / –yes
random/n16-a1-p3-s1spot15.32 µs10.20 µs1.50×– / –yes
random/n256-a2-p8-s1spot552.71 µs291.52 µs1.90×– / –yes
random/n64-a2-p4-s1spot92.36 µs60.14 µs1.54×– / –yes

game/solve

inputreferenceoursreferenceours / referencesize (ours / reference)same result
ladder_10000oink-ppp2.68 ms319.00 µs8.41×10000 / 10000yes
ladder_10000oink-zlk2.68 ms320.00 µs8.39×10000 / 10000yes
ladder_10000spot2.68 ms9.54 ms0.28×10000 / 10000yes
ladder_50000oink-ppp20.61 ms2.24 ms9.21×50000 / 50000yes
ladder_50000oink-zlk20.61 ms2.21 ms9.31×50000 / 50000yes
ladder_50000spot20.61 ms76.42 ms0.27×50000 / 50000yes
random_few_10000oink-ppp7.86 ms1.46 ms5.40×10000 / 10000yes
random_few_10000oink-zlk7.86 ms2.71 ms2.90×10000 / 10000yes
random_few_10000spot7.86 ms21.12 ms0.37×10000 / 10000yes
random_few_50000oink-ppp79.87 ms17.05 ms4.69×50000 / 50000yes
random_few_50000oink-zlk79.87 ms18.62 ms4.29×50000 / 50000yes
random_few_50000spot79.87 ms705.28 ms0.11×50000 / 50000yes
random_many_10000oink-ppp6.92 ms1.53 ms4.53×10000 / 10000yes
random_many_10000oink-zlk6.92 ms2.72 ms2.54×10000 / 10000yes
random_many_10000spot6.92 ms30.90 ms0.22×10000 / 10000yes
random_many_50000oink-ppp76.28 ms30.90 ms2.47×50000 / 50000yes
random_many_50000oink-zlk76.28 ms45.66 ms1.67×50000 / 50000yes
random_many_50000spot76.28 ms706.70 ms0.11×50000 / 50000yes

Log

== head f30631e1b439e865b0fb238377b9803d2cd59c27: building ["scc", "ts", "runs", "dfa", "dpa", "storage", "nondeterministic", "learning", "game"] ["wallclock"]
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
    Finished `bench` profile [optimized] target(s) in 2m 06s
  Executable benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-a84ba41f760e70ed)
  Executable benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-23bb829f074058ad)
  Executable benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-14422f732ec85029)
  Executable benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-ee53c00379271c59)
  Executable benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-f164d05108dbce73)
  Executable benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-85a152aa046a29b4)
  Executable benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-1b22c641fd9d2d89)
  Executable benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-f95e3d4a8c506941)
  Executable benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-4948f6cc6a43d201)
  Executable benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-93fb383f608d3bab)
== head: running scc
    Finished `bench` profile [optimized] target(s) in 0.23s
     Running benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-1b22c641fd9d2d89)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running ts
    Finished `bench` profile [optimized] target(s) in 0.19s
     Running benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-4948f6cc6a43d201)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running runs
    Finished `bench` profile [optimized] target(s) in 0.20s
     Running benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-85a152aa046a29b4)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dfa
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-a84ba41f760e70ed)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dpa
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-23bb829f074058ad)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running storage
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-f95e3d4a8c506941)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running nondeterministic
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-f164d05108dbce73)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running learning
    Finished `bench` profile [optimized] target(s) in 0.20s
     Running benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-ee53c00379271c59)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running game
    Finished `bench` profile [optimized] target(s) in 0.19s
     Running benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-14422f732ec85029)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running wallclock (wall clock)
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-93fb383f608d3bab)
Gnuplot not found, using plotters backend
Benchmarking wallclock/scc/tarjan/scrambled_10000
Benchmarking wallclock/scc/tarjan/scrambled_10000: Analyzing
Benchmarking wallclock/dpa/normalized/scrambled_4000
Benchmarking wallclock/dpa/normalized/scrambled_4000: Analyzing
Benchmarking wallclock/dpa/complement/scrambled_4000
Benchmarking wallclock/dpa/complement/scrambled_4000: Analyzing
Benchmarking wallclock/dfa/minimize/redundant_4000
Benchmarking wallclock/dfa/minimize/redundant_4000: Analyzing
Benchmarking wallclock/dfa/accepts/word_100000
Benchmarking wallclock/dfa/accepts/word_100000: Analyzing
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12: Analyzing
Benchmarking wallclock/nfa/determinize/nth_from_end_12
Benchmarking wallclock/nfa/determinize/nth_from_end_12: Analyzing
Benchmarking wallclock/hoa/parse/ltl
Benchmarking wallclock/hoa/parse/ltl: Analyzing
Benchmarking collect/clone/100000
Benchmarking collect/clone/100000: Analyzing
Benchmarking collect/collect_owned/100000
Benchmarking collect/collect_owned/100000: Analyzing
Benchmarking collect/collect_ref/100000
Benchmarking collect/collect_ref/100000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/100000
Benchmarking collect/map_edge_colors_collect_owned/100000: Analyzing
Benchmarking collect/clone/1000000
Benchmarking collect/clone/1000000: Analyzing
Benchmarking collect/collect_owned/1000000
Benchmarking collect/collect_owned/1000000: Analyzing
Benchmarking collect/collect_ref/1000000
Benchmarking collect/collect_ref/1000000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/1000000
Benchmarking collect/map_edge_colors_collect_owned/1000000: Analyzing
== head: building the Python bindings
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support
🐍 Found CPython 3.13 at /nix/store/vilynzly1pyhawy04ljmrp7n9ffr24qy-python3-3.13.17/bin/python3
📡 Using build options features from pyproject.toml
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
   Compiling automata-py v0.1.0 (/var/lib/automata-bench/work/src-head/python)
    Finished `release` profile [optimized] target(s) in 1m 50s
📦 Built wheel for abi3 Python ≥ 3.11 to /var/lib/automata-bench/work/versus-head/wheels/automatars-0.1.0-cp311-abi3-manylinux_2_34_aarch64.whl
== head: running benches/versus/compare.py (Spot, Oink)
dpa/equivalence/hard/l-dsa-3 vs spot: 0.155 ms / 0.117 ms
dpa/equivalence/hard/l-nba-2 vs spot: 0.068 ms / 0.036 ms
dpa/equivalence/hard/m-nba-3 vs spot: 0.278 ms / 0.136 ms
dpa/equivalence/ltl/and-gf-3 vs spot: 0.012 ms / 0.013 ms
dpa/equivalence/ltl/eh-patterns-0 vs spot: 0.029 ms / 0.021 ms
dpa/equivalence/ltl/gf-equiv-3 vs spot: 0.028 ms / 0.034 ms
dpa/equivalence/ltl/gf-implies-3 vs spot: 0.014 ms / 0.020 ms
dpa/equivalence/ltl/gh-q-3 vs spot: 0.064 ms / 0.030 ms
dpa/equivalence/ltl/gh-r-3 vs spot: 0.031 ms / 0.074 ms
dpa/equivalence/ltl/go-theta-3 vs spot: 0.014 ms / 0.022 ms
dpa/equivalence/ltl/lily-patterns-1 vs spot: 0.015 ms / 0.016 ms
dpa/equivalence/ltl/ms-phi-r-1 vs spot: 0.014 ms / 0.038 ms
dpa/equivalence/ltl/ms-phi-s-1 vs spot: 0.014 ms / 0.039 ms
dpa/equivalence/ltl/rv-counter-3 vs spot: 0.031 ms / 0.013 ms
dpa/equivalence/ltl/sb-patterns-0 vs spot: 0.016 ms / 0.014 ms
dpa/equivalence/ltl/sejk-k-2 vs spot: 0.018 ms / 0.057 ms
dpa/equivalence/ltl/tv-uu-3 vs spot: 0.049 ms / 0.037 ms
dpa/equivalence/ltl/u-left-3 vs spot: 0.013 ms / 0.008 ms
dpa/equivalence/many_aps/and-gf-12 vs spot: 0.179 ms / 0.065 ms
dpa/equivalence/many_aps/gf-equiv-12 vs spot: 0.163 ms / 0.090 ms
dpa/equivalence/many_aps/r-right-10 vs spot: 0.077 ms / 0.026 ms
dpa/equivalence/many_aps/random-n16-a8-p4-s1 vs spot: 0.306 ms / 0.128 ms
dpa/equivalence/many_aps/random-n24-a12-p4-s1 vs spot: 1.288 ms / 0.431 ms
dpa/equivalence/many_aps/random-n32-a16-p4-s1 vs spot: 3.965 ms / 1.158 ms
dpa/equivalence/random/n128-a2-p6-s1 vs spot: 0.359 ms / 0.207 ms
dpa/equivalence/random/n16-a1-p3-s1 vs spot: 0.025 ms / 0.015 ms
dpa/equivalence/random/n256-a2-p8-s1 vs spot: 0.964 ms / 0.474 ms
dpa/equivalence/random/n64-a2-p4-s1 vs spot: 0.156 ms / 0.096 ms
dpa/inclusion/hard/l-dsa-3 vs spot: 0.079 ms / 0.061 ms
dpa/inclusion/hard/l-nba-2 vs spot: 0.040 ms / 0.023 ms
dpa/inclusion/hard/m-nba-3 vs spot: 0.168 ms / 0.084 ms
dpa/inclusion/ltl/and-gf-3 vs spot: 0.007 ms / 0.009 ms
dpa/inclusion/ltl/eh-patterns-0 vs spot: 0.019 ms / 0.018 ms
dpa/inclusion/ltl/gf-equiv-3 vs spot: 0.015 ms / 0.019 ms
dpa/inclusion/ltl/gf-implies-3 vs spot: 0.008 ms / 0.012 ms
dpa/inclusion/ltl/gh-q-3 vs spot: 0.040 ms / 0.027 ms
dpa/inclusion/ltl/gh-r-3 vs spot: 0.016 ms / 0.035 ms
dpa/inclusion/ltl/go-theta-3 vs spot: 0.009 ms / 0.016 ms
dpa/inclusion/ltl/lily-patterns-1 vs spot: 0.010 ms / 0.014 ms
dpa/inclusion/ltl/ms-phi-r-1 vs spot: 0.008 ms / 0.019 ms
dpa/inclusion/ltl/ms-phi-s-1 vs spot: 0.008 ms / 0.025 ms
dpa/inclusion/ltl/rv-counter-3 vs spot: 0.020 ms / 0.014 ms
dpa/inclusion/ltl/sb-patterns-0 vs spot: 0.010 ms / 0.012 ms
dpa/inclusion/ltl/sejk-k-2 vs spot: 0.010 ms / 0.029 ms
dpa/inclusion/ltl/tv-uu-3 vs spot: 0.034 ms / 0.030 ms
dpa/inclusion/ltl/u-left-3 vs spot: 0.009 ms / 0.009 ms
dpa/inclusion/many_aps/and-gf-12 vs spot: 0.152 ms / 0.055 ms
dpa/inclusion/many_aps/gf-equiv-12 vs spot: 0.092 ms / 0.051 ms
dpa/inclusion/many_aps/r-right-10 vs spot: 0.055 ms / 0.023 ms
dpa/inclusion/many_aps/random-n16-a8-p4-s1 vs spot: 0.242 ms / 0.097 ms
dpa/inclusion/many_aps/random-n24-a12-p4-s1 vs spot: 1.233 ms / 0.383 ms
dpa/inclusion/many_aps/random-n32-a16-p4-s1 vs spot: 3.849 ms / 1.089 ms
dpa/inclusion/random/n128-a2-p6-s1 vs spot: 0.213 ms / 0.122 ms
dpa/inclusion/random/n16-a1-p3-s1 vs spot: 0.015 ms / 0.010 ms
dpa/inclusion/random/n256-a2-p8-s1 vs spot: 0.553 ms / 0.292 ms
dpa/inclusion/random/n64-a2-p4-s1 vs spot: 0.092 ms / 0.060 ms
game/solve/random_few_10000 vs spot: 7.855 ms / 21.118 ms
game/solve/random_few_10000 vs oink-zlk: 7.855 ms / 2.710 ms
game/solve/random_few_10000 vs oink-ppp: 7.855 ms / 1.456 ms
game/solve/random_few_50000 vs spot: 79.865 ms / 705.277 ms
game/solve/random_few_50000 vs oink-zlk: 79.865 ms / 18.625 ms
game/solve/random_few_50000 vs oink-ppp: 79.865 ms / 17.046 ms
game/solve/random_many_10000 vs spot: 6.918 ms / 30.903 ms
game/solve/random_many_10000 vs oink-zlk: 6.918 ms / 2.720 ms
game/solve/random_many_10000 vs oink-ppp: 6.918 ms / 1.526 ms
game/solve/random_many_50000 vs spot: 76.281 ms / 706.696 ms
game/solve/random_many_50000 vs oink-zlk: 76.281 ms / 45.658 ms
game/solve/random_many_50000 vs oink-ppp: 76.281 ms / 30.901 ms
game/solve/ladder_10000 vs spot: 2.684 ms / 9.544 ms
game/solve/ladder_10000 vs oink-zlk: 2.684 ms / 0.320 ms
game/solve/ladder_10000 vs oink-ppp: 2.684 ms / 0.319 ms
game/solve/ladder_50000 vs spot: 20.611 ms / 76.424 ms
game/solve/ladder_50000 vs oink-zlk: 20.611 ms / 2.213 ms
game/solve/ladder_50000 vs oink-ppp: 20.611 ms / 2.239 ms
== head: 266 benchmarks, 16 wall-clock measurements, 74 comparisons