automata benchmarks

History · Versus Spot and Oink

PR #162 · d0b8e812

Atoms for propositional alphabets: faster determinization (#157)

status done · queued 2026-11-10 19:13 · started 2026-11-10 19:13 · finished 2026-11-10 19:24

· · rustc 1.95.0 (59807616e 2026-04-14) (built from a source tarball) · valgrind-3.26.0 · pinned to cpu 1

Against merge base f83ff30f — 12 regressions, 6 improvements, 244 unchanged, 4 new

benchmarkbaseheadchangeheap bytes
dpa::dpa::collect_dpa::scrambled_1000290 651299 640+3.09 %
dpa::dpa::collect_dpa::scrambled_40001 133 0441 169 031+3.18 %
dpa::dpa::map_edge_colors_collect::scrambled_1000289 296297 297+2.77 %
dpa::dpa::map_edge_colors_collect::scrambled_40001 125 6911 157 692+2.84 %
dpa::dpa::witness_colors::chain_1004 635 2764 741 540+2.29 %
dpa::dpa::witness_colors::chain_40045 080 18846 520 653+3.20 %
dpa::dpa::witness_colors::scrambled_1004 026 6034 126 372+2.48 %
dpa::dpa::witness_colors::scrambled_40030 619 06531 846 712+4.01 %
dpa::dpa::witnesses_fixtures::synthetic18 835 73619 359 356+2.78 %
ts::ts::add_edges_in_order::k2_50001 362 6711 402 671+2.94 %+0.00 %
ts::ts::add_edges_in_order::k8_50006 509 2856 669 285+2.46 %+0.00 %
ts::ts::add_edges_in_reverse::k2_50001 362 6731 402 671+2.94 %+0.00 %
ts::ts::determinize_propositional::aps_6_n_613 066 7534 333 844-66.83 %+23.34 %
ts::ts::determinize_propositional::aps_7_n_6172 153 5254 525 387-97.37 %-11.82 %
ts::ts::determinize_propositional::aps_8_n_61 736 746 66326 425 542-98.48 %-21.49 %
ts::ts::erase_state_colors_collect::scrambled_100002 887 1662 377 172-17.66 %+0.00 %
ts::ts::map_edge_colors_collect::scrambled_100002 888 7682 368 784-18.00 %+0.00 %
ts::ts::map_edge_colors_collect::scrambled_4000011 495 3919 415 409-18.09 %+0.00 %
dpa::dpa::accepted_word_fixtures::ltl–546 210new
dpa::dpa::accepted_word_fixtures::synthetic–1 402 296new
dpa::dpa::accepted_word_fixtures_by_atoms::ltl–1 734 675new
dpa::dpa::accepted_word_fixtures_by_atoms::synthetic–2 873 256new
dfa::dfa::accepts_many_words::scrambled_1000343 072343 072+0.00 %+0.00 %
dfa::dfa::complement::scrambled_1000366 068366 068+0.00 %+0.00 %
dfa::dfa::complement::scrambled_40001 435 8731 435 873+0.00 %+0.00 %
dfa::dfa::equivalent::deep_difference_10061 66761 667+0.00 %+0.00 %
dfa::dfa::equivalent::deep_difference_400212 158212 158+0.00 %+0.00 %
dfa::dfa::equivalent::different_1009 0479 047+0.00 %+0.00 %
dfa::dfa::equivalent::different_4009 0889 088+0.00 %+0.00 %
dfa::dfa::equivalent::minimized_10035 84935 849+0.00 %+0.00 %
dfa::dfa::equivalent::minimized_400120 587120 587+0.00 %+0.00 %
dfa::dfa::give_word::empty_10001 872 6131 872 613+0.00 %+0.00 %
dfa::dfa::give_word::empty_40006 930 8236 930 823+0.00 %+0.00 %
dfa::dfa::give_word::far_1000164 342164 342+0.00 %+0.00 %
dfa::dfa::give_word::far_4000946 982946 982+0.00 %+0.00 %
dfa::dfa::intersection::scrambled_1003 319 7093 319 709+0.00 %+0.00 %
dfa::dfa::intersection::scrambled_20013 224 87313 224 873+0.00 %+0.00 %
dfa::dfa::intersection_is_empty::empty_10059 26659 266+0.00 %+0.00 %
dfa::dfa::intersection_is_empty::empty_200109 269109 269+0.00 %+0.00 %
dfa::dfa::is_empty_language::empty_10001 872 6151 872 615+0.00 %+0.00 %
dfa::dfa::is_empty_language::empty_40006 930 8256 930 825+0.00 %+0.00 %
dfa::dfa::minimize::minimal_10003 450 9243 450 924+0.00 %+0.00 %
dfa::dfa::minimize::minimal_400013 472 60213 472 602+0.00 %+0.00 %
dfa::dfa::minimize::redundant_10002 292 8072 292 807+0.00 %+0.00 %
dfa::dfa::minimize::redundant_40008 869 1068 869 106+0.00 %+0.00 %
dfa::dfa::minimize::scrambled_10003 670 2563 670 256+0.00 %+0.00 %
dfa::dfa::minimize::scrambled_400014 250 46914 250 469+0.00 %+0.00 %
dfa::dfa::minimize_moore_with_edge_colors::scrambled_10003 669 2523 669 252+0.00 %+0.00 %
dfa::dfa::minimize_moore_with_edge_colors::scrambled_400014 239 07314 239 073+0.00 %+0.00 %
dfa::dfa::union::scrambled_1003 319 7093 319 709+0.00 %+0.00 %
dfa::dfa::union::scrambled_20013 224 87313 224 873+0.00 %+0.00 %
dpa::dpa::accepted_word::chain_100114 174113 482-0.61 %
dpa::dpa::accepted_word::chain_400385 785383 293-0.65 %
dpa::dpa::accepted_word::scrambled_100144 718145 688+0.67 %
dpa::dpa::accepted_word::scrambled_400492 612496 798+0.85 %
dpa::dpa::complement::chain_10004 101 1574 096 940-0.10 %
dpa::dpa::complement::chain_400016 670 17816 653 211-0.10 %
dpa::dpa::complement::scrambled_10004 960 1654 958 494-0.03 %
dpa::dpa::complement::scrambled_400020 044 50520 037 458-0.04 %
dpa::dpa::complement_fixtures::ltl1 481 7991 480 492-0.09 %
dpa::dpa::complement_fixtures::many_aps13 922 08913 919 764-0.02 %
dpa::dpa::complement_fixtures::synthetic9 667 5669 652 572-0.16 %
dpa::dpa::complement_without_normalizing::scrambled_1000565 479574 430+1.58 %
dpa::dpa::complement_without_normalizing::scrambled_40002 223 8502 259 823+1.62 %
dpa::dpa::equivalent_fixtures::many_aps21 636 04221 649 390+0.06 %
dpa::dpa::hoa_parse::ltl5 507 5985 509 161+0.03 %
dpa::dpa::hoa_parse::many_aps88 883 06388 889 281+0.01 %
dpa::dpa::hoa_parse::synthetic25 090 68325 096 681+0.02 %
dpa::dpa::hoa_print::ltl19 037 29319 039 651+0.01 %
dpa::dpa::hoa_print::many_aps19 662 96919 675 345+0.06 %
dpa::dpa::hoa_print::synthetic27 383 65327 397 176+0.05 %
dpa::dpa::included_fixtures::many_aps18 595 18918 597 823+0.01 %
dpa::dpa::included_in::complement_100269 594270 299+0.26 %
dpa::dpa::included_in::complement_2587 75488 075+0.37 %
dpa::dpa::included_in::complement_50151 045151 571+0.35 %
dpa::dpa::included_in::normalized_100542 475543 003+0.10 %
dpa::dpa::included_in::normalized_25184 636184 848+0.11 %
dpa::dpa::included_in::normalized_50313 247313 519+0.09 %
dpa::dpa::language_equivalent::complement_100279 169279 874+0.25 %
dpa::dpa::language_equivalent::complement_2590 27990 600+0.36 %
dpa::dpa::language_equivalent::complement_50155 920156 446+0.34 %
dpa::dpa::language_equivalent::normalized_100992 450993 356+0.09 %
dpa::dpa::language_equivalent::normalized_25326 481326 815+0.10 %
dpa::dpa::language_equivalent::normalized_50580 879581 323+0.08 %
dpa::dpa::normalized::chain_10003 828 6033 824 386-0.11 %
dpa::dpa::normalized::chain_400015 581 62415 564 657-0.11 %
dpa::dpa::normalized::scrambled_10004 687 4134 685 742-0.04 %
dpa::dpa::normalized::scrambled_400018 955 75318 948 706-0.04 %
dpa::dpa::normalized::scrambled_k4_100025 967 23026 000 725+0.13 %
dpa::dpa::normalized_fixtures::ltl1 417 2071 415 889-0.09 %
dpa::dpa::normalized_fixtures::many_aps13 662 04013 659 715-0.02 %
dpa::dpa::normalized_fixtures::synthetic9 351 5409 337 591-0.15 %
dpa::dpa::prefix_partition::chain_163 307 1633 339 069+0.96 %
dpa::dpa::prefix_partition::chain_8853 054861 644+1.01 %
dpa::dpa::prefix_partition::scrambled_163 688 0553 707 909+0.54 %
dpa::dpa::prefix_partition::scrambled_81 015 7011 022 858+0.70 %
dpa::dpa::rejected_word::chain_100922 706938 248+1.68 %
dpa::dpa::rejected_word::chain_4003 582 2343 643 943+1.72 %
dpa::dpa::rejected_word::scrambled_100244 006247 893+1.59 %
dpa::dpa::rejected_word::scrambled_400920 138935 362+1.65 %
dpa::dpa::sccs::chain_1000500 886500 888+0.00 %
dpa::dpa::sccs::chain_40001 928 6131 928 615+0.00 %
dpa::dpa::sccs::scrambled_1000567 843567 845+0.00 %
dpa::dpa::sccs::scrambled_40002 257 4832 257 485+0.00 %
dpa::dpa::witness_inequivalence::complement_100279 168279 873+0.25 %
dpa::dpa::witness_inequivalence::complement_2590 27890 599+0.36 %
dpa::dpa::witness_inequivalence::complement_50155 919156 445+0.34 %
dpa::dpa::witness_not_subset_of::complement_100269 593270 298+0.26 %
dpa::dpa::witness_not_subset_of::complement_2587 75388 074+0.37 %
dpa::dpa::witness_not_subset_of::complement_50151 044151 570+0.35 %
dpa::dpa::witness_not_subset_of::normalized_100542 474543 002+0.10 %
dpa::dpa::witness_not_subset_of::normalized_25184 635184 847+0.11 %
dpa::dpa::witness_not_subset_of::normalized_50313 246313 518+0.09 %
dpa::dpa::witnesses_fixtures::ltl2 217 3892 229 215+0.53 %
game::game::tangle_learning::few_priorities_10002 697 0482 697 048+0.00 %+0.00 %
game::game::tangle_learning::few_priorities_40008 120 6768 120 676+0.00 %+0.00 %
game::game::tangle_learning::ladder_10001 179 3141 179 314+0.00 %+0.00 %
game::game::tangle_learning::ladder_40004 408 1344 408 134+0.00 %+0.00 %
game::game::tangle_learning::many_priorities_10002 018 0282 018 028+0.00 %+0.00 %
game::game::tangle_learning::many_priorities_40006 752 8356 752 835+0.00 %+0.00 %
game::game::tangle_learning_min_odd::few_priorities_10001 521 6151 521 615+0.00 %+0.00 %
game::game::tangle_learning_min_odd::few_priorities_40008 017 7008 017 700+0.00 %+0.00 %
game::game::zielonka::few_priorities_10001 126 1991 126 199+0.00 %+0.00 %
game::game::zielonka::few_priorities_40003 470 2543 470 254+0.00 %+0.00 %
game::game::zielonka::ladder_10005 134 2815 134 281+0.00 %+0.00 %
game::game::zielonka::ladder_400019 816 37319 816 373+0.00 %+0.00 %
game::game::zielonka::many_priorities_10004 103 5184 103 518+0.00 %+0.00 %
game::game::zielonka::many_priorities_400012 909 83712 909 837+0.00 %+0.00 %
game::game::zielonka_min_odd::few_priorities_10002 139 0772 139 077+0.00 %+0.00 %
game::game::zielonka_min_odd::few_priorities_40006 302 0756 302 075+0.00 %+0.00 %
learning::learning::lstar_mealy::counting_161 814 0561 816 655+0.14 %+0.00 %
learning::learning::lstar_mealy::counting_8481 502481 475-0.01 %+0.00 %
learning::learning::sprout_parity::inf_ab_1001 270 7571 270 720-0.00 %+0.00 %
learning::learning::sprout_parity::inf_ab_4002 610 8202 610 857+0.00 %+0.00 %
nondeterministic::nondeterministic::fixture_accepts::ltl762 545762 545+0.00 %
nondeterministic::nondeterministic::nba_accepts::nba_400125 269 894125 269 894+0.00 %
nondeterministic::nondeterministic::nba_witness::nba_400014 689 96114 689 961+0.00 %
nondeterministic::nondeterministic::nca_accepts::nca_400266 229 440266 229 440+0.00 %
nondeterministic::nondeterministic::nca_witness::nca_400019 458 26119 458 261+0.00 %
nondeterministic::nondeterministic::nma_witness::nma_100025 744 46925 744 469+0.00 %
nondeterministic::nondeterministic::npa_accepts::empty_npa_400493 167 601493 167 601+0.00 %
nondeterministic::nondeterministic::npa_accepts::npa_10030 411 62830 411 628+0.00 %
nondeterministic::nondeterministic::npa_accepts::npa_400125 279 542125 279 542+0.00 %
nondeterministic::nondeterministic::npa_witness::empty_npa_10009 121 9529 121 952+0.00 %
nondeterministic::nondeterministic::npa_witness::empty_npa_400037 023 30337 023 303+0.00 %
nondeterministic::nondeterministic::npa_witness::npa_10002 479 5312 479 531+0.00 %
nondeterministic::nondeterministic::npa_witness::npa_400020 354 25520 354 255+0.00 %
nondeterministic::nondeterministic::nra_witness::nra_400020 639 56720 639 567+0.00 %
runs::runs::dfa_accepts_many_words::count_10004 303 2134 303 213+0.00 %+0.00 %
runs::runs::dfa_accepts_one_word::len_1003 4923 492+0.00 %+0.00 %
runs::runs::dfa_accepts_one_word::len_10000266 070266 070+0.00 %+0.00 %
runs::runs::dfa_accepts_one_word::len_1000002 650 7312 650 731+0.00 %+0.00 %
runs::runs::dfa_reached_state::count_10003 570 2113 570 211+0.00 %+0.00 %
runs::runs::dpa_accepts_lassos::long9 150 7459 150 745+0.00 %+0.00 %
runs::runs::dpa_accepts_lassos::short406 490406 490+0.00 %+0.00 %
runs::runs::dpa_recurrent_edge_colors::long15 116 26615 116 266+0.00 %+0.00 %
runs::runs::dpa_recurrent_edge_colors::short1 189 7901 189 790+0.00 %+0.00 %
runs::runs::dpa_recurrent_states::long10 502 98810 502 988+0.00 %+0.00 %
runs::runs::dpa_recurrent_states::short572 037572 037+0.00 %+0.00 %
runs::runs::dpa_visited_sequences::long8 738 2638 738 263+0.00 %+0.00 %
runs::runs::dpa_visited_sequences::short370 678370 678+0.00 %+0.00 %
runs::runs::fixture_accepts_lassos::ltl8 766 2718 766 271+0.00 %
runs::runs::fixture_accepts_lassos::synthetic4 340 1894 340 189+0.00 %
scc::scc::scc_queries::small_cycles_1000906 676906 676+0.00 %+0.00 %
scc::scc::scc_queries::small_cycles_40003 548 9193 548 919+0.00 %+0.00 %
scc::scc::tarjan::back_edges_1000512 099512 099+0.00 %+0.00 %
scc::scc::tarjan::back_edges_40001 987 6271 987 627+0.00 %+0.00 %
scc::scc::tarjan::binary_tree_1000571 217571 217+0.00 %+0.00 %
scc::scc::tarjan::binary_tree_40002 290 4592 290 459+0.00 %+0.00 %
scc::scc::tarjan::fragmented_1000658 522658 522+0.00 %+0.00 %
scc::scc::tarjan::fragmented_40002 534 1082 534 108+0.00 %+0.00 %
scc::scc::tarjan::grid_1024596 080596 080+0.00 %+0.00 %
scc::scc::tarjan::grid_40962 378 4842 378 484+0.00 %+0.00 %
scc::scc::tarjan::path_10001 929 1191 929 119+0.00 %+0.00 %
scc::scc::tarjan::path_40007 632 9087 632 908+0.00 %+0.00 %
scc::scc::tarjan::ring_1000512 099512 099+0.00 %+0.00 %
scc::scc::tarjan::ring_40001 987 6271 987 627+0.00 %+0.00 %
scc::scc::tarjan::scrambled_1000579 056579 056+0.00 %+0.00 %
scc::scc::tarjan::scrambled_40002 316 4972 316 497+0.00 %+0.00 %
scc::scc::tarjan::small_cycles_1000905 658905 658+0.00 %+0.00 %
scc::scc::tarjan::small_cycles_40003 544 9013 544 901+0.00 %+0.00 %
scc::scc::tarjan_from::fragmented_1000242 530242 530+0.00 %+0.00 %
scc::scc::tarjan_from::fragmented_4000907 762907 762+0.00 %+0.00 %
scc::scc::tarjan_from::path_10001 921 1581 921 158+0.00 %+0.00 %
scc::scc::tarjan_from::path_40007 545 5847 545 584+0.00 %+0.00 %
scc::scc::tarjan_from::ring_1000503 645503 645+0.00 %+0.00 %
scc::scc::tarjan_from::scrambled_1000570 602570 602+0.00 %+0.00 %
storage::storage::add_edges_dense::k2_50001 356 4011 356 401+0.00 %+0.00 %
storage::storage::add_edges_dense::k8_50004 247 0594 247 059+0.00 %+0.00 %
storage::storage::add_edges_linked::k2_50001 402 6701 402 670+0.00 %+0.00 %
storage::storage::add_edges_linked::k8_50006 669 2846 669 284+0.00 %+0.00 %
storage::storage::clone_csr::k2_1000030 59030 590+0.00 %+0.00 %
storage::storage::clone_dense::k2_1000015 11315 113+0.00 %+0.00 %
storage::storage::clone_linked::k2_10000370 824370 824+0.00 %+0.00 %
storage::storage::complete_dense::partial_5000155 628155 628+0.00 %+0.00 %
storage::storage::complete_linked::partial_50001 210 8541 210 854+0.00 %+0.00 %
storage::storage::edges_from_all_states_csr::k2_200072 04772 047+0.00 %+0.00 %
storage::storage::edges_from_all_states_csr::k8_2000108 047108 047+0.00 %+0.00 %
storage::storage::edges_from_all_states_dense::k2_2000102 043102 043+0.00 %+0.00 %
storage::storage::edges_from_all_states_dense::k8_2000140 043140 043+0.00 %+0.00 %
storage::storage::edges_from_all_states_linked::k2_200068 03368 033+0.00 %+0.00 %
storage::storage::edges_from_all_states_linked::k8_2000152 033152 033+0.00 %+0.00 %
storage::storage::minimize_csr::redundant_20004 633 7564 633 756+0.00 %+0.00 %
storage::storage::minimize_csr::redundant_800018 159 86118 159 861+0.00 %+0.00 %
storage::storage::minimize_dense::redundant_20004 676 9104 676 910+0.00 %+0.00 %
storage::storage::minimize_dense::redundant_800018 364 89518 364 895+0.00 %+0.00 %
storage::storage::minimize_linked::redundant_20004 505 0384 505 038+0.00 %+0.00 %
storage::storage::minimize_linked::redundant_800017 714 72117 714 721+0.00 %+0.00 %
storage::storage::overwrite_edges_dense::k2_50002 126 4312 126 431+0.00 %+0.00 %
storage::storage::overwrite_edges_linked::k2_50003 790 7753 790 775+0.00 %+0.00 %
storage::storage::predecessors_of_all_states_csr::k2_2000102 046102 046+0.00 %+0.00 %
storage::storage::predecessors_of_all_states_dense::k2_2000290 369290 369+0.00 %+0.00 %
storage::storage::predecessors_of_all_states_linked::k2_200068 03368 033+0.00 %+0.00 %
storage::storage::product_successor_csr::n20064 09664 096+0.00 %+0.00 %
storage::storage::product_successor_dense::n20073 08773 087+0.00 %+0.00 %
storage::storage::product_successor_linked::n20046 54746 547+0.00 %+0.00 %
storage::storage::reachable_states_csr::k2_80001 101 1151 101 115+0.00 %+0.00 %
storage::storage::reachable_states_dense::k2_80001 133 6991 133 699+0.00 %+0.00 %
storage::storage::reachable_states_linked::k2_8000924 279924 279+0.00 %+0.00 %
storage::storage::remove_state_dense::k2_100001 072 2121 072 212+0.00 %+0.00 %
storage::storage::remove_state_linked::k2_100001 094 0121 094 012+0.00 %+0.00 %
storage::storage::run_many_words_csr::count_1000551 036551 036+0.00 %+0.00 %
storage::storage::run_many_words_dense::count_1000558 044558 044+0.00 %+0.00 %
storage::storage::run_many_words_linked::count_1000481 044481 044+0.00 %+0.00 %
storage::storage::run_one_word_csr::len_1000002 600 0542 600 054+0.00 %+0.00 %
storage::storage::run_one_word_dense::len_1000002 800 0502 800 050+0.00 %+0.00 %
storage::storage::run_one_word_linked::len_1000002 350 0392 350 039+0.00 %+0.00 %
storage::storage::successor_all_pairs_csr::k2_2000122 242122 242+0.00 %+0.00 %
storage::storage::successor_all_pairs_csr::k8_2000702 302702 302+0.00 %+0.00 %
storage::storage::successor_all_pairs_dense::k2_2000162 237162 237+0.00 %+0.00 %
storage::storage::successor_all_pairs_dense::k8_2000814 297814 297+0.00 %+0.00 %
storage::storage::successor_all_pairs_linked::k2_2000108 236108 236+0.00 %+0.00 %
storage::storage::successor_all_pairs_linked::k8_2000750 296750 296+0.00 %+0.00 %
ts::ts::builder_with_edges::k2_500081 357 91581 362 972+0.01 %+0.00 %
ts::ts::canonical_naming::scrambled_15003 178 5993 144 090-1.09 %+0.00 %
ts::ts::canonical_naming::scrambled_500984 074972 576-1.17 %+0.00 %
ts::ts::clone::scrambled_10000371 163371 163+0.00 %+0.00 %
ts::ts::clone::scrambled_400001 481 4001 481 400+0.00 %+0.00 %
ts::ts::collect_dts::scrambled_100003 465 1493 455 128-0.29 %+0.00 %
ts::ts::collect_dts::scrambled_4000013 799 62813 759 607-0.29 %+0.00 %
ts::ts::determinize::nth_from_end_104 437 3974 414 320-0.52 %+0.00 %
ts::ts::determinize::nth_from_end_1219 671 28619 628 006-0.22 %+0.00 %
ts::ts::determinize::nth_from_end_81 016 0701 007 268-0.87 %+0.00 %
ts::ts::edges_from_all_states::scrambled_200072 66972 669+0.00 %+0.00 %
ts::ts::edges_from_all_states::scrambled_k8_2000156 601156 601+0.00 %+0.00 %
ts::ts::into_dfa::scrambled_100003 565 1663 635 166+1.96 %+0.00 %
ts::ts::matching_product_successor_index::n20075 66275 662+0.00 %+0.00 %
ts::ts::minimal_representative_of_all_states::scrambled_1002 102 8112 102 811+0.00 %+0.00 %
ts::ts::minimal_representative_of_all_states::scrambled_30016 917 26116 917 261+0.00 %+0.00 %
ts::ts::minimal_representatives::scrambled_20003 386 0403 386 040+0.00 %+0.00 %
ts::ts::minimal_representatives::scrambled_800014 908 24914 908 249+0.00 %+0.00 %
ts::ts::predecessors_of_all_states::scrambled_200072 66972 669+0.00 %+0.00 %
ts::ts::predecessors_of_all_states::scrambled_8000288 601288 601+0.00 %+0.00 %
ts::ts::reachable_states::scrambled_2000290 091290 091+0.00 %+0.00 %
ts::ts::reachable_states::scrambled_80001 148 6961 148 696+0.00 %+0.00 %
ts::ts::remove_edges_from_matching::scrambled_10000567 234566 234-0.18 %+0.00 %
ts::ts::remove_edges_from_matching::scrambled_40000897 234896 234-0.11 %+0.00 %
ts::ts::remove_state::scrambled_100001 094 0121 091 012-0.27 %+0.00 %
ts::ts::remove_state::scrambled_400001 424 0121 421 012-0.21 %+0.00 %
ts::ts::scc_minimal_representatives::small_cycles_100219 916219 042-0.40 %+0.00 %
ts::ts::scc_minimal_representatives::small_cycles_4001 246 1221 242 475-0.29 %+0.00 %
ts::ts::subset_construction::nth_from_end_1010 160 16810 228 206+0.67 %+0.00 %
ts::ts::subset_construction::nth_from_end_1244 980 93445 253 025+0.60 %+0.00 %
ts::ts::subset_construction::nth_from_end_82 216 4732 233 506+0.77 %+0.00 %
ts::ts::successor_index_all_pairs::scrambled_2000108 863108 863+0.00 %+0.00 %
ts::ts::successor_index_all_pairs::scrambled_k8_2000750 855750 855+0.00 %+0.00 %

Wall-clock times

Criterion means of a single run. They depend on the machine and are not compared; use the instruction counts above for that.

benchmarkbasehead
collect/clone/1000004.28 ms (4.19 ms … 4.36 ms)4.11 ms (4.05 ms … 4.18 ms)
collect/clone/100000042.35 ms (42.32 ms … 42.39 ms)40.37 ms (39.55 ms … 41.20 ms)
collect/collect_owned/1000008.32 ms (8.14 ms … 8.50 ms)7.88 ms (7.86 ms … 7.91 ms)
collect/collect_owned/100000090.86 ms (88.52 ms … 93.21 ms)82.06 ms (81.94 ms … 82.18 ms)
collect/collect_ref/1000005.28 ms (5.26 ms … 5.30 ms)5.09 ms (4.95 ms … 5.24 ms)
collect/collect_ref/100000084.37 ms (83.43 ms … 85.31 ms)86.68 ms (85.58 ms … 87.78 ms)
collect/map_edge_colors_collect_owned/1000008.01 ms (7.87 ms … 8.15 ms)8.51 ms (8.27 ms … 8.74 ms)
collect/map_edge_colors_collect_owned/100000086.50 ms (84.47 ms … 88.54 ms)83.84 ms (83.71 ms … 83.97 ms)
wallclock/dfa/accepts/word_100000835.68 µs (828.62 µs … 842.75 µs)901.98 µs (900.48 µs … 903.48 µs)
wallclock/dfa/minimize/redundant_40001.44 ms (1.42 ms … 1.45 ms)1.44 ms (1.44 ms … 1.44 ms)
wallclock/dpa/complement/scrambled_40003.74 ms (3.68 ms … 3.81 ms)3.67 ms (3.63 ms … 3.71 ms)
wallclock/dpa/normalized/scrambled_40003.40 ms (3.40 ms … 3.41 ms)3.62 ms (3.53 ms … 3.71 ms)
wallclock/hoa/parse/ltl953.13 µs (949.84 µs … 956.42 µs)943.60 µs (941.86 µs … 945.34 µs)
wallclock/nfa/determinize/nth_from_end_123.80 ms (3.73 ms … 3.87 ms)3.44 ms (3.42 ms … 3.46 ms)
wallclock/nfa/subset_construction/nth_from_end_1210.31 ms (10.27 ms … 10.35 ms)7.98 ms (7.77 ms … 8.18 ms)
wallclock/scc/tarjan/scrambled_100002.28 ms (2.26 ms … 2.30 ms)2.19 ms (2.17 ms … 2.22 ms)

Versus Spot and Oink

dpa/equivalence

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot137.74 µs114.40 µs1.20×– / –yes
hard/l-nba-2spot63.70 µs35.38 µs1.80×– / –yes
hard/m-nba-3spot263.98 µs132.02 µs2.00×– / –yes
ltl/and-gf-3spot10.26 µs12.56 µs0.82×– / –yes
ltl/eh-patterns-0spot27.08 µs20.70 µs1.31×– / –yes
ltl/gf-equiv-3spot25.06 µs34.22 µs0.73×– / –yes
ltl/gf-implies-3spot13.26 µs20.02 µs0.66×– / –yes
ltl/gh-q-3spot57.84 µs27.72 µs2.09×– / –yes
ltl/gh-r-3spot25.76 µs67.44 µs0.38×– / –yes
ltl/go-theta-3spot12.60 µs20.92 µs0.60×– / –yes
ltl/lily-patterns-1spot13.68 µs15.48 µs0.88×– / –yes
ltl/ms-phi-r-1spot12.12 µs37.28 µs0.33×– / –yes
ltl/ms-phi-s-1spot12.24 µs38.78 µs0.32×– / –yes
ltl/rv-counter-3spot30.12 µs12.90 µs2.33×– / –yes
ltl/sb-patterns-0spot14.34 µs13.84 µs1.04×– / –yes
ltl/sejk-k-2spot16.00 µs56.62 µs0.28×– / –yes
ltl/tv-uu-3spot45.80 µs36.44 µs1.26×– / –yes
ltl/u-left-3spot11.76 µs8.20 µs1.43×– / –yes
many_aps/and-gf-12spot172.30 µs65.40 µs2.63×– / –yes
many_aps/gf-equiv-12spot146.40 µs88.16 µs1.66×– / –yes
many_aps/r-right-10spot72.16 µs24.62 µs2.93×– / –yes
many_aps/random-n16-a8-p4-s1spot289.72 µs126.60 µs2.29×– / –yes
many_aps/random-n24-a12-p4-s1spot1.34 ms438.99 µs3.05×– / –yes
many_aps/random-n32-a16-p4-s1spot4.09 ms1.17 ms3.48×– / –yes
random/n128-a2-p6-s1spot352.16 µs207.08 µs1.70×– / –yes
random/n16-a1-p3-s1spot24.18 µs15.44 µs1.57×– / –yes
random/n256-a2-p8-s1spot951.13 µs465.55 µs2.04×– / –yes
random/n64-a2-p4-s1spot151.26 µs96.38 µs1.57×– / –yes

dpa/inclusion

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot75.54 µs59.46 µs1.27×– / –yes
hard/l-nba-2spot38.24 µs22.40 µs1.71×– / –yes
hard/m-nba-3spot161.00 µs81.36 µs1.98×– / –yes
ltl/and-gf-3spot6.58 µs7.88 µs0.84×– / –yes
ltl/eh-patterns-0spot17.28 µs16.84 µs1.03×– / –yes
ltl/gf-equiv-3spot13.72 µs18.70 µs0.73×– / –yes
ltl/gf-implies-3spot7.16 µs11.08 µs0.65×– / –yes
ltl/gh-q-3spot36.94 µs27.08 µs1.36×– / –yes
ltl/gh-r-3spot14.44 µs33.92 µs0.43×– / –yes
ltl/go-theta-3spot8.08 µs15.34 µs0.53×– / –yes
ltl/lily-patterns-1spot9.12 µs13.00 µs0.70×– / –yes
ltl/ms-phi-r-1spot7.16 µs18.72 µs0.38×– / –yes
ltl/ms-phi-s-1spot7.16 µs25.20 µs0.28×– / –yes
ltl/rv-counter-3spot19.42 µs13.84 µs1.40×– / –yes
ltl/sb-patterns-0spot9.00 µs11.72 µs0.77×– / –yes
ltl/sejk-k-2spot9.24 µs27.58 µs0.34×– / –yes
ltl/tv-uu-3spot31.50 µs28.40 µs1.11×– / –yes
ltl/u-left-3spot7.76 µs8.36 µs0.93×– / –yes
many_aps/and-gf-12spot141.98 µs51.38 µs2.76×– / –yes
many_aps/gf-equiv-12spot81.86 µs49.48 µs1.65×– / –yes
many_aps/r-right-10spot52.60 µs22.96 µs2.29×– / –yes
many_aps/random-n16-a8-p4-s1spot240.96 µs95.20 µs2.53×– / –yes
many_aps/random-n24-a12-p4-s1spot1.16 ms381.46 µs3.05×– / –yes
many_aps/random-n32-a16-p4-s1spot3.92 ms1.09 ms3.60×– / –yes
random/n128-a2-p6-s1spot209.44 µs121.80 µs1.72×– / –yes
random/n16-a1-p3-s1spot14.52 µs9.96 µs1.46×– / –yes
random/n256-a2-p8-s1spot535.27 µs289.76 µs1.85×– / –yes
random/n64-a2-p4-s1spot89.98 µs62.06 µs1.45×– / –yes

game/solve

inputreferenceoursreferenceours / referencesize (ours / reference)same result
ladder_10000oink-ppp2.86 ms322.50 µs8.86×10000 / 10000yes
ladder_10000oink-zlk2.86 ms322.50 µs8.86×10000 / 10000yes
ladder_10000spot2.86 ms9.62 ms0.30×10000 / 10000yes
ladder_50000oink-ppp22.76 ms2.18 ms10.46×50000 / 50000yes
ladder_50000oink-zlk22.76 ms2.29 ms9.94×50000 / 50000yes
ladder_50000spot22.76 ms80.02 ms0.28×50000 / 50000yes
random_few_10000oink-ppp7.81 ms1.46 ms5.37×10000 / 10000yes
random_few_10000oink-zlk7.81 ms2.67 ms2.93×10000 / 10000yes
random_few_10000spot7.81 ms21.14 ms0.37×10000 / 10000yes
random_few_50000oink-ppp91.56 ms17.95 ms5.10×50000 / 50000yes
random_few_50000oink-zlk91.56 ms21.24 ms4.31×50000 / 50000yes
random_few_50000spot91.56 ms777.27 ms0.12×50000 / 50000yes
random_many_10000oink-ppp7.18 ms1.60 ms4.49×10000 / 10000yes
random_many_10000oink-zlk7.18 ms2.87 ms2.50×10000 / 10000yes
random_many_10000spot7.18 ms30.90 ms0.23×10000 / 10000yes
random_many_50000oink-ppp75.52 ms33.60 ms2.25×50000 / 50000yes
random_many_50000oink-zlk75.52 ms42.00 ms1.80×50000 / 50000yes
random_many_50000spot75.52 ms718.00 ms0.11×50000 / 50000yes

Log

== head d0b8e812ec08fa062220a1072a42b3d7f03c7a03: building ["scc", "ts", "runs", "dfa", "dpa", "storage", "nondeterministic", "learning", "game"] ["wallclock"]
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
    Finished `bench` profile [optimized] target(s) in 2m 15s
  Executable benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-a84ba41f760e70ed)
  Executable benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-23bb829f074058ad)
  Executable benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-14422f732ec85029)
  Executable benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-ee53c00379271c59)
  Executable benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-f164d05108dbce73)
  Executable benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-85a152aa046a29b4)
  Executable benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-1b22c641fd9d2d89)
  Executable benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-f95e3d4a8c506941)
  Executable benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-4948f6cc6a43d201)
  Executable benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-93fb383f608d3bab)
== head: running scc
    Finished `bench` profile [optimized] target(s) in 0.27s
     Running benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-1b22c641fd9d2d89)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running ts
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-4948f6cc6a43d201)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running runs
    Finished `bench` profile [optimized] target(s) in 0.23s
     Running benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-85a152aa046a29b4)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dfa
    Finished `bench` profile [optimized] target(s) in 0.26s
     Running benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-a84ba41f760e70ed)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dpa
    Finished `bench` profile [optimized] target(s) in 0.24s
     Running benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-23bb829f074058ad)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running storage
    Finished `bench` profile [optimized] target(s) in 0.20s
     Running benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-f95e3d4a8c506941)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running nondeterministic
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-f164d05108dbce73)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running learning
    Finished `bench` profile [optimized] target(s) in 0.23s
     Running benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-ee53c00379271c59)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running game
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-14422f732ec85029)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running wallclock (wall clock)
    Finished `bench` profile [optimized] target(s) in 0.20s
     Running benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-93fb383f608d3bab)
Gnuplot not found, using plotters backend
Benchmarking wallclock/scc/tarjan/scrambled_10000
Benchmarking wallclock/scc/tarjan/scrambled_10000: Analyzing
Benchmarking wallclock/dpa/normalized/scrambled_4000
Benchmarking wallclock/dpa/normalized/scrambled_4000: Analyzing
Benchmarking wallclock/dpa/complement/scrambled_4000
Benchmarking wallclock/dpa/complement/scrambled_4000: Analyzing
Benchmarking wallclock/dfa/minimize/redundant_4000
Benchmarking wallclock/dfa/minimize/redundant_4000: Analyzing
Benchmarking wallclock/dfa/accepts/word_100000
Benchmarking wallclock/dfa/accepts/word_100000: Analyzing
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12: Analyzing
Benchmarking wallclock/nfa/determinize/nth_from_end_12
Benchmarking wallclock/nfa/determinize/nth_from_end_12: Analyzing
Benchmarking wallclock/hoa/parse/ltl
Benchmarking wallclock/hoa/parse/ltl: Analyzing
Benchmarking collect/clone/100000
Benchmarking collect/clone/100000: Analyzing
Benchmarking collect/collect_owned/100000
Benchmarking collect/collect_owned/100000: Analyzing
Benchmarking collect/collect_ref/100000
Benchmarking collect/collect_ref/100000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/100000
Benchmarking collect/map_edge_colors_collect_owned/100000: Analyzing
Benchmarking collect/clone/1000000
Benchmarking collect/clone/1000000: Analyzing
Benchmarking collect/collect_owned/1000000
Benchmarking collect/collect_owned/1000000: Analyzing
Benchmarking collect/collect_ref/1000000
Benchmarking collect/collect_ref/1000000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/1000000
Benchmarking collect/map_edge_colors_collect_owned/1000000: Analyzing
== head: building the Python bindings
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support
🐍 Found CPython 3.13 at /nix/store/vilynzly1pyhawy04ljmrp7n9ffr24qy-python3-3.13.17/bin/python3
📡 Using build options features from pyproject.toml
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
   Compiling automata-py v0.1.0 (/var/lib/automata-bench/work/src-head/python)
    Finished `release` profile [optimized] target(s) in 1m 57s
📦 Built wheel for abi3 Python ≥ 3.11 to /var/lib/automata-bench/work/versus-head/wheels/automatars-0.1.0-cp311-abi3-manylinux_2_34_aarch64.whl
== head: running benches/versus/compare.py (Spot, Oink)
dpa/equivalence/hard/l-dsa-3 vs spot: 0.138 ms / 0.114 ms
dpa/equivalence/hard/l-nba-2 vs spot: 0.064 ms / 0.035 ms
dpa/equivalence/hard/m-nba-3 vs spot: 0.264 ms / 0.132 ms
dpa/equivalence/ltl/and-gf-3 vs spot: 0.010 ms / 0.013 ms
dpa/equivalence/ltl/eh-patterns-0 vs spot: 0.027 ms / 0.021 ms
dpa/equivalence/ltl/gf-equiv-3 vs spot: 0.025 ms / 0.034 ms
dpa/equivalence/ltl/gf-implies-3 vs spot: 0.013 ms / 0.020 ms
dpa/equivalence/ltl/gh-q-3 vs spot: 0.058 ms / 0.028 ms
dpa/equivalence/ltl/gh-r-3 vs spot: 0.026 ms / 0.067 ms
dpa/equivalence/ltl/go-theta-3 vs spot: 0.013 ms / 0.021 ms
dpa/equivalence/ltl/lily-patterns-1 vs spot: 0.014 ms / 0.015 ms
dpa/equivalence/ltl/ms-phi-r-1 vs spot: 0.012 ms / 0.037 ms
dpa/equivalence/ltl/ms-phi-s-1 vs spot: 0.012 ms / 0.039 ms
dpa/equivalence/ltl/rv-counter-3 vs spot: 0.030 ms / 0.013 ms
dpa/equivalence/ltl/sb-patterns-0 vs spot: 0.014 ms / 0.014 ms
dpa/equivalence/ltl/sejk-k-2 vs spot: 0.016 ms / 0.057 ms
dpa/equivalence/ltl/tv-uu-3 vs spot: 0.046 ms / 0.036 ms
dpa/equivalence/ltl/u-left-3 vs spot: 0.012 ms / 0.008 ms
dpa/equivalence/many_aps/and-gf-12 vs spot: 0.172 ms / 0.065 ms
dpa/equivalence/many_aps/gf-equiv-12 vs spot: 0.146 ms / 0.088 ms
dpa/equivalence/many_aps/r-right-10 vs spot: 0.072 ms / 0.025 ms
dpa/equivalence/many_aps/random-n16-a8-p4-s1 vs spot: 0.290 ms / 0.127 ms
dpa/equivalence/many_aps/random-n24-a12-p4-s1 vs spot: 1.338 ms / 0.439 ms
dpa/equivalence/many_aps/random-n32-a16-p4-s1 vs spot: 4.086 ms / 1.173 ms
dpa/equivalence/random/n128-a2-p6-s1 vs spot: 0.352 ms / 0.207 ms
dpa/equivalence/random/n16-a1-p3-s1 vs spot: 0.024 ms / 0.015 ms
dpa/equivalence/random/n256-a2-p8-s1 vs spot: 0.951 ms / 0.466 ms
dpa/equivalence/random/n64-a2-p4-s1 vs spot: 0.151 ms / 0.096 ms
dpa/inclusion/hard/l-dsa-3 vs spot: 0.076 ms / 0.059 ms
dpa/inclusion/hard/l-nba-2 vs spot: 0.038 ms / 0.022 ms
dpa/inclusion/hard/m-nba-3 vs spot: 0.161 ms / 0.081 ms
dpa/inclusion/ltl/and-gf-3 vs spot: 0.007 ms / 0.008 ms
dpa/inclusion/ltl/eh-patterns-0 vs spot: 0.017 ms / 0.017 ms
dpa/inclusion/ltl/gf-equiv-3 vs spot: 0.014 ms / 0.019 ms
dpa/inclusion/ltl/gf-implies-3 vs spot: 0.007 ms / 0.011 ms
dpa/inclusion/ltl/gh-q-3 vs spot: 0.037 ms / 0.027 ms
dpa/inclusion/ltl/gh-r-3 vs spot: 0.014 ms / 0.034 ms
dpa/inclusion/ltl/go-theta-3 vs spot: 0.008 ms / 0.015 ms
dpa/inclusion/ltl/lily-patterns-1 vs spot: 0.009 ms / 0.013 ms
dpa/inclusion/ltl/ms-phi-r-1 vs spot: 0.007 ms / 0.019 ms
dpa/inclusion/ltl/ms-phi-s-1 vs spot: 0.007 ms / 0.025 ms
dpa/inclusion/ltl/rv-counter-3 vs spot: 0.019 ms / 0.014 ms
dpa/inclusion/ltl/sb-patterns-0 vs spot: 0.009 ms / 0.012 ms
dpa/inclusion/ltl/sejk-k-2 vs spot: 0.009 ms / 0.028 ms
dpa/inclusion/ltl/tv-uu-3 vs spot: 0.032 ms / 0.028 ms
dpa/inclusion/ltl/u-left-3 vs spot: 0.008 ms / 0.008 ms
dpa/inclusion/many_aps/and-gf-12 vs spot: 0.142 ms / 0.051 ms
dpa/inclusion/many_aps/gf-equiv-12 vs spot: 0.082 ms / 0.049 ms
dpa/inclusion/many_aps/r-right-10 vs spot: 0.053 ms / 0.023 ms
dpa/inclusion/many_aps/random-n16-a8-p4-s1 vs spot: 0.241 ms / 0.095 ms
dpa/inclusion/many_aps/random-n24-a12-p4-s1 vs spot: 1.163 ms / 0.381 ms
dpa/inclusion/many_aps/random-n32-a16-p4-s1 vs spot: 3.922 ms / 1.090 ms
dpa/inclusion/random/n128-a2-p6-s1 vs spot: 0.209 ms / 0.122 ms
dpa/inclusion/random/n16-a1-p3-s1 vs spot: 0.015 ms / 0.010 ms
dpa/inclusion/random/n256-a2-p8-s1 vs spot: 0.535 ms / 0.290 ms
dpa/inclusion/random/n64-a2-p4-s1 vs spot: 0.090 ms / 0.062 ms
game/solve/random_few_10000 vs spot: 7.807 ms / 21.135 ms
game/solve/random_few_10000 vs oink-zlk: 7.807 ms / 2.668 ms
game/solve/random_few_10000 vs oink-ppp: 7.807 ms / 1.455 ms
game/solve/random_few_50000 vs spot: 91.558 ms / 777.269 ms
game/solve/random_few_50000 vs oink-zlk: 91.558 ms / 21.244 ms
game/solve/random_few_50000 vs oink-ppp: 91.558 ms / 17.954 ms
game/solve/random_many_10000 vs spot: 7.182 ms / 30.897 ms
game/solve/random_many_10000 vs oink-zlk: 7.182 ms / 2.869 ms
game/solve/random_many_10000 vs oink-ppp: 7.182 ms / 1.598 ms
game/solve/random_many_50000 vs spot: 75.521 ms / 717.995 ms
game/solve/random_many_50000 vs oink-zlk: 75.521 ms / 42.003 ms
game/solve/random_many_50000 vs oink-ppp: 75.521 ms / 33.605 ms
game/solve/ladder_10000 vs spot: 2.857 ms / 9.615 ms
game/solve/ladder_10000 vs oink-zlk: 2.857 ms / 0.323 ms
game/solve/ladder_10000 vs oink-ppp: 2.857 ms / 0.323 ms
game/solve/ladder_50000 vs spot: 22.764 ms / 80.024 ms
game/solve/ladder_50000 vs oink-zlk: 22.764 ms / 2.291 ms
game/solve/ladder_50000 vs oink-ppp: 22.764 ms / 2.175 ms
== head: 266 benchmarks, 16 wall-clock measurements, 74 comparisons