automata benchmarks

History · Versus Spot and Oink

main · 1bf383b2

Benchmarks: fixtures with many atomic propositions (#156, PR A) (#160)

status done · queued 2026-11-09 23:53 · started 2026-11-09 23:53 · finished 2026-11-10 00:04

· · rustc 1.95.0 (59807616e 2026-04-14) (built from a source tarball) · valgrind-3.26.0 · pinned to cpu 1

Results

benchmarkinstructionsest. cyclesL1 hitsLL hitsRAM hitsheap bytesallocations
dfa::dfa::accepts_many_words::scrambled_1000343 072435 476428 9411 2231200
dfa::dfa::complement::scrambled_1000366 068584 230566 3353 39726139 10419
dfa::dfa::complement::scrambled_40001 435 8732 309 0852 223 22516 99725556 89623
dfa::dfa::equivalent::deep_difference_10061 66797 66490 4794419916 19625
dfa::dfa::equivalent::deep_difference_400212 158336 646314 2011 85037765 36433
dfa::dfa::equivalent::different_1009 04715 37112 5717791 80412
dfa::dfa::equivalent::different_4009 08815 57212 52777761 80412
dfa::dfa::equivalent::minimized_10035 84954 44551 890105587 75617
dfa::dfa::equivalent::minimized_400120 587182 449176 6997236131 58023
dfa::dfa::give_word::empty_10001 872 6132 558 7332 531 0284 92588152 8643 738
dfa::dfa::give_word::empty_40006 930 8239 657 0819 554 28620 05572679 80815 828
dfa::dfa::give_word::far_1000164 342231 691227 0315475528 176394
dfa::dfa::give_word::far_4000946 9821 329 4981 307 3883 92571148 4642 233
dfa::dfa::intersection::scrambled_1003 719 7275 957 6185 779 01312 0683 379428 08842
dfa::dfa::intersection::scrambled_20014 824 89124 130 02222 990 72797 30918 6501 708 12049
dfa::dfa::intersection_is_empty::empty_10059 26694 07387 3733118715 70020
dfa::dfa::intersection_is_empty::empty_200109 269175 052161 72761429331 58023
dfa::dfa::is_empty_language::empty_10001 872 6152 558 7372 531 0324 92588152 8643 738
dfa::dfa::is_empty_language::empty_40006 930 8259 657 0519 554 29120 05571679 80815 828
dfa::dfa::minimize::minimal_10003 435 7345 203 5714 825 98139 7555 1091 135 4922 214
dfa::dfa::minimize::minimal_400013 412 36820 482 39118 822 711204 59918 1914 543 7248 254
dfa::dfa::minimize::redundant_10002 284 6413 452 6693 214 85925 9113 093760 7801 201
dfa::dfa::minimize::redundant_40008 836 89613 488 24712 428 632143 6669 7513 044 9004 241
dfa::dfa::minimize::scrambled_10003 647 9265 556 0415 121 11151 1115 1251 116 0522 217
dfa::dfa::minimize::scrambled_400014 161 22721 944 44919 835 394294 15218 2374 470 1408 261
dfa::dfa::minimize_moore_with_edge_colors::scrambled_10003 646 9365 548 7365 119 14650 0925 1181 116 0522 217
dfa::dfa::minimize_moore_with_edge_colors::scrambled_400014 149 84521 911 33119 806 826300 63417 1814 470 1408 261
dfa::dfa::union::scrambled_1003 719 7125 957 5665 778 97612 0793 377428 08842
dfa::dfa::union::scrambled_20014 824 87624 129 92222 990 71797 29118 6501 708 12049
dpa::dpa::accepted_word::chain_100112 530172 038159 79362341––
dpa::dpa::accepted_word::chain_400379 899573 673538 9582 974567––
dpa::dpa::accepted_word::scrambled_100144 738217 060204 26546359––
dpa::dpa::accepted_word::scrambled_400493 406730 290696 7502 823555––
dpa::dpa::collect_dpa::scrambled_1000301 647500 484481 2893 8045––
dpa::dpa::collect_dpa::scrambled_40001 177 0402 005 9251 882 25516 4041 190––
dpa::dpa::complement::chain_10004 108 5846 261 2595 757 84455 1906 499––
dpa::dpa::complement::chain_400016 700 09325 593 46323 352 178261 95926 614––
dpa::dpa::complement::scrambled_10004 971 4537 468 0126 921 10762 8876 642––
dpa::dpa::complement::scrambled_400020 090 46930 567 03927 901 194344 36526 972––
dpa::dpa::complement_fixtures::ltl1 500 4322 171 1762 115 5416 528657––
dpa::dpa::complement_fixtures::many_aps128 792 342173 017 509172 361 199127 146588––
dpa::dpa::complement_fixtures::synthetic9 444 89913 499 05612 952 021105 102615––
dpa::dpa::complement_without_normalizing::scrambled_1000574 449836 894807 7945 18391––
dpa::dpa::complement_without_normalizing::scrambled_40002 259 8533 331 1313 176 57621 9791 276––
dpa::dpa::equivalent_fixtures::many_aps94 848 718126 854 456126 243 651119 214421––
dpa::dpa::hoa_parse::ltl10 979 63415 945 61315 496 79368 4283 048––
dpa::dpa::hoa_parse::many_aps270 019 542368 922 221367 397 221191 55816 206––
dpa::dpa::hoa_parse::synthetic40 738 32959 603 68257 993 437251 86710 026––
dpa::dpa::hoa_print::ltl49 107 21667 193 97266 562 417122 489546––
dpa::dpa::hoa_print::many_aps753 714 8601 018 965 2531 017 570 603275 122544––
dpa::dpa::hoa_print::synthetic143 208 859197 422 360194 336 435613 454533––
dpa::dpa::included_fixtures::many_aps92 011 612122 808 738122 339 94390 777426––
dpa::dpa::included_in::complement_100269 290408 186389 346856416––
dpa::dpa::included_in::complement_2587 716139 297124 95290397––
dpa::dpa::included_in::complement_50150 987231 660216 495303390––
dpa::dpa::included_in::normalized_100541 195797 974769 9041 617571––
dpa::dpa::included_in::normalized_25184 110274 382259 927322367––
dpa::dpa::included_in::normalized_50312 439459 812442 892521409––
dpa::dpa::language_equivalent::complement_100278 865420 208401 288858418––
dpa::dpa::language_equivalent::complement_2590 241142 689128 184101400––
dpa::dpa::language_equivalent::complement_50155 862237 938222 653313392––
dpa::dpa::language_equivalent::normalized_100989 9051 436 2101 399 2552 638679––
dpa::dpa::language_equivalent::normalized_25325 469474 570458 005422413––
dpa::dpa::language_equivalent::normalized_50579 273835 436816 396728440––
dpa::dpa::map_edge_colors_collect::scrambled_1000307 253507 028486 9933 88817––
dpa::dpa::map_edge_colors_collect::scrambled_40001 197 6462 030 2491 902 26417 1831 202––
dpa::dpa::normalized::chain_10003 836 0305 918 7315 428 29652 9586 447––
dpa::dpa::normalized::chain_400015 611 53924 231 77122 036 141253 22026 558––
dpa::dpa::normalized::scrambled_10004 698 7017 125 1076 591 23760 6446 590––
dpa::dpa::normalized::scrambled_400019 001 71729 205 54226 584 692335 75826 916––
dpa::dpa::normalized::scrambled_k4_100025 720 47936 579 26834 425 133379 7907 291––
dpa::dpa::normalized_fixtures::ltl1 409 4552 044 2621 998 1725 340554––
dpa::dpa::normalized_fixtures::many_aps129 810 449174 473 178173 829 593125 875406––
dpa::dpa::normalized_fixtures::synthetic9 128 77713 095 79712 571 462100 877570––
dpa::dpa::prefix_partition::chain_163 310 4145 029 2744 961 0391 9781 667––
dpa::dpa::prefix_partition::chain_8853 8681 309 4521 277 922412842––
dpa::dpa::prefix_partition::scrambled_163 700 1155 664 9965 575 4564 0691 977––
dpa::dpa::prefix_partition::scrambled_81 019 2211 559 9221 527 042570858––
dpa::dpa::rejected_word::chain_100937 9461 371 6841 328 3792 963814––
dpa::dpa::rejected_word::chain_4003 643 6015 469 7665 168 51648 0981 736––
dpa::dpa::rejected_word::scrambled_100247 652384 336350 301528897––
dpa::dpa::rejected_word::scrambled_400934 6861 437 8701 324 06010 2321 790––
dpa::dpa::sccs::chain_1000487 826762 432686 8276 8471 182––
dpa::dpa::sccs::chain_40001 876 5392 968 3892 642 26429 9105 045––
dpa::dpa::sccs::scrambled_1000554 783860 514784 1698 2411 004––
dpa::dpa::sccs::scrambled_40002 205 4093 495 9603 116 68546 2944 223––
dpa::dpa::witness_colors::chain_1004 743 8786 717 3956 594 7608 9032 232––
dpa::dpa::witness_colors::chain_40046 530 82565 834 98864 234 348193 98118 021––
dpa::dpa::witness_colors::scrambled_1004 128 7225 851 2685 756 1189 3211 387––
dpa::dpa::witness_colors::scrambled_40031 854 84645 091 47844 267 468145 8882 702––
dpa::dpa::witness_inequivalence::complement_100278 864420 219401 284861418––
dpa::dpa::witness_inequivalence::complement_2590 240142 672128 18797400––
dpa::dpa::witness_inequivalence::complement_50155 861237 941222 651314392––
dpa::dpa::witness_not_subset_of::complement_100269 289408 113389 348855414––
dpa::dpa::witness_not_subset_of::complement_2587 715139 262124 95290396––
dpa::dpa::witness_not_subset_of::complement_50150 986231 633216 493305389––
dpa::dpa::witness_not_subset_of::normalized_100541 194797 935769 9051 616570––
dpa::dpa::witness_not_subset_of::normalized_25184 109274 339259 929320366––
dpa::dpa::witness_not_subset_of::normalized_50312 438459 815442 890522409––
dpa::dpa::witnesses_fixtures::ltl2 245 9063 223 6863 156 7167 500842––
dpa::dpa::witnesses_fixtures::synthetic19 746 17730 076 83929 470 064115 216877––
game::game::tangle_learning::few_priorities_10002 681 1744 034 0063 684 53135 9244 853745 4001 292
game::game::tangle_learning::few_priorities_40008 074 12612 626 42011 018 890194 96718 0772 407 4163 266
game::game::tangle_learning::ladder_10001 178 8141 829 6121 626 13212 4444 036377 868118
game::game::tangle_learning::ladder_40004 406 5837 080 7766 015 791110 85314 5921 511 668225
game::game::tangle_learning::many_priorities_10002 009 4443 025 8582 742 66325 5454 442566 440848
game::game::tangle_learning::many_priorities_40006 730 56210 478 3529 098 317166 24015 6812 187 3441 620
game::game::tangle_learning_min_odd::few_priorities_10001 520 4012 316 1212 075 52626 3563 109525 072189
game::game::tangle_learning_min_odd::few_priorities_40007 998 15112 324 14710 820 237215 97012 1162 560 0641 933
game::game::zielonka::few_priorities_10001 126 2021 709 1951 520 52512 4643 610356 67695
game::game::zielonka::few_priorities_40003 470 2575 583 7434 684 11381 69514 0331 278 56484
game::game::zielonka::ladder_10005 134 2847 016 1796 609 65950 0984 458499 404182
game::game::zielonka::ladder_400019 816 37627 232 08725 422 837246 14716 5291 998 868212
game::game::zielonka::many_priorities_10004 103 4375 602 1455 264 31538 6984 124764 940339
game::game::zielonka::many_priorities_400012 909 79117 933 44416 554 599176 13814 2332 552 644288
game::game::zielonka_min_odd::few_priorities_10002 139 0593 023 2732 800 56823 7582 969512 012195
game::game::zielonka_min_odd::few_priorities_40006 302 0649 279 6238 309 198131 0439 0061 705 700158
learning::learning::lstar_mealy::counting_161 787 0912 435 5912 389 8967111 204154 5042 633
learning::learning::lstar_mealy::counting_8478 063679 813646 49822392040 472862
learning::learning::sprout_parity::inf_ab_1001 272 7751 856 3531 793 4282 2391 478200 4161 872
learning::learning::sprout_parity::inf_ab_4002 614 1053 794 5643 677 6699 3162 009405 0403 649
nondeterministic::nondeterministic::fixture_accepts::ltl822 0891 281 7661 217 8318 363632––
nondeterministic::nondeterministic::nba_accepts::nba_400125 268 484185 045 746174 141 0161 657 44474 786––
nondeterministic::nondeterministic::nba_witness::nba_400014 586 04421 412 59820 397 658138 4279 223––
nondeterministic::nondeterministic::nca_accepts::nca_400266 215 466395 064 980369 412 6154 559 28781 598––
nondeterministic::nondeterministic::nca_witness::nca_400019 590 60929 462 93927 306 869325 51415 100––
nondeterministic::nondeterministic::nma_witness::nma_100025 727 43037 903 66235 879 082371 4984 774––
nondeterministic::nondeterministic::npa_accepts::empty_npa_400492 867 078734 421 679686 766 9348 907 94289 001––
nondeterministic::nondeterministic::npa_accepts::npa_10030 410 45844 530 78642 655 071286 34812 685––
nondeterministic::nondeterministic::npa_accepts::npa_400125 278 162185 056 942174 150 6671 657 76774 784––
nondeterministic::nondeterministic::npa_witness::empty_npa_10009 115 02413 456 66012 720 745123 5093 382––
nondeterministic::nondeterministic::npa_witness::empty_npa_400036 996 51755 319 49551 353 295657 31419 418––
nondeterministic::nondeterministic::npa_witness::npa_10002 471 0533 673 9233 507 44820 7511 792––
nondeterministic::nondeterministic::npa_witness::npa_400020 162 80429 311 81628 099 431177 9099 224––
nondeterministic::nondeterministic::nra_witness::nra_400020 617 67931 391 26928 841 099337 19024 692––
runs::runs::dfa_accepts_many_words::count_10004 303 2135 381 9615 357 6464 7931000
runs::runs::dfa_accepts_one_word::len_1003 4924 7984 39838600
runs::runs::dfa_accepts_one_word::len_10000266 070329 527325 827698600
runs::runs::dfa_accepts_one_word::len_1000002 650 7313 276 8253 244 7106 3251400
runs::runs::dfa_reached_state::count_10003 570 2114 404 9584 380 6434 7931000
runs::runs::dpa_accepts_lassos::long9 150 74511 671 38511 646 5653 144260724 664500
runs::runs::dpa_accepts_lassos::short406 490540 876535 5863729826 236472
runs::runs::dpa_recurrent_edge_colors::long14 635 82618 270 10918 215 5046 455638922 1442 090
runs::runs::dpa_recurrent_edge_colors::short1 098 1071 482 4881 469 2531 590151168 6841 708
runs::runs::dpa_recurrent_states::long11 749 19315 694 62915 662 6943 944349788 104860
runs::runs::dpa_recurrent_states::short589 492792 388783 93378812981 996782
runs::runs::dpa_visited_sequences::long8 738 26312 087 72612 050 5815 0633382 190 780850
runs::runs::dpa_visited_sequences::short370 678511 671506 1616266829 986545
runs::runs::fixture_accepts_lassos::ltl9 167 91813 375 75013 349 2804 426124––
runs::runs::fixture_accepts_lassos::synthetic4 648 8586 847 5406 775 70013 458130––
scc::scc::scc_queries::small_cycles_1000906 6141 433 1251 272 0209 4223 257261 132559
scc::scc::scc_queries::small_cycles_40003 548 8435 657 9634 973 84350 91312 2731 045 3962 071
scc::scc::tarjan::back_edges_1000512 037837 644726 6995 4732 388170 27245
scc::scc::tarjan::back_edges_40001 987 5513 267 0552 821 29526 4538 957680 63253
scc::scc::tarjan::binary_tree_1000571 155892 378815 6134 0621 613105 76039
scc::scc::tarjan::binary_tree_40002 290 3833 582 1503 276 31521 1275 720419 51245
scc::scc::tarjan::fragmented_1000658 4671 046 449922 6449 6482 159146 084121
scc::scc::tarjan::fragmented_40002 534 0394 125 4283 544 56362 7917 626569 852167
scc::scc::tarjan::grid_1024596 018971 686847 6817 7282 439171 16045
scc::scc::tarjan::grid_40962 378 4084 000 9063 363 60163 6709 113684 18453
scc::scc::tarjan::path_10001 929 0573 028 1582 729 48823 1875 221455 2242 064
scc::scc::tarjan::path_40007 632 83212 001 28310 804 66398 32320 1431 821 5368 076
scc::scc::tarjan::ring_1000512 037841 284725 7896 3832 388170 27245
scc::scc::tarjan::ring_40001 987 5513 284 2992 816 98430 7648 957680 63253
scc::scc::tarjan::scrambled_1000578 994936 142823 9526 9542 212170 27245
scc::scc::tarjan::scrambled_40002 316 4213 790 8003 296 68041 8728 136680 63253
scc::scc::tarjan::small_cycles_1000905 5961 431 7741 270 7349 4303 254261 132559
scc::scc::tarjan::small_cycles_40003 544 8255 650 3504 969 43550 29312 2701 045 3962 071
scc::scc::tarjan_from::fragmented_1000242 363394 053344 6482 5241 05174 98564
scc::scc::tarjan_from::fragmented_4000907 5531 491 7711 296 37615 9233 308299 62578
scc::scc::tarjan_from::path_10001 920 9703 017 7372 721 88223 0375 162468 9932 091
scc::scc::tarjan_from::path_40007 545 35411 928 15210 697 327106 73219 9191 876 7538 109
scc::scc::tarjan_from::ring_1000503 457836 369717 7146 2522 497184 04172
scc::scc::tarjan_from::scrambled_1000570 414929 847816 2226 4782 321184 04172
storage::storage::add_edges_dense::k2_50001 356 4012 030 6561 985 3414 940589180 20037
storage::storage::add_edges_dense::k8_50004 247 0596 412 0746 225 38917 8352 786671 72840
storage::storage::add_edges_linked::k2_50001 402 6702 420 4772 258 38214 2262 5991 113 96026
storage::storage::add_edges_linked::k8_50006 669 28411 260 26110 330 81175 81515 7253 866 49628
storage::storage::clone_csr::k2_1000030 590183 48052 6254 1703 143230 0126
storage::storage::clone_dense::k2_1000015 11381 33425 8042 2511 265110 0084
storage::storage::clone_linked::k2_10000370 822901 735529 79512 4668 846680 0083
storage::storage::complete_dense::partial_5000155 627209 425202 330908731364
storage::storage::complete_linked::partial_50001 210 8541 718 7051 629 4556 9511 55765 67218
storage::storage::edges_from_all_states_csr::k2_200072 04777 21475 859222700
storage::storage::edges_from_all_states_csr::k8_2000108 047116 392111 0721 008800
storage::storage::edges_from_all_states_dense::k2_2000102 043119 287117 842233800
storage::storage::edges_from_all_states_dense::k8_2000140 043160 563155 0381 0351000
storage::storage::edges_from_all_states_linked::k2_200068 03386 72575 9252 132400
storage::storage::edges_from_all_states_linked::k8_2000152 033203 735166 6807 376500
storage::storage::minimize_csr::redundant_20004 617 5826 940 5056 457 91057 8865 5191 505 4202 212
storage::storage::minimize_csr::redundant_800018 095 64327 530 14525 281 710272 64325 2926 023 3008 250
storage::storage::minimize_dense::redundant_20004 660 7366 970 9406 512 97558 1404 7791 522 1562 221
storage::storage::minimize_dense::redundant_800018 300 67727 860 93125 561 001279 02925 8516 090 3408 261
storage::storage::minimize_linked::redundant_20004 488 8646 856 4536 304 65361 9276 9191 522 1562 221
storage::storage::minimize_linked::redundant_800017 650 50327 227 88324 777 018302 85326 7606 090 3408 261
storage::storage::overwrite_edges_dense::k2_50002 126 4333 232 7543 172 4247 915593180 20037
storage::storage::overwrite_edges_linked::k2_50003 790 7766 671 1605 952 01544 77214 1512 031 46427
storage::storage::predecessors_of_all_states_csr::k2_2000102 046121 042117 407678700
storage::storage::predecessors_of_all_states_dense::k2_2000290 369397 967388 2221 6204732 0083
storage::storage::predecessors_of_all_states_linked::k2_200068 03391 69574 6903 366500
storage::storage::product_successor_csr::n20064 09675 54575 16014900
storage::storage::product_successor_dense::n20073 08794 65494 104401000
storage::storage::product_successor_linked::n20046 54760 18757 012579800
storage::storage::reachable_states_csr::k2_80001 101 1151 630 0761 591 8467 4153317 36811
storage::storage::reachable_states_dense::k2_80001 133 6991 684 5391 639 2448 8283317 36811
storage::storage::reachable_states_linked::k2_8000924 2791 485 6351 386 52019 5224317 36811
storage::storage::remove_state_dense::k2_100001 072 2121 565 2441 390 28421 5101 926176 0001 003
storage::storage::remove_state_linked::k2_100001 094 0121 531 4451 492 7357 4204680 0003 000
storage::storage::run_many_words_csr::count_1000551 036653 977644 1621 907800
storage::storage::run_many_words_dense::count_1000558 044690 065680 1651 9101000
storage::storage::run_many_words_linked::count_1000481 044611 149601 1191 957700
storage::storage::run_one_word_csr::len_1000002 600 0543 025 4282 993 8236 272700
storage::storage::run_one_word_dense::len_1000002 800 0503 425 4913 393 8116 273900
storage::storage::run_one_word_linked::len_1000002 350 0392 975 4282 943 7736 296500
storage::storage::successor_all_pairs_csr::k2_2000122 242139 528138 103229881
storage::storage::successor_all_pairs_csr::k8_2000702 302802 770797 3751 0238321
storage::storage::successor_all_pairs_dense::k2_2000162 237193 579192 089235981
storage::storage::successor_all_pairs_dense::k8_2000814 297952 885947 3451 0459321
storage::storage::successor_all_pairs_linked::k2_2000108 236139 124128 1792 140781
storage::storage::successor_all_pairs_linked::k8_2000750 296946 170909 0007 3857321
ts::ts::add_edges_in_order::k2_50001 402 6712 420 3472 258 38714 2272 5951 113 96026
ts::ts::add_edges_in_order::k8_50006 669 28511 260 14710 330 81275 82015 7213 866 49628
ts::ts::add_edges_in_reverse::k2_50001 402 6712 419 5772 258 58714 0262 5961 113 96026
ts::ts::builder_with_edges::k2_500081 367 02398 219 38795 717 447451 6966 9561 739 53245
ts::ts::canonical_naming::scrambled_15003 167 6084 657 7354 467 72021 7982 315463 8085 846
ts::ts::canonical_naming::scrambled_500980 4221 418 5151 374 7053 176798134 3361 831
ts::ts::clone::scrambled_10000371 163902 473530 23812 4838 852680 0083
ts::ts::clone::scrambled_400001 481 4003 817 8822 116 80742 57542 5202 720 0083
ts::ts::collect_dts::scrambled_100003 465 1496 166 1505 606 84537 17110 6702 228 06427
ts::ts::collect_dts::scrambled_4000013 799 62824 683 23422 329 474135 29847 9228 912 73631
ts::ts::determinize::nth_from_end_104 485 5246 644 8696 283 32422 2037 158651 7206 013
ts::ts::determinize::nth_from_end_1219 876 08529 277 12927 766 72999 01029 0102 732 60026 423
ts::ts::determinize::nth_from_end_81 027 3331 524 9111 441 5563 0491 946156 5521 396
ts::ts::edges_from_all_states::scrambled_200072 66991 97980 7742 1571200
ts::ts::edges_from_all_states::scrambled_k8_2000156 601208 928171 4287 4021400
ts::ts::erase_state_colors_collect::scrambled_100003 007 1665 308 2474 748 86237 17310 6722 228 06427
ts::ts::into_dfa::scrambled_100003 565 1656 276 2655 716 87537 18110 6712 228 06427
ts::ts::map_edge_colors_collect::scrambled_100003 008 7685 438 2024 819 54738 45012 1832 490 19227
ts::ts::map_edge_colors_collect::scrambled_4000011 975 39121 772 19419 182 194140 32953 9539 961 29631
ts::ts::matching_product_successor_index::n20075 662111 066106 4666473900
ts::ts::minimal_representative_of_all_states::scrambled_1002 100 8142 967 8562 964 19112103347 5044 679
ts::ts::minimal_representative_of_all_states::scrambled_30016 911 26423 979 43123 966 5111 8421063 116 78438 196
ts::ts::minimal_representatives::scrambled_20003 384 0404 762 1304 715 2358 91067322 9447 709
ts::ts::minimal_representatives::scrambled_800014 900 24920 949 86520 734 91042 592571 464 89632 693
ts::ts::predecessors_of_all_states::scrambled_200072 66996 91579 5403 3911200
ts::ts::predecessors_of_all_states::scrambled_8000288 601389 362313 81215 0191300
ts::ts::reachable_states::scrambled_2000290 091460 935440 0053 605834 3369
ts::ts::reachable_states::scrambled_80001 148 6961 838 5401 739 09019 5195317 36811
ts::ts::remove_edges_from_matching::scrambled_10000566 234795 987771 6474 6583064 0002 000
ts::ts::remove_edges_from_matching::scrambled_40000896 2341 177 4871 126 27210 0333064 0002 000
ts::ts::remove_state::scrambled_100001 091 0121 528 4451 489 7357 4204680 0003 000
ts::ts::remove_state::scrambled_400001 421 0121 912 9091 843 61913 5364680 0003 000
ts::ts::scc_minimal_representatives::small_cycles_100219 944321 099305 5445943644 272309
ts::ts::scc_minimal_representatives::small_cycles_4001 246 3521 775 6451 725 2553 890884354 1481 325
ts::ts::subset_construction::nth_from_end_1010 216 38414 514 16714 019 01245 5097 6461 073 5209 919
ts::ts::subset_construction::nth_from_end_1245 206 09464 008 67361 763 083218 74132 9114 518 97643 857
ts::ts::subset_construction::nth_from_end_82 230 4653 186 9313 078 3816 2052 215257 1042 276
ts::ts::successor_index_all_pairs::scrambled_2000108 863140 297129 0222 1571481
ts::ts::successor_index_all_pairs::scrambled_k8_2000750 855947 282909 7427 40315321

Wall-clock times

Criterion means of a single run. They depend on the machine and are not compared; use the instruction counts above for that.

benchmarkhead
collect/clone/1000003.97 ms (3.95 ms … 3.99 ms)
collect/clone/100000040.52 ms (39.91 ms … 41.14 ms)
collect/collect_owned/1000007.46 ms (7.43 ms … 7.49 ms)
collect/collect_owned/100000090.62 ms (88.64 ms … 92.59 ms)
collect/collect_ref/1000005.44 ms (5.41 ms … 5.48 ms)
collect/collect_ref/100000090.63 ms (88.75 ms … 92.51 ms)
collect/map_edge_colors_collect_owned/1000007.37 ms (7.36 ms … 7.37 ms)
collect/map_edge_colors_collect_owned/100000084.39 ms (83.67 ms … 85.11 ms)
wallclock/dfa/accepts/word_100000937.29 µs (937.18 µs … 937.40 µs)
wallclock/dfa/minimize/redundant_40001.41 ms (1.41 ms … 1.41 ms)
wallclock/dpa/complement/scrambled_40003.60 ms (3.58 ms … 3.63 ms)
wallclock/dpa/normalized/scrambled_40003.41 ms (3.41 ms … 3.42 ms)
wallclock/hoa/parse/ltl1.75 ms (1.74 ms … 1.76 ms)
wallclock/nfa/determinize/nth_from_end_123.43 ms (3.39 ms … 3.46 ms)
wallclock/nfa/subset_construction/nth_from_end_128.85 ms (8.75 ms … 8.95 ms)
wallclock/scc/tarjan/scrambled_100002.58 ms (2.53 ms … 2.64 ms)

Versus Spot and Oink

dpa/equivalence

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot132.08 µs114.12 µs1.16×– / –yes
hard/l-nba-2spot58.08 µs34.52 µs1.68×– / –yes
hard/m-nba-3spot256.82 µs132.98 µs1.93×– / –yes
ltl/and-gf-3spot9.92 µs11.32 µs0.88×– / –yes
ltl/eh-patterns-0spot22.88 µs19.22 µs1.19×– / –yes
ltl/gf-equiv-3spot21.44 µs33.12 µs0.65×– / –yes
ltl/gf-implies-3spot11.04 µs18.34 µs0.60×– / –yes
ltl/gh-q-3spot51.56 µs27.28 µs1.89×– / –yes
ltl/gh-r-3spot23.80 µs67.10 µs0.35×– / –yes
ltl/go-theta-3spot12.16 µs20.12 µs0.60×– / –yes
ltl/lily-patterns-1spot12.72 µs14.96 µs0.85×– / –yes
ltl/ms-phi-r-1spot11.44 µs36.76 µs0.31×– / –yes
ltl/ms-phi-s-1spot11.36 µs37.80 µs0.30×– / –yes
ltl/rv-counter-3spot28.10 µs12.00 µs2.34×– / –yes
ltl/sb-patterns-0spot13.52 µs13.16 µs1.03×– / –yes
ltl/sejk-k-2spot15.08 µs55.36 µs0.27×– / –yes
ltl/tv-uu-3spot41.50 µs36.18 µs1.15×– / –yes
ltl/u-left-3spot10.98 µs7.84 µs1.40×– / –yes
many_aps/and-gf-12spot565.63 µs61.68 µs9.17×– / –yes
many_aps/gf-equiv-12spot134.32 µs89.58 µs1.50×– / –yes
many_aps/r-right-10spot145.96 µs25.40 µs5.75×– / –yes
many_aps/random-n16-a8-p4-s1spot694.21 µs128.78 µs5.39×– / –yes
many_aps/random-n24-a12-p4-s1spot2.95 ms426.93 µs6.92×– / –yes
many_aps/random-n32-a16-p4-s1spot7.08 ms1.23 ms5.76×– / –yes
random/n128-a2-p6-s1spot344.92 µs203.00 µs1.70×– / –yes
random/n16-a1-p3-s1spot22.56 µs15.42 µs1.46×– / –yes
random/n256-a2-p8-s1spot946.25 µs465.19 µs2.03×– / –yes
random/n64-a2-p4-s1spot148.64 µs95.96 µs1.55×– / –yes

dpa/inclusion

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot70.24 µs59.26 µs1.19×– / –yes
hard/l-nba-2spot34.80 µs21.64 µs1.61×– / –yes
hard/m-nba-3spot145.72 µs80.16 µs1.82×– / –yes
ltl/and-gf-3spot5.92 µs7.44 µs0.80×– / –yes
ltl/eh-patterns-0spot14.94 µs16.60 µs0.90×– / –yes
ltl/gf-equiv-3spot12.28 µs17.76 µs0.69×– / –yes
ltl/gf-implies-3spot6.38 µs10.62 µs0.60×– / –yes
ltl/gh-q-3spot33.08 µs28.88 µs1.15×– / –yes
ltl/gh-r-3spot13.68 µs33.80 µs0.40×– / –yes
ltl/go-theta-3spot7.32 µs14.98 µs0.49×– / –yes
ltl/lily-patterns-1spot8.20 µs12.68 µs0.65×– / –yes
ltl/ms-phi-r-1spot6.64 µs18.10 µs0.37×– / –yes
ltl/ms-phi-s-1spot6.52 µs24.28 µs0.27×– / –yes
ltl/rv-counter-3spot18.06 µs13.68 µs1.32×– / –yes
ltl/sb-patterns-0spot8.12 µs11.06 µs0.73×– / –yes
ltl/sejk-k-2spot8.48 µs27.64 µs0.31×– / –yes
ltl/tv-uu-3spot27.70 µs27.88 µs0.99×– / –yes
ltl/u-left-3spot6.92 µs8.00 µs0.86×– / –yes
many_aps/and-gf-12spot545.99 µs51.04 µs10.70×– / –yes
many_aps/gf-equiv-12spot85.60 µs51.30 µs1.67×– / –yes
many_aps/r-right-10spot129.84 µs23.12 µs5.62×– / –yes
many_aps/random-n16-a8-p4-s1spot649.69 µs97.52 µs6.66×– / –yes
many_aps/random-n24-a12-p4-s1spot2.86 ms381.20 µs7.50×– / –yes
many_aps/random-n32-a16-p4-s1spot6.90 ms1.07 ms6.48×– / –yes
random/n128-a2-p6-s1spot202.08 µs121.10 µs1.67×– / –yes
random/n16-a1-p3-s1spot13.16 µs9.44 µs1.39×– / –yes
random/n256-a2-p8-s1spot516.85 µs288.66 µs1.79×– / –yes
random/n64-a2-p4-s1spot85.80 µs61.20 µs1.40×– / –yes

game/solve

inputreferenceoursreferenceours / referencesize (ours / reference)same result
ladder_10000oink-ppp2.79 ms307.00 µs9.08×10000 / 10000yes
ladder_10000oink-zlk2.79 ms306.00 µs9.11×10000 / 10000yes
ladder_10000spot2.79 ms9.64 ms0.29×10000 / 10000yes
ladder_50000oink-ppp21.45 ms2.13 ms10.07×50000 / 50000yes
ladder_50000oink-zlk21.45 ms2.21 ms9.68×50000 / 50000yes
ladder_50000spot21.45 ms70.52 ms0.30×50000 / 50000yes
random_few_10000oink-ppp7.83 ms1.48 ms5.30×10000 / 10000yes
random_few_10000oink-zlk7.83 ms2.76 ms2.83×10000 / 10000yes
random_few_10000spot7.83 ms21.32 ms0.37×10000 / 10000yes
random_few_50000oink-ppp81.24 ms16.97 ms4.79×50000 / 50000yes
random_few_50000oink-zlk81.24 ms20.84 ms3.90×50000 / 50000yes
random_few_50000spot81.24 ms712.15 ms0.11×50000 / 50000yes
random_many_10000oink-ppp7.26 ms1.60 ms4.52×10000 / 10000yes
random_many_10000oink-zlk7.26 ms2.87 ms2.53×10000 / 10000yes
random_many_10000spot7.26 ms30.41 ms0.24×10000 / 10000yes
random_many_50000oink-ppp84.93 ms34.31 ms2.48×50000 / 50000yes
random_many_50000oink-zlk84.93 ms46.33 ms1.83×50000 / 50000yes
random_many_50000spot84.93 ms729.73 ms0.12×50000 / 50000yes

Log

== head 1bf383b20a4b798c37edf559a20e4bdb9ded5c24: building ["scc", "ts", "runs", "dfa", "dpa", "storage", "nondeterministic", "learning", "game"] ["wallclock"]
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
    Finished `bench` profile [optimized] target(s) in 2m 00s
  Executable benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-239ee49a27fc4838)
  Executable benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-ee8d1549550eb250)
  Executable benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-ae02ba3f0e2bb10f)
  Executable benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-b0862ee18af8325a)
  Executable benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-1f493dd18fd6ada5)
  Executable benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-94e3643e0035d2ac)
  Executable benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-960a9d86024978ff)
  Executable benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-3f2fde62ddd01f66)
  Executable benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-df55bed5e680b932)
  Executable benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-00d5b8c4e6931bbf)
== head: running scc
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-960a9d86024978ff)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running ts
    Finished `bench` profile [optimized] target(s) in 0.18s
     Running benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-df55bed5e680b932)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running runs
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-94e3643e0035d2ac)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dfa
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-239ee49a27fc4838)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dpa
    Finished `bench` profile [optimized] target(s) in 0.19s
     Running benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-ee8d1549550eb250)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running storage
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-3f2fde62ddd01f66)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running nondeterministic
    Finished `bench` profile [optimized] target(s) in 0.24s
     Running benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-1f493dd18fd6ada5)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running learning
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-b0862ee18af8325a)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running game
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-ae02ba3f0e2bb10f)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running wallclock (wall clock)
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-00d5b8c4e6931bbf)
Gnuplot not found, using plotters backend
Benchmarking wallclock/scc/tarjan/scrambled_10000
Benchmarking wallclock/scc/tarjan/scrambled_10000: Analyzing
Benchmarking wallclock/dpa/normalized/scrambled_4000
Benchmarking wallclock/dpa/normalized/scrambled_4000: Analyzing
Benchmarking wallclock/dpa/complement/scrambled_4000
Benchmarking wallclock/dpa/complement/scrambled_4000: Analyzing
Benchmarking wallclock/dfa/minimize/redundant_4000
Benchmarking wallclock/dfa/minimize/redundant_4000: Analyzing
Benchmarking wallclock/dfa/accepts/word_100000
Benchmarking wallclock/dfa/accepts/word_100000: Analyzing
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12: Analyzing
Benchmarking wallclock/nfa/determinize/nth_from_end_12
Benchmarking wallclock/nfa/determinize/nth_from_end_12: Analyzing
Benchmarking wallclock/hoa/parse/ltl
Benchmarking wallclock/hoa/parse/ltl: Analyzing
Benchmarking collect/clone/100000
Benchmarking collect/clone/100000: Analyzing
Benchmarking collect/collect_owned/100000
Benchmarking collect/collect_owned/100000: Analyzing
Benchmarking collect/collect_ref/100000
Benchmarking collect/collect_ref/100000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/100000
Benchmarking collect/map_edge_colors_collect_owned/100000: Analyzing
Benchmarking collect/clone/1000000
Benchmarking collect/clone/1000000: Analyzing
Benchmarking collect/collect_owned/1000000
Benchmarking collect/collect_owned/1000000: Analyzing
Benchmarking collect/collect_ref/1000000
Benchmarking collect/collect_ref/1000000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/1000000
Benchmarking collect/map_edge_colors_collect_owned/1000000: Analyzing
== head: building the Python bindings
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support
🐍 Found CPython 3.13 at /nix/store/vilynzly1pyhawy04ljmrp7n9ffr24qy-python3-3.13.17/bin/python3
📡 Using build options features from pyproject.toml
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
   Compiling automata-py v0.1.0 (/var/lib/automata-bench/work/src-head/python)
    Finished `release` profile [optimized] target(s) in 1m 50s
📦 Built wheel for abi3 Python ≥ 3.11 to /var/lib/automata-bench/work/versus-head/wheels/automatars-0.1.0-cp311-abi3-manylinux_2_34_aarch64.whl
== head: running benches/versus/compare.py (Spot, Oink)
dpa/equivalence/hard/l-dsa-3 vs spot: 0.132 ms / 0.114 ms
dpa/equivalence/hard/l-nba-2 vs spot: 0.058 ms / 0.035 ms
dpa/equivalence/hard/m-nba-3 vs spot: 0.257 ms / 0.133 ms
dpa/equivalence/ltl/and-gf-3 vs spot: 0.010 ms / 0.011 ms
dpa/equivalence/ltl/eh-patterns-0 vs spot: 0.023 ms / 0.019 ms
dpa/equivalence/ltl/gf-equiv-3 vs spot: 0.021 ms / 0.033 ms
dpa/equivalence/ltl/gf-implies-3 vs spot: 0.011 ms / 0.018 ms
dpa/equivalence/ltl/gh-q-3 vs spot: 0.052 ms / 0.027 ms
dpa/equivalence/ltl/gh-r-3 vs spot: 0.024 ms / 0.067 ms
dpa/equivalence/ltl/go-theta-3 vs spot: 0.012 ms / 0.020 ms
dpa/equivalence/ltl/lily-patterns-1 vs spot: 0.013 ms / 0.015 ms
dpa/equivalence/ltl/ms-phi-r-1 vs spot: 0.011 ms / 0.037 ms
dpa/equivalence/ltl/ms-phi-s-1 vs spot: 0.011 ms / 0.038 ms
dpa/equivalence/ltl/rv-counter-3 vs spot: 0.028 ms / 0.012 ms
dpa/equivalence/ltl/sb-patterns-0 vs spot: 0.014 ms / 0.013 ms
dpa/equivalence/ltl/sejk-k-2 vs spot: 0.015 ms / 0.055 ms
dpa/equivalence/ltl/tv-uu-3 vs spot: 0.042 ms / 0.036 ms
dpa/equivalence/ltl/u-left-3 vs spot: 0.011 ms / 0.008 ms
dpa/equivalence/many_aps/and-gf-12 vs spot: 0.566 ms / 0.062 ms
dpa/equivalence/many_aps/gf-equiv-12 vs spot: 0.134 ms / 0.090 ms
dpa/equivalence/many_aps/r-right-10 vs spot: 0.146 ms / 0.025 ms
dpa/equivalence/many_aps/random-n16-a8-p4-s1 vs spot: 0.694 ms / 0.129 ms
dpa/equivalence/many_aps/random-n24-a12-p4-s1 vs spot: 2.953 ms / 0.427 ms
dpa/equivalence/many_aps/random-n32-a16-p4-s1 vs spot: 7.078 ms / 1.228 ms
dpa/equivalence/random/n128-a2-p6-s1 vs spot: 0.345 ms / 0.203 ms
dpa/equivalence/random/n16-a1-p3-s1 vs spot: 0.023 ms / 0.015 ms
dpa/equivalence/random/n256-a2-p8-s1 vs spot: 0.946 ms / 0.465 ms
dpa/equivalence/random/n64-a2-p4-s1 vs spot: 0.149 ms / 0.096 ms
dpa/inclusion/hard/l-dsa-3 vs spot: 0.070 ms / 0.059 ms
dpa/inclusion/hard/l-nba-2 vs spot: 0.035 ms / 0.022 ms
dpa/inclusion/hard/m-nba-3 vs spot: 0.146 ms / 0.080 ms
dpa/inclusion/ltl/and-gf-3 vs spot: 0.006 ms / 0.007 ms
dpa/inclusion/ltl/eh-patterns-0 vs spot: 0.015 ms / 0.017 ms
dpa/inclusion/ltl/gf-equiv-3 vs spot: 0.012 ms / 0.018 ms
dpa/inclusion/ltl/gf-implies-3 vs spot: 0.006 ms / 0.011 ms
dpa/inclusion/ltl/gh-q-3 vs spot: 0.033 ms / 0.029 ms
dpa/inclusion/ltl/gh-r-3 vs spot: 0.014 ms / 0.034 ms
dpa/inclusion/ltl/go-theta-3 vs spot: 0.007 ms / 0.015 ms
dpa/inclusion/ltl/lily-patterns-1 vs spot: 0.008 ms / 0.013 ms
dpa/inclusion/ltl/ms-phi-r-1 vs spot: 0.007 ms / 0.018 ms
dpa/inclusion/ltl/ms-phi-s-1 vs spot: 0.007 ms / 0.024 ms
dpa/inclusion/ltl/rv-counter-3 vs spot: 0.018 ms / 0.014 ms
dpa/inclusion/ltl/sb-patterns-0 vs spot: 0.008 ms / 0.011 ms
dpa/inclusion/ltl/sejk-k-2 vs spot: 0.008 ms / 0.028 ms
dpa/inclusion/ltl/tv-uu-3 vs spot: 0.028 ms / 0.028 ms
dpa/inclusion/ltl/u-left-3 vs spot: 0.007 ms / 0.008 ms
dpa/inclusion/many_aps/and-gf-12 vs spot: 0.546 ms / 0.051 ms
dpa/inclusion/many_aps/gf-equiv-12 vs spot: 0.086 ms / 0.051 ms
dpa/inclusion/many_aps/r-right-10 vs spot: 0.130 ms / 0.023 ms
dpa/inclusion/many_aps/random-n16-a8-p4-s1 vs spot: 0.650 ms / 0.098 ms
dpa/inclusion/many_aps/random-n24-a12-p4-s1 vs spot: 2.858 ms / 0.381 ms
dpa/inclusion/many_aps/random-n32-a16-p4-s1 vs spot: 6.901 ms / 1.065 ms
dpa/inclusion/random/n128-a2-p6-s1 vs spot: 0.202 ms / 0.121 ms
dpa/inclusion/random/n16-a1-p3-s1 vs spot: 0.013 ms / 0.009 ms
dpa/inclusion/random/n256-a2-p8-s1 vs spot: 0.517 ms / 0.289 ms
dpa/inclusion/random/n64-a2-p4-s1 vs spot: 0.086 ms / 0.061 ms
game/solve/random_few_10000 vs spot: 7.827 ms / 21.323 ms
game/solve/random_few_10000 vs oink-zlk: 7.827 ms / 2.764 ms
game/solve/random_few_10000 vs oink-ppp: 7.827 ms / 1.478 ms
game/solve/random_few_50000 vs spot: 81.239 ms / 712.146 ms
game/solve/random_few_50000 vs oink-zlk: 81.239 ms / 20.835 ms
game/solve/random_few_50000 vs oink-ppp: 81.239 ms / 16.969 ms
game/solve/random_many_10000 vs spot: 7.256 ms / 30.414 ms
game/solve/random_many_10000 vs oink-zlk: 7.256 ms / 2.873 ms
game/solve/random_many_10000 vs oink-ppp: 7.256 ms / 1.604 ms
game/solve/random_many_50000 vs spot: 84.932 ms / 729.727 ms
game/solve/random_many_50000 vs oink-zlk: 84.932 ms / 46.334 ms
game/solve/random_many_50000 vs oink-ppp: 84.932 ms / 34.306 ms
game/solve/ladder_10000 vs spot: 2.788 ms / 9.637 ms
game/solve/ladder_10000 vs oink-zlk: 2.788 ms / 0.306 ms
game/solve/ladder_10000 vs oink-ppp: 2.788 ms / 0.307 ms
game/solve/ladder_50000 vs spot: 21.447 ms / 70.517 ms
game/solve/ladder_50000 vs oink-zlk: 21.447 ms / 2.215 ms
game/solve/ladder_50000 vs oink-ppp: 21.447 ms / 2.130 ms
== head: 259 benchmarks, 16 wall-clock measurements, 74 comparisons