automata benchmarks

History · Versus Spot and Oink

main · 8ce15b69

PropAlphabet: oxidd BDDs instead of biodivine-lib-bdd (#156, PR C) (#161)

status done · queued 2026-11-10 12:43 · started 2026-11-10 12:43 · finished 2026-11-10 12:53

· · rustc 1.95.0 (59807616e 2026-04-14) (built from a source tarball) · valgrind-3.26.0 · pinned to cpu 1

Results

benchmarkinstructionsest. cyclesL1 hitsLL hitsRAM hitsheap bytesallocations
dfa::dfa::accepts_many_words::scrambled_1000343 072435 464428 9441 2201200
dfa::dfa::complement::scrambled_1000366 068584 238566 3333 39926139 10419
dfa::dfa::complement::scrambled_40001 435 8732 309 0932 223 22316 99925556 89623
dfa::dfa::equivalent::deep_difference_10061 66797 78690 4714920216 19625
dfa::dfa::equivalent::deep_difference_400212 158336 680314 2001 85037865 36433
dfa::dfa::equivalent::different_1009 04715 44312 5688811 80412
dfa::dfa::equivalent::different_4009 08815 64412 52478781 80412
dfa::dfa::equivalent::minimized_10035 84954 51351 888105607 75617
dfa::dfa::equivalent::minimized_400120 587182 487176 6977246231 58023
dfa::dfa::give_word::empty_10001 872 6132 558 6052 531 0304 92784152 8643 738
dfa::dfa::give_word::empty_40006 930 8239 656 9619 554 28620 05968679 80815 828
dfa::dfa::give_word::far_1000164 342231 589227 0345475228 176394
dfa::dfa::give_word::far_4000946 9821 329 3961 307 3913 92568148 4642 233
dfa::dfa::intersection::scrambled_1003 319 7095 217 4015 038 98612 0653 374428 08842
dfa::dfa::intersection::scrambled_20013 224 87321 169 19120 030 86197 14418 6461 708 12049
dfa::dfa::intersection_is_empty::empty_10059 26694 00187 3763018515 70020
dfa::dfa::intersection_is_empty::empty_200109 269175 048161 72861329331 58023
dfa::dfa::is_empty_language::empty_10001 872 6152 558 6772 531 0324 92786152 8643 738
dfa::dfa::is_empty_language::empty_40006 930 8259 656 9659 554 29020 05968679 80815 828
dfa::dfa::minimize::minimal_10003 450 9245 260 2624 882 20739 7715 1201 135 4922 214
dfa::dfa::minimize::minimal_400013 472 60220 707 23819 046 968204 64718 2014 543 7248 254
dfa::dfa::minimize::redundant_10002 292 8073 485 4273 247 03225 9373 106760 7801 201
dfa::dfa::minimize::redundant_40008 869 10613 616 89312 556 903143 6579 7633 044 9004 241
dfa::dfa::minimize::scrambled_10003 670 2565 623 8715 188 59651 1105 1351 116 0522 217
dfa::dfa::minimize::scrambled_400014 250 46922 214 47420 104 804294 19818 2484 470 1408 261
dfa::dfa::minimize_moore_with_edge_colors::scrambled_10003 669 2525 616 6885 186 58350 1115 1301 116 0522 217
dfa::dfa::minimize_moore_with_edge_colors::scrambled_400014 239 07322 181 42820 076 193300 69617 1934 470 1408 261
dfa::dfa::union::scrambled_1003 319 7095 217 4055 038 98512 0663 374428 08842
dfa::dfa::union::scrambled_20013 224 87321 169 15720 030 86297 14418 6451 708 12049
dpa::dpa::accepted_word::chain_100114 174175 422162 92756349––
dpa::dpa::accepted_word::chain_400385 785585 537550 5472 973575––
dpa::dpa::accepted_word::scrambled_100144 718217 848204 73340369––
dpa::dpa::accepted_word::scrambled_400492 612731 552697 6922 817565––
dpa::dpa::collect_dpa::scrambled_1000290 651469 464450 2943 8064––
dpa::dpa::collect_dpa::scrambled_40001 133 0441 881 8971 758 26216 4041 189––
dpa::dpa::complement::chain_10004 101 1576 256 2115 752 47155 1576 513––
dpa::dpa::complement::chain_400016 670 17825 571 92123 330 326261 92326 628––
dpa::dpa::complement::scrambled_10004 960 1657 464 4156 918 01562 7516 647––
dpa::dpa::complement::scrambled_400020 044 50530 552 41427 887 289344 19326 976––
dpa::dpa::complement_fixtures::ltl1 481 8322 154 1142 111 4545 200476––
dpa::dpa::complement_fixtures::many_aps13 922 08920 576 30020 085 67087 3531 539––
dpa::dpa::complement_fixtures::synthetic9 666 53213 875 64413 366 01498 426500––
dpa::dpa::complement_without_normalizing::scrambled_1000565 457809 029779 8045 18095––
dpa::dpa::complement_without_normalizing::scrambled_40002 223 8503 219 2383 064 57821 9721 280––
dpa::dpa::equivalent_fixtures::many_aps21 636 04232 269 83531 594 205122 3231 829––
dpa::dpa::hoa_parse::ltl5 507 5988 383 9968 054 07141 6043 483––
dpa::dpa::hoa_parse::many_aps88 883 054144 042 697126 819 632291 918450 385––
dpa::dpa::hoa_parse::synthetic25 090 68338 164 20136 749 261180 51514 639––
dpa::dpa::hoa_print::ltl19 037 29336 368 57524 186 6855 502347 268––
dpa::dpa::hoa_print::many_aps19 662 96928 400 86828 320 34313 228411––
dpa::dpa::hoa_print::synthetic27 383 65348 636 13536 383 9656 454349 140––
dpa::dpa::included_fixtures::many_aps18 595 18927 913 20127 369 96195 8101 834––
dpa::dpa::included_in::complement_100269 594409 087390 467882406––
dpa::dpa::included_in::complement_2587 754139 251125 14199389––
dpa::dpa::included_in::complement_50151 045231 793216 883322380––
dpa::dpa::included_in::normalized_100542 475802 092774 5671 522569––
dpa::dpa::included_in::normalized_25184 636275 643261 728235364––
dpa::dpa::included_in::normalized_50313 247462 152445 827444403––
dpa::dpa::language_equivalent::complement_100279 169420 971402 421875405––
dpa::dpa::language_equivalent::complement_2590 279142 461128 39690389––
dpa::dpa::language_equivalent::complement_50155 920237 913223 058318379––
dpa::dpa::language_equivalent::normalized_100992 4501 444 6071 408 4772 543669––
dpa::dpa::language_equivalent::normalized_25326 481477 097461 497264408––
dpa::dpa::language_equivalent::normalized_50580 879840 373822 208623430––
dpa::dpa::map_edge_colors_collect::scrambled_1000289 296465 295445 0653 89921––
dpa::dpa::map_edge_colors_collect::scrambled_40001 125 6911 862 4781 734 34817 1841 206––
dpa::dpa::normalized::chain_10003 828 6035 913 7475 422 92252 9246 463––
dpa::dpa::normalized::chain_400015 581 62424 210 35522 014 280253 19026 575––
dpa::dpa::normalized::scrambled_10004 687 4137 121 7126 588 10260 5506 596––
dpa::dpa::normalized::scrambled_400018 955 75329 191 05526 570 775335 59526 923––
dpa::dpa::normalized::scrambled_k4_100025 967 23037 085 68634 932 476379 5497 299––
dpa::dpa::normalized_fixtures::ltl1 417 2072 070 1412 029 9865 098419––
dpa::dpa::normalized_fixtures::many_aps13 662 04020 252 27519 776 59084 6021 505––
dpa::dpa::normalized_fixtures::synthetic9 351 54013 468 34112 987 01693 010465––
dpa::dpa::prefix_partition::chain_163 307 1635 026 9254 954 0602 8761 671––
dpa::dpa::prefix_partition::chain_8853 0541 309 3891 275 914773846––
dpa::dpa::prefix_partition::scrambled_163 688 0555 640 4265 546 1514 9741 983––
dpa::dpa::prefix_partition::scrambled_81 015 7011 554 1781 518 4381 114862––
dpa::dpa::rejected_word::chain_100922 7061 350 6341 307 0292 981820––
dpa::dpa::rejected_word::chain_4003 582 2235 383 7965 082 46648 0791 741––
dpa::dpa::rejected_word::scrambled_100244 006379 043345 013527897––
dpa::dpa::rejected_word::scrambled_400920 1381 417 0481 303 06310 2251 796––
dpa::dpa::sccs::chain_1000500 886788 923712 9036 8461 194––
dpa::dpa::sccs::chain_40001 928 6133 072 8962 746 36129 9085 057––
dpa::dpa::sccs::scrambled_1000567 843887 009810 2448 2411 016––
dpa::dpa::sccs::scrambled_40002 257 4833 600 4333 220 78346 2924 234––
dpa::dpa::witness_colors::chain_1004 635 2876 541 6676 417 6479 1242 240––
dpa::dpa::witness_colors::chain_40045 080 18863 452 81061 849 340194 50518 027––
dpa::dpa::witness_colors::scrambled_1004 026 6035 681 8845 586 2949 3391 397––
dpa::dpa::witness_colors::scrambled_40030 619 06543 036 14942 208 714146 5102 711––
dpa::dpa::witness_inequivalence::complement_100279 168420 996402 421873406––
dpa::dpa::witness_inequivalence::complement_2590 278142 456128 39689389––
dpa::dpa::witness_inequivalence::complement_50155 919237 938223 058316380––
dpa::dpa::witness_not_subset_of::complement_100269 593409 090390 465883406––
dpa::dpa::witness_not_subset_of::complement_2587 753139 280125 14098390––
dpa::dpa::witness_not_subset_of::complement_50151 044231 814216 884319381––
dpa::dpa::witness_not_subset_of::normalized_100542 474802 087774 5671 521569––
dpa::dpa::witness_not_subset_of::normalized_25184 635275 680261 725236365––
dpa::dpa::witness_not_subset_of::normalized_50313 246462 143445 828442403––
dpa::dpa::witnesses_fixtures::ltl2 217 3893 204 7073 135 3427 951846––
dpa::dpa::witnesses_fixtures::synthetic18 835 73628 666 81828 119 628103 327873––
game::game::tangle_learning::few_priorities_10002 697 0484 062 9303 713 17535 9104 863745 4001 292
game::game::tangle_learning::few_priorities_40008 120 67612 712 28611 104 496194 95618 0862 407 4163 266
game::game::tangle_learning::ladder_10001 179 3141 831 7261 627 97112 4434 044377 868118
game::game::tangle_learning::ladder_40004 408 1347 087 2366 022 281110 82614 5951 511 668225
game::game::tangle_learning::many_priorities_10002 018 0283 042 0902 758 58525 5234 454566 440848
game::game::tangle_learning::many_priorities_40006 752 83510 526 4309 145 870166 28915 6892 187 3441 620
game::game::tangle_learning_min_odd::few_priorities_10001 521 6152 322 1342 081 28426 3513 117525 072189
game::game::tangle_learning_min_odd::few_priorities_40008 017 70012 367 40010 863 250215 95512 1252 560 0641 933
game::game::zielonka::few_priorities_10001 126 1991 709 0211 520 53612 4553 606356 67695
game::game::zielonka::few_priorities_40003 470 2545 583 6874 684 11781 69014 0321 278 56484
game::game::zielonka::ladder_10005 134 2817 016 0776 609 66750 0904 456499 404182
game::game::zielonka::ladder_400019 816 37327 231 95125 422 846246 13916 5261 998 868212
game::game::zielonka::many_priorities_10004 103 5185 602 3035 264 43838 6914 126764 940339
game::game::zielonka::many_priorities_400012 909 83717 933 47616 554 676176 12914 2332 552 644288
game::game::zielonka_min_odd::few_priorities_10002 139 0773 023 3732 800 59323 7592 971512 012195
game::game::zielonka_min_odd::few_priorities_40006 302 0759 279 6878 309 227131 0299 0091 705 700158
learning::learning::lstar_mealy::counting_161 816 4872 477 8522 431 8727611 205154 5042 633
learning::learning::lstar_mealy::counting_8481 429684 428650 92322692540 472862
learning::learning::sprout_parity::inf_ab_1001 270 7201 850 1851 786 8802 2241 491200 4161 872
learning::learning::sprout_parity::inf_ab_4002 610 8203 782 7013 665 5919 2682 022405 0403 649
nondeterministic::nondeterministic::fixture_accepts::ltl762 5451 185 1021 120 5628 449637––
nondeterministic::nondeterministic::nba_accepts::nba_400125 269 894185 047 774174 142 9491 657 47074 785––
nondeterministic::nondeterministic::nba_witness::nba_400014 689 96121 476 74620 461 706138 4129 228––
nondeterministic::nondeterministic::nca_accepts::nca_400266 229 440395 082 740369 431 0054 559 08481 609––
nondeterministic::nondeterministic::nca_witness::nca_400019 458 26129 273 29527 117 250325 48815 103––
nondeterministic::nondeterministic::nma_witness::nma_100025 744 46937 890 67835 867 768371 1504 776––
nondeterministic::nondeterministic::npa_accepts::empty_npa_400493 167 601734 838 886687 188 0268 907 14489 004––
nondeterministic::nondeterministic::npa_accepts::npa_10030 411 62844 532 16242 656 897286 24412 687––
nondeterministic::nondeterministic::npa_accepts::npa_400125 279 542185 057 918174 152 9531 657 50574 784––
nondeterministic::nondeterministic::npa_witness::empty_npa_10009 121 95213 459 23712 723 917123 4323 376––
nondeterministic::nondeterministic::npa_witness::empty_npa_400037 023 30355 330 02651 364 426657 23619 412––
nondeterministic::nondeterministic::npa_witness::npa_10002 479 5313 680 1793 513 79420 7401 791––
nondeterministic::nondeterministic::npa_witness::npa_400020 354 25529 430 89628 218 496177 8989 226––
nondeterministic::nondeterministic::nra_witness::nra_400020 639 56731 400 77128 850 876337 11424 695––
runs::runs::dfa_accepts_many_words::count_10004 303 2135 381 9795 357 6494 7891100
runs::runs::dfa_accepts_one_word::len_1003 4924 8664 39638800
runs::runs::dfa_accepts_one_word::len_10000266 070329 557325 827697700
runs::runs::dfa_accepts_one_word::len_1000002 650 7313 276 8893 244 7096 3241600
runs::runs::dfa_reached_state::count_10003 570 2114 404 9084 380 6484 789900
runs::runs::dpa_accepts_lassos::long9 150 74511 671 08311 646 6333 077259724 664500
runs::runs::dpa_accepts_lassos::short406 490540 866535 5963619926 236472
runs::runs::dpa_recurrent_edge_colors::long15 116 26619 231 17719 179 4625 835644922 1442 090
runs::runs::dpa_recurrent_edge_colors::short1 189 7901 666 3231 655 0081 157158168 6841 708
runs::runs::dpa_recurrent_states::long10 502 98813 660 28413 630 4043 519351788 104860
runs::runs::dpa_recurrent_states::short572 037764 556757 26653413281 996782
runs::runs::dpa_visited_sequences::long8 738 26312 085 90812 050 9984 6513332 190 780850
runs::runs::dpa_visited_sequences::short370 678510 989506 2944986329 986545
runs::runs::fixture_accepts_lassos::ltl8 766 27112 652 15612 620 1765 542122––
runs::runs::fixture_accepts_lassos::synthetic4 340 1896 192 9936 139 1539 907123––
scc::scc::scc_queries::small_cycles_1000906 6761 433 5441 272 0949 4283 266261 132559
scc::scc::scc_queries::small_cycles_40003 548 9195 658 3944 973 93950 91712 2821 045 3962 071
scc::scc::tarjan::back_edges_1000512 099838 107726 7775 4732 399170 27245
scc::scc::tarjan::back_edges_40001 987 6273 267 5342 821 39426 4528 968680 63253
scc::scc::tarjan::binary_tree_1000571 217892 769815 6944 0611 622105 76039
scc::scc::tarjan::binary_tree_40002 290 4593 582 5733 276 41321 1295 729419 51245
scc::scc::tarjan::fragmented_1000658 5221 046 852922 7179 6442 169146 084121
scc::scc::tarjan::fragmented_40002 534 1084 125 8593 544 65462 7897 636569 852167
scc::scc::tarjan::grid_1024596 080972 081847 7617 7282 448171 16045
scc::scc::tarjan::grid_40962 378 4844 001 3513 363 70163 6699 123684 18453
scc::scc::tarjan::path_10001 929 1193 028 5232 729 56823 1885 229455 2242 064
scc::scc::tarjan::path_40007 632 90812 001 66410 804 76498 32320 1511 821 5368 076
scc::scc::tarjan::ring_1000512 099841 747725 8676 3832 399170 27245
scc::scc::tarjan::ring_40001 987 6273 284 7822 817 08230 7648 968680 63253
scc::scc::tarjan::scrambled_1000579 056936 579824 0296 9562 222170 27245
scc::scc::tarjan::scrambled_40002 316 4973 791 2493 296 77941 8728 146680 63253
scc::scc::tarjan::small_cycles_1000905 6581 432 2191 270 8099 4343 264261 132559
scc::scc::tarjan::small_cycles_40003 544 9015 650 7854 969 53050 29812 2791 045 3962 071
scc::scc::tarjan_from::fragmented_1000242 530394 598344 8782 5241 06074 98564
scc::scc::tarjan_from::fragmented_4000907 7621 492 3421 296 66715 9233 316299 62578
scc::scc::tarjan_from::path_10001 921 1583 018 2442 722 14423 0375 169468 9932 091
scc::scc::tarjan_from::path_40007 545 58411 928 68110 697 651106 73119 9251 876 7538 109
scc::scc::tarjan_from::ring_1000503 645836 974717 9746 2512 507184 04172
scc::scc::tarjan_from::scrambled_1000570 602930 380816 4856 4762 329184 04172
storage::storage::add_edges_dense::k2_50001 356 4012 030 4901 985 3454 941584180 20037
storage::storage::add_edges_dense::k8_50004 247 0596 411 8746 225 39417 8362 780671 72840
storage::storage::add_edges_linked::k2_50001 402 6702 420 4692 258 38414 2242 5991 113 96026
storage::storage::add_edges_linked::k8_50006 669 28411 260 27310 330 80875 81815 7253 866 49628
storage::storage::clone_csr::k2_1000030 590183 48052 6254 1703 143230 0126
storage::storage::clone_dense::k2_1000015 11381 34225 8022 2531 265110 0084
storage::storage::clone_linked::k2_10000370 824901 703529 79812 4668 845680 0083
storage::storage::complete_dense::partial_5000155 628209 494202 329908751364
storage::storage::complete_linked::partial_50001 210 8541 718 8451 629 4506 9521 56165 67218
storage::storage::edges_from_all_states_csr::k2_200072 04777 21475 859222700
storage::storage::edges_from_all_states_csr::k8_2000108 047116 392111 0721 008800
storage::storage::edges_from_all_states_dense::k2_2000102 043119 287117 842233800
storage::storage::edges_from_all_states_dense::k8_2000140 043160 529155 0391 035900
storage::storage::edges_from_all_states_linked::k2_200068 03386 72575 9252 132400
storage::storage::edges_from_all_states_linked::k8_2000152 033203 701166 6817 376400
storage::storage::minimize_csr::redundant_20004 633 7567 005 0566 522 11157 8935 5281 505 4202 212
storage::storage::minimize_csr::redundant_800018 159 86127 786 74025 537 970272 65425 3006 023 3008 250
storage::storage::minimize_dense::redundant_20004 676 9107 035 6516 577 16658 1534 7921 522 1562 221
storage::storage::minimize_dense::redundant_800018 364 89528 117 70225 817 247279 05025 8636 090 3408 261
storage::storage::minimize_linked::redundant_20004 505 0386 920 7686 368 86861 9266 9221 522 1562 221
storage::storage::minimize_linked::redundant_800017 714 72127 484 31425 033 274302 87426 7626 090 3408 261
storage::storage::overwrite_edges_dense::k2_50002 126 4313 232 5183 172 4287 916586180 20037
storage::storage::overwrite_edges_linked::k2_50003 790 7756 671 1135 952 01844 76914 1502 031 46427
storage::storage::predecessors_of_all_states_csr::k2_2000102 046121 042117 407678700
storage::storage::predecessors_of_all_states_dense::k2_2000290 369397 907388 2221 6224532 0083
storage::storage::predecessors_of_all_states_linked::k2_200068 03391 72974 6893 366600
storage::storage::product_successor_csr::n20064 09675 54575 16014900
storage::storage::product_successor_dense::n20073 08794 62094 10540900
storage::storage::product_successor_linked::n20046 54760 17557 015576800
storage::storage::reachable_states_csr::k2_80001 101 1151 630 1181 591 8437 4173417 36811
storage::storage::reachable_states_dense::k2_80001 133 6991 684 5051 639 2458 8283217 36811
storage::storage::reachable_states_linked::k2_8000924 2791 485 5951 386 51519 5294117 36811
storage::storage::remove_state_dense::k2_100001 072 2121 565 0861 390 28621 5131 921176 0001 003
storage::storage::remove_state_linked::k2_100001 094 0121 531 4791 492 7347 4204780 0003 000
storage::storage::run_many_words_csr::count_1000551 036653 943644 1631 907700
storage::storage::run_many_words_dense::count_1000558 044690 031680 1661 910900
storage::storage::run_many_words_linked::count_1000481 044611 149601 1191 957700
storage::storage::run_one_word_csr::len_1000002 600 0543 025 4282 993 8236 272700
storage::storage::run_one_word_dense::len_1000002 800 0503 425 4573 393 8126 273800
storage::storage::run_one_word_linked::len_1000002 350 0392 975 4282 943 7736 296500
storage::storage::successor_all_pairs_csr::k2_2000122 242139 664138 0992291281
storage::storage::successor_all_pairs_csr::k8_2000702 302802 872797 3721 02311321
storage::storage::successor_all_pairs_dense::k2_2000162 237193 545192 090235881
storage::storage::successor_all_pairs_dense::k8_2000814 297952 851947 3461 0458321
storage::storage::successor_all_pairs_linked::k2_2000108 236139 158128 1782 140881
storage::storage::successor_all_pairs_linked::k8_2000750 296946 204908 9997 3858321
ts::ts::add_edges_in_order::k2_50001 402 6712 420 3052 258 39014 2252 5941 113 96026
ts::ts::add_edges_in_order::k8_50006 669 28511 260 12510 330 81075 82315 7203 866 49628
ts::ts::add_edges_in_reverse::k2_50001 402 6712 419 5012 258 59114 0242 5941 113 96026
ts::ts::builder_with_edges::k2_500081 363 41498 215 82995 713 504451 7106 9651 739 53245
ts::ts::canonical_naming::scrambled_15003 167 6084 657 7734 467 71821 7992 316463 8085 846
ts::ts::canonical_naming::scrambled_500980 4221 418 5491 374 7043 176799134 3361 831
ts::ts::clone::scrambled_10000371 163902 473530 23812 4838 852680 0083
ts::ts::clone::scrambled_400001 481 4003 817 8822 116 80742 57542 5202 720 0083
ts::ts::collect_dts::scrambled_100003 465 1496 166 0825 606 84737 17110 6682 228 06427
ts::ts::collect_dts::scrambled_4000013 799 62824 683 17022 329 475135 29947 9208 912 73631
ts::ts::determinize::nth_from_end_104 485 5246 645 1096 283 27922 2467 160651 7206 013
ts::ts::determinize::nth_from_end_1219 876 08529 277 43727 766 66799 07029 0122 732 60026 423
ts::ts::determinize::nth_from_end_81 027 3331 525 0091 441 5393 0651 947156 5521 396
ts::ts::edges_from_all_states::scrambled_200072 66991 97980 7742 1571200
ts::ts::edges_from_all_states::scrambled_k8_2000156 601208 894171 4297 4021300
ts::ts::erase_state_colors_collect::scrambled_100003 007 1665 308 3274 748 85737 17610 6742 228 06427
ts::ts::into_dfa::scrambled_100003 565 1656 276 2995 716 87437 18110 6722 228 06427
ts::ts::map_edge_colors_collect::scrambled_100003 008 7685 438 0624 819 55238 44912 1792 490 19227
ts::ts::map_edge_colors_collect::scrambled_4000011 975 39121 772 10019 182 195140 33153 9509 961 29631
ts::ts::matching_product_successor_index::n20075 662111 062106 4676463900
ts::ts::minimal_representative_of_all_states::scrambled_1002 100 8142 967 8222 964 19212102347 5044 679
ts::ts::minimal_representative_of_all_states::scrambled_30016 911 26423 979 43123 966 5111 8421063 116 78438 196
ts::ts::minimal_representatives::scrambled_20003 384 0404 762 0964 715 2368 91066322 9447 709
ts::ts::minimal_representatives::scrambled_800014 900 24920 949 86520 734 91042 592571 464 89632 693
ts::ts::predecessors_of_all_states::scrambled_200072 66996 94979 5393 3911300
ts::ts::predecessors_of_all_states::scrambled_8000288 601389 362313 81215 0191300
ts::ts::reachable_states::scrambled_2000290 091460 905440 0053 606824 3369
ts::ts::reachable_states::scrambled_80001 148 6961 838 5141 739 08919 5215217 36811
ts::ts::remove_edges_from_matching::scrambled_10000566 234796 055771 6454 6583264 0002 000
ts::ts::remove_edges_from_matching::scrambled_40000896 2341 177 5551 126 27010 0333264 0002 000
ts::ts::remove_state::scrambled_100001 091 0121 528 4451 489 7357 4204680 0003 000
ts::ts::remove_state::scrambled_400001 421 0121 912 9131 843 61813 5374680 0003 000
ts::ts::scc_minimal_representatives::small_cycles_100219 985321 260305 6005943944 272309
ts::ts::scc_minimal_representatives::small_cycles_4001 246 4071 775 8601 725 3303 890888354 1481 325
ts::ts::subset_construction::nth_from_end_1010 216 43914 514 36214 019 09245 5047 6501 073 5209 919
ts::ts::subset_construction::nth_from_end_1245 206 16364 008 87861 763 193218 72532 9164 518 97643 857
ts::ts::subset_construction::nth_from_end_82 230 5063 187 0943 078 4446 1972 219257 1042 276
ts::ts::successor_index_all_pairs::scrambled_2000108 863140 297129 0222 1571481
ts::ts::successor_index_all_pairs::scrambled_k8_2000750 855947 316909 7417 40316321

Wall-clock times

Criterion means of a single run. They depend on the machine and are not compared; use the instruction counts above for that.

benchmarkhead
collect/clone/1000003.38 ms (3.32 ms … 3.45 ms)
collect/clone/100000040.98 ms (40.91 ms … 41.06 ms)
collect/collect_owned/1000007.67 ms (7.64 ms … 7.70 ms)
collect/collect_owned/100000078.68 ms (76.46 ms … 80.90 ms)
collect/collect_ref/1000005.04 ms (5.00 ms … 5.08 ms)
collect/collect_ref/100000082.49 ms (81.81 ms … 83.18 ms)
collect/map_edge_colors_collect_owned/1000007.90 ms (7.86 ms … 7.94 ms)
collect/map_edge_colors_collect_owned/100000080.26 ms (78.46 ms … 82.06 ms)
wallclock/dfa/accepts/word_100000827.59 µs (827.04 µs … 828.14 µs)
wallclock/dfa/minimize/redundant_40001.42 ms (1.42 ms … 1.43 ms)
wallclock/dpa/complement/scrambled_40003.71 ms (3.67 ms … 3.75 ms)
wallclock/dpa/normalized/scrambled_40003.32 ms (3.31 ms … 3.33 ms)
wallclock/hoa/parse/ltl962.50 µs (939.95 µs … 985.05 µs)
wallclock/nfa/determinize/nth_from_end_123.67 ms (3.60 ms … 3.75 ms)
wallclock/nfa/subset_construction/nth_from_end_128.36 ms (8.12 ms … 8.59 ms)
wallclock/scc/tarjan/scrambled_100002.42 ms (2.40 ms … 2.43 ms)

Versus Spot and Oink

dpa/equivalence

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot140.18 µs113.02 µs1.24×– / –yes
hard/l-nba-2spot64.58 µs35.16 µs1.84×– / –yes
hard/m-nba-3spot265.02 µs133.02 µs1.99×– / –yes
ltl/and-gf-3spot10.24 µs11.56 µs0.89×– / –yes
ltl/eh-patterns-0spot26.70 µs19.56 µs1.37×– / –yes
ltl/gf-equiv-3spot24.36 µs32.92 µs0.74×– / –yes
ltl/gf-implies-3spot12.76 µs18.82 µs0.68×– / –yes
ltl/gh-q-3spot59.48 µs29.56 µs2.01×– / –yes
ltl/gh-r-3spot25.68 µs67.14 µs0.38×– / –yes
ltl/go-theta-3spot12.60 µs20.40 µs0.62×– / –yes
ltl/lily-patterns-1spot13.78 µs15.46 µs0.89×– / –yes
ltl/ms-phi-r-1spot12.08 µs37.18 µs0.32×– / –yes
ltl/ms-phi-s-1spot12.04 µs38.48 µs0.31×– / –yes
ltl/rv-counter-3spot30.18 µs12.68 µs2.38×– / –yes
ltl/sb-patterns-0spot14.36 µs13.36 µs1.07×– / –yes
ltl/sejk-k-2spot15.84 µs56.28 µs0.28×– / –yes
ltl/tv-uu-3spot46.24 µs36.28 µs1.27×– / –yes
ltl/u-left-3spot11.88 µs8.16 µs1.46×– / –yes
many_aps/and-gf-12spot175.14 µs62.70 µs2.79×– / –yes
many_aps/gf-equiv-12spot146.90 µs90.50 µs1.62×– / –yes
many_aps/r-right-10spot73.82 µs24.82 µs2.97×– / –yes
many_aps/random-n16-a8-p4-s1spot303.52 µs130.40 µs2.33×– / –yes
many_aps/random-n24-a12-p4-s1spot1.47 ms433.68 µs3.39×– / –yes
many_aps/random-n32-a16-p4-s1spot4.51 ms1.16 ms3.89×– / –yes
random/n128-a2-p6-s1spot363.50 µs204.38 µs1.78×– / –yes
random/n16-a1-p3-s1spot24.18 µs15.48 µs1.56×– / –yes
random/n256-a2-p8-s1spot971.83 µs469.35 µs2.07×– / –yes
random/n64-a2-p4-s1spot153.52 µs96.12 µs1.60×– / –yes

dpa/inclusion

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot77.00 µs59.64 µs1.29×– / –yes
hard/l-nba-2spot38.92 µs22.24 µs1.75×– / –yes
hard/m-nba-3spot162.60 µs81.46 µs2.00×– / –yes
ltl/and-gf-3spot6.64 µs7.80 µs0.85×– / –yes
ltl/eh-patterns-0spot17.50 µs16.84 µs1.04×– / –yes
ltl/gf-equiv-3spot13.78 µs18.12 µs0.76×– / –yes
ltl/gf-implies-3spot7.20 µs10.94 µs0.66×– / –yes
ltl/gh-q-3spot37.20 µs26.96 µs1.38×– / –yes
ltl/gh-r-3spot14.58 µs33.46 µs0.44×– / –yes
ltl/go-theta-3spot8.12 µs15.50 µs0.52×– / –yes
ltl/lily-patterns-1spot9.24 µs12.86 µs0.72×– / –yes
ltl/ms-phi-r-1spot7.16 µs18.32 µs0.39×– / –yes
ltl/ms-phi-s-1spot7.28 µs24.24 µs0.30×– / –yes
ltl/rv-counter-3spot19.32 µs13.76 µs1.40×– / –yes
ltl/sb-patterns-0spot9.08 µs11.56 µs0.79×– / –yes
ltl/sejk-k-2spot9.14 µs27.60 µs0.33×– / –yes
ltl/tv-uu-3spot31.72 µs28.94 µs1.10×– / –yes
ltl/u-left-3spot7.80 µs8.36 µs0.93×– / –yes
many_aps/and-gf-12spot144.72 µs52.82 µs2.74×– / –yes
many_aps/gf-equiv-12spot82.48 µs49.42 µs1.67×– / –yes
many_aps/r-right-10spot53.14 µs22.80 µs2.33×– / –yes
many_aps/random-n16-a8-p4-s1spot244.26 µs94.46 µs2.59×– / –yes
many_aps/random-n24-a12-p4-s1spot1.38 ms370.72 µs3.73×– / –yes
many_aps/random-n32-a16-p4-s1spot4.40 ms1.08 ms4.09×– / –yes
random/n128-a2-p6-s1spot214.32 µs122.38 µs1.75×– / –yes
random/n16-a1-p3-s1spot14.84 µs10.12 µs1.47×– / –yes
random/n256-a2-p8-s1spot552.31 µs291.06 µs1.90×– / –yes
random/n64-a2-p4-s1spot91.62 µs60.16 µs1.52×– / –yes

game/solve

inputreferenceoursreferenceours / referencesize (ours / reference)same result
ladder_10000oink-ppp2.77 ms297.00 µs9.31×10000 / 10000yes
ladder_10000oink-zlk2.77 ms309.00 µs8.95×10000 / 10000yes
ladder_10000spot2.77 ms9.56 ms0.29×10000 / 10000yes
ladder_50000oink-ppp20.35 ms2.01 ms10.12×50000 / 50000yes
ladder_50000oink-zlk20.35 ms2.06 ms9.89×50000 / 50000yes
ladder_50000spot20.35 ms70.13 ms0.29×50000 / 50000yes
random_few_10000oink-ppp8.02 ms1.46 ms5.50×10000 / 10000yes
random_few_10000oink-zlk8.02 ms2.61 ms3.08×10000 / 10000yes
random_few_10000spot8.02 ms20.94 ms0.38×10000 / 10000yes
random_few_50000oink-ppp82.48 ms18.61 ms4.43×50000 / 50000yes
random_few_50000oink-zlk82.48 ms21.39 ms3.86×50000 / 50000yes
random_few_50000spot82.48 ms746.29 ms0.11×50000 / 50000yes
random_many_10000oink-ppp7.51 ms1.56 ms4.80×10000 / 10000yes
random_many_10000oink-zlk7.51 ms2.98 ms2.52×10000 / 10000yes
random_many_10000spot7.51 ms31.05 ms0.24×10000 / 10000yes
random_many_50000oink-ppp80.44 ms28.59 ms2.81×50000 / 50000yes
random_many_50000oink-zlk80.44 ms49.23 ms1.63×50000 / 50000yes
random_many_50000spot80.44 ms719.23 ms0.11×50000 / 50000yes

Log

== head 8ce15b69a7257040d49284ac22df55ff79b8a15d: building ["scc", "ts", "runs", "dfa", "dpa", "storage", "nondeterministic", "learning", "game"] ["wallclock"]
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
    Finished `bench` profile [optimized] target(s) in 2m 07s
  Executable benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-a84ba41f760e70ed)
  Executable benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-23bb829f074058ad)
  Executable benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-14422f732ec85029)
  Executable benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-ee53c00379271c59)
  Executable benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-f164d05108dbce73)
  Executable benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-85a152aa046a29b4)
  Executable benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-1b22c641fd9d2d89)
  Executable benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-f95e3d4a8c506941)
  Executable benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-4948f6cc6a43d201)
  Executable benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-93fb383f608d3bab)
== head: running scc
    Finished `bench` profile [optimized] target(s) in 0.23s
     Running benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-1b22c641fd9d2d89)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running ts
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-4948f6cc6a43d201)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running runs
    Finished `bench` profile [optimized] target(s) in 0.23s
     Running benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-85a152aa046a29b4)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dfa
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-a84ba41f760e70ed)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dpa
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-23bb829f074058ad)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running storage
    Finished `bench` profile [optimized] target(s) in 0.27s
     Running benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-f95e3d4a8c506941)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running nondeterministic
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-f164d05108dbce73)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running learning
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-ee53c00379271c59)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running game
    Finished `bench` profile [optimized] target(s) in 0.20s
     Running benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-14422f732ec85029)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running wallclock (wall clock)
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-93fb383f608d3bab)
Gnuplot not found, using plotters backend
Benchmarking wallclock/scc/tarjan/scrambled_10000
Benchmarking wallclock/scc/tarjan/scrambled_10000: Analyzing
Benchmarking wallclock/dpa/normalized/scrambled_4000
Benchmarking wallclock/dpa/normalized/scrambled_4000: Analyzing
Benchmarking wallclock/dpa/complement/scrambled_4000
Benchmarking wallclock/dpa/complement/scrambled_4000: Analyzing
Benchmarking wallclock/dfa/minimize/redundant_4000
Benchmarking wallclock/dfa/minimize/redundant_4000: Analyzing
Benchmarking wallclock/dfa/accepts/word_100000
Benchmarking wallclock/dfa/accepts/word_100000: Analyzing
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12: Analyzing
Benchmarking wallclock/nfa/determinize/nth_from_end_12
Benchmarking wallclock/nfa/determinize/nth_from_end_12: Analyzing
Benchmarking wallclock/hoa/parse/ltl
Benchmarking wallclock/hoa/parse/ltl: Analyzing
Benchmarking collect/clone/100000
Benchmarking collect/clone/100000: Analyzing
Benchmarking collect/collect_owned/100000
Benchmarking collect/collect_owned/100000: Analyzing
Benchmarking collect/collect_ref/100000
Benchmarking collect/collect_ref/100000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/100000
Benchmarking collect/map_edge_colors_collect_owned/100000: Analyzing
Benchmarking collect/clone/1000000
Benchmarking collect/clone/1000000: Analyzing
Benchmarking collect/collect_owned/1000000
Benchmarking collect/collect_owned/1000000: Analyzing
Benchmarking collect/collect_ref/1000000
Benchmarking collect/collect_ref/1000000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/1000000
Benchmarking collect/map_edge_colors_collect_owned/1000000: Analyzing
== head: building the Python bindings
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support
🐍 Found CPython 3.13 at /nix/store/vilynzly1pyhawy04ljmrp7n9ffr24qy-python3-3.13.17/bin/python3
📡 Using build options features from pyproject.toml
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
   Compiling automata-py v0.1.0 (/var/lib/automata-bench/work/src-head/python)
    Finished `release` profile [optimized] target(s) in 1m 53s
📦 Built wheel for abi3 Python ≥ 3.11 to /var/lib/automata-bench/work/versus-head/wheels/automatars-0.1.0-cp311-abi3-manylinux_2_34_aarch64.whl
== head: running benches/versus/compare.py (Spot, Oink)
dpa/equivalence/hard/l-dsa-3 vs spot: 0.140 ms / 0.113 ms
dpa/equivalence/hard/l-nba-2 vs spot: 0.065 ms / 0.035 ms
dpa/equivalence/hard/m-nba-3 vs spot: 0.265 ms / 0.133 ms
dpa/equivalence/ltl/and-gf-3 vs spot: 0.010 ms / 0.012 ms
dpa/equivalence/ltl/eh-patterns-0 vs spot: 0.027 ms / 0.020 ms
dpa/equivalence/ltl/gf-equiv-3 vs spot: 0.024 ms / 0.033 ms
dpa/equivalence/ltl/gf-implies-3 vs spot: 0.013 ms / 0.019 ms
dpa/equivalence/ltl/gh-q-3 vs spot: 0.059 ms / 0.030 ms
dpa/equivalence/ltl/gh-r-3 vs spot: 0.026 ms / 0.067 ms
dpa/equivalence/ltl/go-theta-3 vs spot: 0.013 ms / 0.020 ms
dpa/equivalence/ltl/lily-patterns-1 vs spot: 0.014 ms / 0.015 ms
dpa/equivalence/ltl/ms-phi-r-1 vs spot: 0.012 ms / 0.037 ms
dpa/equivalence/ltl/ms-phi-s-1 vs spot: 0.012 ms / 0.038 ms
dpa/equivalence/ltl/rv-counter-3 vs spot: 0.030 ms / 0.013 ms
dpa/equivalence/ltl/sb-patterns-0 vs spot: 0.014 ms / 0.013 ms
dpa/equivalence/ltl/sejk-k-2 vs spot: 0.016 ms / 0.056 ms
dpa/equivalence/ltl/tv-uu-3 vs spot: 0.046 ms / 0.036 ms
dpa/equivalence/ltl/u-left-3 vs spot: 0.012 ms / 0.008 ms
dpa/equivalence/many_aps/and-gf-12 vs spot: 0.175 ms / 0.063 ms
dpa/equivalence/many_aps/gf-equiv-12 vs spot: 0.147 ms / 0.091 ms
dpa/equivalence/many_aps/r-right-10 vs spot: 0.074 ms / 0.025 ms
dpa/equivalence/many_aps/random-n16-a8-p4-s1 vs spot: 0.304 ms / 0.130 ms
dpa/equivalence/many_aps/random-n24-a12-p4-s1 vs spot: 1.472 ms / 0.434 ms
dpa/equivalence/many_aps/random-n32-a16-p4-s1 vs spot: 4.505 ms / 1.159 ms
dpa/equivalence/random/n128-a2-p6-s1 vs spot: 0.364 ms / 0.204 ms
dpa/equivalence/random/n16-a1-p3-s1 vs spot: 0.024 ms / 0.015 ms
dpa/equivalence/random/n256-a2-p8-s1 vs spot: 0.972 ms / 0.469 ms
dpa/equivalence/random/n64-a2-p4-s1 vs spot: 0.154 ms / 0.096 ms
dpa/inclusion/hard/l-dsa-3 vs spot: 0.077 ms / 0.060 ms
dpa/inclusion/hard/l-nba-2 vs spot: 0.039 ms / 0.022 ms
dpa/inclusion/hard/m-nba-3 vs spot: 0.163 ms / 0.081 ms
dpa/inclusion/ltl/and-gf-3 vs spot: 0.007 ms / 0.008 ms
dpa/inclusion/ltl/eh-patterns-0 vs spot: 0.018 ms / 0.017 ms
dpa/inclusion/ltl/gf-equiv-3 vs spot: 0.014 ms / 0.018 ms
dpa/inclusion/ltl/gf-implies-3 vs spot: 0.007 ms / 0.011 ms
dpa/inclusion/ltl/gh-q-3 vs spot: 0.037 ms / 0.027 ms
dpa/inclusion/ltl/gh-r-3 vs spot: 0.015 ms / 0.033 ms
dpa/inclusion/ltl/go-theta-3 vs spot: 0.008 ms / 0.016 ms
dpa/inclusion/ltl/lily-patterns-1 vs spot: 0.009 ms / 0.013 ms
dpa/inclusion/ltl/ms-phi-r-1 vs spot: 0.007 ms / 0.018 ms
dpa/inclusion/ltl/ms-phi-s-1 vs spot: 0.007 ms / 0.024 ms
dpa/inclusion/ltl/rv-counter-3 vs spot: 0.019 ms / 0.014 ms
dpa/inclusion/ltl/sb-patterns-0 vs spot: 0.009 ms / 0.012 ms
dpa/inclusion/ltl/sejk-k-2 vs spot: 0.009 ms / 0.028 ms
dpa/inclusion/ltl/tv-uu-3 vs spot: 0.032 ms / 0.029 ms
dpa/inclusion/ltl/u-left-3 vs spot: 0.008 ms / 0.008 ms
dpa/inclusion/many_aps/and-gf-12 vs spot: 0.145 ms / 0.053 ms
dpa/inclusion/many_aps/gf-equiv-12 vs spot: 0.082 ms / 0.049 ms
dpa/inclusion/many_aps/r-right-10 vs spot: 0.053 ms / 0.023 ms
dpa/inclusion/many_aps/random-n16-a8-p4-s1 vs spot: 0.244 ms / 0.094 ms
dpa/inclusion/many_aps/random-n24-a12-p4-s1 vs spot: 1.383 ms / 0.371 ms
dpa/inclusion/many_aps/random-n32-a16-p4-s1 vs spot: 4.395 ms / 1.076 ms
dpa/inclusion/random/n128-a2-p6-s1 vs spot: 0.214 ms / 0.122 ms
dpa/inclusion/random/n16-a1-p3-s1 vs spot: 0.015 ms / 0.010 ms
dpa/inclusion/random/n256-a2-p8-s1 vs spot: 0.552 ms / 0.291 ms
dpa/inclusion/random/n64-a2-p4-s1 vs spot: 0.092 ms / 0.060 ms
game/solve/random_few_10000 vs spot: 8.020 ms / 20.943 ms
game/solve/random_few_10000 vs oink-zlk: 8.020 ms / 2.606 ms
game/solve/random_few_10000 vs oink-ppp: 8.020 ms / 1.458 ms
game/solve/random_few_50000 vs spot: 82.481 ms / 746.287 ms
game/solve/random_few_50000 vs oink-zlk: 82.481 ms / 21.391 ms
game/solve/random_few_50000 vs oink-ppp: 82.481 ms / 18.607 ms
game/solve/random_many_10000 vs spot: 7.508 ms / 31.055 ms
game/solve/random_many_10000 vs oink-zlk: 7.508 ms / 2.976 ms
game/solve/random_many_10000 vs oink-ppp: 7.508 ms / 1.564 ms
game/solve/random_many_50000 vs spot: 80.437 ms / 719.231 ms
game/solve/random_many_50000 vs oink-zlk: 80.437 ms / 49.232 ms
game/solve/random_many_50000 vs oink-ppp: 80.437 ms / 28.590 ms
game/solve/ladder_10000 vs spot: 2.765 ms / 9.557 ms
game/solve/ladder_10000 vs oink-zlk: 2.765 ms / 0.309 ms
game/solve/ladder_10000 vs oink-ppp: 2.765 ms / 0.297 ms
game/solve/ladder_50000 vs spot: 20.346 ms / 70.129 ms
game/solve/ladder_50000 vs oink-zlk: 20.346 ms / 2.057 ms
game/solve/ladder_50000 vs oink-ppp: 20.346 ms / 2.010 ms
== head: 259 benchmarks, 16 wall-clock measurements, 74 comparisons