automata benchmarks

History · Versus Spot and Oink

main · f83ff30f

Benchmark: determinization over a propositional alphabet (#163)

status done · queued 2026-11-10 18:07 · started 2026-11-10 18:09 · finished 2026-11-10 18:21

· · rustc 1.95.0 (59807616e 2026-04-14) (built from a source tarball) · valgrind-3.26.0 · pinned to cpu 1

Results

benchmarkinstructionsest. cyclesL1 hitsLL hitsRAM hitsheap bytesallocations
dfa::dfa::accepts_many_words::scrambled_1000343 072435 464428 9441 2201200
dfa::dfa::complement::scrambled_1000366 068584 238566 3333 39926139 10419
dfa::dfa::complement::scrambled_40001 435 8732 309 0932 223 22316 99925556 89623
dfa::dfa::equivalent::deep_difference_10061 66797 78690 4714920216 19625
dfa::dfa::equivalent::deep_difference_400212 158336 680314 2001 85037865 36433
dfa::dfa::equivalent::different_1009 04715 44312 5688811 80412
dfa::dfa::equivalent::different_4009 08815 64412 52478781 80412
dfa::dfa::equivalent::minimized_10035 84954 51351 888105607 75617
dfa::dfa::equivalent::minimized_400120 587182 487176 6977246231 58023
dfa::dfa::give_word::empty_10001 872 6132 558 6052 531 0304 92784152 8643 738
dfa::dfa::give_word::empty_40006 930 8239 656 9619 554 28620 05968679 80815 828
dfa::dfa::give_word::far_1000164 342231 589227 0345475228 176394
dfa::dfa::give_word::far_4000946 9821 329 3961 307 3913 92568148 4642 233
dfa::dfa::intersection::scrambled_1003 319 7095 217 4015 038 98612 0653 374428 08842
dfa::dfa::intersection::scrambled_20013 224 87321 169 19120 030 86197 14418 6461 708 12049
dfa::dfa::intersection_is_empty::empty_10059 26694 00187 3763018515 70020
dfa::dfa::intersection_is_empty::empty_200109 269175 048161 72861329331 58023
dfa::dfa::is_empty_language::empty_10001 872 6152 558 6772 531 0324 92786152 8643 738
dfa::dfa::is_empty_language::empty_40006 930 8259 656 9659 554 29020 05968679 80815 828
dfa::dfa::minimize::minimal_10003 450 9245 260 2624 882 20739 7715 1201 135 4922 214
dfa::dfa::minimize::minimal_400013 472 60220 707 23819 046 968204 64718 2014 543 7248 254
dfa::dfa::minimize::redundant_10002 292 8073 485 4273 247 03225 9373 106760 7801 201
dfa::dfa::minimize::redundant_40008 869 10613 616 89312 556 903143 6579 7633 044 9004 241
dfa::dfa::minimize::scrambled_10003 670 2565 623 8715 188 59651 1105 1351 116 0522 217
dfa::dfa::minimize::scrambled_400014 250 46922 214 47420 104 804294 19818 2484 470 1408 261
dfa::dfa::minimize_moore_with_edge_colors::scrambled_10003 669 2525 616 6885 186 58350 1115 1301 116 0522 217
dfa::dfa::minimize_moore_with_edge_colors::scrambled_400014 239 07322 181 42820 076 193300 69617 1934 470 1408 261
dfa::dfa::union::scrambled_1003 319 7095 217 4055 038 98512 0663 374428 08842
dfa::dfa::union::scrambled_20013 224 87321 169 15720 030 86297 14418 6451 708 12049
dpa::dpa::accepted_word::chain_100114 174175 422162 92756349––
dpa::dpa::accepted_word::chain_400385 785585 537550 5472 973575––
dpa::dpa::accepted_word::scrambled_100144 718217 848204 73340369––
dpa::dpa::accepted_word::scrambled_400492 612731 552697 6922 817565––
dpa::dpa::collect_dpa::scrambled_1000290 651469 464450 2943 8064––
dpa::dpa::collect_dpa::scrambled_40001 133 0441 881 8971 758 26216 4041 189––
dpa::dpa::complement::chain_10004 101 1576 256 2115 752 47155 1576 513––
dpa::dpa::complement::chain_400016 670 17825 571 92123 330 326261 92326 628––
dpa::dpa::complement::scrambled_10004 960 1657 464 4156 918 01562 7516 647––
dpa::dpa::complement::scrambled_400020 044 50530 552 41427 887 289344 19326 976––
dpa::dpa::complement_fixtures::ltl1 481 7992 154 0782 111 4185 200476––
dpa::dpa::complement_fixtures::many_aps13 922 08920 576 21020 085 67087 3561 536––
dpa::dpa::complement_fixtures::synthetic9 667 56613 876 77213 367 14298 426500––
dpa::dpa::complement_without_normalizing::scrambled_1000565 479809 053779 8285 18095––
dpa::dpa::complement_without_normalizing::scrambled_40002 223 8503 219 2383 064 57821 9721 280––
dpa::dpa::equivalent_fixtures::many_aps21 636 04232 269 80531 594 205122 3241 828––
dpa::dpa::hoa_parse::ltl5 507 5988 383 9968 054 07141 6043 483––
dpa::dpa::hoa_parse::many_aps88 883 063144 042 882126 819 642291 925450 389––
dpa::dpa::hoa_parse::synthetic25 090 68338 164 20136 749 261180 51514 639––
dpa::dpa::hoa_print::ltl19 037 29336 368 51324 186 6935 495347 267––
dpa::dpa::hoa_print::many_aps19 662 96928 400 83828 320 34313 229410––
dpa::dpa::hoa_print::synthetic27 383 65348 636 21536 383 9606 457349 142––
dpa::dpa::included_fixtures::many_aps18 595 18927 913 20127 369 96195 8101 834––
dpa::dpa::included_in::complement_100269 594409 087390 467882406––
dpa::dpa::included_in::complement_2587 754139 251125 14199389––
dpa::dpa::included_in::complement_50151 045231 793216 883322380––
dpa::dpa::included_in::normalized_100542 475802 092774 5671 522569––
dpa::dpa::included_in::normalized_25184 636275 643261 728235364––
dpa::dpa::included_in::normalized_50313 247462 152445 827444403––
dpa::dpa::language_equivalent::complement_100279 169420 971402 421875405––
dpa::dpa::language_equivalent::complement_2590 279142 461128 39690389––
dpa::dpa::language_equivalent::complement_50155 920237 913223 058318379––
dpa::dpa::language_equivalent::normalized_100992 4501 444 6071 408 4772 543669––
dpa::dpa::language_equivalent::normalized_25326 481477 097461 497264408––
dpa::dpa::language_equivalent::normalized_50580 879840 373822 208623430––
dpa::dpa::map_edge_colors_collect::scrambled_1000289 296465 295445 0653 89921––
dpa::dpa::map_edge_colors_collect::scrambled_40001 125 6911 862 4781 734 34817 1841 206––
dpa::dpa::normalized::chain_10003 828 6035 913 7475 422 92252 9246 463––
dpa::dpa::normalized::chain_400015 581 62424 210 35522 014 280253 19026 575––
dpa::dpa::normalized::scrambled_10004 687 4137 121 7126 588 10260 5506 596––
dpa::dpa::normalized::scrambled_400018 955 75329 191 05526 570 775335 59526 923––
dpa::dpa::normalized::scrambled_k4_100025 967 23037 085 68634 932 476379 5497 299––
dpa::dpa::normalized_fixtures::ltl1 417 2072 070 1412 029 9865 098419––
dpa::dpa::normalized_fixtures::many_aps13 662 04020 252 21519 776 59084 6041 503––
dpa::dpa::normalized_fixtures::synthetic9 351 54013 468 34112 987 01693 010465––
dpa::dpa::prefix_partition::chain_163 307 1635 026 9254 954 0602 8761 671––
dpa::dpa::prefix_partition::chain_8853 0541 309 3891 275 914773846––
dpa::dpa::prefix_partition::scrambled_163 688 0555 640 4265 546 1514 9741 983––
dpa::dpa::prefix_partition::scrambled_81 015 7011 554 1781 518 4381 114862––
dpa::dpa::rejected_word::chain_100922 7061 350 6341 307 0292 981820––
dpa::dpa::rejected_word::chain_4003 582 2345 383 8085 082 47848 0791 741––
dpa::dpa::rejected_word::scrambled_100244 006379 043345 013527897––
dpa::dpa::rejected_word::scrambled_400920 1381 417 0481 303 06310 2251 796––
dpa::dpa::sccs::chain_1000500 886788 923712 9036 8461 194––
dpa::dpa::sccs::chain_40001 928 6133 072 8962 746 36129 9085 057––
dpa::dpa::sccs::scrambled_1000567 843887 009810 2448 2411 016––
dpa::dpa::sccs::scrambled_40002 257 4833 600 4333 220 78346 2924 234––
dpa::dpa::witness_colors::chain_1004 635 2766 541 6556 417 6359 1242 240––
dpa::dpa::witness_colors::chain_40045 080 18863 452 81061 849 340194 50518 027––
dpa::dpa::witness_colors::scrambled_1004 026 6035 681 8845 586 2949 3391 397––
dpa::dpa::witness_colors::scrambled_40030 619 06543 036 14942 208 714146 5102 711––
dpa::dpa::witness_inequivalence::complement_100279 168420 996402 421873406––
dpa::dpa::witness_inequivalence::complement_2590 278142 456128 39689389––
dpa::dpa::witness_inequivalence::complement_50155 919237 938223 058316380––
dpa::dpa::witness_not_subset_of::complement_100269 593409 090390 465883406––
dpa::dpa::witness_not_subset_of::complement_2587 753139 280125 14098390––
dpa::dpa::witness_not_subset_of::complement_50151 044231 814216 884319381––
dpa::dpa::witness_not_subset_of::normalized_100542 474802 087774 5671 521569––
dpa::dpa::witness_not_subset_of::normalized_25184 635275 680261 725236365––
dpa::dpa::witness_not_subset_of::normalized_50313 246462 143445 828442403––
dpa::dpa::witnesses_fixtures::ltl2 217 3893 204 7073 135 3427 951846––
dpa::dpa::witnesses_fixtures::synthetic18 835 73628 666 81828 119 628103 327873––
game::game::tangle_learning::few_priorities_10002 697 0484 062 9303 713 17535 9104 863745 4001 292
game::game::tangle_learning::few_priorities_40008 120 67612 712 28611 104 496194 95618 0862 407 4163 266
game::game::tangle_learning::ladder_10001 179 3141 831 7261 627 97112 4434 044377 868118
game::game::tangle_learning::ladder_40004 408 1347 087 2366 022 281110 82614 5951 511 668225
game::game::tangle_learning::many_priorities_10002 018 0283 042 0902 758 58525 5234 454566 440848
game::game::tangle_learning::many_priorities_40006 752 83510 526 4309 145 870166 28915 6892 187 3441 620
game::game::tangle_learning_min_odd::few_priorities_10001 521 6152 322 1342 081 28426 3513 117525 072189
game::game::tangle_learning_min_odd::few_priorities_40008 017 70012 367 40010 863 250215 95512 1252 560 0641 933
game::game::zielonka::few_priorities_10001 126 1991 709 0211 520 53612 4553 606356 67695
game::game::zielonka::few_priorities_40003 470 2545 583 6874 684 11781 69014 0321 278 56484
game::game::zielonka::ladder_10005 134 2817 016 0776 609 66750 0904 456499 404182
game::game::zielonka::ladder_400019 816 37327 231 95125 422 846246 13916 5261 998 868212
game::game::zielonka::many_priorities_10004 103 5185 602 3035 264 43838 6914 126764 940339
game::game::zielonka::many_priorities_400012 909 83717 933 47616 554 676176 12914 2332 552 644288
game::game::zielonka_min_odd::few_priorities_10002 139 0773 023 3732 800 59323 7592 971512 012195
game::game::zielonka_min_odd::few_priorities_40006 302 0759 279 6878 309 227131 0299 0091 705 700158
learning::learning::lstar_mealy::counting_161 814 0562 474 5152 428 6957571 201154 5042 633
learning::learning::lstar_mealy::counting_8481 502684 445651 02522392340 472862
learning::learning::sprout_parity::inf_ab_1001 270 7571 850 2401 786 9352 2241 491200 4161 872
learning::learning::sprout_parity::inf_ab_4002 610 8203 782 7013 665 5919 2682 022405 0403 649
nondeterministic::nondeterministic::fixture_accepts::ltl762 5451 185 1021 120 5628 449637––
nondeterministic::nondeterministic::nba_accepts::nba_400125 269 894185 047 774174 142 9491 657 47074 785––
nondeterministic::nondeterministic::nba_witness::nba_400014 689 96121 476 74620 461 706138 4129 228––
nondeterministic::nondeterministic::nca_accepts::nca_400266 229 440395 082 740369 431 0054 559 08481 609––
nondeterministic::nondeterministic::nca_witness::nca_400019 458 26129 273 29527 117 250325 48815 103––
nondeterministic::nondeterministic::nma_witness::nma_100025 744 46937 890 67835 867 768371 1504 776––
nondeterministic::nondeterministic::npa_accepts::empty_npa_400493 167 601734 838 886687 188 0268 907 14489 004––
nondeterministic::nondeterministic::npa_accepts::npa_10030 411 62844 532 16242 656 897286 24412 687––
nondeterministic::nondeterministic::npa_accepts::npa_400125 279 542185 057 918174 152 9531 657 50574 784––
nondeterministic::nondeterministic::npa_witness::empty_npa_10009 121 95213 459 23712 723 917123 4323 376––
nondeterministic::nondeterministic::npa_witness::empty_npa_400037 023 30355 330 02651 364 426657 23619 412––
nondeterministic::nondeterministic::npa_witness::npa_10002 479 5313 680 1793 513 79420 7401 791––
nondeterministic::nondeterministic::npa_witness::npa_400020 354 25529 430 89628 218 496177 8989 226––
nondeterministic::nondeterministic::nra_witness::nra_400020 639 56731 400 77128 850 876337 11424 695––
runs::runs::dfa_accepts_many_words::count_10004 303 2135 381 9795 357 6494 7891100
runs::runs::dfa_accepts_one_word::len_1003 4924 8664 39638800
runs::runs::dfa_accepts_one_word::len_10000266 070329 557325 827697700
runs::runs::dfa_accepts_one_word::len_1000002 650 7313 276 8893 244 7096 3241600
runs::runs::dfa_reached_state::count_10003 570 2114 404 9084 380 6484 789900
runs::runs::dpa_accepts_lassos::long9 150 74511 671 08311 646 6333 077259724 664500
runs::runs::dpa_accepts_lassos::short406 490540 866535 5963619926 236472
runs::runs::dpa_recurrent_edge_colors::long15 116 26619 231 17719 179 4625 835644922 1442 090
runs::runs::dpa_recurrent_edge_colors::short1 189 7901 666 3231 655 0081 157158168 6841 708
runs::runs::dpa_recurrent_states::long10 502 98813 660 28413 630 4043 519351788 104860
runs::runs::dpa_recurrent_states::short572 037764 556757 26653413281 996782
runs::runs::dpa_visited_sequences::long8 738 26312 085 90812 050 9984 6513332 190 780850
runs::runs::dpa_visited_sequences::short370 678510 989506 2944986329 986545
runs::runs::fixture_accepts_lassos::ltl8 766 27112 652 15612 620 1765 542122––
runs::runs::fixture_accepts_lassos::synthetic4 340 1896 192 9936 139 1539 907123––
scc::scc::scc_queries::small_cycles_1000906 6761 433 5441 272 0949 4283 266261 132559
scc::scc::scc_queries::small_cycles_40003 548 9195 658 3944 973 93950 91712 2821 045 3962 071
scc::scc::tarjan::back_edges_1000512 099838 107726 7775 4732 399170 27245
scc::scc::tarjan::back_edges_40001 987 6273 267 5342 821 39426 4528 968680 63253
scc::scc::tarjan::binary_tree_1000571 217892 769815 6944 0611 622105 76039
scc::scc::tarjan::binary_tree_40002 290 4593 582 5733 276 41321 1295 729419 51245
scc::scc::tarjan::fragmented_1000658 5221 046 852922 7179 6442 169146 084121
scc::scc::tarjan::fragmented_40002 534 1084 125 8593 544 65462 7897 636569 852167
scc::scc::tarjan::grid_1024596 080972 081847 7617 7282 448171 16045
scc::scc::tarjan::grid_40962 378 4844 001 3513 363 70163 6699 123684 18453
scc::scc::tarjan::path_10001 929 1193 028 5232 729 56823 1885 229455 2242 064
scc::scc::tarjan::path_40007 632 90812 001 66410 804 76498 32320 1511 821 5368 076
scc::scc::tarjan::ring_1000512 099841 747725 8676 3832 399170 27245
scc::scc::tarjan::ring_40001 987 6273 284 7822 817 08230 7648 968680 63253
scc::scc::tarjan::scrambled_1000579 056936 579824 0296 9562 222170 27245
scc::scc::tarjan::scrambled_40002 316 4973 791 2493 296 77941 8728 146680 63253
scc::scc::tarjan::small_cycles_1000905 6581 432 2191 270 8099 4343 264261 132559
scc::scc::tarjan::small_cycles_40003 544 9015 650 7854 969 53050 29812 2791 045 3962 071
scc::scc::tarjan_from::fragmented_1000242 530394 598344 8782 5241 06074 98564
scc::scc::tarjan_from::fragmented_4000907 7621 492 3421 296 66715 9233 316299 62578
scc::scc::tarjan_from::path_10001 921 1583 018 2442 722 14423 0375 169468 9932 091
scc::scc::tarjan_from::path_40007 545 58411 928 68110 697 651106 73119 9251 876 7538 109
scc::scc::tarjan_from::ring_1000503 645836 974717 9746 2512 507184 04172
scc::scc::tarjan_from::scrambled_1000570 602930 380816 4856 4762 329184 04172
storage::storage::add_edges_dense::k2_50001 356 4012 030 4901 985 3454 941584180 20037
storage::storage::add_edges_dense::k8_50004 247 0596 411 8746 225 39417 8362 780671 72840
storage::storage::add_edges_linked::k2_50001 402 6702 420 4692 258 38414 2242 5991 113 96026
storage::storage::add_edges_linked::k8_50006 669 28411 260 27310 330 80875 81815 7253 866 49628
storage::storage::clone_csr::k2_1000030 590183 48052 6254 1703 143230 0126
storage::storage::clone_dense::k2_1000015 11381 34225 8022 2531 265110 0084
storage::storage::clone_linked::k2_10000370 824901 703529 79812 4668 845680 0083
storage::storage::complete_dense::partial_5000155 628209 494202 329908751364
storage::storage::complete_linked::partial_50001 210 8541 718 8451 629 4506 9521 56165 67218
storage::storage::edges_from_all_states_csr::k2_200072 04777 21475 859222700
storage::storage::edges_from_all_states_csr::k8_2000108 047116 392111 0721 008800
storage::storage::edges_from_all_states_dense::k2_2000102 043119 287117 842233800
storage::storage::edges_from_all_states_dense::k8_2000140 043160 529155 0391 035900
storage::storage::edges_from_all_states_linked::k2_200068 03386 72575 9252 132400
storage::storage::edges_from_all_states_linked::k8_2000152 033203 701166 6817 376400
storage::storage::minimize_csr::redundant_20004 633 7567 005 0566 522 11157 8935 5281 505 4202 212
storage::storage::minimize_csr::redundant_800018 159 86127 786 74025 537 970272 65425 3006 023 3008 250
storage::storage::minimize_dense::redundant_20004 676 9107 035 6516 577 16658 1534 7921 522 1562 221
storage::storage::minimize_dense::redundant_800018 364 89528 117 70225 817 247279 05025 8636 090 3408 261
storage::storage::minimize_linked::redundant_20004 505 0386 920 7686 368 86861 9266 9221 522 1562 221
storage::storage::minimize_linked::redundant_800017 714 72127 484 31425 033 274302 87426 7626 090 3408 261
storage::storage::overwrite_edges_dense::k2_50002 126 4313 232 5183 172 4287 916586180 20037
storage::storage::overwrite_edges_linked::k2_50003 790 7756 671 1135 952 01844 76914 1502 031 46427
storage::storage::predecessors_of_all_states_csr::k2_2000102 046121 042117 407678700
storage::storage::predecessors_of_all_states_dense::k2_2000290 369397 907388 2221 6224532 0083
storage::storage::predecessors_of_all_states_linked::k2_200068 03391 72974 6893 366600
storage::storage::product_successor_csr::n20064 09675 54575 16014900
storage::storage::product_successor_dense::n20073 08794 62094 10540900
storage::storage::product_successor_linked::n20046 54760 17557 015576800
storage::storage::reachable_states_csr::k2_80001 101 1151 630 1181 591 8437 4173417 36811
storage::storage::reachable_states_dense::k2_80001 133 6991 684 5051 639 2458 8283217 36811
storage::storage::reachable_states_linked::k2_8000924 2791 485 5951 386 51519 5294117 36811
storage::storage::remove_state_dense::k2_100001 072 2121 565 0861 390 28621 5131 921176 0001 003
storage::storage::remove_state_linked::k2_100001 094 0121 531 4791 492 7347 4204780 0003 000
storage::storage::run_many_words_csr::count_1000551 036653 943644 1631 907700
storage::storage::run_many_words_dense::count_1000558 044690 031680 1661 910900
storage::storage::run_many_words_linked::count_1000481 044611 149601 1191 957700
storage::storage::run_one_word_csr::len_1000002 600 0543 025 4282 993 8236 272700
storage::storage::run_one_word_dense::len_1000002 800 0503 425 4573 393 8126 273800
storage::storage::run_one_word_linked::len_1000002 350 0392 975 4282 943 7736 296500
storage::storage::successor_all_pairs_csr::k2_2000122 242139 664138 0992291281
storage::storage::successor_all_pairs_csr::k8_2000702 302802 872797 3721 02311321
storage::storage::successor_all_pairs_dense::k2_2000162 237193 545192 090235881
storage::storage::successor_all_pairs_dense::k8_2000814 297952 851947 3461 0458321
storage::storage::successor_all_pairs_linked::k2_2000108 236139 158128 1782 140881
storage::storage::successor_all_pairs_linked::k8_2000750 296946 204908 9997 3858321
ts::ts::add_edges_in_order::k2_50001 362 6712 340 4412 178 38614 2252 5981 113 96026
ts::ts::add_edges_in_order::k8_50006 509 28510 940 22910 010 81475 81515 7243 866 49628
ts::ts::add_edges_in_reverse::k2_50001 362 6732 339 6392 178 58914 0242 5981 113 96026
ts::ts::builder_with_edges::k2_500081 357 91598 210 23295 707 647451 7836 9621 739 53245
ts::ts::canonical_naming::scrambled_15003 178 5994 676 7424 486 54221 8072 319463 8085 846
ts::ts::canonical_naming::scrambled_500984 0741 424 7541 380 9643 158800134 3361 831
ts::ts::clone::scrambled_10000371 163902 473530 23812 4838 852680 0083
ts::ts::clone::scrambled_400001 481 4003 817 8862 116 80642 57642 5202 720 0083
ts::ts::collect_dts::scrambled_100003 465 1496 166 0485 606 84837 17110 6672 228 06427
ts::ts::collect_dts::scrambled_4000013 799 62824 683 15422 329 479135 29547 9208 912 73631
ts::ts::determinize::nth_from_end_104 437 3976 586 4156 224 98022 1817 158651 7206 013
ts::ts::determinize::nth_from_end_1219 671 28629 027 03127 516 93698 94929 0102 732 60026 423
ts::ts::determinize::nth_from_end_81 016 0701 511 3351 428 0003 0381 947156 5521 396
ts::ts::determinize_propositional::aps_6_n_613 066 75319 949 75519 523 57015 8259 9161 080 6924 064
ts::ts::determinize_propositional::aps_7_n_6172 153 525254 472 146250 938 991576 55718 5821 357 63421 321
ts::ts::determinize_propositional::aps_8_n_61 736 746 6632 578 731 4722 529 136 3429 367 23778 8276 652 416163 407
ts::ts::edges_from_all_states::scrambled_200072 66991 97580 7752 1561200
ts::ts::edges_from_all_states::scrambled_k8_2000156 601208 890171 4307 4011300
ts::ts::erase_state_colors_collect::scrambled_100002 887 1665 148 2714 588 85637 17910 6722 228 06427
ts::ts::into_dfa::scrambled_100003 565 1666 276 2025 716 87737 18210 6692 228 06427
ts::ts::map_edge_colors_collect::scrambled_100002 888 7685 278 0924 659 55238 44812 1802 490 19227
ts::ts::map_edge_colors_collect::scrambled_4000011 495 39121 132 09618 542 196140 33053 9509 961 29631
ts::ts::matching_product_successor_index::n20075 662111 104106 4646484000
ts::ts::minimal_representative_of_all_states::scrambled_1002 102 8112 971 4192 967 67913105347 5044 679
ts::ts::minimal_representative_of_all_states::scrambled_30016 917 26123 990 26223 976 9321 9101083 116 78438 196
ts::ts::minimal_representatives::scrambled_20003 386 0404 764 0224 717 2478 90065322 9447 709
ts::ts::minimal_representatives::scrambled_800014 908 24920 957 76720 742 92742 576561 464 89632 693
ts::ts::predecessors_of_all_states::scrambled_200072 66996 94979 5393 3911300
ts::ts::predecessors_of_all_states::scrambled_8000288 601389 396313 81115 0191400
ts::ts::reachable_states::scrambled_2000290 091461 017439 9923 617844 3369
ts::ts::reachable_states::scrambled_80001 148 6961 838 5681 739 08319 5265317 36811
ts::ts::remove_edges_from_matching::scrambled_10000567 234796 995772 6454 6603064 0002 000
ts::ts::remove_edges_from_matching::scrambled_40000897 2341 178 4951 127 27010 0353064 0002 000
ts::ts::remove_state::scrambled_100001 094 0121 531 4111 492 7367 4204580 0003 000
ts::ts::remove_state::scrambled_400001 424 0121 915 8751 846 62013 5364580 0003 000
ts::ts::scc_minimal_representatives::small_cycles_100219 916321 711306 0816043844 272309
ts::ts::scc_minimal_representatives::small_cycles_4001 246 1221 777 6751 727 2703 872887354 1481 325
ts::ts::subset_construction::nth_from_end_1010 160 16814 437 34113 942 40145 4947 6421 073 5209 919
ts::ts::subset_construction::nth_from_end_1244 980 93463 701 67761 456 052218 76932 9084 518 97643 857
ts::ts::subset_construction::nth_from_end_82 216 4733 167 8093 059 3296 2052 213257 1042 276
ts::ts::successor_index_all_pairs::scrambled_2000108 863140 293129 0232 1561481
ts::ts::successor_index_all_pairs::scrambled_k8_2000750 855947 278909 7437 40215321

Wall-clock times

Criterion means of a single run. They depend on the machine and are not compared; use the instruction counts above for that.

benchmarkhead
collect/clone/1000004.28 ms (4.19 ms … 4.36 ms)
collect/clone/100000042.35 ms (42.32 ms … 42.39 ms)
collect/collect_owned/1000008.32 ms (8.14 ms … 8.50 ms)
collect/collect_owned/100000090.86 ms (88.52 ms … 93.21 ms)
collect/collect_ref/1000005.28 ms (5.26 ms … 5.30 ms)
collect/collect_ref/100000084.37 ms (83.43 ms … 85.31 ms)
collect/map_edge_colors_collect_owned/1000008.01 ms (7.87 ms … 8.15 ms)
collect/map_edge_colors_collect_owned/100000086.50 ms (84.47 ms … 88.54 ms)
wallclock/dfa/accepts/word_100000835.68 µs (828.62 µs … 842.75 µs)
wallclock/dfa/minimize/redundant_40001.44 ms (1.42 ms … 1.45 ms)
wallclock/dpa/complement/scrambled_40003.74 ms (3.68 ms … 3.81 ms)
wallclock/dpa/normalized/scrambled_40003.40 ms (3.40 ms … 3.41 ms)
wallclock/hoa/parse/ltl953.13 µs (949.84 µs … 956.42 µs)
wallclock/nfa/determinize/nth_from_end_123.80 ms (3.73 ms … 3.87 ms)
wallclock/nfa/subset_construction/nth_from_end_1210.31 ms (10.27 ms … 10.35 ms)
wallclock/scc/tarjan/scrambled_100002.28 ms (2.26 ms … 2.30 ms)

Versus Spot and Oink

dpa/equivalence

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot146.38 µs115.34 µs1.27×– / –yes
hard/l-nba-2spot67.98 µs35.42 µs1.92×– / –yes
hard/m-nba-3spot278.94 µs135.14 µs2.06×– / –yes
ltl/and-gf-3spot11.20 µs11.90 µs0.94×– / –yes
ltl/eh-patterns-0spot29.02 µs19.96 µs1.45×– / –yes
ltl/gf-equiv-3spot26.28 µs33.44 µs0.79×– / –yes
ltl/gf-implies-3spot13.74 µs18.98 µs0.72×– / –yes
ltl/gh-q-3spot62.22 µs28.06 µs2.22×– / –yes
ltl/gh-r-3spot27.70 µs68.36 µs0.41×– / –yes
ltl/go-theta-3spot13.90 µs21.00 µs0.66×– / –yes
ltl/lily-patterns-1spot14.96 µs15.76 µs0.95×– / –yes
ltl/ms-phi-r-1spot13.16 µs37.56 µs0.35×– / –yes
ltl/ms-phi-s-1spot13.08 µs38.82 µs0.34×– / –yes
ltl/rv-counter-3spot31.90 µs12.86 µs2.48×– / –yes
ltl/sb-patterns-0spot15.68 µs13.40 µs1.17×– / –yes
ltl/sejk-k-2spot17.16 µs57.14 µs0.30×– / –yes
ltl/tv-uu-3spot48.38 µs36.40 µs1.33×– / –yes
ltl/u-left-3spot12.76 µs8.24 µs1.55×– / –yes
many_aps/and-gf-12spot181.72 µs66.54 µs2.73×– / –yes
many_aps/gf-equiv-12spot152.88 µs89.58 µs1.71×– / –yes
many_aps/r-right-10spot75.48 µs24.82 µs3.04×– / –yes
many_aps/random-n16-a8-p4-s1spot306.46 µs128.46 µs2.39×– / –yes
many_aps/random-n24-a12-p4-s1spot1.55 ms454.40 µs3.42×– / –yes
many_aps/random-n32-a16-p4-s1spot4.74 ms1.27 ms3.75×– / –yes
random/n128-a2-p6-s1spot371.04 µs206.56 µs1.80×– / –yes
random/n16-a1-p3-s1spot25.28 µs15.20 µs1.66×– / –yes
random/n256-a2-p8-s1spot999.79 µs477.40 µs2.09×– / –yes
random/n64-a2-p4-s1spot156.98 µs98.36 µs1.60×– / –yes

dpa/inclusion

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot80.20 µs60.18 µs1.33×– / –yes
hard/l-nba-2spot40.60 µs22.72 µs1.79×– / –yes
hard/m-nba-3spot166.60 µs81.84 µs2.04×– / –yes
ltl/and-gf-3spot7.12 µs8.02 µs0.89×– / –yes
ltl/eh-patterns-0spot18.22 µs16.84 µs1.08×– / –yes
ltl/gf-equiv-3spot14.56 µs18.28 µs0.80×– / –yes
ltl/gf-implies-3spot7.64 µs11.00 µs0.69×– / –yes
ltl/gh-q-3spot38.82 µs26.96 µs1.44×– / –yes
ltl/gh-r-3spot15.32 µs34.16 µs0.45×– / –yes
ltl/go-theta-3spot8.96 µs15.56 µs0.58×– / –yes
ltl/lily-patterns-1spot9.96 µs13.26 µs0.75×– / –yes
ltl/ms-phi-r-1spot7.76 µs18.62 µs0.42×– / –yes
ltl/ms-phi-s-1spot7.72 µs24.38 µs0.32×– / –yes
ltl/rv-counter-3spot19.98 µs13.82 µs1.45×– / –yes
ltl/sb-patterns-0spot9.76 µs11.68 µs0.84×– / –yes
ltl/sejk-k-2spot10.02 µs27.68 µs0.36×– / –yes
ltl/tv-uu-3spot32.78 µs28.68 µs1.14×– / –yes
ltl/u-left-3spot8.32 µs8.56 µs0.97×– / –yes
many_aps/and-gf-12spot151.40 µs53.32 µs2.84×– / –yes
many_aps/gf-equiv-12spot86.34 µs50.00 µs1.73×– / –yes
many_aps/r-right-10spot54.58 µs22.86 µs2.39×– / –yes
many_aps/random-n16-a8-p4-s1spot247.72 µs97.50 µs2.54×– / –yes
many_aps/random-n24-a12-p4-s1spot1.58 ms399.12 µs3.95×– / –yes
many_aps/random-n32-a16-p4-s1spot4.64 ms1.16 ms4.01×– / –yes
random/n128-a2-p6-s1spot219.48 µs126.24 µs1.74×– / –yes
random/n16-a1-p3-s1spot15.32 µs9.92 µs1.54×– / –yes
random/n256-a2-p8-s1spot574.59 µs307.50 µs1.87×– / –yes
random/n64-a2-p4-s1spot94.60 µs61.50 µs1.54×– / –yes

game/solve

inputreferenceoursreferenceours / referencesize (ours / reference)same result
ladder_10000oink-ppp3.20 ms315.00 µs10.17×10000 / 10000yes
ladder_10000oink-zlk3.20 ms322.50 µs9.94×10000 / 10000yes
ladder_10000spot3.20 ms9.78 ms0.33×10000 / 10000yes
ladder_50000oink-ppp23.57 ms2.15 ms10.98×50000 / 50000yes
ladder_50000oink-zlk23.57 ms2.20 ms10.73×50000 / 50000yes
ladder_50000spot23.57 ms75.82 ms0.31×50000 / 50000yes
random_few_10000oink-ppp8.66 ms1.53 ms5.66×10000 / 10000yes
random_few_10000oink-zlk8.66 ms2.83 ms3.05×10000 / 10000yes
random_few_10000spot8.66 ms21.77 ms0.40×10000 / 10000yes
random_few_50000oink-ppp92.60 ms19.75 ms4.69×50000 / 50000yes
random_few_50000oink-zlk92.60 ms22.23 ms4.17×50000 / 50000yes
random_few_50000spot92.60 ms799.18 ms0.12×50000 / 50000yes
random_many_10000oink-ppp7.83 ms1.71 ms4.57×10000 / 10000yes
random_many_10000oink-zlk7.83 ms3.06 ms2.56×10000 / 10000yes
random_many_10000spot7.83 ms31.62 ms0.25×10000 / 10000yes
random_many_50000oink-ppp89.28 ms37.97 ms2.35×50000 / 50000yes
random_many_50000oink-zlk89.28 ms51.08 ms1.75×50000 / 50000yes
random_many_50000spot89.28 ms806.32 ms0.11×50000 / 50000yes

Log

== head f83ff30f95eb8e9d5a3346d6a33cd69eb4027dbb: building ["scc", "ts", "runs", "dfa", "dpa", "storage", "nondeterministic", "learning", "game"] ["wallclock"]
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
    Finished `bench` profile [optimized] target(s) in 2m 11s
  Executable benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-a84ba41f760e70ed)
  Executable benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-23bb829f074058ad)
  Executable benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-14422f732ec85029)
  Executable benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-ee53c00379271c59)
  Executable benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-f164d05108dbce73)
  Executable benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-85a152aa046a29b4)
  Executable benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-1b22c641fd9d2d89)
  Executable benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-f95e3d4a8c506941)
  Executable benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-4948f6cc6a43d201)
  Executable benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-93fb383f608d3bab)
== head: running scc
    Finished `bench` profile [optimized] target(s) in 0.23s
     Running benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-1b22c641fd9d2d89)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running ts
    Finished `bench` profile [optimized] target(s) in 0.19s
     Running benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-4948f6cc6a43d201)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running runs
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-85a152aa046a29b4)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dfa
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-a84ba41f760e70ed)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dpa
    Finished `bench` profile [optimized] target(s) in 0.27s
     Running benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-23bb829f074058ad)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running storage
    Finished `bench` profile [optimized] target(s) in 0.23s
     Running benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-f95e3d4a8c506941)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running nondeterministic
    Finished `bench` profile [optimized] target(s) in 0.20s
     Running benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-f164d05108dbce73)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running learning
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-ee53c00379271c59)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running game
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-14422f732ec85029)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running wallclock (wall clock)
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-93fb383f608d3bab)
Gnuplot not found, using plotters backend
Benchmarking wallclock/scc/tarjan/scrambled_10000
Benchmarking wallclock/scc/tarjan/scrambled_10000: Analyzing
Benchmarking wallclock/dpa/normalized/scrambled_4000
Benchmarking wallclock/dpa/normalized/scrambled_4000: Analyzing
Benchmarking wallclock/dpa/complement/scrambled_4000
Benchmarking wallclock/dpa/complement/scrambled_4000: Analyzing
Benchmarking wallclock/dfa/minimize/redundant_4000
Benchmarking wallclock/dfa/minimize/redundant_4000: Analyzing
Benchmarking wallclock/dfa/accepts/word_100000
Benchmarking wallclock/dfa/accepts/word_100000: Analyzing
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12: Analyzing
Benchmarking wallclock/nfa/determinize/nth_from_end_12
Benchmarking wallclock/nfa/determinize/nth_from_end_12: Analyzing
Benchmarking wallclock/hoa/parse/ltl
Benchmarking wallclock/hoa/parse/ltl: Analyzing
Benchmarking collect/clone/100000
Benchmarking collect/clone/100000: Analyzing
Benchmarking collect/collect_owned/100000
Benchmarking collect/collect_owned/100000: Analyzing
Benchmarking collect/collect_ref/100000
Benchmarking collect/collect_ref/100000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/100000
Benchmarking collect/map_edge_colors_collect_owned/100000: Analyzing
Benchmarking collect/clone/1000000
Benchmarking collect/clone/1000000: Analyzing
Benchmarking collect/collect_owned/1000000
Benchmarking collect/collect_owned/1000000: Analyzing
Benchmarking collect/collect_ref/1000000
Benchmarking collect/collect_ref/1000000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/1000000
Benchmarking collect/map_edge_colors_collect_owned/1000000: Analyzing
== head: building the Python bindings
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support
🐍 Found CPython 3.13 at /nix/store/vilynzly1pyhawy04ljmrp7n9ffr24qy-python3-3.13.17/bin/python3
📡 Using build options features from pyproject.toml
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
   Compiling automata-py v0.1.0 (/var/lib/automata-bench/work/src-head/python)
    Finished `release` profile [optimized] target(s) in 1m 54s
📦 Built wheel for abi3 Python ≥ 3.11 to /var/lib/automata-bench/work/versus-head/wheels/automatars-0.1.0-cp311-abi3-manylinux_2_34_aarch64.whl
== head: running benches/versus/compare.py (Spot, Oink)
dpa/equivalence/hard/l-dsa-3 vs spot: 0.146 ms / 0.115 ms
dpa/equivalence/hard/l-nba-2 vs spot: 0.068 ms / 0.035 ms
dpa/equivalence/hard/m-nba-3 vs spot: 0.279 ms / 0.135 ms
dpa/equivalence/ltl/and-gf-3 vs spot: 0.011 ms / 0.012 ms
dpa/equivalence/ltl/eh-patterns-0 vs spot: 0.029 ms / 0.020 ms
dpa/equivalence/ltl/gf-equiv-3 vs spot: 0.026 ms / 0.033 ms
dpa/equivalence/ltl/gf-implies-3 vs spot: 0.014 ms / 0.019 ms
dpa/equivalence/ltl/gh-q-3 vs spot: 0.062 ms / 0.028 ms
dpa/equivalence/ltl/gh-r-3 vs spot: 0.028 ms / 0.068 ms
dpa/equivalence/ltl/go-theta-3 vs spot: 0.014 ms / 0.021 ms
dpa/equivalence/ltl/lily-patterns-1 vs spot: 0.015 ms / 0.016 ms
dpa/equivalence/ltl/ms-phi-r-1 vs spot: 0.013 ms / 0.038 ms
dpa/equivalence/ltl/ms-phi-s-1 vs spot: 0.013 ms / 0.039 ms
dpa/equivalence/ltl/rv-counter-3 vs spot: 0.032 ms / 0.013 ms
dpa/equivalence/ltl/sb-patterns-0 vs spot: 0.016 ms / 0.013 ms
dpa/equivalence/ltl/sejk-k-2 vs spot: 0.017 ms / 0.057 ms
dpa/equivalence/ltl/tv-uu-3 vs spot: 0.048 ms / 0.036 ms
dpa/equivalence/ltl/u-left-3 vs spot: 0.013 ms / 0.008 ms
dpa/equivalence/many_aps/and-gf-12 vs spot: 0.182 ms / 0.067 ms
dpa/equivalence/many_aps/gf-equiv-12 vs spot: 0.153 ms / 0.090 ms
dpa/equivalence/many_aps/r-right-10 vs spot: 0.075 ms / 0.025 ms
dpa/equivalence/many_aps/random-n16-a8-p4-s1 vs spot: 0.306 ms / 0.128 ms
dpa/equivalence/many_aps/random-n24-a12-p4-s1 vs spot: 1.554 ms / 0.454 ms
dpa/equivalence/many_aps/random-n32-a16-p4-s1 vs spot: 4.741 ms / 1.266 ms
dpa/equivalence/random/n128-a2-p6-s1 vs spot: 0.371 ms / 0.207 ms
dpa/equivalence/random/n16-a1-p3-s1 vs spot: 0.025 ms / 0.015 ms
dpa/equivalence/random/n256-a2-p8-s1 vs spot: 1.000 ms / 0.477 ms
dpa/equivalence/random/n64-a2-p4-s1 vs spot: 0.157 ms / 0.098 ms
dpa/inclusion/hard/l-dsa-3 vs spot: 0.080 ms / 0.060 ms
dpa/inclusion/hard/l-nba-2 vs spot: 0.041 ms / 0.023 ms
dpa/inclusion/hard/m-nba-3 vs spot: 0.167 ms / 0.082 ms
dpa/inclusion/ltl/and-gf-3 vs spot: 0.007 ms / 0.008 ms
dpa/inclusion/ltl/eh-patterns-0 vs spot: 0.018 ms / 0.017 ms
dpa/inclusion/ltl/gf-equiv-3 vs spot: 0.015 ms / 0.018 ms
dpa/inclusion/ltl/gf-implies-3 vs spot: 0.008 ms / 0.011 ms
dpa/inclusion/ltl/gh-q-3 vs spot: 0.039 ms / 0.027 ms
dpa/inclusion/ltl/gh-r-3 vs spot: 0.015 ms / 0.034 ms
dpa/inclusion/ltl/go-theta-3 vs spot: 0.009 ms / 0.016 ms
dpa/inclusion/ltl/lily-patterns-1 vs spot: 0.010 ms / 0.013 ms
dpa/inclusion/ltl/ms-phi-r-1 vs spot: 0.008 ms / 0.019 ms
dpa/inclusion/ltl/ms-phi-s-1 vs spot: 0.008 ms / 0.024 ms
dpa/inclusion/ltl/rv-counter-3 vs spot: 0.020 ms / 0.014 ms
dpa/inclusion/ltl/sb-patterns-0 vs spot: 0.010 ms / 0.012 ms
dpa/inclusion/ltl/sejk-k-2 vs spot: 0.010 ms / 0.028 ms
dpa/inclusion/ltl/tv-uu-3 vs spot: 0.033 ms / 0.029 ms
dpa/inclusion/ltl/u-left-3 vs spot: 0.008 ms / 0.009 ms
dpa/inclusion/many_aps/and-gf-12 vs spot: 0.151 ms / 0.053 ms
dpa/inclusion/many_aps/gf-equiv-12 vs spot: 0.086 ms / 0.050 ms
dpa/inclusion/many_aps/r-right-10 vs spot: 0.055 ms / 0.023 ms
dpa/inclusion/many_aps/random-n16-a8-p4-s1 vs spot: 0.248 ms / 0.098 ms
dpa/inclusion/many_aps/random-n24-a12-p4-s1 vs spot: 1.578 ms / 0.399 ms
dpa/inclusion/many_aps/random-n32-a16-p4-s1 vs spot: 4.638 ms / 1.157 ms
dpa/inclusion/random/n128-a2-p6-s1 vs spot: 0.219 ms / 0.126 ms
dpa/inclusion/random/n16-a1-p3-s1 vs spot: 0.015 ms / 0.010 ms
dpa/inclusion/random/n256-a2-p8-s1 vs spot: 0.575 ms / 0.308 ms
dpa/inclusion/random/n64-a2-p4-s1 vs spot: 0.095 ms / 0.062 ms
game/solve/random_few_10000 vs spot: 8.658 ms / 21.766 ms
game/solve/random_few_10000 vs oink-zlk: 8.658 ms / 2.834 ms
game/solve/random_few_10000 vs oink-ppp: 8.658 ms / 1.530 ms
game/solve/random_few_50000 vs spot: 92.603 ms / 799.181 ms
game/solve/random_few_50000 vs oink-zlk: 92.603 ms / 22.232 ms
game/solve/random_few_50000 vs oink-ppp: 92.603 ms / 19.748 ms
game/solve/random_many_10000 vs spot: 7.831 ms / 31.617 ms
game/solve/random_many_10000 vs oink-zlk: 7.831 ms / 3.055 ms
game/solve/random_many_10000 vs oink-ppp: 7.831 ms / 1.714 ms
game/solve/random_many_50000 vs spot: 89.280 ms / 806.317 ms
game/solve/random_many_50000 vs oink-zlk: 89.280 ms / 51.084 ms
game/solve/random_many_50000 vs oink-ppp: 89.280 ms / 37.973 ms
game/solve/ladder_10000 vs spot: 3.205 ms / 9.780 ms
game/solve/ladder_10000 vs oink-zlk: 3.205 ms / 0.323 ms
game/solve/ladder_10000 vs oink-ppp: 3.205 ms / 0.315 ms
game/solve/ladder_50000 vs spot: 23.566 ms / 75.820 ms
game/solve/ladder_50000 vs oink-zlk: 23.566 ms / 2.196 ms
game/solve/ladder_50000 vs oink-ppp: 23.566 ms / 2.147 ms
== head: 262 benchmarks, 16 wall-clock measurements, 74 comparisons