automata benchmarks

History · Versus Spot and Oink

main · 38e9f0d4

Atoms for propositional alphabets: faster determinization (#157) (#162)

status done · queued 2026-11-11 13:37 · started 2026-11-11 13:37 · finished 2026-11-11 13:48

· · rustc 1.95.0 (59807616e 2026-04-14) (built from a source tarball) · valgrind-3.26.0 · pinned to cpu 1

Results

benchmarkinstructionsest. cyclesL1 hitsLL hitsRAM hitsheap bytesallocations
dfa::dfa::accepts_many_words::scrambled_1000343 072435 464428 9441 2201200
dfa::dfa::complement::scrambled_1000366 068584 238566 3333 39926139 10419
dfa::dfa::complement::scrambled_40001 435 8732 309 0932 223 22316 99925556 89623
dfa::dfa::equivalent::deep_difference_10061 66797 78690 4714920216 19625
dfa::dfa::equivalent::deep_difference_400212 158336 680314 2001 85037865 36433
dfa::dfa::equivalent::different_1009 04715 44312 5688811 80412
dfa::dfa::equivalent::different_4009 08815 64412 52478781 80412
dfa::dfa::equivalent::minimized_10035 84954 51351 888105607 75617
dfa::dfa::equivalent::minimized_400120 587182 487176 6977246231 58023
dfa::dfa::give_word::empty_10001 872 6132 558 6052 531 0304 92784152 8643 738
dfa::dfa::give_word::empty_40006 930 8239 656 9619 554 28620 05968679 80815 828
dfa::dfa::give_word::far_1000164 342231 589227 0345475228 176394
dfa::dfa::give_word::far_4000946 9821 329 3961 307 3913 92568148 4642 233
dfa::dfa::intersection::scrambled_1003 319 7095 217 4015 038 98612 0653 374428 08842
dfa::dfa::intersection::scrambled_20013 224 87321 169 19120 030 86197 14418 6461 708 12049
dfa::dfa::intersection_is_empty::empty_10059 26694 00187 3763018515 70020
dfa::dfa::intersection_is_empty::empty_200109 269175 048161 72861329331 58023
dfa::dfa::is_empty_language::empty_10001 872 6152 558 6772 531 0324 92786152 8643 738
dfa::dfa::is_empty_language::empty_40006 930 8259 656 9659 554 29020 05968679 80815 828
dfa::dfa::minimize::minimal_10003 450 9245 260 2624 882 20739 7715 1201 135 4922 214
dfa::dfa::minimize::minimal_400013 472 60220 707 23819 046 968204 64718 2014 543 7248 254
dfa::dfa::minimize::redundant_10002 292 8073 485 4273 247 03225 9373 106760 7801 201
dfa::dfa::minimize::redundant_40008 869 10613 616 89312 556 903143 6579 7633 044 9004 241
dfa::dfa::minimize::scrambled_10003 670 2565 623 8715 188 59651 1105 1351 116 0522 217
dfa::dfa::minimize::scrambled_400014 250 46922 214 47420 104 804294 19818 2484 470 1408 261
dfa::dfa::minimize_moore_with_edge_colors::scrambled_10003 669 2525 616 6885 186 58350 1115 1301 116 0522 217
dfa::dfa::minimize_moore_with_edge_colors::scrambled_400014 239 07322 181 42820 076 193300 69617 1934 470 1408 261
dfa::dfa::union::scrambled_1003 319 7095 217 4055 038 98512 0663 374428 08842
dfa::dfa::union::scrambled_20013 224 87321 169 15720 030 86297 14418 6451 708 12049
dpa::dpa::accepted_word::chain_100113 482174 665162 01053354––
dpa::dpa::accepted_word::chain_400383 293582 364547 2342 966580––
dpa::dpa::accepted_word::scrambled_100145 688219 606206 48142369––
dpa::dpa::accepted_word::scrambled_400496 798738 926705 0262 818566––
dpa::dpa::accepted_word_fixtures::ltl546 210807 833778 2182 717458––
dpa::dpa::accepted_word_fixtures::synthetic1 402 2962 089 6622 013 63212 441395––
dpa::dpa::accepted_word_fixtures_by_atoms::ltl1 734 6752 593 6312 518 22610 496655––
dpa::dpa::accepted_word_fixtures_by_atoms::synthetic2 873 2564 313 7444 220 06914 220645––
dpa::dpa::collect_dpa::scrambled_1000299 640481 513462 2833 8046––
dpa::dpa::collect_dpa::scrambled_40001 169 0311 929 9181 806 24816 4041 190––
dpa::dpa::complement::chain_10004 096 9406 248 5245 745 04455 1546 506––
dpa::dpa::complement::chain_400016 653 21125 541 84823 300 378261 94026 622––
dpa::dpa::complement::scrambled_10004 958 4947 462 3116 915 74162 8346 640––
dpa::dpa::complement::scrambled_400020 037 45830 543 21427 877 684344 30926 971––
dpa::dpa::complement_fixtures::ltl1 480 5032 153 2992 109 0745 555470––
dpa::dpa::complement_fixtures::many_aps13 919 76420 572 84020 081 70087 2031 575––
dpa::dpa::complement_fixtures::synthetic9 652 58313 851 25513 340 56598 659497––
dpa::dpa::complement_without_normalizing::scrambled_1000574 441820 766791 7715 17689––
dpa::dpa::complement_without_normalizing::scrambled_40002 259 8233 267 0113 112 52121 9801 274––
dpa::dpa::equivalent_fixtures::many_aps21 649 39032 289 80331 617 648121 6421 827––
dpa::dpa::hoa_parse::ltl5 509 1618 394 7048 055 61943 1843 519––
dpa::dpa::hoa_parse::many_aps88 889 281144 103 380126 816 815303 428450 555––
dpa::dpa::hoa_parse::synthetic25 096 68138 185 15336 756 203183 06514 675––
dpa::dpa::hoa_print::ltl19 039 65136 379 12324 186 7587 485347 284––
dpa::dpa::hoa_print::many_aps19 675 34528 413 98128 332 28113 386422––
dpa::dpa::hoa_print::synthetic27 397 17648 704 41936 382 23920 330349 158––
dpa::dpa::included_fixtures::many_aps18 597 82327 918 98827 376 57395 6241 837––
dpa::dpa::included_in::complement_100270 299410 428391 668861413––
dpa::dpa::included_in::complement_2588 075140 043125 70396396––
dpa::dpa::included_in::complement_50151 571232 902217 782308388––
dpa::dpa::included_in::normalized_100543 003803 274776 5941 360568––
dpa::dpa::included_in::normalized_25184 848275 786262 531103364––
dpa::dpa::included_in::normalized_50313 519462 832446 897352405––
dpa::dpa::language_equivalent::complement_100279 874422 356403 611865412––
dpa::dpa::language_equivalent::complement_2590 600143 209128 95493394––
dpa::dpa::language_equivalent::complement_50156 446238 990223 950313385––
dpa::dpa::language_equivalent::normalized_100993 3561 447 2061 412 2612 285672––
dpa::dpa::language_equivalent::normalized_25326 815477 816462 811124411––
dpa::dpa::language_equivalent::normalized_50581 323841 604824 179461432––
dpa::dpa::map_edge_colors_collect::scrambled_1000297 297476 256456 0713 89720––
dpa::dpa::map_edge_colors_collect::scrambled_40001 157 6921 906 4471 778 35217 1841 205––
dpa::dpa::normalized::chain_10003 824 3865 906 0565 415 49652 9206 456––
dpa::dpa::normalized::chain_400015 564 65724 180 16021 984 355253 18526 568––
dpa::dpa::normalized::scrambled_10004 685 7427 119 6726 585 82760 6326 591––
dpa::dpa::normalized::scrambled_400018 948 70629 181 68526 561 205335 67726 917––
dpa::dpa::normalized::scrambled_k4_100026 000 72537 157 84135 003 006379 9377 290––
dpa::dpa::normalized_fixtures::ltl1 415 8892 067 1342 028 1844 885415––
dpa::dpa::normalized_fixtures::many_aps13 659 71520 248 78119 772 54684 5371 530––
dpa::dpa::normalized_fixtures::synthetic9 337 59113 443 50012 961 68093 130462––
dpa::dpa::prefix_partition::chain_163 339 0695 086 3055 008 5403 8281 675––
dpa::dpa::prefix_partition::chain_8861 6441 327 0891 290 1741 461846––
dpa::dpa::prefix_partition::scrambled_163 707 9095 679 6455 579 3956 1901 980––
dpa::dpa::prefix_partition::scrambled_81 022 8581 569 6481 529 9831 913860––
dpa::dpa::rejected_word::chain_100938 2481 376 6761 332 7463 018824––
dpa::dpa::rejected_word::chain_4003 643 9435 486 5685 184 98348 1021 745––
dpa::dpa::rejected_word::scrambled_100247 893385 958351 708522904––
dpa::dpa::rejected_word::scrambled_400935 3621 443 2361 329 34110 2211 794––
dpa::dpa::sccs::chain_1000500 888788 721712 9216 8371 189––
dpa::dpa::sccs::chain_40001 928 6153 072 6982 746 37829 9005 052––
dpa::dpa::sccs::scrambled_1000567 845886 821810 2668 2271 012––
dpa::dpa::sccs::scrambled_40002 257 4853 600 2793 220 80446 2784 231––
dpa::dpa::witness_colors::chain_1004 741 5406 720 5356 596 7409 1282 233––
dpa::dpa::witness_colors::chain_40046 520 65365 842 74464 242 184193 95818 022––
dpa::dpa::witness_colors::scrambled_1004 126 3725 852 9865 757 3819 4051 388––
dpa::dpa::witness_colors::scrambled_40031 846 70145 099 68544 276 665145 6832 703––
dpa::dpa::witness_inequivalence::complement_100279 873422 291403 611866410––
dpa::dpa::witness_inequivalence::complement_2590 599143 204128 95492394––
dpa::dpa::witness_inequivalence::complement_50156 445238 959223 949314384––
dpa::dpa::witness_not_subset_of::complement_100270 298410 431391 666862413––
dpa::dpa::witness_not_subset_of::complement_2588 074140 076125 70196397––
dpa::dpa::witness_not_subset_of::complement_50151 570232 879217 779311387––
dpa::dpa::witness_not_subset_of::normalized_100543 002803 231776 5961 358567––
dpa::dpa::witness_not_subset_of::normalized_25184 847275 793262 528105364––
dpa::dpa::witness_not_subset_of::normalized_50313 518462 809446 894355404––
dpa::dpa::witnesses_fixtures::ltl2 229 2153 231 7443 163 2497 840837––
dpa::dpa::witnesses_fixtures::synthetic19 359 35629 661 01829 124 838101 230858––
game::game::tangle_learning::few_priorities_10002 697 0484 062 9303 713 17535 9104 863745 4001 292
game::game::tangle_learning::few_priorities_40008 120 67612 712 28611 104 496194 95618 0862 407 4163 266
game::game::tangle_learning::ladder_10001 179 3141 831 7261 627 97112 4434 044377 868118
game::game::tangle_learning::ladder_40004 408 1347 087 2366 022 281110 82614 5951 511 668225
game::game::tangle_learning::many_priorities_10002 018 0283 042 0902 758 58525 5234 454566 440848
game::game::tangle_learning::many_priorities_40006 752 83510 526 4309 145 870166 28915 6892 187 3441 620
game::game::tangle_learning_min_odd::few_priorities_10001 521 6152 322 1342 081 28426 3513 117525 072189
game::game::tangle_learning_min_odd::few_priorities_40008 017 70012 367 40010 863 250215 95512 1252 560 0641 933
game::game::zielonka::few_priorities_10001 126 1991 709 0211 520 53612 4553 606356 67695
game::game::zielonka::few_priorities_40003 470 2545 583 6874 684 11781 69014 0321 278 56484
game::game::zielonka::ladder_10005 134 2817 016 0776 609 66750 0904 456499 404182
game::game::zielonka::ladder_400019 816 37327 231 95125 422 846246 13916 5261 998 868212
game::game::zielonka::many_priorities_10004 103 5185 602 3035 264 43838 6914 126764 940339
game::game::zielonka::many_priorities_400012 909 83717 933 47616 554 676176 12914 2332 552 644288
game::game::zielonka_min_odd::few_priorities_10002 139 0773 023 3732 800 59323 7592 971512 012195
game::game::zielonka_min_odd::few_priorities_40006 302 0759 279 6878 309 227131 0299 0091 705 700158
learning::learning::lstar_mealy::counting_161 816 8472 478 3092 432 2447571 208154 5042 633
learning::learning::lstar_mealy::counting_8481 469684 456650 97122292540 472862
learning::learning::sprout_parity::inf_ab_1001 270 7571 850 2401 786 9352 2241 491200 4161 872
learning::learning::sprout_parity::inf_ab_4002 610 8203 782 7133 665 5889 2712 022405 0403 649
nondeterministic::nondeterministic::fixture_accepts::ltl762 5451 185 1021 120 5628 449637––
nondeterministic::nondeterministic::nba_accepts::nba_400125 269 894185 047 774174 142 9491 657 47074 785––
nondeterministic::nondeterministic::nba_witness::nba_400014 689 96121 476 74620 461 706138 4129 228––
nondeterministic::nondeterministic::nca_accepts::nca_400266 229 440395 082 740369 431 0054 559 08481 609––
nondeterministic::nondeterministic::nca_witness::nca_400019 458 26129 273 29527 117 250325 48815 103––
nondeterministic::nondeterministic::nma_witness::nma_100025 744 46937 890 67835 867 768371 1504 776––
nondeterministic::nondeterministic::npa_accepts::empty_npa_400493 167 601734 838 886687 188 0268 907 14489 004––
nondeterministic::nondeterministic::npa_accepts::npa_10030 411 62844 532 16242 656 897286 24412 687––
nondeterministic::nondeterministic::npa_accepts::npa_400125 279 542185 057 918174 152 9531 657 50574 784––
nondeterministic::nondeterministic::npa_witness::empty_npa_10009 121 95213 459 23712 723 917123 4323 376––
nondeterministic::nondeterministic::npa_witness::empty_npa_400037 023 30355 330 02651 364 426657 23619 412––
nondeterministic::nondeterministic::npa_witness::npa_10002 479 5313 680 1793 513 79420 7401 791––
nondeterministic::nondeterministic::npa_witness::npa_400020 354 25529 430 89628 218 496177 8989 226––
nondeterministic::nondeterministic::nra_witness::nra_400020 639 56731 400 77128 850 876337 11424 695––
runs::runs::dfa_accepts_many_words::count_10004 303 2135 381 9795 357 6494 7891100
runs::runs::dfa_accepts_one_word::len_1003 4924 8664 39638800
runs::runs::dfa_accepts_one_word::len_10000266 070329 557325 827697700
runs::runs::dfa_accepts_one_word::len_1000002 650 7313 276 8893 244 7096 3241600
runs::runs::dfa_reached_state::count_10003 570 2114 404 9084 380 6484 789900
runs::runs::dpa_accepts_lassos::long9 150 74511 671 08311 646 6333 077259724 664500
runs::runs::dpa_accepts_lassos::short406 490540 866535 5963619926 236472
runs::runs::dpa_recurrent_edge_colors::long15 116 26619 231 17719 179 4625 835644922 1442 090
runs::runs::dpa_recurrent_edge_colors::short1 189 7901 666 3231 655 0081 157158168 6841 708
runs::runs::dpa_recurrent_states::long10 502 98813 660 28413 630 4043 519351788 104860
runs::runs::dpa_recurrent_states::short572 037764 556757 26653413281 996782
runs::runs::dpa_visited_sequences::long8 738 26312 085 90812 050 9984 6513332 190 780850
runs::runs::dpa_visited_sequences::short370 678510 989506 2944986329 986545
runs::runs::fixture_accepts_lassos::ltl8 766 27112 652 15612 620 1765 542122––
runs::runs::fixture_accepts_lassos::synthetic4 340 1896 192 9936 139 1539 907123––
scc::scc::scc_queries::small_cycles_1000906 6761 433 5441 272 0949 4283 266261 132559
scc::scc::scc_queries::small_cycles_40003 548 9195 658 3944 973 93950 91712 2821 045 3962 071
scc::scc::tarjan::back_edges_1000512 099838 107726 7775 4732 399170 27245
scc::scc::tarjan::back_edges_40001 987 6273 267 5342 821 39426 4528 968680 63253
scc::scc::tarjan::binary_tree_1000571 217892 769815 6944 0611 622105 76039
scc::scc::tarjan::binary_tree_40002 290 4593 582 5733 276 41321 1295 729419 51245
scc::scc::tarjan::fragmented_1000658 5221 046 852922 7179 6442 169146 084121
scc::scc::tarjan::fragmented_40002 534 1084 125 8593 544 65462 7897 636569 852167
scc::scc::tarjan::grid_1024596 080972 081847 7617 7282 448171 16045
scc::scc::tarjan::grid_40962 378 4844 001 3513 363 70163 6699 123684 18453
scc::scc::tarjan::path_10001 929 1193 028 5232 729 56823 1885 229455 2242 064
scc::scc::tarjan::path_40007 632 90812 001 66410 804 76498 32320 1511 821 5368 076
scc::scc::tarjan::ring_1000512 099841 747725 8676 3832 399170 27245
scc::scc::tarjan::ring_40001 987 6273 284 7822 817 08230 7648 968680 63253
scc::scc::tarjan::scrambled_1000579 056936 579824 0296 9562 222170 27245
scc::scc::tarjan::scrambled_40002 316 4973 791 2493 296 77941 8728 146680 63253
scc::scc::tarjan::small_cycles_1000905 6581 432 2191 270 8099 4343 264261 132559
scc::scc::tarjan::small_cycles_40003 544 9015 650 7854 969 53050 29812 2791 045 3962 071
scc::scc::tarjan_from::fragmented_1000242 530394 598344 8782 5241 06074 98564
scc::scc::tarjan_from::fragmented_4000907 7621 492 3421 296 66715 9233 316299 62578
scc::scc::tarjan_from::path_10001 921 1583 018 2442 722 14423 0375 169468 9932 091
scc::scc::tarjan_from::path_40007 545 58411 928 68110 697 651106 73119 9251 876 7538 109
scc::scc::tarjan_from::ring_1000503 645836 974717 9746 2512 507184 04172
scc::scc::tarjan_from::scrambled_1000570 602930 380816 4856 4762 329184 04172
storage::storage::add_edges_dense::k2_50001 356 4012 030 4901 985 3454 941584180 20037
storage::storage::add_edges_dense::k8_50004 247 0596 411 8746 225 39417 8362 780671 72840
storage::storage::add_edges_linked::k2_50001 402 6702 420 4692 258 38414 2242 5991 113 96026
storage::storage::add_edges_linked::k8_50006 669 28411 260 27310 330 80875 81815 7253 866 49628
storage::storage::clone_csr::k2_1000030 590183 48052 6254 1703 143230 0126
storage::storage::clone_dense::k2_1000015 11381 34225 8022 2531 265110 0084
storage::storage::clone_linked::k2_10000370 824901 703529 79812 4668 845680 0083
storage::storage::complete_dense::partial_5000155 628209 494202 329908751364
storage::storage::complete_linked::partial_50001 210 8541 718 8451 629 4506 9521 56165 67218
storage::storage::edges_from_all_states_csr::k2_200072 04777 21475 859222700
storage::storage::edges_from_all_states_csr::k8_2000108 047116 392111 0721 008800
storage::storage::edges_from_all_states_dense::k2_2000102 043119 287117 842233800
storage::storage::edges_from_all_states_dense::k8_2000140 043160 529155 0391 035900
storage::storage::edges_from_all_states_linked::k2_200068 03386 72575 9252 132400
storage::storage::edges_from_all_states_linked::k8_2000152 033203 701166 6817 376400
storage::storage::minimize_csr::redundant_20004 633 7567 005 0566 522 11157 8935 5281 505 4202 212
storage::storage::minimize_csr::redundant_800018 159 86127 786 74025 537 970272 65425 3006 023 3008 250
storage::storage::minimize_dense::redundant_20004 676 9107 035 6516 577 16658 1534 7921 522 1562 221
storage::storage::minimize_dense::redundant_800018 364 89528 117 70225 817 247279 05025 8636 090 3408 261
storage::storage::minimize_linked::redundant_20004 505 0386 920 7686 368 86861 9266 9221 522 1562 221
storage::storage::minimize_linked::redundant_800017 714 72127 484 31425 033 274302 87426 7626 090 3408 261
storage::storage::overwrite_edges_dense::k2_50002 126 4313 232 5183 172 4287 916586180 20037
storage::storage::overwrite_edges_linked::k2_50003 790 7756 671 1135 952 01844 76914 1502 031 46427
storage::storage::predecessors_of_all_states_csr::k2_2000102 046121 042117 407678700
storage::storage::predecessors_of_all_states_dense::k2_2000290 369397 907388 2221 6224532 0083
storage::storage::predecessors_of_all_states_linked::k2_200068 03391 72974 6893 366600
storage::storage::product_successor_csr::n20064 09675 54575 16014900
storage::storage::product_successor_dense::n20073 08794 62094 10540900
storage::storage::product_successor_linked::n20046 54760 17557 015576800
storage::storage::reachable_states_csr::k2_80001 101 1151 630 1181 591 8437 4173417 36811
storage::storage::reachable_states_dense::k2_80001 133 6991 684 5051 639 2458 8283217 36811
storage::storage::reachable_states_linked::k2_8000924 2791 485 5951 386 51519 5294117 36811
storage::storage::remove_state_dense::k2_100001 072 2121 565 0861 390 28621 5131 921176 0001 003
storage::storage::remove_state_linked::k2_100001 094 0121 531 4791 492 7347 4204780 0003 000
storage::storage::run_many_words_csr::count_1000551 036653 943644 1631 907700
storage::storage::run_many_words_dense::count_1000558 044690 031680 1661 910900
storage::storage::run_many_words_linked::count_1000481 044611 149601 1191 957700
storage::storage::run_one_word_csr::len_1000002 600 0543 025 4282 993 8236 272700
storage::storage::run_one_word_dense::len_1000002 800 0503 425 4573 393 8126 273800
storage::storage::run_one_word_linked::len_1000002 350 0392 975 4282 943 7736 296500
storage::storage::successor_all_pairs_csr::k2_2000122 242139 664138 0992291281
storage::storage::successor_all_pairs_csr::k8_2000702 302802 872797 3721 02311321
storage::storage::successor_all_pairs_dense::k2_2000162 237193 545192 090235881
storage::storage::successor_all_pairs_dense::k8_2000814 297952 851947 3461 0458321
storage::storage::successor_all_pairs_linked::k2_2000108 236139 158128 1782 140881
storage::storage::successor_all_pairs_linked::k8_2000750 296946 204908 9997 3858321
ts::ts::add_edges_in_order::k2_50001 402 6712 420 3392 258 38914 2252 5951 113 96026
ts::ts::add_edges_in_order::k8_50006 669 28511 260 12710 330 81775 81515 7213 866 49628
ts::ts::add_edges_in_reverse::k2_50001 402 6712 419 5392 258 58914 0252 5951 113 96026
ts::ts::builder_with_edges::k2_500081 365 96798 218 37795 716 412451 6106 9691 739 53245
ts::ts::canonical_naming::scrambled_15003 144 0904 592 0874 402 17721 7982 312463 8085 846
ts::ts::canonical_naming::scrambled_500972 5761 396 5231 352 8383 172795134 3361 831
ts::ts::clone::scrambled_10000371 163902 511530 23612 4848 853680 0083
ts::ts::clone::scrambled_400001 481 4003 817 8822 116 80742 57542 5202 720 0083
ts::ts::collect_dts::scrambled_100003 455 1286 155 9595 596 81437 16710 6662 228 06427
ts::ts::collect_dts::scrambled_4000013 759 60724 643 03522 289 445135 29247 9188 912 73631
ts::ts::determinize::nth_from_end_104 414 3206 505 4016 144 54122 1297 149651 7206 013
ts::ts::determinize::nth_from_end_1219 628 00628 773 78127 264 53198 83629 0022 732 60026 423
ts::ts::determinize::nth_from_end_81 007 2681 486 4871 403 6173 0151 937156 5521 396
ts::ts::determinize_propositional::aps_6_n_64 333 8446 819 4306 334 37019 55011 0661 332 8964 208
ts::ts::determinize_propositional::aps_7_n_64 525 3877 097 8346 557 72925 35111 8101 197 1621 965
ts::ts::determinize_propositional::aps_8_n_626 425 54240 592 59737 675 092167 84859 3795 222 79213 670
ts::ts::edges_from_all_states::scrambled_200072 66991 97580 7752 1561200
ts::ts::edges_from_all_states::scrambled_k8_2000156 601208 856171 4317 4011200
ts::ts::erase_state_colors_collect::scrambled_100002 377 1724 108 2513 548 88137 16310 6732 228 06427
ts::ts::into_dfa::scrambled_100003 635 1666 416 2195 856 87937 17810 6702 228 06427
ts::ts::map_edge_colors_collect::scrambled_100002 368 7844 158 0513 539 57138 45012 1782 490 19227
ts::ts::map_edge_colors_collect::scrambled_400009 415 40916 652 01514 062 220140 33053 9479 961 29631
ts::ts::matching_product_successor_index::n20075 662111 078106 4636503900
ts::ts::minimal_representative_of_all_states::scrambled_1002 102 8112 971 3132 967 68312102347 5044 679
ts::ts::minimal_representative_of_all_states::scrambled_30016 917 26123 990 16823 976 9331 9121053 116 78438 196
ts::ts::minimal_representatives::scrambled_20003 386 0404 763 9764 717 2518 89764322 9447 709
ts::ts::minimal_representatives::scrambled_800014 908 24920 957 74120 742 92642 578551 464 89632 693
ts::ts::predecessors_of_all_states::scrambled_200072 66996 91579 5403 3911200
ts::ts::predecessors_of_all_states::scrambled_8000288 601389 328313 81315 0191200
ts::ts::reachable_states::scrambled_2000290 091460 911439 9963 616814 3369
ts::ts::reachable_states::scrambled_80001 148 6961 838 4921 739 08719 5245117 36811
ts::ts::remove_edges_from_matching::scrambled_10000566 234795 995771 6454 6603064 0002 000
ts::ts::remove_edges_from_matching::scrambled_40000896 2341 177 4951 126 27010 0353064 0002 000
ts::ts::remove_state::scrambled_100001 091 0121 528 4151 489 7357 4214580 0003 000
ts::ts::remove_state::scrambled_400001 421 0121 912 8791 843 61913 5374580 0003 000
ts::ts::scc_minimal_representatives::small_cycles_100219 042318 864303 2495743844 272309
ts::ts::scc_minimal_representatives::small_cycles_4001 242 4751 766 0001 715 5803 882886354 1481 325
ts::ts::subset_construction::nth_from_end_1010 228 20614 537 46214 042 03745 5637 6461 073 5209 919
ts::ts::subset_construction::nth_from_end_1245 253 02564 100 57461 854 764218 77832 9124 518 97643 857
ts::ts::subset_construction::nth_from_end_82 233 5063 193 1683 084 2236 2842 215257 1042 276
ts::ts::successor_index_all_pairs::scrambled_2000108 863140 327129 0222 1561581
ts::ts::successor_index_all_pairs::scrambled_k8_2000750 855947 312909 7427 40216321

Wall-clock times

Criterion means of a single run. They depend on the machine and are not compared; use the instruction counts above for that.

benchmarkhead
collect/clone/1000003.93 ms (3.84 ms … 4.03 ms)
collect/clone/100000044.28 ms (43.85 ms … 44.72 ms)
collect/collect_owned/1000007.23 ms (7.19 ms … 7.27 ms)
collect/collect_owned/100000081.41 ms (79.65 ms … 83.17 ms)
collect/collect_ref/1000005.21 ms (5.19 ms … 5.22 ms)
collect/collect_ref/100000089.22 ms (88.40 ms … 90.04 ms)
collect/map_edge_colors_collect_owned/1000008.02 ms (7.96 ms … 8.07 ms)
collect/map_edge_colors_collect_owned/100000089.84 ms (87.48 ms … 92.21 ms)
wallclock/dfa/accepts/word_100000927.76 µs (909.78 µs … 945.75 µs)
wallclock/dfa/minimize/redundant_40001.40 ms (1.39 ms … 1.42 ms)
wallclock/dpa/complement/scrambled_40003.59 ms (3.59 ms … 3.60 ms)
wallclock/dpa/normalized/scrambled_40003.43 ms (3.36 ms … 3.50 ms)
wallclock/hoa/parse/ltl944.29 µs (940.39 µs … 948.18 µs)
wallclock/nfa/determinize/nth_from_end_123.45 ms (3.43 ms … 3.48 ms)
wallclock/nfa/subset_construction/nth_from_end_128.96 ms (8.78 ms … 9.15 ms)
wallclock/scc/tarjan/scrambled_100002.21 ms (2.19 ms … 2.24 ms)

Versus Spot and Oink

dpa/equivalence

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot140.18 µs113.74 µs1.23×– / –yes
hard/l-nba-2spot64.66 µs35.76 µs1.81×– / –yes
hard/m-nba-3spot275.62 µs138.58 µs1.99×– / –yes
ltl/and-gf-3spot10.38 µs11.94 µs0.87×– / –yes
ltl/eh-patterns-0spot26.76 µs19.84 µs1.35×– / –yes
ltl/gf-equiv-3spot24.48 µs33.44 µs0.73×– / –yes
ltl/gf-implies-3spot12.82 µs19.08 µs0.67×– / –yes
ltl/gh-q-3spot58.74 µs28.46 µs2.06×– / –yes
ltl/gh-r-3spot25.72 µs69.20 µs0.37×– / –yes
ltl/go-theta-3spot12.58 µs20.70 µs0.61×– / –yes
ltl/lily-patterns-1spot13.68 µs15.76 µs0.87×– / –yes
ltl/ms-phi-r-1spot12.14 µs37.60 µs0.32×– / –yes
ltl/ms-phi-s-1spot12.18 µs39.14 µs0.31×– / –yes
ltl/rv-counter-3spot30.40 µs13.48 µs2.26×– / –yes
ltl/sb-patterns-0spot14.52 µs14.10 µs1.03×– / –yes
ltl/sejk-k-2spot15.98 µs57.56 µs0.28×– / –yes
ltl/tv-uu-3spot46.68 µs38.64 µs1.21×– / –yes
ltl/u-left-3spot11.76 µs8.34 µs1.41×– / –yes
many_aps/and-gf-12spot180.02 µs66.32 µs2.71×– / –yes
many_aps/gf-equiv-12spot151.04 µs96.12 µs1.57×– / –yes
many_aps/r-right-10spot75.56 µs25.50 µs2.96×– / –yes
many_aps/random-n16-a8-p4-s1spot306.68 µs128.56 µs2.39×– / –yes
many_aps/random-n24-a12-p4-s1spot1.49 ms448.40 µs3.33×– / –yes
many_aps/random-n32-a16-p4-s1spot4.54 ms1.23 ms3.68×– / –yes
random/n128-a2-p6-s1spot357.96 µs207.52 µs1.72×– / –yes
random/n16-a1-p3-s1spot24.08 µs16.10 µs1.50×– / –yes
random/n256-a2-p8-s1spot967.89 µs474.99 µs2.04×– / –yes
random/n64-a2-p4-s1spot152.10 µs100.38 µs1.52×– / –yes

dpa/inclusion

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot76.24 µs59.84 µs1.27×– / –yes
hard/l-nba-2spot38.60 µs22.74 µs1.70×– / –yes
hard/m-nba-3spot163.12 µs84.98 µs1.92×– / –yes
ltl/and-gf-3spot6.78 µs8.10 µs0.84×– / –yes
ltl/eh-patterns-0spot17.58 µs17.76 µs0.99×– / –yes
ltl/gf-equiv-3spot14.04 µs19.22 µs0.73×– / –yes
ltl/gf-implies-3spot7.38 µs12.04 µs0.61×– / –yes
ltl/gh-q-3spot37.52 µs27.64 µs1.36×– / –yes
ltl/gh-r-3spot14.50 µs34.58 µs0.42×– / –yes
ltl/go-theta-3spot8.36 µs16.16 µs0.52×– / –yes
ltl/lily-patterns-1spot9.48 µs13.48 µs0.70×– / –yes
ltl/ms-phi-r-1spot7.36 µs18.78 µs0.39×– / –yes
ltl/ms-phi-s-1spot7.24 µs24.74 µs0.29×– / –yes
ltl/rv-counter-3spot19.36 µs14.00 µs1.38×– / –yes
ltl/sb-patterns-0spot9.00 µs11.84 µs0.76×– / –yes
ltl/sejk-k-2spot9.44 µs27.68 µs0.34×– / –yes
ltl/tv-uu-3spot31.48 µs28.88 µs1.09×– / –yes
ltl/u-left-3spot8.00 µs9.26 µs0.86×– / –yes
many_aps/and-gf-12spot142.92 µs52.74 µs2.71×– / –yes
many_aps/gf-equiv-12spot82.04 µs50.30 µs1.63×– / –yes
many_aps/r-right-10spot52.90 µs23.16 µs2.28×– / –yes
many_aps/random-n16-a8-p4-s1spot257.92 µs104.62 µs2.47×– / –yes
many_aps/random-n24-a12-p4-s1spot1.31 ms403.14 µs3.26×– / –yes
many_aps/random-n32-a16-p4-s1spot4.14 ms1.13 ms3.66×– / –yes
random/n128-a2-p6-s1spot213.28 µs124.96 µs1.71×– / –yes
random/n16-a1-p3-s1spot14.68 µs9.92 µs1.48×– / –yes
random/n256-a2-p8-s1spot550.03 µs295.64 µs1.86×– / –yes
random/n64-a2-p4-s1spot90.20 µs60.04 µs1.50×– / –yes

game/solve

inputreferenceoursreferenceours / referencesize (ours / reference)same result
ladder_10000oink-ppp2.61 ms310.00 µs8.41×10000 / 10000yes
ladder_10000oink-zlk2.61 ms307.00 µs8.49×10000 / 10000yes
ladder_10000spot2.61 ms9.47 ms0.28×10000 / 10000yes
ladder_50000oink-ppp20.44 ms2.21 ms9.23×50000 / 50000yes
ladder_50000oink-zlk20.44 ms1.98 ms10.31×50000 / 50000yes
ladder_50000spot20.44 ms71.23 ms0.29×50000 / 50000yes
random_few_10000oink-ppp7.79 ms1.47 ms5.31×10000 / 10000yes
random_few_10000oink-zlk7.79 ms2.74 ms2.84×10000 / 10000yes
random_few_10000spot7.79 ms21.49 ms0.36×10000 / 10000yes
random_few_50000oink-ppp80.47 ms17.62 ms4.57×50000 / 50000yes
random_few_50000oink-zlk80.47 ms20.55 ms3.92×50000 / 50000yes
random_few_50000spot80.47 ms684.01 ms0.12×50000 / 50000yes
random_many_10000oink-ppp7.03 ms1.59 ms4.41×10000 / 10000yes
random_many_10000oink-zlk7.03 ms2.81 ms2.50×10000 / 10000yes
random_many_10000spot7.03 ms31.17 ms0.23×10000 / 10000yes
random_many_50000oink-ppp81.71 ms26.64 ms3.07×50000 / 50000yes
random_many_50000oink-zlk81.71 ms35.94 ms2.27×50000 / 50000yes
random_many_50000spot81.71 ms667.50 ms0.12×50000 / 50000yes

Log

== head 38e9f0d4ee9208f4d9f2210d954d92f1b9eaa42c: building ["scc", "ts", "runs", "dfa", "dpa", "storage", "nondeterministic", "learning", "game"] ["wallclock"]
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
    Finished `bench` profile [optimized] target(s) in 2m 09s
  Executable benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-a84ba41f760e70ed)
  Executable benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-23bb829f074058ad)
  Executable benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-14422f732ec85029)
  Executable benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-ee53c00379271c59)
  Executable benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-f164d05108dbce73)
  Executable benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-85a152aa046a29b4)
  Executable benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-1b22c641fd9d2d89)
  Executable benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-f95e3d4a8c506941)
  Executable benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-4948f6cc6a43d201)
  Executable benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-93fb383f608d3bab)
== head: running scc
    Finished `bench` profile [optimized] target(s) in 0.26s
     Running benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-1b22c641fd9d2d89)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running ts
    Finished `bench` profile [optimized] target(s) in 0.24s
     Running benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-4948f6cc6a43d201)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running runs
    Finished `bench` profile [optimized] target(s) in 0.19s
     Running benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-85a152aa046a29b4)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dfa
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-a84ba41f760e70ed)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dpa
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-23bb829f074058ad)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running storage
    Finished `bench` profile [optimized] target(s) in 0.23s
     Running benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-f95e3d4a8c506941)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running nondeterministic
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-f164d05108dbce73)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running learning
    Finished `bench` profile [optimized] target(s) in 0.19s
     Running benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-ee53c00379271c59)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running game
    Finished `bench` profile [optimized] target(s) in 0.19s
     Running benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-14422f732ec85029)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running wallclock (wall clock)
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-93fb383f608d3bab)
Gnuplot not found, using plotters backend
Benchmarking wallclock/scc/tarjan/scrambled_10000
Benchmarking wallclock/scc/tarjan/scrambled_10000: Analyzing
Benchmarking wallclock/dpa/normalized/scrambled_4000
Benchmarking wallclock/dpa/normalized/scrambled_4000: Analyzing
Benchmarking wallclock/dpa/complement/scrambled_4000
Benchmarking wallclock/dpa/complement/scrambled_4000: Analyzing
Benchmarking wallclock/dfa/minimize/redundant_4000
Benchmarking wallclock/dfa/minimize/redundant_4000: Analyzing
Benchmarking wallclock/dfa/accepts/word_100000
Benchmarking wallclock/dfa/accepts/word_100000: Analyzing
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12: Analyzing
Benchmarking wallclock/nfa/determinize/nth_from_end_12
Benchmarking wallclock/nfa/determinize/nth_from_end_12: Analyzing
Benchmarking wallclock/hoa/parse/ltl
Benchmarking wallclock/hoa/parse/ltl: Analyzing
Benchmarking collect/clone/100000
Benchmarking collect/clone/100000: Analyzing
Benchmarking collect/collect_owned/100000
Benchmarking collect/collect_owned/100000: Analyzing
Benchmarking collect/collect_ref/100000
Benchmarking collect/collect_ref/100000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/100000
Benchmarking collect/map_edge_colors_collect_owned/100000: Analyzing
Benchmarking collect/clone/1000000
Benchmarking collect/clone/1000000: Analyzing
Benchmarking collect/collect_owned/1000000
Benchmarking collect/collect_owned/1000000: Analyzing
Benchmarking collect/collect_ref/1000000
Benchmarking collect/collect_ref/1000000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/1000000
Benchmarking collect/map_edge_colors_collect_owned/1000000: Analyzing
== head: building the Python bindings
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support
🐍 Found CPython 3.13 at /nix/store/vilynzly1pyhawy04ljmrp7n9ffr24qy-python3-3.13.17/bin/python3
📡 Using build options features from pyproject.toml
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
   Compiling automata-py v0.1.0 (/var/lib/automata-bench/work/src-head/python)
    Finished `release` profile [optimized] target(s) in 1m 55s
📦 Built wheel for abi3 Python ≥ 3.11 to /var/lib/automata-bench/work/versus-head/wheels/automatars-0.1.0-cp311-abi3-manylinux_2_34_aarch64.whl
== head: running benches/versus/compare.py (Spot, Oink)
dpa/equivalence/hard/l-dsa-3 vs spot: 0.140 ms / 0.114 ms
dpa/equivalence/hard/l-nba-2 vs spot: 0.065 ms / 0.036 ms
dpa/equivalence/hard/m-nba-3 vs spot: 0.276 ms / 0.139 ms
dpa/equivalence/ltl/and-gf-3 vs spot: 0.010 ms / 0.012 ms
dpa/equivalence/ltl/eh-patterns-0 vs spot: 0.027 ms / 0.020 ms
dpa/equivalence/ltl/gf-equiv-3 vs spot: 0.024 ms / 0.033 ms
dpa/equivalence/ltl/gf-implies-3 vs spot: 0.013 ms / 0.019 ms
dpa/equivalence/ltl/gh-q-3 vs spot: 0.059 ms / 0.028 ms
dpa/equivalence/ltl/gh-r-3 vs spot: 0.026 ms / 0.069 ms
dpa/equivalence/ltl/go-theta-3 vs spot: 0.013 ms / 0.021 ms
dpa/equivalence/ltl/lily-patterns-1 vs spot: 0.014 ms / 0.016 ms
dpa/equivalence/ltl/ms-phi-r-1 vs spot: 0.012 ms / 0.038 ms
dpa/equivalence/ltl/ms-phi-s-1 vs spot: 0.012 ms / 0.039 ms
dpa/equivalence/ltl/rv-counter-3 vs spot: 0.030 ms / 0.013 ms
dpa/equivalence/ltl/sb-patterns-0 vs spot: 0.015 ms / 0.014 ms
dpa/equivalence/ltl/sejk-k-2 vs spot: 0.016 ms / 0.058 ms
dpa/equivalence/ltl/tv-uu-3 vs spot: 0.047 ms / 0.039 ms
dpa/equivalence/ltl/u-left-3 vs spot: 0.012 ms / 0.008 ms
dpa/equivalence/many_aps/and-gf-12 vs spot: 0.180 ms / 0.066 ms
dpa/equivalence/many_aps/gf-equiv-12 vs spot: 0.151 ms / 0.096 ms
dpa/equivalence/many_aps/r-right-10 vs spot: 0.076 ms / 0.025 ms
dpa/equivalence/many_aps/random-n16-a8-p4-s1 vs spot: 0.307 ms / 0.129 ms
dpa/equivalence/many_aps/random-n24-a12-p4-s1 vs spot: 1.494 ms / 0.448 ms
dpa/equivalence/many_aps/random-n32-a16-p4-s1 vs spot: 4.540 ms / 1.234 ms
dpa/equivalence/random/n128-a2-p6-s1 vs spot: 0.358 ms / 0.208 ms
dpa/equivalence/random/n16-a1-p3-s1 vs spot: 0.024 ms / 0.016 ms
dpa/equivalence/random/n256-a2-p8-s1 vs spot: 0.968 ms / 0.475 ms
dpa/equivalence/random/n64-a2-p4-s1 vs spot: 0.152 ms / 0.100 ms
dpa/inclusion/hard/l-dsa-3 vs spot: 0.076 ms / 0.060 ms
dpa/inclusion/hard/l-nba-2 vs spot: 0.039 ms / 0.023 ms
dpa/inclusion/hard/m-nba-3 vs spot: 0.163 ms / 0.085 ms
dpa/inclusion/ltl/and-gf-3 vs spot: 0.007 ms / 0.008 ms
dpa/inclusion/ltl/eh-patterns-0 vs spot: 0.018 ms / 0.018 ms
dpa/inclusion/ltl/gf-equiv-3 vs spot: 0.014 ms / 0.019 ms
dpa/inclusion/ltl/gf-implies-3 vs spot: 0.007 ms / 0.012 ms
dpa/inclusion/ltl/gh-q-3 vs spot: 0.038 ms / 0.028 ms
dpa/inclusion/ltl/gh-r-3 vs spot: 0.015 ms / 0.035 ms
dpa/inclusion/ltl/go-theta-3 vs spot: 0.008 ms / 0.016 ms
dpa/inclusion/ltl/lily-patterns-1 vs spot: 0.009 ms / 0.013 ms
dpa/inclusion/ltl/ms-phi-r-1 vs spot: 0.007 ms / 0.019 ms
dpa/inclusion/ltl/ms-phi-s-1 vs spot: 0.007 ms / 0.025 ms
dpa/inclusion/ltl/rv-counter-3 vs spot: 0.019 ms / 0.014 ms
dpa/inclusion/ltl/sb-patterns-0 vs spot: 0.009 ms / 0.012 ms
dpa/inclusion/ltl/sejk-k-2 vs spot: 0.009 ms / 0.028 ms
dpa/inclusion/ltl/tv-uu-3 vs spot: 0.031 ms / 0.029 ms
dpa/inclusion/ltl/u-left-3 vs spot: 0.008 ms / 0.009 ms
dpa/inclusion/many_aps/and-gf-12 vs spot: 0.143 ms / 0.053 ms
dpa/inclusion/many_aps/gf-equiv-12 vs spot: 0.082 ms / 0.050 ms
dpa/inclusion/many_aps/r-right-10 vs spot: 0.053 ms / 0.023 ms
dpa/inclusion/many_aps/random-n16-a8-p4-s1 vs spot: 0.258 ms / 0.105 ms
dpa/inclusion/many_aps/random-n24-a12-p4-s1 vs spot: 1.314 ms / 0.403 ms
dpa/inclusion/many_aps/random-n32-a16-p4-s1 vs spot: 4.144 ms / 1.133 ms
dpa/inclusion/random/n128-a2-p6-s1 vs spot: 0.213 ms / 0.125 ms
dpa/inclusion/random/n16-a1-p3-s1 vs spot: 0.015 ms / 0.010 ms
dpa/inclusion/random/n256-a2-p8-s1 vs spot: 0.550 ms / 0.296 ms
dpa/inclusion/random/n64-a2-p4-s1 vs spot: 0.090 ms / 0.060 ms
game/solve/random_few_10000 vs spot: 7.791 ms / 21.490 ms
game/solve/random_few_10000 vs oink-zlk: 7.791 ms / 2.744 ms
game/solve/random_few_10000 vs oink-ppp: 7.791 ms / 1.467 ms
game/solve/random_few_50000 vs spot: 80.470 ms / 684.008 ms
game/solve/random_few_50000 vs oink-zlk: 80.470 ms / 20.552 ms
game/solve/random_few_50000 vs oink-ppp: 80.470 ms / 17.620 ms
game/solve/random_many_10000 vs spot: 7.027 ms / 31.170 ms
game/solve/random_many_10000 vs oink-zlk: 7.027 ms / 2.808 ms
game/solve/random_many_10000 vs oink-ppp: 7.027 ms / 1.592 ms
game/solve/random_many_50000 vs spot: 81.713 ms / 667.496 ms
game/solve/random_many_50000 vs oink-zlk: 81.713 ms / 35.941 ms
game/solve/random_many_50000 vs oink-ppp: 81.713 ms / 26.636 ms
game/solve/ladder_10000 vs spot: 2.608 ms / 9.470 ms
game/solve/ladder_10000 vs oink-zlk: 2.608 ms / 0.307 ms
game/solve/ladder_10000 vs oink-ppp: 2.608 ms / 0.310 ms
game/solve/ladder_50000 vs spot: 20.438 ms / 71.234 ms
game/solve/ladder_50000 vs oink-zlk: 20.438 ms / 1.982 ms
game/solve/ladder_50000 vs oink-ppp: 20.438 ms / 2.215 ms
== head: 266 benchmarks, 16 wall-clock measurements, 74 comparisons