automata benchmarks

History · Versus Spot and Oink

main · 2b5473e7

docs: drop obsolete entries from PERFORMANCE.md (#165)

status done · queued 2026-11-11 17:18 · started 2026-11-11 17:18 · finished 2026-11-11 17:38

· · rustc 1.95.0 (59807616e 2026-04-14) (built from a source tarball) · valgrind-3.26.0 · pinned to cpu 1

Results

benchmarkinstructionsest. cyclesL1 hitsLL hitsRAM hitsheap bytesallocations
dfa::dfa::accepts_many_words::scrambled_1000319 438399 885393 3851 2231100
dfa::dfa::complement::scrambled_1000318 874512 713495 0333 37523139 10419
dfa::dfa::complement::scrambled_40001 247 6492 024 5981 938 85316 99522556 89623
dfa::dfa::equivalent::deep_difference_10052 83182 61975 4994519716 19625
dfa::dfa::equivalent::deep_difference_400177 852277 913255 5681 85137465 36433
dfa::dfa::equivalent::different_1008 34914 14811 4887751 80412
dfa::dfa::equivalent::different_4008 39014 35311 44378721 80412
dfa::dfa::equivalent::minimized_10031 31246 50844 273104497 75617
dfa::dfa::equivalent::minimized_400103 504153 082147 6577285131 58023
dfa::dfa::give_word::empty_10001 795 2342 450 0352 422 4804 91685152 8643 738
dfa::dfa::give_word::empty_40006 603 9839 194 1339 091 42820 06568679 80815 828
dfa::dfa::give_word::far_1000157 743223 352218 8275485128 176394
dfa::dfa::give_word::far_4000909 2251 280 3471 258 3373 93367148 4642 233
dfa::dfa::intersection::scrambled_1002 716 0483 995 2733 817 11312 0143 374428 08842
dfa::dfa::intersection::scrambled_20010 809 51116 277 72415 140 77496 86818 6461 708 12049
dfa::dfa::intersection_is_empty::empty_10050 65979 04172 5313518115 70020
dfa::dfa::intersection_is_empty::empty_20092 418145 704132 52961928831 58023
dfa::dfa::is_empty_language::empty_10001 795 2342 450 0692 422 4794 91686152 8643 738
dfa::dfa::is_empty_language::empty_40006 603 9839 194 1679 091 42720 06569679 80815 828
dfa::dfa::minimize::minimal_10003 298 9794 979 3184 604 64339 7325 0291 135 4922 214
dfa::dfa::minimize::minimal_400012 876 08919 609 31417 952 604204 57218 1104 543 7248 254
dfa::dfa::minimize::redundant_10002 174 5513 263 1683 028 27825 8873 013760 7801 201
dfa::dfa::minimize::redundant_40008 406 75412 753 34411 696 774143 6179 6713 044 9004 241
dfa::dfa::minimize::scrambled_10003 502 2565 309 2174 877 65251 0405 0391 116 0522 217
dfa::dfa::minimize::scrambled_400013 587 25820 978 23018 872 885294 02618 1494 470 1408 261
dfa::dfa::minimize_moore_with_edge_colors::scrambled_10003 500 3175 293 0044 866 63950 0985 0251 116 0522 217
dfa::dfa::minimize_moore_with_edge_colors::scrambled_400013 571 89120 909 14018 808 240300 57817 0864 470 1408 261
dfa::dfa::union::scrambled_1002 716 0483 995 2733 817 11312 0143 374428 08842
dfa::dfa::union::scrambled_20010 809 51116 277 72015 140 77596 86718 6461 708 12049
dpa::dpa::accepted_word::chain_100114 774174 778162 97850330––
dpa::dpa::accepted_word::chain_400391 265589 357555 0622 967556––
dpa::dpa::accepted_word::scrambled_100145 660218 233206 02840343––
dpa::dpa::accepted_word::scrambled_400499 945740 793707 7732 817541––
dpa::dpa::accepted_word_fixtures::ltl495 442716 515690 4702 248423––
dpa::dpa::accepted_word_fixtures::synthetic1 255 4201 811 2261 739 74111 798357––
dpa::dpa::accepted_word_fixtures_by_atoms::ltl1 620 5282 381 5472 324 0477 671547––
dpa::dpa::accepted_word_fixtures_by_atoms::synthetic2 741 0794 051 5213 966 98113 128540––
dpa::dpa::collect_dpa::scrambled_1000258 437421 129402 0243 8003––
dpa::dpa::collect_dpa::scrambled_40001 004 7821 689 4741 565 92916 3931 188––
dpa::dpa::complement::chain_10003 806 7135 721 4445 221 06955 1636 416––
dpa::dpa::complement::chain_400015 498 94023 452 27621 213 691262 01426 529––
dpa::dpa::complement::scrambled_10004 640 2806 873 5866 331 48662 5146 558––
dpa::dpa::complement::scrambled_400018 790 50128 240 19425 579 294344 02726 879––
dpa::dpa::complement_fixtures::ltl1 318 7521 878 3201 840 3404 810398––
dpa::dpa::complement_fixtures::many_aps12 439 59317 914 29517 431 66586 3761 450––
dpa::dpa::complement_fixtures::synthetic9 051 76712 765 56612 268 55696 441423––
dpa::dpa::complement_without_normalizing::scrambled_1000542 888786 588757 8635 15784––
dpa::dpa::complement_without_normalizing::scrambled_40002 135 2353 133 7992 979 55421 9661 269––
dpa::dpa::equivalent_fixtures::many_aps18 698 03427 049 25026 383 895120 3241 821––
dpa::dpa::hoa_parse::ltl5 102 0067 693 3737 405 21335 6173 145––
dpa::dpa::hoa_parse::many_aps80 066 835132 200 341115 009 426290 122449 723––
dpa::dpa::hoa_parse::synthetic23 303 31935 262 96033 885 730175 32514 303––
dpa::dpa::hoa_print::ltl14 449 24831 516 46219 332 3327 343347 069––
dpa::dpa::hoa_print::many_aps17 708 49225 469 85525 318 21027 928343––
dpa::dpa::hoa_print::synthetic21 972 18342 634 43330 233 39837 557348 950––
dpa::dpa::included_fixtures::many_aps15 907 78323 119 51322 585 73394 4011 765––
dpa::dpa::included_in::complement_100260 739388 049371 329845357––
dpa::dpa::included_in::complement_2584 611131 494119 15980341––
dpa::dpa::included_in::complement_50146 080219 879206 694313332––
dpa::dpa::included_in::normalized_100524 060764 006738 4411 396531––
dpa::dpa::included_in::normalized_25177 925261 399249 524156317––
dpa::dpa::included_in::normalized_50302 723439 883425 518374357––
dpa::dpa::language_equivalent::complement_100268 661397 813380 478870371––
dpa::dpa::language_equivalent::complement_2586 600134 349121 47989355––
dpa::dpa::language_equivalent::complement_50149 885224 791211 036329346––
dpa::dpa::language_equivalent::normalized_100963 9591 387 3411 353 4462 215652––
dpa::dpa::language_equivalent::normalized_25315 935456 707442 182238381––
dpa::dpa::language_equivalent::normalized_50564 774808 108791 073502415––
dpa::dpa::map_edge_colors_collect::scrambled_1000263 002419 699399 6593 87519––
dpa::dpa::map_edge_colors_collect::scrambled_40001 021 3471 681 8541 553 85917 1711 204––
dpa::dpa::normalized::chain_10003 513 1885 339 0084 851 57852 9246 366––
dpa::dpa::normalized::chain_400014 326 41521 930 60419 737 759253 21626 479––
dpa::dpa::normalized::scrambled_10004 346 5726 490 8235 961 71360 2666 508––
dpa::dpa::normalized::scrambled_400017 617 79326 718 12324 103 098335 20226 829––
dpa::dpa::normalized::scrambled_k4_100025 249 99935 407 07433 257 969379 4077 202––
dpa::dpa::normalized_fixtures::ltl1 272 9301 821 4011 786 3414 646338––
dpa::dpa::normalized_fixtures::many_aps12 275 18317 721 63017 254 67083 5921 400––
dpa::dpa::normalized_fixtures::synthetic8 843 73312 507 29512 040 14090 743384––
dpa::dpa::prefix_partition::chain_163 574 1785 456 7065 390 3512 1131 594––
dpa::dpa::prefix_partition::chain_8916 1071 409 0571 379 877432772––
dpa::dpa::prefix_partition::scrambled_163 999 6986 155 4246 070 4443 7451 893––
dpa::dpa::prefix_partition::scrambled_81 086 5101 669 6131 639 248599782––
dpa::dpa::rejected_word::chain_100885 2311 282 0251 241 3952 834756––
dpa::dpa::rejected_word::chain_4003 441 5845 130 4884 832 08347 9771 672––
dpa::dpa::rejected_word::scrambled_100235 103360 768329 643506817––
dpa::dpa::rejected_word::scrambled_400887 0761 355 3101 244 36010 2341 708––
dpa::dpa::sccs::chain_1000511 023774 930699 6556 8301 175––
dpa::dpa::sccs::chain_40001 971 6303 022 7192 696 93929 8905 038––
dpa::dpa::sccs::scrambled_1000576 822871 308795 3438 221996––
dpa::dpa::sccs::scrambled_40002 295 9453 543 7433 164 88846 2734 214––
dpa::dpa::witness_colors::chain_1004 471 3306 366 0486 245 8889 0102 146––
dpa::dpa::witness_colors::chain_40043 697 42462 813 21261 212 837194 57917 928––
dpa::dpa::witness_colors::scrambled_1003 895 1885 531 8515 442 5718 7351 303––
dpa::dpa::witness_colors::scrambled_40029 949 98942 974 13442 154 394145 6852 609––
dpa::dpa::witness_inequivalence::complement_100268 661397 839380 479868372––
dpa::dpa::witness_inequivalence::complement_2586 600134 391121 47691356––
dpa::dpa::witness_inequivalence::complement_50149 885224 829211 034330347––
dpa::dpa::witness_not_subset_of::complement_100260 739388 101371 331841359––
dpa::dpa::witness_not_subset_of::complement_2584 611131 562119 15780343––
dpa::dpa::witness_not_subset_of::complement_50146 080219 943206 693312334––
dpa::dpa::witness_not_subset_of::normalized_100524 060764 046738 4461 389533––
dpa::dpa::witness_not_subset_of::normalized_25177 925261 455249 525153319––
dpa::dpa::witness_not_subset_of::normalized_50302 723439 927425 522368359––
dpa::dpa::witnesses_fixtures::ltl1 970 3492 739 3712 683 5415 769771––
dpa::dpa::witnesses_fixtures::synthetic15 821 11722 785 10122 257 96199 968780––
game::game::tangle_learning::few_priorities_10002 598 0813 917 5333 570 78335 9674 769745 4001 292
game::game::tangle_learning::few_priorities_40007 882 20712 361 12310 759 633194 36117 9912 407 4163 266
game::game::tangle_learning::ladder_10001 169 4601 811 0711 611 13112 2753 959377 868118
game::game::tangle_learning::ladder_40004 381 8167 035 9965 976 831110 24914 5121 511 668225
game::game::tangle_learning::many_priorities_10001 963 2522 956 1722 676 22725 5184 353566 440848
game::game::tangle_learning::many_priorities_40006 594 92110 279 4938 905 703165 65615 5862 187 3441 620
game::game::tangle_learning_min_odd::few_priorities_10001 500 2132 281 4772 044 14726 2703 028525 072189
game::game::tangle_learning_min_odd::few_priorities_40007 799 99212 020 31810 520 538215 73212 0322 560 0641 933
game::game::zielonka::few_priorities_10001 107 8761 656 0561 469 42112 4213 558356 67695
game::game::zielonka::few_priorities_40003 498 8975 565 1974 670 22281 12113 9821 278 56484
game::game::zielonka::ladder_10004 708 2536 180 8325 779 89749 8214 338549 716211
game::game::zielonka::ladder_400018 161 86723 979 39022 178 395245 35016 4072 200 404247
game::game::zielonka::many_priorities_10003 821 9635 020 0384 685 58338 3734 074841 196367
game::game::zielonka::many_priorities_400012 101 96916 234 07514 860 860175 38314 1802 808 868310
game::game::zielonka_min_odd::few_priorities_10002 040 9082 803 9652 583 83023 5522 925512 012195
game::game::zielonka_min_odd::few_priorities_40006 089 9658 778 7117 810 441130 9278 9611 705 700158
learning::learning::lstar_mealy::counting_161 758 2682 387 2242 349 0843761 036154 5042 633
learning::learning::lstar_mealy::counting_8450 608631 355604 9757574340 472862
learning::learning::sprout_parity::inf_ab_1001 147 9441 643 1321 584 5421 9321 398200 4161 872
learning::learning::sprout_parity::inf_ab_4002 354 5943 354 8633 243 6838 7891 921405 0403 649
nondeterministic::nondeterministic::fixture_accepts::ltl731 4871 133 7321 084 3076 245520––
nondeterministic::nondeterministic::nba_accepts::nba_400122 996 323179 087 159168 150 7591 663 47074 830––
nondeterministic::nondeterministic::nba_witness::nba_400017 383 33326 028 56425 014 714138 5249 178––
nondeterministic::nondeterministic::nca_accepts::nca_400254 927 534371 569 218345 867 6134 569 13581 598––
nondeterministic::nondeterministic::nca_witness::nca_400021 904 94133 241 98731 087 887325 67315 021––
nondeterministic::nondeterministic::nma_witness::nma_100025 383 66637 527 28935 512 844370 1084 683––
nondeterministic::nondeterministic::npa_accepts::empty_npa_400458 821 236670 095 587622 425 2328 911 20488 981––
nondeterministic::nondeterministic::npa_accepts::npa_10029 740 29842 889 06941 005 489287 78112 705––
nondeterministic::nondeterministic::npa_accepts::npa_400122 955 922179 060 796168 131 0861 662 05574 841––
nondeterministic::nondeterministic::npa_witness::empty_npa_10009 427 73713 979 00513 247 685123 2343 290––
nondeterministic::nondeterministic::npa_witness::empty_npa_400038 265 19257 449 62853 485 923657 36819 339––
nondeterministic::nondeterministic::npa_witness::npa_10002 979 0724 478 2454 313 25520 7831 745––
nondeterministic::nondeterministic::npa_witness::npa_400023 816 12935 551 42134 340 101178 0119 179––
nondeterministic::nondeterministic::nra_witness::nra_400021 455 58332 709 99330 163 398337 20324 588––
runs::runs::dfa_accepts_many_words::count_10004 053 1825 005 9024 981 6024 7901000
runs::runs::dfa_accepts_one_word::len_1003 5614 8524 45238600
runs::runs::dfa_accepts_one_word::len_10000276 039339 481335 781698600
runs::runs::dfa_accepts_one_word::len_1000002 750 7003 376 7793 344 6646 3251400
runs::runs::dfa_reached_state::count_10003 442 1804 272 8314 248 6014 790800
runs::runs::dpa_accepts_lassos::long8 225 40510 785 65410 761 0993 133254724 664500
runs::runs::dpa_accepts_lassos::short376 801502 316497 0864029226 236472
runs::runs::dpa_recurrent_edge_colors::long15 221 63619 508 02219 456 3875 938627922 1442 090
runs::runs::dpa_recurrent_edge_colors::short1 058 6241 435 3981 424 3081 238140168 6841 708
runs::runs::dpa_recurrent_states::long13 767 74019 256 02819 225 9333 597346788 104860
runs::runs::dpa_recurrent_states::short596 810814 493807 32355212681 996782
runs::runs::dpa_visited_sequences::long8 707 28312 042 60112 009 5714 2823322 190 780850
runs::runs::dpa_visited_sequences::short350 418483 868479 1835036229 986545
runs::runs::fixture_accepts_lassos::ltl7 833 94711 053 58611 022 4065 55098––
runs::runs::fixture_accepts_lassos::synthetic4 309 0576 188 5146 136 8599 624101––
scc::scc::scc_queries::small_cycles_1000855 9461 334 6741 174 0249 4363 242261 132559
scc::scc::scc_queries::small_cycles_40003 348 7615 268 9204 585 40050 89112 2591 045 3962 071
scc::scc::tarjan::back_edges_1000493 252795 135684 5705 4672 378170 27245
scc::scc::tarjan::back_edges_40001 914 6603 101 3902 656 01026 4478 947680 63253
scc::scc::tarjan::binary_tree_1000551 319848 324771 9344 0781 600105 76039
scc::scc::tarjan::binary_tree_40002 212 9413 410 0143 104 54421 1385 708419 51245
scc::scc::tarjan::fragmented_1000626 713981 364858 1849 6562 140146 084121
scc::scc::tarjan::fragmented_40002 413 5153 877 5493 297 11462 8457 606569 852167
scc::scc::tarjan::grid_1024575 645926 563802 9037 7432 427171 16045
scc::scc::tarjan::grid_40962 299 2053 824 7593 187 82963 6729 102684 18453
scc::scc::tarjan::path_10001 735 1372 678 4602 380 25523 2065 205455 2242 064
scc::scc::tarjan::path_40006 859 74810 607 2659 411 22098 32020 1271 821 5368 076
scc::scc::tarjan::ring_1000493 252798 771683 6616 3762 378170 27245
scc::scc::tarjan::ring_40001 914 6603 118 6382 651 69830 7598 947680 63253
scc::scc::tarjan::scrambled_1000559 051892 093780 1286 9862 201170 27245
scc::scc::tarjan::scrambled_40002 238 9753 618 6543 124 91441 8738 125680 63253
scc::scc::tarjan::small_cycles_1000854 9281 333 2991 172 7399 4463 238261 132559
scc::scc::tarjan::small_cycles_40003 344 7435 261 1294 581 02450 24312 2541 045 3962 071
scc::scc::tarjan_from::fragmented_1000232 901373 990324 8802 5491 03974 98564
scc::scc::tarjan_from::fragmented_4000872 5191 416 6221 221 48215 9563 296299 62578
scc::scc::tarjan_from::path_10001 724 7402 664 4852 369 14023 0615 144468 9932 091
scc::scc::tarjan_from::path_40006 763 89010 521 5729 291 232106 76119 9011 876 7538 109
scc::scc::tarjan_from::ring_1000482 362790 400672 0606 2662 486184 04172
scc::scc::tarjan_from::scrambled_1000548 161882 310768 8806 5232 309184 04172
storage::storage::add_edges_dense::k2_50001 360 8142 034 5921 989 5424 936582180 20037
storage::storage::add_edges_dense::k8_50004 281 4306 445 9126 259 53217 8302 778671 72840
storage::storage::add_edges_linked::k2_50001 252 4512 165 0362 003 04114 2202 5971 113 96026
storage::storage::add_edges_linked::k8_50006 144 06310 329 5989 400 50875 75715 7233 866 49628
storage::storage::clone_csr::k2_1000030 473183 13452 5044 1673 137230 0126
storage::storage::clone_dense::k2_1000015 04281 08025 7302 2501 260110 0084
storage::storage::clone_linked::k2_10000380 760911 527539 73212 4658 842680 0083
storage::storage::complete_dense::partial_5000238 753355 489348 674901661364
storage::storage::complete_linked::partial_50001 202 0091 723 9381 635 3436 9461 53965 67218
storage::storage::edges_from_all_states_csr::k2_200072 04577 20475 854221700
storage::storage::edges_from_all_states_csr::k8_2000108 045116 394111 0641 010800
storage::storage::edges_from_all_states_dense::k2_200092 047101 29199 846233800
storage::storage::edges_from_all_states_dense::k8_2000128 047140 545135 0401 038900
storage::storage::edges_from_all_states_linked::k2_200070 03388 75577 9252 131500
storage::storage::edges_from_all_states_linked::k8_2000154 033205 731168 6817 375500
storage::storage::minimize_csr::redundant_20004 399 8166 585 6946 106 64457 7935 4311 505 4202 212
storage::storage::minimize_csr::redundant_800017 235 33426 136 15523 891 730272 47825 2016 023 3008 250
storage::storage::minimize_dense::redundant_20004 434 8236 603 8576 149 50758 0474 6891 522 1562 221
storage::storage::minimize_dense::redundant_800017 408 18526 418 73724 122 742278 89325 7586 090 3408 261
storage::storage::minimize_linked::redundant_20004 305 9576 557 2696 009 17461 8446 8251 522 1562 221
storage::storage::minimize_linked::redundant_800016 930 01726 057 46023 610 780302 69526 6636 090 3408 261
storage::storage::overwrite_edges_dense::k2_50002 140 8453 246 6433 186 6287 908585180 20037
storage::storage::overwrite_edges_linked::k2_50003 410 5536 005 5895 286 67944 75314 1472 031 46427
storage::storage::predecessors_of_all_states_csr::k2_2000102 062121 152117 427682900
storage::storage::predecessors_of_all_states_dense::k2_2000232 309311 675302 1601 6164132 0083
storage::storage::predecessors_of_all_states_linked::k2_200070 03393 69176 6913 365500
storage::storage::product_successor_csr::n20065 08876 49376 14813800
storage::storage::product_successor_dense::n20065 08876 60176 12140800
storage::storage::product_successor_linked::n20046 54060 17256 992587700
storage::storage::reachable_states_csr::k2_80001 124 9181 621 4451 583 5757 4062417 36811
storage::storage::reachable_states_dense::k2_80001 173 5071 691 9441 646 9648 8352317 36811
storage::storage::reachable_states_linked::k2_8000980 0941 508 9951 410 26019 5163317 36811
storage::storage::remove_state_dense::k2_10000924 1621 347 8341 173 24421 5131 915176 0001 003
storage::storage::remove_state_linked::k2_10000997 0191 358 2811 319 7617 4174180 0003 000
storage::storage::run_many_words_csr::count_1000512 039601 912592 1621 908600
storage::storage::run_many_words_dense::count_1000512 042601 929592 1641 911600
storage::storage::run_many_words_linked::count_1000481 040611 107601 1121 957600
storage::storage::run_one_word_csr::len_1000002 500 0402 925 2992 893 7996 272400
storage::storage::run_one_word_dense::len_1000002 600 0423 025 3452 993 8006 274500
storage::storage::run_one_word_linked::len_1000002 350 0392 975 3942 943 7746 296400
storage::storage::successor_all_pairs_csr::k2_2000122 208139 477138 057228881
storage::storage::successor_all_pairs_csr::k8_2000702 262802 721797 3211 0248321
storage::storage::successor_all_pairs_dense::k2_2000150 205169 512168 047237881
storage::storage::successor_all_pairs_dense::k8_2000778 259880 816875 2961 0488321
storage::storage::successor_all_pairs_linked::k2_2000110 202141 070130 1352 138781
storage::storage::successor_all_pairs_linked::k8_2000752 256948 110910 9507 3837321
ts::ts::add_edges_in_order::k2_50001 062 4461 774 9721 613 03714 2152 5961 113 96026
ts::ts::add_edges_in_order::k8_50005 384 0588 769 4627 840 52275 73415 7223 866 49628
ts::ts::add_edges_in_reverse::k2_50001 062 4461 774 1801 613 23514 0172 5961 113 96026
ts::ts::builder_with_edges::k2_500079 800 78195 801 97093 298 580451 7766 9861 739 53245
ts::ts::canonical_naming::scrambled_15003 034 5224 445 1094 255 46921 8562 296463 8085 846
ts::ts::canonical_naming::scrambled_500938 2661 350 7681 307 5883 169781134 3361 831
ts::ts::clone::scrambled_10000381 079912 234540 13912 4838 848680 0083
ts::ts::clone::scrambled_400001 521 3163 857 6472 156 70742 57642 5162 720 0083
ts::ts::collect_dts::scrambled_100002 934 9255 185 5844 626 49937 16910 6642 228 06427
ts::ts::collect_dts::scrambled_4000011 679 37220 762 63218 409 072135 30047 9168 912 73631
ts::ts::determinize::nth_from_end_104 305 6806 339 2805 978 94522 0947 139651 7206 013
ts::ts::determinize::nth_from_end_1219 177 58828 064 76226 556 08798 79828 9912 732 60026 423
ts::ts::determinize::nth_from_end_8980 4401 446 6331 364 2282 9921 927156 5521 396
ts::ts::determinize_propositional::aps_6_n_64 088 2486 427 2935 941 66820 04811 0111 332 8964 208
ts::ts::determinize_propositional::aps_7_n_64 295 5876 716 4446 179 15425 32711 7331 197 9301 967
ts::ts::determinize_propositional::aps_8_n_625 052 41338 429 66935 513 084168 39259 2755 240 45613 678
ts::ts::edges_from_all_states::scrambled_200074 64693 91082 7402 1571100
ts::ts::edges_from_all_states::scrambled_k8_2000158 578210 791173 3967 4021100
ts::ts::erase_state_colors_collect::scrambled_100001 846 8553 177 6992 618 44437 17510 6682 228 06427
ts::ts::into_dfa::scrambled_100003 054 9835 375 8184 816 59337 18310 6662 228 06427
ts::ts::map_edge_colors_collect::scrambled_100001 928 4683 407 6062 789 15638 45112 1772 490 19227
ts::ts::map_edge_colors_collect::scrambled_400007 655 04713 651 55811 061 738140 33553 9479 961 29631
ts::ts::matching_product_successor_index::n20047 60262 66258 4225963600
ts::ts::minimal_representative_of_all_states::scrambled_1002 002 0682 856 2242 852 7641397347 5044 679
ts::ts::minimal_representative_of_all_states::scrambled_30016 088 40423 069 35623 054 0862 3541003 116 78438 196
ts::ts::minimal_representatives::scrambled_20003 248 7964 585 4984 538 7288 92062322 9447 709
ts::ts::minimal_representatives::scrambled_800014 321 19120 180 41519 965 69042 574531 464 89632 693
ts::ts::predecessors_of_all_states::scrambled_200074 64698 85081 5053 3921100
ts::ts::predecessors_of_all_states::scrambled_8000296 578397 297321 77715 0201200
ts::ts::reachable_states::scrambled_2000243 901374 261353 7313 609714 3369
ts::ts::reachable_states::scrambled_8000964 4801 493 8281 394 77819 5234117 36811
ts::ts::remove_edges_from_matching::scrambled_10000446 232589 776565 6564 6562464 0002 000
ts::ts::remove_edges_from_matching::scrambled_40000746 232941 276890 28110 0312464 0002 000
ts::ts::remove_state::scrambled_10000997 0191 358 3191 319 7597 4184280 0003 000
ts::ts::remove_state::scrambled_400001 357 0191 772 7831 703 64313 5344280 0003 000
ts::ts::scc_minimal_representatives::small_cycles_100210 321304 345289 5604541644 272309
ts::ts::scc_minimal_representatives::small_cycles_4001 192 0861 679 6431 629 9233 889865354 1481 325
ts::ts::subset_construction::nth_from_end_109 439 13313 173 46512 681 49545 2787 5881 073 5209 919
ts::ts::subset_construction::nth_from_end_1242 061 90058 466 83156 225 971218 19432 8544 518 97643 857
ts::ts::subset_construction::nth_from_end_82 040 0112 862 5422 756 7526 0522 158257 1042 276
ts::ts::successor_index_all_pairs::scrambled_2000110 814142 198130 9532 1581381
ts::ts::successor_index_all_pairs::scrambled_k8_2000752 800949 177911 6677 40414321

Wall-clock times

Criterion means of a single run. They depend on the machine and are not compared; use the instruction counts above for that.

benchmarkhead
collect/clone/1000004.10 ms (3.99 ms … 4.21 ms)
collect/clone/100000040.33 ms (39.93 ms … 40.73 ms)
collect/collect_owned/1000006.99 ms (6.92 ms … 7.06 ms)
collect/collect_owned/100000079.28 ms (77.83 ms … 80.72 ms)
collect/collect_ref/1000004.30 ms (4.22 ms … 4.38 ms)
collect/collect_ref/100000077.79 ms (76.40 ms … 79.17 ms)
collect/map_edge_colors_collect_owned/1000006.19 ms (6.07 ms … 6.31 ms)
collect/map_edge_colors_collect_owned/100000070.78 ms (69.81 ms … 71.75 ms)
wallclock/dfa/accepts/word_100000927.11 µs (909.93 µs … 944.29 µs)
wallclock/dfa/minimize/redundant_40001.36 ms (1.35 ms … 1.37 ms)
wallclock/dpa/complement/scrambled_40003.30 ms (3.29 ms … 3.31 ms)
wallclock/dpa/normalized/scrambled_40003.03 ms (3.02 ms … 3.05 ms)
wallclock/hoa/parse/ltl852.65 µs (847.42 µs … 857.89 µs)
wallclock/nfa/determinize/nth_from_end_123.48 ms (3.42 ms … 3.54 ms)
wallclock/nfa/subset_construction/nth_from_end_128.23 ms (8.05 ms … 8.42 ms)
wallclock/scc/tarjan/scrambled_100002.12 ms (2.07 ms … 2.18 ms)

Versus Spot and Oink

dpa/equivalence

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot121.96 µs114.74 µs1.06×– / –yes
hard/l-nba-2spot55.74 µs36.38 µs1.53×– / –yes
hard/m-nba-3spot237.68 µs138.18 µs1.72×– / –yes
ltl/and-gf-3spot9.06 µs11.84 µs0.77×– / –yes
ltl/eh-patterns-0spot21.52 µs20.00 µs1.08×– / –yes
ltl/gf-equiv-3spot19.84 µs33.32 µs0.60×– / –yes
ltl/gf-implies-3spot10.48 µs19.12 µs0.55×– / –yes
ltl/gh-q-3spot50.06 µs29.24 µs1.71×– / –yes
ltl/gh-r-3spot20.80 µs67.54 µs0.31×– / –yes
ltl/go-theta-3spot10.92 µs21.14 µs0.52×– / –yes
ltl/lily-patterns-1spot11.98 µs15.78 µs0.76×– / –yes
ltl/ms-phi-r-1spot10.20 µs38.20 µs0.27×– / –yes
ltl/ms-phi-s-1spot10.32 µs39.74 µs0.26×– / –yes
ltl/rv-counter-3spot27.58 µs13.24 µs2.08×– / –yes
ltl/sb-patterns-0spot12.52 µs14.18 µs0.88×– / –yes
ltl/sejk-k-2spot13.40 µs57.36 µs0.23×– / –yes
ltl/tv-uu-3spot40.96 µs37.76 µs1.08×– / –yes
ltl/u-left-3spot10.44 µs8.56 µs1.22×– / –yes
many_aps/and-gf-12spot154.04 µs65.06 µs2.37×– / –yes
many_aps/gf-equiv-12spot130.24 µs90.52 µs1.44×– / –yes
many_aps/r-right-10spot64.54 µs25.76 µs2.51×– / –yes
many_aps/random-n16-a8-p4-s1spot258.82 µs130.84 µs1.98×– / –yes
many_aps/random-n24-a12-p4-s1spot1.20 ms449.32 µs2.67×– / –yes
many_aps/random-n32-a16-p4-s1spot3.91 ms1.19 ms3.29×– / –yes
random/n128-a2-p6-s1spot330.06 µs207.90 µs1.59×– / –yes
random/n16-a1-p3-s1spot21.40 µs15.74 µs1.36×– / –yes
random/n256-a2-p8-s1spot883.57 µs476.76 µs1.85×– / –yes
random/n64-a2-p4-s1spot141.74 µs101.78 µs1.39×– / –yes

dpa/inclusion

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot72.30 µs61.26 µs1.18×– / –yes
hard/l-nba-2spot35.06 µs23.66 µs1.48×– / –yes
hard/m-nba-3spot147.06 µs87.64 µs1.68×– / –yes
ltl/and-gf-3spot5.76 µs8.16 µs0.71×– / –yes
ltl/eh-patterns-0spot14.44 µs17.66 µs0.82×– / –yes
ltl/gf-equiv-3spot11.28 µs18.78 µs0.60×– / –yes
ltl/gf-implies-3spot6.12 µs11.80 µs0.52×– / –yes
ltl/gh-q-3spot32.60 µs28.52 µs1.14×– / –yes
ltl/gh-r-3spot12.34 µs34.34 µs0.36×– / –yes
ltl/go-theta-3spot6.84 µs16.10 µs0.42×– / –yes
ltl/lily-patterns-1spot8.04 µs13.40 µs0.60×– / –yes
ltl/ms-phi-r-1spot6.08 µs19.00 µs0.32×– / –yes
ltl/ms-phi-s-1spot6.04 µs24.82 µs0.24×– / –yes
ltl/rv-counter-3spot17.90 µs14.70 µs1.22×– / –yes
ltl/sb-patterns-0spot7.68 µs12.26 µs0.63×– / –yes
ltl/sejk-k-2spot7.80 µs27.92 µs0.28×– / –yes
ltl/tv-uu-3spot28.36 µs30.42 µs0.93×– / –yes
ltl/u-left-3spot6.80 µs8.80 µs0.77×– / –yes
many_aps/and-gf-12spot124.22 µs54.24 µs2.29×– / –yes
many_aps/gf-equiv-12spot73.64 µs51.30 µs1.44×– / –yes
many_aps/r-right-10spot47.52 µs24.44 µs1.94×– / –yes
many_aps/random-n16-a8-p4-s1spot214.50 µs96.78 µs2.22×– / –yes
many_aps/random-n24-a12-p4-s1spot1.05 ms380.50 µs2.75×– / –yes
many_aps/random-n32-a16-p4-s1spot3.68 ms1.13 ms3.26×– / –yes
random/n128-a2-p6-s1spot196.90 µs125.32 µs1.57×– / –yes
random/n16-a1-p3-s1spot13.14 µs10.28 µs1.28×– / –yes
random/n256-a2-p8-s1spot508.26 µs305.30 µs1.66×– / –yes
random/n64-a2-p4-s1spot86.14 µs65.84 µs1.31×– / –yes

game/solve

inputreferenceoursreferenceours / referencesize (ours / reference)same result
ladder_10000oink-ppp2.70 ms322.50 µs8.38×10000 / 10000yes
ladder_10000oink-zlk2.70 ms320.50 µs8.43×10000 / 10000yes
ladder_10000spot2.70 ms9.65 ms0.28×10000 / 10000yes
ladder_50000oink-ppp20.00 ms2.15 ms9.30×50000 / 50000yes
ladder_50000oink-zlk20.00 ms2.18 ms9.19×50000 / 50000yes
ladder_50000spot20.00 ms76.47 ms0.26×50000 / 50000yes
random_few_10000oink-ppp7.27 ms1.50 ms4.84×10000 / 10000yes
random_few_10000oink-zlk7.27 ms2.67 ms2.73×10000 / 10000yes
random_few_10000spot7.27 ms20.54 ms0.35×10000 / 10000yes
random_few_50000oink-ppp82.02 ms16.01 ms5.12×50000 / 50000yes
random_few_50000oink-zlk82.02 ms19.11 ms4.29×50000 / 50000yes
random_few_50000spot82.02 ms691.48 ms0.12×50000 / 50000yes
random_many_10000oink-ppp6.30 ms1.62 ms3.90×10000 / 10000yes
random_many_10000oink-zlk6.30 ms2.70 ms2.34×10000 / 10000yes
random_many_10000spot6.30 ms33.62 ms0.19×10000 / 10000yes
random_many_50000oink-ppp81.10 ms31.50 ms2.57×50000 / 50000yes
random_many_50000oink-zlk81.10 ms48.23 ms1.68×50000 / 50000yes
random_many_50000spot81.10 ms712.79 ms0.11×50000 / 50000yes

Log

== head 2b5473e7fce6730c7a92d0f24df9525438296779: building ["scc", "ts", "runs", "dfa", "dpa", "storage", "nondeterministic", "learning", "game"] ["wallclock"]
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
    Finished `bench` profile [optimized] target(s) in 5m 37s
  Executable benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-c46c79a89c1c2941)
  Executable benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-0b05141996aaf468)
  Executable benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-fc166083591c2309)
  Executable benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-72c8d077fb858ea9)
  Executable benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-e95514e1aa52b38c)
  Executable benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-3b37aa113a4e75ae)
  Executable benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-b88e81c5af8ec1fc)
  Executable benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-fffb4b0cac4d81af)
  Executable benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-bb6ee25ff0b06f0f)
  Executable benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-3b599e26071f9e69)
== head: running scc
    Finished `bench` profile [optimized] target(s) in 0.25s
     Running benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-b88e81c5af8ec1fc)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running ts
    Finished `bench` profile [optimized] target(s) in 0.18s
     Running benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-bb6ee25ff0b06f0f)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running runs
    Finished `bench` profile [optimized] target(s) in 0.18s
     Running benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-3b37aa113a4e75ae)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dfa
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-c46c79a89c1c2941)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dpa
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-0b05141996aaf468)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running storage
    Finished `bench` profile [optimized] target(s) in 0.23s
     Running benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-fffb4b0cac4d81af)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running nondeterministic
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-e95514e1aa52b38c)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running learning
    Finished `bench` profile [optimized] target(s) in 0.20s
     Running benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-72c8d077fb858ea9)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running game
    Finished `bench` profile [optimized] target(s) in 0.20s
     Running benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-fc166083591c2309)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running wallclock (wall clock)
    Finished `bench` profile [optimized] target(s) in 0.23s
     Running benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-3b599e26071f9e69)
Gnuplot not found, using plotters backend
Benchmarking wallclock/scc/tarjan/scrambled_10000
Benchmarking wallclock/scc/tarjan/scrambled_10000: Analyzing
Benchmarking wallclock/dpa/normalized/scrambled_4000
Benchmarking wallclock/dpa/normalized/scrambled_4000: Analyzing
Benchmarking wallclock/dpa/complement/scrambled_4000
Benchmarking wallclock/dpa/complement/scrambled_4000: Analyzing
Benchmarking wallclock/dfa/minimize/redundant_4000
Benchmarking wallclock/dfa/minimize/redundant_4000: Analyzing
Benchmarking wallclock/dfa/accepts/word_100000
Benchmarking wallclock/dfa/accepts/word_100000: Analyzing
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12: Analyzing
Benchmarking wallclock/nfa/determinize/nth_from_end_12
Benchmarking wallclock/nfa/determinize/nth_from_end_12: Analyzing
Benchmarking wallclock/hoa/parse/ltl
Benchmarking wallclock/hoa/parse/ltl: Analyzing
Benchmarking collect/clone/100000
Benchmarking collect/clone/100000: Analyzing
Benchmarking collect/collect_owned/100000
Benchmarking collect/collect_owned/100000: Analyzing
Benchmarking collect/collect_ref/100000
Benchmarking collect/collect_ref/100000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/100000
Benchmarking collect/map_edge_colors_collect_owned/100000: Analyzing
Benchmarking collect/clone/1000000
Benchmarking collect/clone/1000000: Analyzing
Benchmarking collect/collect_owned/1000000
Benchmarking collect/collect_owned/1000000: Analyzing
Benchmarking collect/collect_ref/1000000
Benchmarking collect/collect_ref/1000000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/1000000
Benchmarking collect/map_edge_colors_collect_owned/1000000: Analyzing
== head: building the Python bindings
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support
🐍 Found CPython 3.13 at /nix/store/vilynzly1pyhawy04ljmrp7n9ffr24qy-python3-3.13.17/bin/python3
📡 Using build options features from pyproject.toml
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
   Compiling automata-py v0.1.0 (/var/lib/automata-bench/work/src-head/python)
    Finished `release` profile [optimized] target(s) in 7m 41s
📦 Built wheel for abi3 Python ≥ 3.11 to /var/lib/automata-bench/work/versus-head/wheels/automatars-0.1.0-cp311-abi3-manylinux_2_34_aarch64.whl
== head: running benches/versus/compare.py (Spot, Oink)
dpa/equivalence/hard/l-dsa-3 vs spot: 0.122 ms / 0.115 ms
dpa/equivalence/hard/l-nba-2 vs spot: 0.056 ms / 0.036 ms
dpa/equivalence/hard/m-nba-3 vs spot: 0.238 ms / 0.138 ms
dpa/equivalence/ltl/and-gf-3 vs spot: 0.009 ms / 0.012 ms
dpa/equivalence/ltl/eh-patterns-0 vs spot: 0.022 ms / 0.020 ms
dpa/equivalence/ltl/gf-equiv-3 vs spot: 0.020 ms / 0.033 ms
dpa/equivalence/ltl/gf-implies-3 vs spot: 0.010 ms / 0.019 ms
dpa/equivalence/ltl/gh-q-3 vs spot: 0.050 ms / 0.029 ms
dpa/equivalence/ltl/gh-r-3 vs spot: 0.021 ms / 0.068 ms
dpa/equivalence/ltl/go-theta-3 vs spot: 0.011 ms / 0.021 ms
dpa/equivalence/ltl/lily-patterns-1 vs spot: 0.012 ms / 0.016 ms
dpa/equivalence/ltl/ms-phi-r-1 vs spot: 0.010 ms / 0.038 ms
dpa/equivalence/ltl/ms-phi-s-1 vs spot: 0.010 ms / 0.040 ms
dpa/equivalence/ltl/rv-counter-3 vs spot: 0.028 ms / 0.013 ms
dpa/equivalence/ltl/sb-patterns-0 vs spot: 0.013 ms / 0.014 ms
dpa/equivalence/ltl/sejk-k-2 vs spot: 0.013 ms / 0.057 ms
dpa/equivalence/ltl/tv-uu-3 vs spot: 0.041 ms / 0.038 ms
dpa/equivalence/ltl/u-left-3 vs spot: 0.010 ms / 0.009 ms
dpa/equivalence/many_aps/and-gf-12 vs spot: 0.154 ms / 0.065 ms
dpa/equivalence/many_aps/gf-equiv-12 vs spot: 0.130 ms / 0.091 ms
dpa/equivalence/many_aps/r-right-10 vs spot: 0.065 ms / 0.026 ms
dpa/equivalence/many_aps/random-n16-a8-p4-s1 vs spot: 0.259 ms / 0.131 ms
dpa/equivalence/many_aps/random-n24-a12-p4-s1 vs spot: 1.200 ms / 0.449 ms
dpa/equivalence/many_aps/random-n32-a16-p4-s1 vs spot: 3.914 ms / 1.191 ms
dpa/equivalence/random/n128-a2-p6-s1 vs spot: 0.330 ms / 0.208 ms
dpa/equivalence/random/n16-a1-p3-s1 vs spot: 0.021 ms / 0.016 ms
dpa/equivalence/random/n256-a2-p8-s1 vs spot: 0.884 ms / 0.477 ms
dpa/equivalence/random/n64-a2-p4-s1 vs spot: 0.142 ms / 0.102 ms
dpa/inclusion/hard/l-dsa-3 vs spot: 0.072 ms / 0.061 ms
dpa/inclusion/hard/l-nba-2 vs spot: 0.035 ms / 0.024 ms
dpa/inclusion/hard/m-nba-3 vs spot: 0.147 ms / 0.088 ms
dpa/inclusion/ltl/and-gf-3 vs spot: 0.006 ms / 0.008 ms
dpa/inclusion/ltl/eh-patterns-0 vs spot: 0.014 ms / 0.018 ms
dpa/inclusion/ltl/gf-equiv-3 vs spot: 0.011 ms / 0.019 ms
dpa/inclusion/ltl/gf-implies-3 vs spot: 0.006 ms / 0.012 ms
dpa/inclusion/ltl/gh-q-3 vs spot: 0.033 ms / 0.029 ms
dpa/inclusion/ltl/gh-r-3 vs spot: 0.012 ms / 0.034 ms
dpa/inclusion/ltl/go-theta-3 vs spot: 0.007 ms / 0.016 ms
dpa/inclusion/ltl/lily-patterns-1 vs spot: 0.008 ms / 0.013 ms
dpa/inclusion/ltl/ms-phi-r-1 vs spot: 0.006 ms / 0.019 ms
dpa/inclusion/ltl/ms-phi-s-1 vs spot: 0.006 ms / 0.025 ms
dpa/inclusion/ltl/rv-counter-3 vs spot: 0.018 ms / 0.015 ms
dpa/inclusion/ltl/sb-patterns-0 vs spot: 0.008 ms / 0.012 ms
dpa/inclusion/ltl/sejk-k-2 vs spot: 0.008 ms / 0.028 ms
dpa/inclusion/ltl/tv-uu-3 vs spot: 0.028 ms / 0.030 ms
dpa/inclusion/ltl/u-left-3 vs spot: 0.007 ms / 0.009 ms
dpa/inclusion/many_aps/and-gf-12 vs spot: 0.124 ms / 0.054 ms
dpa/inclusion/many_aps/gf-equiv-12 vs spot: 0.074 ms / 0.051 ms
dpa/inclusion/many_aps/r-right-10 vs spot: 0.048 ms / 0.024 ms
dpa/inclusion/many_aps/random-n16-a8-p4-s1 vs spot: 0.215 ms / 0.097 ms
dpa/inclusion/many_aps/random-n24-a12-p4-s1 vs spot: 1.047 ms / 0.381 ms
dpa/inclusion/many_aps/random-n32-a16-p4-s1 vs spot: 3.685 ms / 1.130 ms
dpa/inclusion/random/n128-a2-p6-s1 vs spot: 0.197 ms / 0.125 ms
dpa/inclusion/random/n16-a1-p3-s1 vs spot: 0.013 ms / 0.010 ms
dpa/inclusion/random/n256-a2-p8-s1 vs spot: 0.508 ms / 0.305 ms
dpa/inclusion/random/n64-a2-p4-s1 vs spot: 0.086 ms / 0.066 ms
game/solve/random_few_10000 vs spot: 7.270 ms / 20.535 ms
game/solve/random_few_10000 vs oink-zlk: 7.270 ms / 2.665 ms
game/solve/random_few_10000 vs oink-ppp: 7.270 ms / 1.503 ms
game/solve/random_few_50000 vs spot: 82.019 ms / 691.476 ms
game/solve/random_few_50000 vs oink-zlk: 82.019 ms / 19.112 ms
game/solve/random_few_50000 vs oink-ppp: 82.019 ms / 16.011 ms
game/solve/random_many_10000 vs spot: 6.298 ms / 33.620 ms
game/solve/random_many_10000 vs oink-zlk: 6.298 ms / 2.696 ms
game/solve/random_many_10000 vs oink-ppp: 6.298 ms / 1.616 ms
game/solve/random_many_50000 vs spot: 81.096 ms / 712.790 ms
game/solve/random_many_50000 vs oink-zlk: 81.096 ms / 48.230 ms
game/solve/random_many_50000 vs oink-ppp: 81.096 ms / 31.504 ms
game/solve/ladder_10000 vs spot: 2.702 ms / 9.647 ms
game/solve/ladder_10000 vs oink-zlk: 2.702 ms / 0.321 ms
game/solve/ladder_10000 vs oink-ppp: 2.702 ms / 0.323 ms
game/solve/ladder_50000 vs spot: 19.998 ms / 76.468 ms
game/solve/ladder_50000 vs oink-zlk: 19.998 ms / 2.176 ms
game/solve/ladder_50000 vs oink-ppp: 19.998 ms / 2.151 ms
== head: 266 benchmarks, 16 wall-clock measurements, 74 comparisons