automata benchmarks

History · Versus Spot and Oink

main · f002431b

Release profile: one codegen unit and fat LTO (#164)

status done · queued 2026-11-11 16:51 · started 2026-11-11 16:51 · finished 2026-11-11 17:11

· · rustc 1.95.0 (59807616e 2026-04-14) (built from a source tarball) · valgrind-3.26.0 · pinned to cpu 1

Results

benchmarkinstructionsest. cyclesL1 hitsLL hitsRAM hitsheap bytesallocations
dfa::dfa::accepts_many_words::scrambled_1000319 438399 885393 3851 2231100
dfa::dfa::complement::scrambled_1000318 874512 713495 0333 37523139 10419
dfa::dfa::complement::scrambled_40001 247 6492 024 5981 938 85316 99522556 89623
dfa::dfa::equivalent::deep_difference_10052 83182 61975 4994519716 19625
dfa::dfa::equivalent::deep_difference_400177 852277 913255 5681 85137465 36433
dfa::dfa::equivalent::different_1008 34914 14811 4887751 80412
dfa::dfa::equivalent::different_4008 39014 35311 44378721 80412
dfa::dfa::equivalent::minimized_10031 31246 50844 273104497 75617
dfa::dfa::equivalent::minimized_400103 504153 082147 6577285131 58023
dfa::dfa::give_word::empty_10001 795 2342 450 0352 422 4804 91685152 8643 738
dfa::dfa::give_word::empty_40006 603 9839 194 1339 091 42820 06568679 80815 828
dfa::dfa::give_word::far_1000157 743223 352218 8275485128 176394
dfa::dfa::give_word::far_4000909 2251 280 3471 258 3373 93367148 4642 233
dfa::dfa::intersection::scrambled_1002 716 0483 995 2733 817 11312 0143 374428 08842
dfa::dfa::intersection::scrambled_20010 809 51116 277 72415 140 77496 86818 6461 708 12049
dfa::dfa::intersection_is_empty::empty_10050 65979 04172 5313518115 70020
dfa::dfa::intersection_is_empty::empty_20092 418145 704132 52961928831 58023
dfa::dfa::is_empty_language::empty_10001 795 2342 450 0692 422 4794 91686152 8643 738
dfa::dfa::is_empty_language::empty_40006 603 9839 194 1679 091 42720 06569679 80815 828
dfa::dfa::minimize::minimal_10003 298 9794 979 3184 604 64339 7325 0291 135 4922 214
dfa::dfa::minimize::minimal_400012 876 08919 609 31417 952 604204 57218 1104 543 7248 254
dfa::dfa::minimize::redundant_10002 174 5513 263 1683 028 27825 8873 013760 7801 201
dfa::dfa::minimize::redundant_40008 406 75412 753 34411 696 774143 6179 6713 044 9004 241
dfa::dfa::minimize::scrambled_10003 502 2565 309 2174 877 65251 0405 0391 116 0522 217
dfa::dfa::minimize::scrambled_400013 587 25820 978 23018 872 885294 02618 1494 470 1408 261
dfa::dfa::minimize_moore_with_edge_colors::scrambled_10003 500 3175 293 0044 866 63950 0985 0251 116 0522 217
dfa::dfa::minimize_moore_with_edge_colors::scrambled_400013 571 89120 909 14018 808 240300 57817 0864 470 1408 261
dfa::dfa::union::scrambled_1002 716 0483 995 2733 817 11312 0143 374428 08842
dfa::dfa::union::scrambled_20010 809 51116 277 72015 140 77596 86718 6461 708 12049
dpa::dpa::accepted_word::chain_100114 774174 778162 97850330––
dpa::dpa::accepted_word::chain_400391 265589 357555 0622 967556––
dpa::dpa::accepted_word::scrambled_100145 660218 233206 02840343––
dpa::dpa::accepted_word::scrambled_400499 945740 793707 7732 817541––
dpa::dpa::accepted_word_fixtures::ltl495 442716 515690 4702 248423––
dpa::dpa::accepted_word_fixtures::synthetic1 255 4201 811 2261 739 74111 798357––
dpa::dpa::accepted_word_fixtures_by_atoms::ltl1 620 5282 381 5472 324 0477 671547––
dpa::dpa::accepted_word_fixtures_by_atoms::synthetic2 741 0794 051 5213 966 98113 128540––
dpa::dpa::collect_dpa::scrambled_1000258 437421 129402 0243 8003––
dpa::dpa::collect_dpa::scrambled_40001 004 7821 689 4741 565 92916 3931 188––
dpa::dpa::complement::chain_10003 806 7135 721 4445 221 06955 1636 416––
dpa::dpa::complement::chain_400015 498 94023 452 27621 213 691262 01426 529––
dpa::dpa::complement::scrambled_10004 640 2806 873 5866 331 48662 5146 558––
dpa::dpa::complement::scrambled_400018 790 51228 240 20625 579 306344 02726 879––
dpa::dpa::complement_fixtures::ltl1 318 7521 878 3201 840 3404 810398––
dpa::dpa::complement_fixtures::many_aps12 439 59317 914 26517 431 66586 3771 449––
dpa::dpa::complement_fixtures::synthetic9 051 76712 765 56612 268 55696 441423––
dpa::dpa::complement_without_normalizing::scrambled_1000542 899786 600757 8755 15784––
dpa::dpa::complement_without_normalizing::scrambled_40002 135 2353 133 7992 979 55421 9661 269––
dpa::dpa::equivalent_fixtures::many_aps18 698 03427 049 25026 383 895120 3241 821––
dpa::dpa::hoa_parse::ltl5 102 0067 693 3737 405 21335 6173 145––
dpa::dpa::hoa_parse::many_aps80 066 835132 200 317115 009 432290 116449 723––
dpa::dpa::hoa_parse::synthetic23 303 31935 262 96033 885 730175 32514 303––
dpa::dpa::hoa_print::ltl14 449 24831 516 41219 332 3377 339347 068––
dpa::dpa::hoa_print::many_aps17 708 49225 469 85525 318 21027 928343––
dpa::dpa::hoa_print::synthetic21 972 18342 634 32330 233 40337 555348 947––
dpa::dpa::included_fixtures::many_aps15 907 78323 119 54322 585 73394 4001 766––
dpa::dpa::included_in::complement_100260 739388 049371 329845357––
dpa::dpa::included_in::complement_2584 611131 494119 15980341––
dpa::dpa::included_in::complement_50146 080219 879206 694313332––
dpa::dpa::included_in::normalized_100524 060764 006738 4411 396531––
dpa::dpa::included_in::normalized_25177 925261 399249 524156317––
dpa::dpa::included_in::normalized_50302 723439 883425 518374357––
dpa::dpa::language_equivalent::complement_100268 661397 813380 478870371––
dpa::dpa::language_equivalent::complement_2586 600134 349121 47989355––
dpa::dpa::language_equivalent::complement_50149 885224 791211 036329346––
dpa::dpa::language_equivalent::normalized_100963 9591 387 3411 353 4462 215652––
dpa::dpa::language_equivalent::normalized_25315 935456 707442 182238381––
dpa::dpa::language_equivalent::normalized_50564 774808 108791 073502415––
dpa::dpa::map_edge_colors_collect::scrambled_1000263 002419 699399 6593 87519––
dpa::dpa::map_edge_colors_collect::scrambled_40001 021 3471 681 8541 553 85917 1711 204––
dpa::dpa::normalized::chain_10003 513 1885 339 0084 851 57852 9246 366––
dpa::dpa::normalized::chain_400014 326 41521 930 60419 737 759253 21626 479––
dpa::dpa::normalized::scrambled_10004 346 5726 490 8235 961 71360 2666 508––
dpa::dpa::normalized::scrambled_400017 617 79326 718 12324 103 098335 20226 829––
dpa::dpa::normalized::scrambled_k4_100025 249 99935 407 07433 257 969379 4077 202––
dpa::dpa::normalized_fixtures::ltl1 272 9301 821 4011 786 3414 646338––
dpa::dpa::normalized_fixtures::many_aps12 275 18317 721 72017 254 67083 5891 403––
dpa::dpa::normalized_fixtures::synthetic8 843 73312 507 29512 040 14090 743384––
dpa::dpa::prefix_partition::chain_163 574 1785 456 7065 390 3512 1131 594––
dpa::dpa::prefix_partition::chain_8916 1071 409 0571 379 877432772––
dpa::dpa::prefix_partition::scrambled_163 999 6986 155 4246 070 4443 7451 893––
dpa::dpa::prefix_partition::scrambled_81 086 5101 669 6131 639 248599782––
dpa::dpa::rejected_word::chain_100885 2311 282 0251 241 3952 834756––
dpa::dpa::rejected_word::chain_4003 441 5735 130 4764 832 07147 9771 672––
dpa::dpa::rejected_word::scrambled_100235 103360 768329 643506817––
dpa::dpa::rejected_word::scrambled_400887 0761 355 3101 244 36010 2341 708––
dpa::dpa::sccs::chain_1000511 023774 930699 6556 8301 175––
dpa::dpa::sccs::chain_40001 971 6303 022 7192 696 93929 8905 038––
dpa::dpa::sccs::scrambled_1000576 822871 308795 3438 221996––
dpa::dpa::sccs::scrambled_40002 295 9453 543 7433 164 88846 2734 214––
dpa::dpa::witness_colors::chain_1004 471 3306 366 0486 245 8889 0102 146––
dpa::dpa::witness_colors::chain_40043 697 43562 813 19661 212 856194 57217 928––
dpa::dpa::witness_colors::scrambled_1003 895 1885 531 8515 442 5718 7351 303––
dpa::dpa::witness_colors::scrambled_40029 949 98942 974 13442 154 394145 6852 609––
dpa::dpa::witness_inequivalence::complement_100268 661397 839380 479868372––
dpa::dpa::witness_inequivalence::complement_2586 600134 391121 47691356––
dpa::dpa::witness_inequivalence::complement_50149 885224 829211 034330347––
dpa::dpa::witness_not_subset_of::complement_100260 739388 101371 331841359––
dpa::dpa::witness_not_subset_of::complement_2584 611131 562119 15780343––
dpa::dpa::witness_not_subset_of::complement_50146 080219 943206 693312334––
dpa::dpa::witness_not_subset_of::normalized_100524 060764 046738 4461 389533––
dpa::dpa::witness_not_subset_of::normalized_25177 925261 455249 525153319––
dpa::dpa::witness_not_subset_of::normalized_50302 723439 927425 522368359––
dpa::dpa::witnesses_fixtures::ltl1 970 3492 739 3712 683 5415 769771––
dpa::dpa::witnesses_fixtures::synthetic15 821 11722 785 10122 257 96199 968780––
game::game::tangle_learning::few_priorities_10002 598 0813 917 5333 570 78335 9674 769745 4001 292
game::game::tangle_learning::few_priorities_40007 882 20712 361 12310 759 633194 36117 9912 407 4163 266
game::game::tangle_learning::ladder_10001 169 4601 811 0711 611 13112 2753 959377 868118
game::game::tangle_learning::ladder_40004 381 8167 035 9965 976 831110 24914 5121 511 668225
game::game::tangle_learning::many_priorities_10001 963 2522 956 1722 676 22725 5184 353566 440848
game::game::tangle_learning::many_priorities_40006 594 92110 279 4938 905 703165 65615 5862 187 3441 620
game::game::tangle_learning_min_odd::few_priorities_10001 500 2132 281 4772 044 14726 2703 028525 072189
game::game::tangle_learning_min_odd::few_priorities_40007 799 99212 020 31810 520 538215 73212 0322 560 0641 933
game::game::zielonka::few_priorities_10001 107 8761 656 0561 469 42112 4213 558356 67695
game::game::zielonka::few_priorities_40003 498 8975 565 1974 670 22281 12113 9821 278 56484
game::game::zielonka::ladder_10004 708 2536 180 8325 779 89749 8214 338549 716211
game::game::zielonka::ladder_400018 161 86723 979 39022 178 395245 35016 4072 200 404247
game::game::zielonka::many_priorities_10003 821 9635 020 0384 685 58338 3734 074841 196367
game::game::zielonka::many_priorities_400012 101 96916 234 07514 860 860175 38314 1802 808 868310
game::game::zielonka_min_odd::few_priorities_10002 040 9082 803 9652 583 83023 5522 925512 012195
game::game::zielonka_min_odd::few_priorities_40006 089 9658 778 7117 810 441130 9278 9611 705 700158
learning::learning::lstar_mealy::counting_161 759 3952 389 0232 350 6184011 040154 5042 633
learning::learning::lstar_mealy::counting_8450 878631 740605 2857674540 472862
learning::learning::sprout_parity::inf_ab_1001 147 9881 643 1761 584 5911 9311 398200 4161 872
learning::learning::sprout_parity::inf_ab_4002 354 5733 354 8063 243 6618 7891 920405 0403 649
nondeterministic::nondeterministic::fixture_accepts::ltl731 4871 133 7321 084 3076 245520––
nondeterministic::nondeterministic::nba_accepts::nba_400122 996 323179 087 159168 150 7591 663 47074 830––
nondeterministic::nondeterministic::nba_witness::nba_400017 383 33326 028 56425 014 714138 5249 178––
nondeterministic::nondeterministic::nca_accepts::nca_400254 927 534371 569 218345 867 6134 569 13581 598––
nondeterministic::nondeterministic::nca_witness::nca_400021 904 94133 241 98731 087 887325 67315 021––
nondeterministic::nondeterministic::nma_witness::nma_100025 383 66637 527 28935 512 844370 1084 683––
nondeterministic::nondeterministic::npa_accepts::empty_npa_400458 821 236670 095 587622 425 2328 911 20488 981––
nondeterministic::nondeterministic::npa_accepts::npa_10029 740 29842 889 06941 005 489287 78112 705––
nondeterministic::nondeterministic::npa_accepts::npa_400122 955 922179 060 796168 131 0861 662 05574 841––
nondeterministic::nondeterministic::npa_witness::empty_npa_10009 427 73713 979 00513 247 685123 2343 290––
nondeterministic::nondeterministic::npa_witness::empty_npa_400038 265 19257 449 62853 485 923657 36819 339––
nondeterministic::nondeterministic::npa_witness::npa_10002 979 0724 478 2454 313 25520 7831 745––
nondeterministic::nondeterministic::npa_witness::npa_400023 816 12935 551 42134 340 101178 0119 179––
nondeterministic::nondeterministic::nra_witness::nra_400021 455 58332 709 99330 163 398337 20324 588––
runs::runs::dfa_accepts_many_words::count_10004 053 1825 005 9024 981 6024 7901000
runs::runs::dfa_accepts_one_word::len_1003 5614 8524 45238600
runs::runs::dfa_accepts_one_word::len_10000276 039339 481335 781698600
runs::runs::dfa_accepts_one_word::len_1000002 750 7003 376 7793 344 6646 3251400
runs::runs::dfa_reached_state::count_10003 442 1804 272 8314 248 6014 790800
runs::runs::dpa_accepts_lassos::long8 225 40510 785 65410 761 0993 133254724 664500
runs::runs::dpa_accepts_lassos::short376 801502 316497 0864029226 236472
runs::runs::dpa_recurrent_edge_colors::long15 221 63619 508 02219 456 3875 938627922 1442 090
runs::runs::dpa_recurrent_edge_colors::short1 058 6241 435 3981 424 3081 238140168 6841 708
runs::runs::dpa_recurrent_states::long13 767 74019 256 02819 225 9333 597346788 104860
runs::runs::dpa_recurrent_states::short596 810814 493807 32355212681 996782
runs::runs::dpa_visited_sequences::long8 707 28312 042 60112 009 5714 2823322 190 780850
runs::runs::dpa_visited_sequences::short350 418483 868479 1835036229 986545
runs::runs::fixture_accepts_lassos::ltl7 833 94711 053 58611 022 4065 55098––
runs::runs::fixture_accepts_lassos::synthetic4 309 0576 188 5146 136 8599 624101––
scc::scc::scc_queries::small_cycles_1000855 9461 334 6741 174 0249 4363 242261 132559
scc::scc::scc_queries::small_cycles_40003 348 7615 268 9204 585 40050 89112 2591 045 3962 071
scc::scc::tarjan::back_edges_1000493 252795 135684 5705 4672 378170 27245
scc::scc::tarjan::back_edges_40001 914 6603 101 3902 656 01026 4478 947680 63253
scc::scc::tarjan::binary_tree_1000551 319848 324771 9344 0781 600105 76039
scc::scc::tarjan::binary_tree_40002 212 9413 410 0143 104 54421 1385 708419 51245
scc::scc::tarjan::fragmented_1000626 713981 364858 1849 6562 140146 084121
scc::scc::tarjan::fragmented_40002 413 5153 877 5493 297 11462 8457 606569 852167
scc::scc::tarjan::grid_1024575 645926 563802 9037 7432 427171 16045
scc::scc::tarjan::grid_40962 299 2053 824 7593 187 82963 6729 102684 18453
scc::scc::tarjan::path_10001 735 1372 678 4602 380 25523 2065 205455 2242 064
scc::scc::tarjan::path_40006 859 74810 607 2659 411 22098 32020 1271 821 5368 076
scc::scc::tarjan::ring_1000493 252798 771683 6616 3762 378170 27245
scc::scc::tarjan::ring_40001 914 6603 118 6382 651 69830 7598 947680 63253
scc::scc::tarjan::scrambled_1000559 051892 093780 1286 9862 201170 27245
scc::scc::tarjan::scrambled_40002 238 9753 618 6543 124 91441 8738 125680 63253
scc::scc::tarjan::small_cycles_1000854 9281 333 2991 172 7399 4463 238261 132559
scc::scc::tarjan::small_cycles_40003 344 7435 261 1294 581 02450 24312 2541 045 3962 071
scc::scc::tarjan_from::fragmented_1000232 901373 990324 8802 5491 03974 98564
scc::scc::tarjan_from::fragmented_4000872 5191 416 6221 221 48215 9563 296299 62578
scc::scc::tarjan_from::path_10001 724 7402 664 4852 369 14023 0615 144468 9932 091
scc::scc::tarjan_from::path_40006 763 89010 521 5729 291 232106 76119 9011 876 7538 109
scc::scc::tarjan_from::ring_1000482 362790 400672 0606 2662 486184 04172
scc::scc::tarjan_from::scrambled_1000548 161882 310768 8806 5232 309184 04172
storage::storage::add_edges_dense::k2_50001 360 8142 034 5921 989 5424 936582180 20037
storage::storage::add_edges_dense::k8_50004 281 4306 445 9126 259 53217 8302 778671 72840
storage::storage::add_edges_linked::k2_50001 252 4512 165 0362 003 04114 2202 5971 113 96026
storage::storage::add_edges_linked::k8_50006 144 06310 329 5989 400 50875 75715 7233 866 49628
storage::storage::clone_csr::k2_1000030 473183 13452 5044 1673 137230 0126
storage::storage::clone_dense::k2_1000015 04281 08025 7302 2501 260110 0084
storage::storage::clone_linked::k2_10000380 760911 527539 73212 4658 842680 0083
storage::storage::complete_dense::partial_5000238 753355 489348 674901661364
storage::storage::complete_linked::partial_50001 202 0091 723 9381 635 3436 9461 53965 67218
storage::storage::edges_from_all_states_csr::k2_200072 04577 20475 854221700
storage::storage::edges_from_all_states_csr::k8_2000108 045116 394111 0641 010800
storage::storage::edges_from_all_states_dense::k2_200092 047101 29199 846233800
storage::storage::edges_from_all_states_dense::k8_2000128 047140 545135 0401 038900
storage::storage::edges_from_all_states_linked::k2_200070 03388 75577 9252 131500
storage::storage::edges_from_all_states_linked::k8_2000154 033205 731168 6817 375500
storage::storage::minimize_csr::redundant_20004 399 8166 585 6946 106 64457 7935 4311 505 4202 212
storage::storage::minimize_csr::redundant_800017 235 33426 136 15523 891 730272 47825 2016 023 3008 250
storage::storage::minimize_dense::redundant_20004 434 8236 603 8576 149 50758 0474 6891 522 1562 221
storage::storage::minimize_dense::redundant_800017 408 18526 418 73724 122 742278 89325 7586 090 3408 261
storage::storage::minimize_linked::redundant_20004 305 9576 557 2696 009 17461 8446 8251 522 1562 221
storage::storage::minimize_linked::redundant_800016 930 01726 057 46023 610 780302 69526 6636 090 3408 261
storage::storage::overwrite_edges_dense::k2_50002 140 8453 246 6433 186 6287 908585180 20037
storage::storage::overwrite_edges_linked::k2_50003 410 5536 005 5895 286 67944 75314 1472 031 46427
storage::storage::predecessors_of_all_states_csr::k2_2000102 062121 152117 427682900
storage::storage::predecessors_of_all_states_dense::k2_2000232 309311 675302 1601 6164132 0083
storage::storage::predecessors_of_all_states_linked::k2_200070 03393 69176 6913 365500
storage::storage::product_successor_csr::n20065 08876 49376 14813800
storage::storage::product_successor_dense::n20065 08876 60176 12140800
storage::storage::product_successor_linked::n20046 54060 17256 992587700
storage::storage::reachable_states_csr::k2_80001 124 9181 621 4451 583 5757 4062417 36811
storage::storage::reachable_states_dense::k2_80001 173 5071 691 9441 646 9648 8352317 36811
storage::storage::reachable_states_linked::k2_8000980 0941 508 9951 410 26019 5163317 36811
storage::storage::remove_state_dense::k2_10000924 1621 347 8341 173 24421 5131 915176 0001 003
storage::storage::remove_state_linked::k2_10000997 0191 358 2811 319 7617 4174180 0003 000
storage::storage::run_many_words_csr::count_1000512 039601 912592 1621 908600
storage::storage::run_many_words_dense::count_1000512 042601 929592 1641 911600
storage::storage::run_many_words_linked::count_1000481 040611 107601 1121 957600
storage::storage::run_one_word_csr::len_1000002 500 0402 925 2992 893 7996 272400
storage::storage::run_one_word_dense::len_1000002 600 0423 025 3452 993 8006 274500
storage::storage::run_one_word_linked::len_1000002 350 0392 975 3942 943 7746 296400
storage::storage::successor_all_pairs_csr::k2_2000122 208139 477138 057228881
storage::storage::successor_all_pairs_csr::k8_2000702 262802 721797 3211 0248321
storage::storage::successor_all_pairs_dense::k2_2000150 205169 512168 047237881
storage::storage::successor_all_pairs_dense::k8_2000778 259880 816875 2961 0488321
storage::storage::successor_all_pairs_linked::k2_2000110 202141 070130 1352 138781
storage::storage::successor_all_pairs_linked::k8_2000752 256948 110910 9507 3837321
ts::ts::add_edges_in_order::k2_50001 062 4461 774 9721 613 03714 2152 5961 113 96026
ts::ts::add_edges_in_order::k8_50005 384 0588 769 4627 840 52275 73415 7223 866 49628
ts::ts::add_edges_in_reverse::k2_50001 062 4461 774 1801 613 23514 0172 5961 113 96026
ts::ts::builder_with_edges::k2_500079 804 19595 805 48793 302 297451 6946 9921 739 53245
ts::ts::canonical_naming::scrambled_15003 034 5224 445 1094 255 46921 8562 296463 8085 846
ts::ts::canonical_naming::scrambled_500938 2661 350 7681 307 5883 169781134 3361 831
ts::ts::clone::scrambled_10000381 079912 234540 13912 4838 848680 0083
ts::ts::clone::scrambled_400001 521 3163 857 6472 156 70742 57642 5162 720 0083
ts::ts::collect_dts::scrambled_100002 934 9255 185 5844 626 49937 16910 6642 228 06427
ts::ts::collect_dts::scrambled_4000011 679 37220 762 63218 409 072135 30047 9168 912 73631
ts::ts::determinize::nth_from_end_104 305 6806 339 2805 978 94522 0947 139651 7206 013
ts::ts::determinize::nth_from_end_1219 177 58828 064 76226 556 08798 79828 9912 732 60026 423
ts::ts::determinize::nth_from_end_8980 4401 446 6331 364 2282 9921 927156 5521 396
ts::ts::determinize_propositional::aps_6_n_64 088 2486 427 2935 941 66820 04811 0111 332 8964 208
ts::ts::determinize_propositional::aps_7_n_64 295 5876 716 4446 179 15425 32711 7331 197 9301 967
ts::ts::determinize_propositional::aps_8_n_625 052 41338 429 56535 513 095168 38359 2735 240 45613 678
ts::ts::edges_from_all_states::scrambled_200074 64693 91082 7402 1571100
ts::ts::edges_from_all_states::scrambled_k8_2000158 578210 791173 3967 4021100
ts::ts::erase_state_colors_collect::scrambled_100001 846 8553 177 6992 618 44437 17510 6682 228 06427
ts::ts::into_dfa::scrambled_100003 054 9835 375 8184 816 59337 18310 6662 228 06427
ts::ts::map_edge_colors_collect::scrambled_100001 928 4683 407 6062 789 15638 45112 1772 490 19227
ts::ts::map_edge_colors_collect::scrambled_400007 655 04713 651 55811 061 738140 33553 9479 961 29631
ts::ts::matching_product_successor_index::n20047 60262 66258 4225963600
ts::ts::minimal_representative_of_all_states::scrambled_1002 002 0682 856 2242 852 7641397347 5044 679
ts::ts::minimal_representative_of_all_states::scrambled_30016 088 40423 069 35623 054 0862 3541003 116 78438 196
ts::ts::minimal_representatives::scrambled_20003 248 7964 585 4984 538 7288 92062322 9447 709
ts::ts::minimal_representatives::scrambled_800014 321 19120 180 41519 965 69042 574531 464 89632 693
ts::ts::predecessors_of_all_states::scrambled_200074 64698 85081 5053 3921100
ts::ts::predecessors_of_all_states::scrambled_8000296 578397 297321 77715 0201200
ts::ts::reachable_states::scrambled_2000243 901374 261353 7313 609714 3369
ts::ts::reachable_states::scrambled_8000964 4801 493 8281 394 77819 5234117 36811
ts::ts::remove_edges_from_matching::scrambled_10000446 232589 776565 6564 6562464 0002 000
ts::ts::remove_edges_from_matching::scrambled_40000746 232941 276890 28110 0312464 0002 000
ts::ts::remove_state::scrambled_10000997 0191 358 3191 319 7597 4184280 0003 000
ts::ts::remove_state::scrambled_400001 357 0191 772 7831 703 64313 5344280 0003 000
ts::ts::scc_minimal_representatives::small_cycles_100210 321304 345289 5604541644 272309
ts::ts::scc_minimal_representatives::small_cycles_4001 192 0861 679 6431 629 9233 889865354 1481 325
ts::ts::subset_construction::nth_from_end_109 439 13313 173 46512 681 49545 2787 5881 073 5209 919
ts::ts::subset_construction::nth_from_end_1242 061 90058 466 83156 225 971218 19432 8544 518 97643 857
ts::ts::subset_construction::nth_from_end_82 040 0112 862 5422 756 7526 0522 158257 1042 276
ts::ts::successor_index_all_pairs::scrambled_2000110 814142 198130 9532 1581381
ts::ts::successor_index_all_pairs::scrambled_k8_2000752 800949 177911 6677 40414321

Wall-clock times

Criterion means of a single run. They depend on the machine and are not compared; use the instruction counts above for that.

benchmarkhead
collect/clone/1000004.96 ms (4.92 ms … 4.99 ms)
collect/clone/100000041.19 ms (41.00 ms … 41.38 ms)
collect/collect_owned/1000008.23 ms (8.17 ms … 8.28 ms)
collect/collect_owned/100000084.06 ms (83.00 ms … 85.13 ms)
collect/collect_ref/1000004.40 ms (4.29 ms … 4.50 ms)
collect/collect_ref/100000077.99 ms (76.68 ms … 79.30 ms)
collect/map_edge_colors_collect_owned/1000006.07 ms (6.05 ms … 6.09 ms)
collect/map_edge_colors_collect_owned/100000072.38 ms (70.87 ms … 73.90 ms)
wallclock/dfa/accepts/word_100000920.86 µs (896.36 µs … 945.36 µs)
wallclock/dfa/minimize/redundant_40001.36 ms (1.35 ms … 1.36 ms)
wallclock/dpa/complement/scrambled_40003.41 ms (3.35 ms … 3.46 ms)
wallclock/dpa/normalized/scrambled_40003.15 ms (3.14 ms … 3.16 ms)
wallclock/hoa/parse/ltl872.60 µs (849.47 µs … 895.74 µs)
wallclock/nfa/determinize/nth_from_end_123.55 ms (3.50 ms … 3.61 ms)
wallclock/nfa/subset_construction/nth_from_end_127.58 ms (7.51 ms … 7.65 ms)
wallclock/scc/tarjan/scrambled_100002.40 ms (2.38 ms … 2.42 ms)

Versus Spot and Oink

dpa/equivalence

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot123.60 µs113.66 µs1.09×– / –yes
hard/l-nba-2spot55.44 µs35.34 µs1.57×– / –yes
hard/m-nba-3spot236.78 µs133.24 µs1.78×– / –yes
ltl/and-gf-3spot9.04 µs11.76 µs0.77×– / –yes
ltl/eh-patterns-0spot21.94 µs19.56 µs1.12×– / –yes
ltl/gf-equiv-3spot20.24 µs33.32 µs0.61×– / –yes
ltl/gf-implies-3spot10.68 µs18.92 µs0.56×– / –yes
ltl/gh-q-3spot50.52 µs28.16 µs1.79×– / –yes
ltl/gh-r-3spot21.00 µs66.94 µs0.31×– / –yes
ltl/go-theta-3spot10.98 µs20.60 µs0.53×– / –yes
ltl/lily-patterns-1spot12.06 µs15.36 µs0.79×– / –yes
ltl/ms-phi-r-1spot10.32 µs37.48 µs0.28×– / –yes
ltl/ms-phi-s-1spot10.32 µs39.22 µs0.26×– / –yes
ltl/rv-counter-3spot27.88 µs12.88 µs2.16×– / –yes
ltl/sb-patterns-0spot12.44 µs13.52 µs0.92×– / –yes
ltl/sejk-k-2spot13.64 µs56.86 µs0.24×– / –yes
ltl/tv-uu-3spot41.18 µs36.56 µs1.13×– / –yes
ltl/u-left-3spot10.36 µs8.32 µs1.25×– / –yes
many_aps/and-gf-12spot154.88 µs63.08 µs2.46×– / –yes
many_aps/gf-equiv-12spot130.08 µs88.46 µs1.47×– / –yes
many_aps/r-right-10spot65.08 µs24.82 µs2.62×– / –yes
many_aps/random-n16-a8-p4-s1spot259.12 µs127.12 µs2.04×– / –yes
many_aps/random-n24-a12-p4-s1spot1.30 ms437.90 µs2.97×– / –yes
many_aps/random-n32-a16-p4-s1spot3.97 ms1.18 ms3.38×– / –yes
random/n128-a2-p6-s1spot332.60 µs205.80 µs1.62×– / –yes
random/n16-a1-p3-s1spot22.12 µs15.24 µs1.45×– / –yes
random/n256-a2-p8-s1spot884.65 µs477.79 µs1.85×– / –yes
random/n64-a2-p4-s1spot139.92 µs98.22 µs1.42×– / –yes

dpa/inclusion

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot71.64 µs59.74 µs1.20×– / –yes
hard/l-nba-2spot35.04 µs22.52 µs1.56×– / –yes
hard/m-nba-3spot149.18 µs81.44 µs1.83×– / –yes
ltl/and-gf-3spot5.76 µs7.96 µs0.72×– / –yes
ltl/eh-patterns-0spot14.32 µs17.04 µs0.84×– / –yes
ltl/gf-equiv-3spot11.44 µs18.38 µs0.62×– / –yes
ltl/gf-implies-3spot6.24 µs11.16 µs0.56×– / –yes
ltl/gh-q-3spot33.20 µs27.20 µs1.22×– / –yes
ltl/gh-r-3spot12.40 µs34.10 µs0.36×– / –yes
ltl/go-theta-3spot6.88 µs15.94 µs0.43×– / –yes
ltl/lily-patterns-1spot8.08 µs13.32 µs0.61×– / –yes
ltl/ms-phi-r-1spot6.08 µs18.86 µs0.32×– / –yes
ltl/ms-phi-s-1spot6.06 µs24.56 µs0.25×– / –yes
ltl/rv-counter-3spot19.24 µs13.76 µs1.40×– / –yes
ltl/sb-patterns-0spot8.08 µs12.46 µs0.65×– / –yes
ltl/sejk-k-2spot8.06 µs27.66 µs0.29×– / –yes
ltl/tv-uu-3spot28.42 µs29.34 µs0.97×– / –yes
ltl/u-left-3spot6.92 µs8.50 µs0.81×– / –yes
many_aps/and-gf-12spot130.32 µs54.92 µs2.37×– / –yes
many_aps/gf-equiv-12spot74.22 µs50.56 µs1.47×– / –yes
many_aps/r-right-10spot47.18 µs23.34 µs2.02×– / –yes
many_aps/random-n16-a8-p4-s1spot216.46 µs98.44 µs2.20×– / –yes
many_aps/random-n24-a12-p4-s1spot1.12 ms368.98 µs3.04×– / –yes
many_aps/random-n32-a16-p4-s1spot3.79 ms1.09 ms3.48×– / –yes
random/n128-a2-p6-s1spot198.44 µs122.86 µs1.62×– / –yes
random/n16-a1-p3-s1spot13.08 µs10.04 µs1.30×– / –yes
random/n256-a2-p8-s1spot510.07 µs296.90 µs1.72×– / –yes
random/n64-a2-p4-s1spot84.42 µs60.44 µs1.40×– / –yes

game/solve

inputreferenceoursreferenceours / referencesize (ours / reference)same result
ladder_10000oink-ppp2.60 ms325.00 µs8.00×10000 / 10000yes
ladder_10000oink-zlk2.60 ms323.00 µs8.05×10000 / 10000yes
ladder_10000spot2.60 ms9.61 ms0.27×10000 / 10000yes
ladder_50000oink-ppp20.51 ms2.27 ms9.05×50000 / 50000yes
ladder_50000oink-zlk20.51 ms2.12 ms9.70×50000 / 50000yes
ladder_50000spot20.51 ms72.37 ms0.28×50000 / 50000yes
random_few_10000oink-ppp7.62 ms1.50 ms5.07×10000 / 10000yes
random_few_10000oink-zlk7.62 ms2.83 ms2.69×10000 / 10000yes
random_few_10000spot7.62 ms22.07 ms0.35×10000 / 10000yes
random_few_50000oink-ppp84.48 ms17.01 ms4.97×50000 / 50000yes
random_few_50000oink-zlk84.48 ms19.94 ms4.24×50000 / 50000yes
random_few_50000spot84.48 ms749.61 ms0.11×50000 / 50000yes
random_many_10000oink-ppp6.65 ms1.65 ms4.03×10000 / 10000yes
random_many_10000oink-zlk6.65 ms3.08 ms2.16×10000 / 10000yes
random_many_10000spot6.65 ms30.66 ms0.22×10000 / 10000yes
random_many_50000oink-ppp77.97 ms32.55 ms2.40×50000 / 50000yes
random_many_50000oink-zlk77.97 ms46.48 ms1.68×50000 / 50000yes
random_many_50000spot77.97 ms720.39 ms0.11×50000 / 50000yes

Log

== head f002431b0f7a095fcff96fb34cd40992fb493e7b: building ["scc", "ts", "runs", "dfa", "dpa", "storage", "nondeterministic", "learning", "game"] ["wallclock"]
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
    Finished `bench` profile [optimized] target(s) in 5m 42s
  Executable benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-c46c79a89c1c2941)
  Executable benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-0b05141996aaf468)
  Executable benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-fc166083591c2309)
  Executable benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-72c8d077fb858ea9)
  Executable benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-e95514e1aa52b38c)
  Executable benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-3b37aa113a4e75ae)
  Executable benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-b88e81c5af8ec1fc)
  Executable benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-fffb4b0cac4d81af)
  Executable benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-bb6ee25ff0b06f0f)
  Executable benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-3b599e26071f9e69)
== head: running scc
    Finished `bench` profile [optimized] target(s) in 0.26s
     Running benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-b88e81c5af8ec1fc)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running ts
    Finished `bench` profile [optimized] target(s) in 0.20s
     Running benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-bb6ee25ff0b06f0f)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running runs
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-3b37aa113a4e75ae)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dfa
    Finished `bench` profile [optimized] target(s) in 0.20s
     Running benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-c46c79a89c1c2941)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dpa
    Finished `bench` profile [optimized] target(s) in 0.23s
     Running benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-0b05141996aaf468)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running storage
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-fffb4b0cac4d81af)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running nondeterministic
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-e95514e1aa52b38c)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running learning
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-72c8d077fb858ea9)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running game
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-fc166083591c2309)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running wallclock (wall clock)
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-3b599e26071f9e69)
Gnuplot not found, using plotters backend
Benchmarking wallclock/scc/tarjan/scrambled_10000
Benchmarking wallclock/scc/tarjan/scrambled_10000: Analyzing
Benchmarking wallclock/dpa/normalized/scrambled_4000
Benchmarking wallclock/dpa/normalized/scrambled_4000: Analyzing
Benchmarking wallclock/dpa/complement/scrambled_4000
Benchmarking wallclock/dpa/complement/scrambled_4000: Analyzing
Benchmarking wallclock/dfa/minimize/redundant_4000
Benchmarking wallclock/dfa/minimize/redundant_4000: Analyzing
Benchmarking wallclock/dfa/accepts/word_100000
Benchmarking wallclock/dfa/accepts/word_100000: Analyzing
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12: Analyzing
Benchmarking wallclock/nfa/determinize/nth_from_end_12
Benchmarking wallclock/nfa/determinize/nth_from_end_12: Analyzing
Benchmarking wallclock/hoa/parse/ltl
Benchmarking wallclock/hoa/parse/ltl: Analyzing
Benchmarking collect/clone/100000
Benchmarking collect/clone/100000: Analyzing
Benchmarking collect/collect_owned/100000
Benchmarking collect/collect_owned/100000: Analyzing
Benchmarking collect/collect_ref/100000
Benchmarking collect/collect_ref/100000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/100000
Benchmarking collect/map_edge_colors_collect_owned/100000: Analyzing
Benchmarking collect/clone/1000000
Benchmarking collect/clone/1000000: Analyzing
Benchmarking collect/collect_owned/1000000
Benchmarking collect/collect_owned/1000000: Analyzing
Benchmarking collect/collect_ref/1000000
Benchmarking collect/collect_ref/1000000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/1000000
Benchmarking collect/map_edge_colors_collect_owned/1000000: Analyzing
== head: building the Python bindings
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support
🐍 Found CPython 3.13 at /nix/store/vilynzly1pyhawy04ljmrp7n9ffr24qy-python3-3.13.17/bin/python3
📡 Using build options features from pyproject.toml
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
   Compiling automata-py v0.1.0 (/var/lib/automata-bench/work/src-head/python)
    Finished `release` profile [optimized] target(s) in 7m 42s
📦 Built wheel for abi3 Python ≥ 3.11 to /var/lib/automata-bench/work/versus-head/wheels/automatars-0.1.0-cp311-abi3-manylinux_2_34_aarch64.whl
== head: running benches/versus/compare.py (Spot, Oink)
dpa/equivalence/hard/l-dsa-3 vs spot: 0.124 ms / 0.114 ms
dpa/equivalence/hard/l-nba-2 vs spot: 0.055 ms / 0.035 ms
dpa/equivalence/hard/m-nba-3 vs spot: 0.237 ms / 0.133 ms
dpa/equivalence/ltl/and-gf-3 vs spot: 0.009 ms / 0.012 ms
dpa/equivalence/ltl/eh-patterns-0 vs spot: 0.022 ms / 0.020 ms
dpa/equivalence/ltl/gf-equiv-3 vs spot: 0.020 ms / 0.033 ms
dpa/equivalence/ltl/gf-implies-3 vs spot: 0.011 ms / 0.019 ms
dpa/equivalence/ltl/gh-q-3 vs spot: 0.051 ms / 0.028 ms
dpa/equivalence/ltl/gh-r-3 vs spot: 0.021 ms / 0.067 ms
dpa/equivalence/ltl/go-theta-3 vs spot: 0.011 ms / 0.021 ms
dpa/equivalence/ltl/lily-patterns-1 vs spot: 0.012 ms / 0.015 ms
dpa/equivalence/ltl/ms-phi-r-1 vs spot: 0.010 ms / 0.037 ms
dpa/equivalence/ltl/ms-phi-s-1 vs spot: 0.010 ms / 0.039 ms
dpa/equivalence/ltl/rv-counter-3 vs spot: 0.028 ms / 0.013 ms
dpa/equivalence/ltl/sb-patterns-0 vs spot: 0.012 ms / 0.014 ms
dpa/equivalence/ltl/sejk-k-2 vs spot: 0.014 ms / 0.057 ms
dpa/equivalence/ltl/tv-uu-3 vs spot: 0.041 ms / 0.037 ms
dpa/equivalence/ltl/u-left-3 vs spot: 0.010 ms / 0.008 ms
dpa/equivalence/many_aps/and-gf-12 vs spot: 0.155 ms / 0.063 ms
dpa/equivalence/many_aps/gf-equiv-12 vs spot: 0.130 ms / 0.088 ms
dpa/equivalence/many_aps/r-right-10 vs spot: 0.065 ms / 0.025 ms
dpa/equivalence/many_aps/random-n16-a8-p4-s1 vs spot: 0.259 ms / 0.127 ms
dpa/equivalence/many_aps/random-n24-a12-p4-s1 vs spot: 1.301 ms / 0.438 ms
dpa/equivalence/many_aps/random-n32-a16-p4-s1 vs spot: 3.973 ms / 1.176 ms
dpa/equivalence/random/n128-a2-p6-s1 vs spot: 0.333 ms / 0.206 ms
dpa/equivalence/random/n16-a1-p3-s1 vs spot: 0.022 ms / 0.015 ms
dpa/equivalence/random/n256-a2-p8-s1 vs spot: 0.885 ms / 0.478 ms
dpa/equivalence/random/n64-a2-p4-s1 vs spot: 0.140 ms / 0.098 ms
dpa/inclusion/hard/l-dsa-3 vs spot: 0.072 ms / 0.060 ms
dpa/inclusion/hard/l-nba-2 vs spot: 0.035 ms / 0.023 ms
dpa/inclusion/hard/m-nba-3 vs spot: 0.149 ms / 0.081 ms
dpa/inclusion/ltl/and-gf-3 vs spot: 0.006 ms / 0.008 ms
dpa/inclusion/ltl/eh-patterns-0 vs spot: 0.014 ms / 0.017 ms
dpa/inclusion/ltl/gf-equiv-3 vs spot: 0.011 ms / 0.018 ms
dpa/inclusion/ltl/gf-implies-3 vs spot: 0.006 ms / 0.011 ms
dpa/inclusion/ltl/gh-q-3 vs spot: 0.033 ms / 0.027 ms
dpa/inclusion/ltl/gh-r-3 vs spot: 0.012 ms / 0.034 ms
dpa/inclusion/ltl/go-theta-3 vs spot: 0.007 ms / 0.016 ms
dpa/inclusion/ltl/lily-patterns-1 vs spot: 0.008 ms / 0.013 ms
dpa/inclusion/ltl/ms-phi-r-1 vs spot: 0.006 ms / 0.019 ms
dpa/inclusion/ltl/ms-phi-s-1 vs spot: 0.006 ms / 0.025 ms
dpa/inclusion/ltl/rv-counter-3 vs spot: 0.019 ms / 0.014 ms
dpa/inclusion/ltl/sb-patterns-0 vs spot: 0.008 ms / 0.012 ms
dpa/inclusion/ltl/sejk-k-2 vs spot: 0.008 ms / 0.028 ms
dpa/inclusion/ltl/tv-uu-3 vs spot: 0.028 ms / 0.029 ms
dpa/inclusion/ltl/u-left-3 vs spot: 0.007 ms / 0.009 ms
dpa/inclusion/many_aps/and-gf-12 vs spot: 0.130 ms / 0.055 ms
dpa/inclusion/many_aps/gf-equiv-12 vs spot: 0.074 ms / 0.051 ms
dpa/inclusion/many_aps/r-right-10 vs spot: 0.047 ms / 0.023 ms
dpa/inclusion/many_aps/random-n16-a8-p4-s1 vs spot: 0.216 ms / 0.098 ms
dpa/inclusion/many_aps/random-n24-a12-p4-s1 vs spot: 1.122 ms / 0.369 ms
dpa/inclusion/many_aps/random-n32-a16-p4-s1 vs spot: 3.794 ms / 1.091 ms
dpa/inclusion/random/n128-a2-p6-s1 vs spot: 0.198 ms / 0.123 ms
dpa/inclusion/random/n16-a1-p3-s1 vs spot: 0.013 ms / 0.010 ms
dpa/inclusion/random/n256-a2-p8-s1 vs spot: 0.510 ms / 0.297 ms
dpa/inclusion/random/n64-a2-p4-s1 vs spot: 0.084 ms / 0.060 ms
game/solve/random_few_10000 vs spot: 7.621 ms / 22.066 ms
game/solve/random_few_10000 vs oink-zlk: 7.621 ms / 2.834 ms
game/solve/random_few_10000 vs oink-ppp: 7.621 ms / 1.503 ms
game/solve/random_few_50000 vs spot: 84.483 ms / 749.608 ms
game/solve/random_few_50000 vs oink-zlk: 84.483 ms / 19.939 ms
game/solve/random_few_50000 vs oink-ppp: 84.483 ms / 17.006 ms
game/solve/random_many_10000 vs spot: 6.651 ms / 30.660 ms
game/solve/random_many_10000 vs oink-zlk: 6.651 ms / 3.083 ms
game/solve/random_many_10000 vs oink-ppp: 6.651 ms / 1.651 ms
game/solve/random_many_50000 vs spot: 77.968 ms / 720.387 ms
game/solve/random_many_50000 vs oink-zlk: 77.968 ms / 46.485 ms
game/solve/random_many_50000 vs oink-ppp: 77.968 ms / 32.548 ms
game/solve/ladder_10000 vs spot: 2.601 ms / 9.611 ms
game/solve/ladder_10000 vs oink-zlk: 2.601 ms / 0.323 ms
game/solve/ladder_10000 vs oink-ppp: 2.601 ms / 0.325 ms
game/solve/ladder_50000 vs spot: 20.513 ms / 72.372 ms
game/solve/ladder_50000 vs oink-zlk: 20.513 ms / 2.115 ms
game/solve/ladder_50000 vs oink-ppp: 20.513 ms / 2.265 ms
== head: 266 benchmarks, 16 wall-clock measurements, 74 comparisons