automata benchmarks

History · Versus Spot and Oink

main · c432f068

HOA: read parity edges without acceptance set (#151) (#167)

status done · queued 2026-11-11 19:29 · started 2026-11-11 19:29 · finished 2026-11-11 19:47

· · rustc 1.95.0 (59807616e 2026-04-14) (built from a source tarball) · valgrind-3.26.0 · pinned to cpu 1

Results

benchmarkinstructionsest. cyclesL1 hitsLL hitsRAM hitsheap bytesallocations
dfa::dfa::accepts_many_words::scrambled_1000319 438399 881393 3861 2221100
dfa::dfa::complement::scrambled_1000318 874512 713495 0333 37523139 10419
dfa::dfa::complement::scrambled_40001 247 6492 024 5981 938 85316 99522556 89623
dfa::dfa::equivalent::deep_difference_10052 83182 61975 4994519716 19625
dfa::dfa::equivalent::deep_difference_400177 852277 913255 5681 85137465 36433
dfa::dfa::equivalent::different_1008 34914 14811 4887751 80412
dfa::dfa::equivalent::different_4008 39014 35311 44378721 80412
dfa::dfa::equivalent::minimized_10031 31246 50844 273104497 75617
dfa::dfa::equivalent::minimized_400103 504153 074147 6597265131 58023
dfa::dfa::give_word::empty_10001 795 2342 450 0352 422 4804 91685152 8643 738
dfa::dfa::give_word::empty_40006 603 9839 194 1339 091 42820 06568679 80815 828
dfa::dfa::give_word::far_1000157 743223 352218 8275485128 176394
dfa::dfa::give_word::far_4000909 2251 280 3471 258 3373 93367148 4642 233
dfa::dfa::intersection::scrambled_1002 716 0483 995 2953 817 11512 0113 375428 08842
dfa::dfa::intersection::scrambled_20010 809 51116 277 75415 140 77496 86718 6471 708 12049
dfa::dfa::intersection_is_empty::empty_10050 65979 03372 5333318115 70020
dfa::dfa::intersection_is_empty::empty_20092 418145 704132 52961928831 58023
dfa::dfa::is_empty_language::empty_10001 795 2342 450 0692 422 4794 91686152 8643 738
dfa::dfa::is_empty_language::empty_40006 603 9839 194 1679 091 42720 06569679 80815 828
dfa::dfa::minimize::minimal_10003 298 9794 979 3904 604 64039 7335 0311 135 4922 214
dfa::dfa::minimize::minimal_400012 876 08919 609 36617 952 606204 56818 1124 543 7248 254
dfa::dfa::minimize::redundant_10002 174 5513 263 2243 028 27925 8843 015760 7801 201
dfa::dfa::minimize::redundant_40008 406 75412 753 39211 696 777143 6129 6733 044 9004 241
dfa::dfa::minimize::scrambled_10003 502 2565 309 2954 877 65551 0345 0421 116 0522 217
dfa::dfa::minimize::scrambled_400013 587 25820 978 31618 872 886294 02218 1524 470 1408 261
dfa::dfa::minimize_moore_with_edge_colors::scrambled_10003 500 3175 293 0704 866 64550 0895 0281 116 0522 217
dfa::dfa::minimize_moore_with_edge_colors::scrambled_400013 571 89120 909 24618 808 236300 57917 0894 470 1408 261
dfa::dfa::union::scrambled_1002 716 0483 995 2953 817 11512 0113 375428 08842
dfa::dfa::union::scrambled_20010 809 51116 277 75415 140 77496 86718 6471 708 12049
dpa::dpa::accepted_word::chain_100114 774174 736162 98148329––
dpa::dpa::accepted_word::chain_400391 265589 315555 0652 965555––
dpa::dpa::accepted_word::scrambled_100145 660218 263206 02839344––
dpa::dpa::accepted_word::scrambled_400499 945740 789707 7742 816541––
dpa::dpa::accepted_word_fixtures::ltl495 442717 367690 2722 444425––
dpa::dpa::accepted_word_fixtures::synthetic1 255 4201 812 7921 739 38712 147362––
dpa::dpa::accepted_word_fixtures_by_atoms::ltl1 620 5282 382 7892 323 7297 990546––
dpa::dpa::accepted_word_fixtures_by_atoms::synthetic2 741 0794 051 6713 966 93613 174539––
dpa::dpa::collect_dpa::scrambled_1000258 437421 129402 0243 8003––
dpa::dpa::collect_dpa::scrambled_40001 004 7821 689 4741 565 92916 3931 188––
dpa::dpa::complement::chain_10003 806 7135 721 6245 221 06955 1576 422––
dpa::dpa::complement::chain_400015 498 94023 452 46421 213 689262 01026 535––
dpa::dpa::complement::scrambled_10004 640 2806 873 6386 331 50362 4936 562––
dpa::dpa::complement::scrambled_400018 790 50128 240 49625 579 271344 04326 886––
dpa::dpa::complement_fixtures::ltl1 318 7521 877 4761 840 5064 650392––
dpa::dpa::complement_fixtures::many_aps12 439 58217 915 44517 431 25086 7941 435––
dpa::dpa::complement_fixtures::synthetic9 051 77812 768 42412 267 86497 144424––
dpa::dpa::complement_without_normalizing::scrambled_1000542 899786 482757 8825 15381––
dpa::dpa::complement_without_normalizing::scrambled_40002 135 2463 133 6892 979 57421 9611 266––
dpa::dpa::equivalent_fixtures::many_aps18 698 03427 050 09026 383 505120 7381 797––
dpa::dpa::hoa_parse::ltl5 180 3327 808 0137 519 90335 7543 124––
dpa::dpa::hoa_parse::many_aps80 468 714132 800 512115 595 187293 060449 715––
dpa::dpa::hoa_parse::synthetic23 750 49035 938 77134 536 016180 57714 282––
dpa::dpa::hoa_print::ltl14 450 35331 507 11319 336 7334 446347 090––
dpa::dpa::hoa_print::many_aps17 708 49225 411 12525 332 97513 152354––
dpa::dpa::hoa_print::synthetic21 972 18342 584 78130 245 82625 127348 952––
dpa::dpa::included_fixtures::many_aps15 907 78323 120 68322 585 40394 7361 760––
dpa::dpa::included_in::complement_100260 739388 107371 322851358––
dpa::dpa::included_in::complement_2584 611131 576119 14692342––
dpa::dpa::included_in::complement_50146 080219 989206 674332333––
dpa::dpa::included_in::normalized_100524 060763 764738 5541 276538––
dpa::dpa::included_in::normalized_25177 925261 373249 568107322––
dpa::dpa::included_in::normalized_50302 723439 869425 574311364––
dpa::dpa::language_equivalent::complement_100268 661397 843380 478869372––
dpa::dpa::language_equivalent::complement_2586 600134 407121 47295356––
dpa::dpa::language_equivalent::complement_50149 885224 881211 021343347––
dpa::dpa::language_equivalent::normalized_100963 9591 387 5111 353 4712 181661––
dpa::dpa::language_equivalent::normalized_25315 935456 429442 274143384––
dpa::dpa::language_equivalent::normalized_50564 774808 298791 093473424––
dpa::dpa::map_edge_colors_collect::scrambled_1000263 002419 699399 6593 87519––
dpa::dpa::map_edge_colors_collect::scrambled_40001 021 3471 681 8541 553 85917 1711 204––
dpa::dpa::normalized::chain_10003 513 1885 339 2224 851 57752 9186 373––
dpa::dpa::normalized::chain_400014 326 41521 930 82219 737 757253 21126 486––
dpa::dpa::normalized::scrambled_10004 346 5726 490 9215 961 72660 2486 513––
dpa::dpa::normalized::scrambled_400017 617 79326 718 45924 103 074335 21826 837––
dpa::dpa::normalized::scrambled_k4_100025 249 99935 407 37233 257 947379 4227 209––
dpa::dpa::normalized_fixtures::ltl1 272 9301 820 3371 786 5624 431332––
dpa::dpa::normalized_fixtures::many_aps12 275 18317 722 94017 254 29083 9791 393––
dpa::dpa::normalized_fixtures::synthetic8 843 73312 509 86512 039 50591 377385––
dpa::dpa::prefix_partition::chain_163 574 1785 459 4725 389 6822 7791 597––
dpa::dpa::prefix_partition::chain_8916 1071 410 2171 379 602705774––
dpa::dpa::prefix_partition::scrambled_163 999 6986 158 3466 069 7514 4331 898––
dpa::dpa::prefix_partition::scrambled_81 086 5101 671 1371 638 897946786––
dpa::dpa::rejected_word::chain_100885 2311 282 1311 241 4212 801763––
dpa::dpa::rejected_word::chain_4003 442 1235 131 1964 832 70147 9391 680––
dpa::dpa::rejected_word::scrambled_100235 103360 904329 639506821––
dpa::dpa::rejected_word::scrambled_400887 0761 355 5181 244 35310 2351 714––
dpa::dpa::sccs::chain_1000511 023774 938699 6536 8321 175––
dpa::dpa::sccs::chain_40001 971 6303 022 7232 696 93829 8915 038––
dpa::dpa::sccs::scrambled_1000576 822871 380795 3408 222998––
dpa::dpa::sccs::scrambled_40002 295 9453 543 8153 164 88546 2744 216––
dpa::dpa::witness_colors::chain_1004 471 3306 365 8026 245 9578 9402 147––
dpa::dpa::witness_colors::chain_40043 697 42462 812 93861 212 943194 46817 933––
dpa::dpa::witness_colors::scrambled_1003 895 1995 532 0395 442 5398 7791 303––
dpa::dpa::witness_colors::scrambled_40029 949 98942 973 99442 154 459145 6162 613––
dpa::dpa::witness_inequivalence::complement_100268 661397 853380 483863373––
dpa::dpa::witness_inequivalence::complement_2586 600134 421121 47690357––
dpa::dpa::witness_inequivalence::complement_50149 885224 895211 025338348––
dpa::dpa::witness_not_subset_of::complement_100260 739388 163371 323848360––
dpa::dpa::witness_not_subset_of::complement_2584 611131 628119 14888344––
dpa::dpa::witness_not_subset_of::complement_50146 080220 041206 676328335––
dpa::dpa::witness_not_subset_of::normalized_100524 060763 820738 5551 273540––
dpa::dpa::witness_not_subset_of::normalized_25177 925261 433249 568105324––
dpa::dpa::witness_not_subset_of::normalized_50302 723439 921425 576307366––
dpa::dpa::witnesses_fixtures::ltl1 970 3492 740 0212 683 2966 025760––
dpa::dpa::witnesses_fixtures::synthetic15 821 11722 786 95722 257 512100 415782––
game::game::tangle_learning::few_priorities_10002 598 0813 917 5333 570 78335 9674 769745 4001 292
game::game::tangle_learning::few_priorities_40007 882 20712 361 12310 759 633194 36117 9912 407 4163 266
game::game::tangle_learning::ladder_10001 169 4601 811 0711 611 13112 2753 959377 868118
game::game::tangle_learning::ladder_40004 381 8167 035 9965 976 831110 24914 5121 511 668225
game::game::tangle_learning::many_priorities_10001 963 2522 956 1722 676 22725 5184 353566 440848
game::game::tangle_learning::many_priorities_40006 594 92110 279 4938 905 703165 65615 5862 187 3441 620
game::game::tangle_learning_min_odd::few_priorities_10001 500 2132 281 4772 044 14726 2703 028525 072189
game::game::tangle_learning_min_odd::few_priorities_40007 799 99212 020 31810 520 538215 73212 0322 560 0641 933
game::game::zielonka::few_priorities_10001 107 8761 656 0561 469 42112 4213 558356 67695
game::game::zielonka::few_priorities_40003 498 8975 565 1974 670 22281 12113 9821 278 56484
game::game::zielonka::ladder_10004 708 2536 180 8325 779 89749 8214 338549 716211
game::game::zielonka::ladder_400018 161 86723 979 39022 178 395245 35016 4072 200 404247
game::game::zielonka::many_priorities_10003 821 9635 020 0384 685 58338 3734 074841 196367
game::game::zielonka::many_priorities_400012 101 96916 234 07514 860 860175 38314 1802 808 868310
game::game::zielonka_min_odd::few_priorities_10002 040 9082 803 9652 583 83023 5522 925512 012195
game::game::zielonka_min_odd::few_priorities_40006 089 9658 778 7117 810 441130 9278 9611 705 700158
learning::learning::lstar_mealy::counting_161 751 7622 376 8552 339 5354361 004154 1842 615
learning::learning::lstar_mealy::counting_8446 460624 621598 88113671640 280852
learning::learning::sprout_parity::inf_ab_1001 146 5081 639 5971 582 4421 9741 351200 3201 868
learning::learning::sprout_parity::inf_ab_4002 353 1063 351 3403 241 4958 8581 873404 9443 645
nondeterministic::nondeterministic::fixture_accepts::ltl731 4871 132 6141 084 5945 957521––
nondeterministic::nondeterministic::nba_accepts::nba_400122 996 323179 087 273168 150 7231 663 50774 829––
nondeterministic::nondeterministic::nba_witness::nba_400017 383 33326 028 58225 014 717138 5209 179––
nondeterministic::nondeterministic::nca_accepts::nca_400254 927 534371 569 756345 867 4714 569 27881 597––
nondeterministic::nondeterministic::nca_witness::nca_400021 904 94133 242 09731 087 867325 69215 022––
nondeterministic::nondeterministic::nma_witness::nma_100025 383 66637 527 53135 512 776370 1774 682––
nondeterministic::nondeterministic::npa_accepts::empty_npa_400458 821 236670 094 773622 425 4288 911 00988 980––
nondeterministic::nondeterministic::npa_accepts::npa_10029 740 29842 888 66741 005 582287 68912 704––
nondeterministic::nondeterministic::npa_accepts::npa_400122 955 922179 060 414168 131 1741 661 96874 840––
nondeterministic::nondeterministic::npa_witness::empty_npa_10009 427 73713 979 09513 247 670123 2483 291––
nondeterministic::nondeterministic::npa_witness::empty_npa_400038 265 19257 449 78653 485 891657 39919 340––
nondeterministic::nondeterministic::npa_witness::npa_10002 979 0724 478 2754 313 25520 7821 746––
nondeterministic::nondeterministic::npa_witness::npa_400023 816 12935 551 46334 340 098178 0139 180––
nondeterministic::nondeterministic::nra_witness::nra_400021 455 58332 710 08930 163 374337 22724 588––
runs::runs::dfa_accepts_many_words::count_10004 053 1825 005 8684 981 6034 790900
runs::runs::dfa_accepts_one_word::len_1003 5614 8864 45138700
runs::runs::dfa_accepts_one_word::len_10000276 039339 515335 780698700
runs::runs::dfa_accepts_one_word::len_1000002 750 7003 376 8133 344 6636 3251500
runs::runs::dfa_reached_state::count_10003 442 1804 272 8654 248 6004 790900
runs::runs::dpa_accepts_lassos::long8 225 40510 785 67810 761 1083 122256724 664500
runs::runs::dpa_accepts_lassos::short376 801502 418497 0834029526 236472
runs::runs::dpa_recurrent_edge_colors::long15 221 63619 507 71019 456 4805 843629922 1442 090
runs::runs::dpa_recurrent_edge_colors::short1 058 6241 435 5481 424 2931 250143168 6841 708
runs::runs::dpa_recurrent_states::long13 767 74019 256 42219 225 8723 653351788 104860
runs::runs::dpa_recurrent_states::short596 810814 701807 31655313281 996782
runs::runs::dpa_visited_sequences::long8 707 28312 042 55512 009 5754 2793312 190 780850
runs::runs::dpa_visited_sequences::short350 418483 834479 1845036129 986545
runs::runs::fixture_accepts_lassos::ltl7 833 94711 053 54811 022 4085 54997––
runs::runs::fixture_accepts_lassos::synthetic4 309 0576 188 4926 136 8579 627100––
scc::scc::scc_queries::small_cycles_1000855 9461 334 6741 174 0249 4363 242261 132559
scc::scc::scc_queries::small_cycles_40003 348 7615 268 9204 585 40050 89112 2591 045 3962 071
scc::scc::tarjan::back_edges_1000493 252795 135684 5705 4672 378170 27245
scc::scc::tarjan::back_edges_40001 914 6603 101 3902 656 01026 4478 947680 63253
scc::scc::tarjan::binary_tree_1000551 319848 324771 9344 0781 600105 76039
scc::scc::tarjan::binary_tree_40002 212 9413 410 0143 104 54421 1385 708419 51245
scc::scc::tarjan::fragmented_1000626 713981 364858 1849 6562 140146 084121
scc::scc::tarjan::fragmented_40002 413 5153 877 5493 297 11462 8457 606569 852167
scc::scc::tarjan::grid_1024575 645926 563802 9037 7432 427171 16045
scc::scc::tarjan::grid_40962 299 2053 824 7593 187 82963 6729 102684 18453
scc::scc::tarjan::path_10001 735 1372 678 4602 380 25523 2065 205455 2242 064
scc::scc::tarjan::path_40006 859 74810 607 2659 411 22098 32020 1271 821 5368 076
scc::scc::tarjan::ring_1000493 252798 771683 6616 3762 378170 27245
scc::scc::tarjan::ring_40001 914 6603 118 6382 651 69830 7598 947680 63253
scc::scc::tarjan::scrambled_1000559 051892 093780 1286 9862 201170 27245
scc::scc::tarjan::scrambled_40002 238 9753 618 6543 124 91441 8738 125680 63253
scc::scc::tarjan::small_cycles_1000854 9281 333 2991 172 7399 4463 238261 132559
scc::scc::tarjan::small_cycles_40003 344 7435 261 1294 581 02450 24312 2541 045 3962 071
scc::scc::tarjan_from::fragmented_1000232 901373 990324 8802 5491 03974 98564
scc::scc::tarjan_from::fragmented_4000872 5191 416 6221 221 48215 9563 296299 62578
scc::scc::tarjan_from::path_10001 724 7402 664 4852 369 14023 0615 144468 9932 091
scc::scc::tarjan_from::path_40006 763 89010 521 5729 291 232106 76119 9011 876 7538 109
scc::scc::tarjan_from::ring_1000482 362790 400672 0606 2662 486184 04172
scc::scc::tarjan_from::scrambled_1000548 161882 310768 8806 5232 309184 04172
storage::storage::add_edges_dense::k2_50001 390 8122 084 7262 039 5364 936586180 20037
storage::storage::add_edges_dense::k8_50004 311 4276 496 0496 309 52417 8312 782671 72840
storage::storage::add_edges_linked::k2_50001 252 4512 165 0622 003 04214 2182 5981 113 96026
storage::storage::add_edges_linked::k8_50006 144 06310 329 6249 400 50975 75515 7243 866 49628
storage::storage::clone_csr::k2_1000022 53776 08542 5756 61812210 0124
storage::storage::clone_dense::k2_1000015 04281 07225 7322 2481 260110 0084
storage::storage::clone_linked::k2_10000380 760911 493539 73312 4658 841680 0083
storage::storage::complete_dense::partial_5000230 702348 722343 45788524561
storage::storage::complete_linked::partial_50001 183 1291 690 4331 602 9686 9441 50765 59215
storage::storage::edges_from_all_states_csr::k2_200034 03838 32838 04329400
storage::storage::edges_from_all_states_csr::k8_200034 03838 62837 968104400
storage::storage::edges_from_all_states_dense::k2_200092 047101 25799 847233700
storage::storage::edges_from_all_states_dense::k8_2000128 047140 511135 0411 038800
storage::storage::edges_from_all_states_linked::k2_200070 03388 75577 9252 131500
storage::storage::edges_from_all_states_linked::k8_2000154 033205 731168 6817 375500
storage::storage::minimize_csr::redundant_20004 240 6456 376 5295 902 60960 6034 8831 505 4202 212
storage::storage::minimize_csr::redundant_800016 584 07325 284 39723 034 757275 48824 9206 023 3008 250
storage::storage::minimize_dense::redundant_20004 434 8236 603 9816 149 50658 0444 6931 522 1562 221
storage::storage::minimize_dense::redundant_800017 408 18526 418 89924 122 739278 89125 7636 090 3408 261
storage::storage::minimize_linked::redundant_20004 305 9576 557 5316 009 16161 8506 8321 522 1562 221
storage::storage::minimize_linked::redundant_800016 930 01726 057 75223 610 767302 70026 6716 090 3408 261
storage::storage::overwrite_edges_dense::k2_50002 170 8433 296 7813 236 6217 909589180 20037
storage::storage::overwrite_edges_linked::k2_50003 410 5536 005 6495 286 67944 75114 1492 031 46427
storage::storage::predecessors_of_all_states_csr::k2_2000251 025342 244316 5394 9592648 0083
storage::storage::predecessors_of_all_states_dense::k2_2000232 309311 709302 1591 6164232 0083
storage::storage::predecessors_of_all_states_linked::k2_200070 03393 65776 6923 365400
storage::storage::product_successor_csr::n20048 08258 50758 10738600
storage::storage::product_successor_dense::n20065 08876 56776 12240700
storage::storage::product_successor_linked::n20046 54060 13056 995585600
storage::storage::reachable_states_csr::k2_8000988 7091 471 2871 410 82711 8962817 36811
storage::storage::reachable_states_dense::k2_80001 173 5071 692 0121 646 9628 8352517 36811
storage::storage::reachable_states_linked::k2_8000980 0941 509 0631 410 25819 5163517 36811
storage::storage::remove_state_dense::k2_10000924 1621 347 8261 173 24621 5111 915176 0001 003
storage::storage::remove_state_linked::k2_10000997 0191 358 2131 319 7637 4173980 0003 000
storage::storage::run_many_words_csr::count_1000504 041624 024614 1441 934600
storage::storage::run_many_words_dense::count_1000512 042601 929592 1641 911600
storage::storage::run_many_words_linked::count_1000481 040611 107601 1121 957600
storage::storage::run_one_word_csr::len_1000002 450 0423 025 3952 993 7956 285500
storage::storage::run_one_word_dense::len_1000002 600 0423 025 3452 993 8006 274500
storage::storage::run_one_word_linked::len_1000002 350 0392 975 4282 943 7736 296500
storage::storage::successor_all_pairs_csr::k2_200096 205117 081113 641646681
storage::storage::successor_all_pairs_csr::k8_2000510 259627 149616 1992 1486321
storage::storage::successor_all_pairs_dense::k2_2000150 205169 478168 048237781
storage::storage::successor_all_pairs_dense::k8_2000778 259880 782875 2971 0487321
storage::storage::successor_all_pairs_linked::k2_2000110 202141 070130 1352 138781
storage::storage::successor_all_pairs_linked::k8_2000752 256948 110910 9507 3837321
ts::ts::add_edges_in_order::k2_50001 062 4461 774 9721 613 03714 2152 5961 113 96026
ts::ts::add_edges_in_order::k8_50005 384 0588 769 4627 840 52275 73415 7223 866 49628
ts::ts::add_edges_in_reverse::k2_50001 062 4461 774 1801 613 23514 0172 5961 113 96026
ts::ts::builder_with_edges::k2_500079 938 74095 907 15193 436 991450 4856 2211 739 53245
ts::ts::canonical_naming::scrambled_15003 034 5224 445 1694 255 45421 8712 296463 8085 846
ts::ts::canonical_naming::scrambled_500938 2661 350 7581 307 5833 175780134 3361 831
ts::ts::clone::scrambled_10000381 079912 268540 13812 4838 849680 0083
ts::ts::clone::scrambled_400001 521 3163 857 6812 156 70642 57642 5172 720 0083
ts::ts::collect_dts::scrambled_100002 934 9255 185 5844 626 49937 16910 6642 228 06427
ts::ts::collect_dts::scrambled_4000011 679 37220 762 63218 409 072135 30047 9168 912 73631
ts::ts::determinize::nth_from_end_104 305 6806 339 2425 978 94722 0937 138651 7206 013
ts::ts::determinize::nth_from_end_1219 177 58828 064 70026 556 09598 79128 9902 732 60026 423
ts::ts::determinize::nth_from_end_8980 4401 446 5991 364 2292 9921 926156 5521 396
ts::ts::determinize_propositional::aps_6_n_64 088 2486 427 4595 941 67920 03011 0181 332 8964 208
ts::ts::determinize_propositional::aps_7_n_64 295 5876 716 4686 179 16325 31611 7351 197 9301 967
ts::ts::determinize_propositional::aps_8_n_625 052 41338 429 95535 513 095168 37059 2865 240 45613 678
ts::ts::edges_from_all_states::scrambled_200074 64693 97882 7382 1571300
ts::ts::edges_from_all_states::scrambled_k8_2000158 578210 859173 3947 4021300
ts::ts::erase_state_colors_collect::scrambled_100001 846 8553 177 5972 618 44737 17510 6652 228 06427
ts::ts::into_dfa::scrambled_100003 054 9835 375 9204 816 59037 18310 6692 228 06427
ts::ts::map_edge_colors_collect::scrambled_100001 928 4683 407 5042 789 15938 45112 1742 490 19227
ts::ts::map_edge_colors_collect::scrambled_400007 655 04713 651 45611 061 741140 33553 9449 961 29631
ts::ts::matching_product_successor_index::n20047 60262 59858 4235973400
ts::ts::minimal_representative_of_all_states::scrambled_1002 002 0682 856 1222 852 7671394347 5044 679
ts::ts::minimal_representative_of_all_states::scrambled_30016 088 40423 069 25423 054 0892 354973 116 78438 196
ts::ts::minimal_representatives::scrambled_20003 248 7964 585 5324 538 7278 92063322 9447 709
ts::ts::minimal_representatives::scrambled_800014 321 19120 180 45319 965 68842 575541 464 89632 693
ts::ts::predecessors_of_all_states::scrambled_200074 64698 85081 5053 3921100
ts::ts::predecessors_of_all_states::scrambled_8000296 578397 297321 77715 0201200
ts::ts::reachable_states::scrambled_2000243 901374 295353 7303 609724 3369
ts::ts::reachable_states::scrambled_8000964 4801 493 8621 394 77719 5234217 36811
ts::ts::remove_edges_from_matching::scrambled_10000446 232589 810565 6554 6562564 0002 000
ts::ts::remove_edges_from_matching::scrambled_40000746 232941 310890 28010 0312564 0002 000
ts::ts::remove_state::scrambled_10000997 0191 358 3191 319 7597 4184280 0003 000
ts::ts::remove_state::scrambled_400001 357 0191 772 7831 703 64313 5344280 0003 000
ts::ts::scc_minimal_representatives::small_cycles_100210 321304 069289 5694440844 272309
ts::ts::scc_minimal_representatives::small_cycles_4001 192 0861 679 3711 629 9313 889857354 1481 325
ts::ts::subset_construction::nth_from_end_109 439 13313 173 43512 681 49545 2797 5871 073 5209 919
ts::ts::subset_construction::nth_from_end_1242 061 90058 466 83756 225 962218 20432 8534 518 97643 857
ts::ts::subset_construction::nth_from_end_82 040 0112 862 5042 756 7546 0512 157257 1042 276
ts::ts::successor_index_all_pairs::scrambled_2000110 814142 232130 9522 1581481
ts::ts::successor_index_all_pairs::scrambled_k8_2000752 800949 211911 6667 40415321

Wall-clock times

Criterion means of a single run. They depend on the machine and are not compared; use the instruction counts above for that.

benchmarkhead
collect/clone/1000003.66 ms (3.63 ms … 3.70 ms)
collect/clone/100000037.76 ms (36.73 ms … 38.79 ms)
collect/collect_owned/1000006.43 ms (6.43 ms … 6.43 ms)
collect/collect_owned/100000074.59 ms (73.17 ms … 76.00 ms)
collect/collect_ref/1000004.10 ms (3.99 ms … 4.22 ms)
collect/collect_ref/100000080.30 ms (78.53 ms … 82.07 ms)
collect/map_edge_colors_collect_owned/1000007.01 ms (6.97 ms … 7.05 ms)
collect/map_edge_colors_collect_owned/100000071.56 ms (70.21 ms … 72.91 ms)
wallclock/dfa/accepts/word_100000828.28 µs (827.51 µs … 829.05 µs)
wallclock/dfa/minimize/redundant_40001.32 ms (1.30 ms … 1.34 ms)
wallclock/dpa/complement/scrambled_40003.36 ms (3.34 ms … 3.39 ms)
wallclock/dpa/normalized/scrambled_40003.08 ms (3.05 ms … 3.11 ms)
wallclock/hoa/parse/ltl776.52 µs (775.95 µs … 777.10 µs)
wallclock/nfa/determinize/nth_from_end_123.01 ms (2.97 ms … 3.05 ms)
wallclock/nfa/subset_construction/nth_from_end_126.45 ms (6.32 ms … 6.59 ms)
wallclock/scc/tarjan/scrambled_100001.93 ms (1.88 ms … 1.98 ms)

Versus Spot and Oink

dpa/equivalence

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot122.58 µs117.70 µs1.04×– / –yes
hard/l-nba-2spot56.20 µs37.44 µs1.50×– / –yes
hard/m-nba-3spot238.82 µs140.78 µs1.70×– / –yes
ltl/and-gf-3spot9.16 µs12.98 µs0.71×– / –yes
ltl/eh-patterns-0spot21.78 µs20.94 µs1.04×– / –yes
ltl/gf-equiv-3spot20.04 µs34.12 µs0.59×– / –yes
ltl/gf-implies-3spot10.68 µs19.90 µs0.54×– / –yes
ltl/gh-q-3spot50.76 µs30.14 µs1.68×– / –yes
ltl/gh-r-3spot20.76 µs68.50 µs0.30×– / –yes
ltl/go-theta-3spot11.20 µs22.00 µs0.51×– / –yes
ltl/lily-patterns-1spot12.34 µs16.58 µs0.74×– / –yes
ltl/ms-phi-r-1spot10.22 µs38.62 µs0.26×– / –yes
ltl/ms-phi-s-1spot10.48 µs41.02 µs0.26×– / –yes
ltl/rv-counter-3spot28.78 µs14.28 µs2.02×– / –yes
ltl/sb-patterns-0spot12.46 µs14.68 µs0.85×– / –yes
ltl/sejk-k-2spot13.48 µs58.24 µs0.23×– / –yes
ltl/tv-uu-3spot41.32 µs39.36 µs1.05×– / –yes
ltl/u-left-3spot10.56 µs9.36 µs1.13×– / –yes
many_aps/and-gf-12spot168.96 µs76.56 µs2.21×– / –yes
many_aps/gf-equiv-12spot131.12 µs91.96 µs1.43×– / –yes
many_aps/r-right-10spot66.88 µs26.88 µs2.49×– / –yes
many_aps/random-n16-a8-p4-s1spot269.88 µs136.08 µs1.98×– / –yes
many_aps/random-n24-a12-p4-s1spot1.18 ms448.70 µs2.62×– / –yes
many_aps/random-n32-a16-p4-s1spot3.86 ms1.23 ms3.13×– / –yes
random/n128-a2-p6-s1spot338.82 µs215.86 µs1.57×– / –yes
random/n16-a1-p3-s1spot22.88 µs16.76 µs1.37×– / –yes
random/n256-a2-p8-s1spot891.45 µs489.03 µs1.82×– / –yes
random/n64-a2-p4-s1spot141.46 µs101.86 µs1.39×– / –yes

dpa/inclusion

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot71.66 µs62.14 µs1.15×– / –yes
hard/l-nba-2spot35.82 µs24.40 µs1.47×– / –yes
hard/m-nba-3spot148.08 µs91.18 µs1.62×– / –yes
ltl/and-gf-3spot5.90 µs8.66 µs0.68×– / –yes
ltl/eh-patterns-0spot14.28 µs17.90 µs0.80×– / –yes
ltl/gf-equiv-3spot11.42 µs19.08 µs0.60×– / –yes
ltl/gf-implies-3spot6.32 µs11.58 µs0.55×– / –yes
ltl/gh-q-3spot33.12 µs29.06 µs1.14×– / –yes
ltl/gh-r-3spot12.32 µs34.86 µs0.35×– / –yes
ltl/go-theta-3spot6.86 µs16.04 µs0.43×– / –yes
ltl/lily-patterns-1spot8.04 µs13.98 µs0.58×– / –yes
ltl/ms-phi-r-1spot6.04 µs19.24 µs0.31×– / –yes
ltl/ms-phi-s-1spot6.12 µs25.32 µs0.24×– / –yes
ltl/rv-counter-3spot18.08 µs14.52 µs1.25×– / –yes
ltl/sb-patterns-0spot7.64 µs12.12 µs0.63×– / –yes
ltl/sejk-k-2spot7.84 µs28.00 µs0.28×– / –yes
ltl/tv-uu-3spot28.46 µs30.08 µs0.95×– / –yes
ltl/u-left-3spot6.72 µs8.68 µs0.77×– / –yes
many_aps/and-gf-12spot125.80 µs54.50 µs2.31×– / –yes
many_aps/gf-equiv-12spot74.80 µs53.08 µs1.41×– / –yes
many_aps/r-right-10spot49.00 µs25.78 µs1.90×– / –yes
many_aps/random-n16-a8-p4-s1spot223.70 µs100.84 µs2.22×– / –yes
many_aps/random-n24-a12-p4-s1spot1.12 ms396.64 µs2.83×– / –yes
many_aps/random-n32-a16-p4-s1spot3.80 ms1.12 ms3.39×– / –yes
random/n128-a2-p6-s1spot198.46 µs128.34 µs1.55×– / –yes
random/n16-a1-p3-s1spot13.46 µs10.80 µs1.25×– / –yes
random/n256-a2-p8-s1spot512.25 µs317.64 µs1.61×– / –yes
random/n64-a2-p4-s1spot87.22 µs65.72 µs1.33×– / –yes

game/solve

inputreferenceoursreferenceours / referencesize (ours / reference)same result
ladder_10000oink-ppp2.75 ms328.00 µs8.39×10000 / 10000yes
ladder_10000oink-zlk2.75 ms335.00 µs8.21×10000 / 10000yes
ladder_10000spot2.75 ms10.17 ms0.27×10000 / 10000yes
ladder_50000oink-ppp18.93 ms2.12 ms8.92×50000 / 50000yes
ladder_50000oink-zlk18.93 ms2.47 ms7.66×50000 / 50000yes
ladder_50000spot18.93 ms77.59 ms0.24×50000 / 50000yes
random_few_10000oink-ppp6.98 ms1.46 ms4.80×10000 / 10000yes
random_few_10000oink-zlk6.98 ms2.76 ms2.53×10000 / 10000yes
random_few_10000spot6.98 ms21.11 ms0.33×10000 / 10000yes
random_few_50000oink-ppp77.03 ms16.00 ms4.81×50000 / 50000yes
random_few_50000oink-zlk77.03 ms20.71 ms3.72×50000 / 50000yes
random_few_50000spot77.03 ms679.94 ms0.11×50000 / 50000yes
random_many_10000oink-ppp6.31 ms1.64 ms3.84×10000 / 10000yes
random_many_10000oink-zlk6.31 ms3.04 ms2.07×10000 / 10000yes
random_many_10000spot6.31 ms29.88 ms0.21×10000 / 10000yes
random_many_50000oink-ppp82.90 ms35.20 ms2.35×50000 / 50000yes
random_many_50000oink-zlk82.90 ms51.28 ms1.62×50000 / 50000yes
random_many_50000spot82.90 ms779.68 ms0.11×50000 / 50000yes

Log

== head c432f0685d5caae9cbf90860f1c1bfc53ac51ae5: building ["scc", "ts", "runs", "dfa", "dpa", "storage", "nondeterministic", "learning", "game"] ["wallclock"]
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
    Finished `bench` profile [optimized] target(s) in 4m 14s
  Executable benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-c46c79a89c1c2941)
  Executable benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-0b05141996aaf468)
  Executable benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-fc166083591c2309)
  Executable benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-72c8d077fb858ea9)
  Executable benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-e95514e1aa52b38c)
  Executable benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-3b37aa113a4e75ae)
  Executable benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-b88e81c5af8ec1fc)
  Executable benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-fffb4b0cac4d81af)
  Executable benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-bb6ee25ff0b06f0f)
  Executable benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-3b599e26071f9e69)
== head: running scc
    Finished `bench` profile [optimized] target(s) in 0.63s
     Running benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-b88e81c5af8ec1fc)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running ts
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-bb6ee25ff0b06f0f)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running runs
    Finished `bench` profile [optimized] target(s) in 0.26s
     Running benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-3b37aa113a4e75ae)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dfa
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-c46c79a89c1c2941)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dpa
    Finished `bench` profile [optimized] target(s) in 0.18s
     Running benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-0b05141996aaf468)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running storage
    Finished `bench` profile [optimized] target(s) in 0.20s
     Running benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-fffb4b0cac4d81af)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running nondeterministic
    Finished `bench` profile [optimized] target(s) in 0.18s
     Running benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-e95514e1aa52b38c)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running learning
    Finished `bench` profile [optimized] target(s) in 0.19s
     Running benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-72c8d077fb858ea9)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running game
    Finished `bench` profile [optimized] target(s) in 0.19s
     Running benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-fc166083591c2309)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running wallclock (wall clock)
    Finished `bench` profile [optimized] target(s) in 0.19s
     Running benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-3b599e26071f9e69)
Gnuplot not found, using plotters backend
Benchmarking wallclock/scc/tarjan/scrambled_10000
Benchmarking wallclock/scc/tarjan/scrambled_10000: Analyzing
Benchmarking wallclock/dpa/normalized/scrambled_4000
Benchmarking wallclock/dpa/normalized/scrambled_4000: Analyzing
Benchmarking wallclock/dpa/complement/scrambled_4000
Benchmarking wallclock/dpa/complement/scrambled_4000: Analyzing
Benchmarking wallclock/dfa/minimize/redundant_4000
Benchmarking wallclock/dfa/minimize/redundant_4000: Analyzing
Benchmarking wallclock/dfa/accepts/word_100000
Benchmarking wallclock/dfa/accepts/word_100000: Analyzing
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12: Analyzing
Benchmarking wallclock/nfa/determinize/nth_from_end_12
Benchmarking wallclock/nfa/determinize/nth_from_end_12: Analyzing
Benchmarking wallclock/hoa/parse/ltl
Benchmarking wallclock/hoa/parse/ltl: Analyzing
Benchmarking collect/clone/100000
Benchmarking collect/clone/100000: Analyzing
Benchmarking collect/collect_owned/100000
Benchmarking collect/collect_owned/100000: Analyzing
Benchmarking collect/collect_ref/100000
Benchmarking collect/collect_ref/100000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/100000
Benchmarking collect/map_edge_colors_collect_owned/100000: Analyzing
Benchmarking collect/clone/1000000
Benchmarking collect/clone/1000000: Analyzing
Benchmarking collect/collect_owned/1000000
Benchmarking collect/collect_owned/1000000: Analyzing
Benchmarking collect/collect_ref/1000000
Benchmarking collect/collect_ref/1000000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/1000000
Benchmarking collect/map_edge_colors_collect_owned/1000000: Analyzing
== head: building the Python bindings
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support
🐍 Found CPython 3.13 at /nix/store/vilynzly1pyhawy04ljmrp7n9ffr24qy-python3-3.13.17/bin/python3
📡 Using build options features from pyproject.toml
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
   Compiling automata-py v0.1.0 (/var/lib/automata-bench/work/src-head/python)
    Finished `release` profile [optimized] target(s) in 7m 33s
📦 Built wheel for abi3 Python ≥ 3.11 to /var/lib/automata-bench/work/versus-head/wheels/automatars-0.1.0-cp311-abi3-manylinux_2_34_aarch64.whl
== head: running benches/versus/compare.py (Spot, Oink)
dpa/equivalence/hard/l-dsa-3 vs spot: 0.123 ms / 0.118 ms
dpa/equivalence/hard/l-nba-2 vs spot: 0.056 ms / 0.037 ms
dpa/equivalence/hard/m-nba-3 vs spot: 0.239 ms / 0.141 ms
dpa/equivalence/ltl/and-gf-3 vs spot: 0.009 ms / 0.013 ms
dpa/equivalence/ltl/eh-patterns-0 vs spot: 0.022 ms / 0.021 ms
dpa/equivalence/ltl/gf-equiv-3 vs spot: 0.020 ms / 0.034 ms
dpa/equivalence/ltl/gf-implies-3 vs spot: 0.011 ms / 0.020 ms
dpa/equivalence/ltl/gh-q-3 vs spot: 0.051 ms / 0.030 ms
dpa/equivalence/ltl/gh-r-3 vs spot: 0.021 ms / 0.069 ms
dpa/equivalence/ltl/go-theta-3 vs spot: 0.011 ms / 0.022 ms
dpa/equivalence/ltl/lily-patterns-1 vs spot: 0.012 ms / 0.017 ms
dpa/equivalence/ltl/ms-phi-r-1 vs spot: 0.010 ms / 0.039 ms
dpa/equivalence/ltl/ms-phi-s-1 vs spot: 0.010 ms / 0.041 ms
dpa/equivalence/ltl/rv-counter-3 vs spot: 0.029 ms / 0.014 ms
dpa/equivalence/ltl/sb-patterns-0 vs spot: 0.012 ms / 0.015 ms
dpa/equivalence/ltl/sejk-k-2 vs spot: 0.013 ms / 0.058 ms
dpa/equivalence/ltl/tv-uu-3 vs spot: 0.041 ms / 0.039 ms
dpa/equivalence/ltl/u-left-3 vs spot: 0.011 ms / 0.009 ms
dpa/equivalence/many_aps/and-gf-12 vs spot: 0.169 ms / 0.077 ms
dpa/equivalence/many_aps/gf-equiv-12 vs spot: 0.131 ms / 0.092 ms
dpa/equivalence/many_aps/r-right-10 vs spot: 0.067 ms / 0.027 ms
dpa/equivalence/many_aps/random-n16-a8-p4-s1 vs spot: 0.270 ms / 0.136 ms
dpa/equivalence/many_aps/random-n24-a12-p4-s1 vs spot: 1.176 ms / 0.449 ms
dpa/equivalence/many_aps/random-n32-a16-p4-s1 vs spot: 3.860 ms / 1.232 ms
dpa/equivalence/random/n128-a2-p6-s1 vs spot: 0.339 ms / 0.216 ms
dpa/equivalence/random/n16-a1-p3-s1 vs spot: 0.023 ms / 0.017 ms
dpa/equivalence/random/n256-a2-p8-s1 vs spot: 0.891 ms / 0.489 ms
dpa/equivalence/random/n64-a2-p4-s1 vs spot: 0.141 ms / 0.102 ms
dpa/inclusion/hard/l-dsa-3 vs spot: 0.072 ms / 0.062 ms
dpa/inclusion/hard/l-nba-2 vs spot: 0.036 ms / 0.024 ms
dpa/inclusion/hard/m-nba-3 vs spot: 0.148 ms / 0.091 ms
dpa/inclusion/ltl/and-gf-3 vs spot: 0.006 ms / 0.009 ms
dpa/inclusion/ltl/eh-patterns-0 vs spot: 0.014 ms / 0.018 ms
dpa/inclusion/ltl/gf-equiv-3 vs spot: 0.011 ms / 0.019 ms
dpa/inclusion/ltl/gf-implies-3 vs spot: 0.006 ms / 0.012 ms
dpa/inclusion/ltl/gh-q-3 vs spot: 0.033 ms / 0.029 ms
dpa/inclusion/ltl/gh-r-3 vs spot: 0.012 ms / 0.035 ms
dpa/inclusion/ltl/go-theta-3 vs spot: 0.007 ms / 0.016 ms
dpa/inclusion/ltl/lily-patterns-1 vs spot: 0.008 ms / 0.014 ms
dpa/inclusion/ltl/ms-phi-r-1 vs spot: 0.006 ms / 0.019 ms
dpa/inclusion/ltl/ms-phi-s-1 vs spot: 0.006 ms / 0.025 ms
dpa/inclusion/ltl/rv-counter-3 vs spot: 0.018 ms / 0.015 ms
dpa/inclusion/ltl/sb-patterns-0 vs spot: 0.008 ms / 0.012 ms
dpa/inclusion/ltl/sejk-k-2 vs spot: 0.008 ms / 0.028 ms
dpa/inclusion/ltl/tv-uu-3 vs spot: 0.028 ms / 0.030 ms
dpa/inclusion/ltl/u-left-3 vs spot: 0.007 ms / 0.009 ms
dpa/inclusion/many_aps/and-gf-12 vs spot: 0.126 ms / 0.055 ms
dpa/inclusion/many_aps/gf-equiv-12 vs spot: 0.075 ms / 0.053 ms
dpa/inclusion/many_aps/r-right-10 vs spot: 0.049 ms / 0.026 ms
dpa/inclusion/many_aps/random-n16-a8-p4-s1 vs spot: 0.224 ms / 0.101 ms
dpa/inclusion/many_aps/random-n24-a12-p4-s1 vs spot: 1.121 ms / 0.397 ms
dpa/inclusion/many_aps/random-n32-a16-p4-s1 vs spot: 3.803 ms / 1.123 ms
dpa/inclusion/random/n128-a2-p6-s1 vs spot: 0.198 ms / 0.128 ms
dpa/inclusion/random/n16-a1-p3-s1 vs spot: 0.013 ms / 0.011 ms
dpa/inclusion/random/n256-a2-p8-s1 vs spot: 0.512 ms / 0.318 ms
dpa/inclusion/random/n64-a2-p4-s1 vs spot: 0.087 ms / 0.066 ms
game/solve/random_few_10000 vs spot: 6.982 ms / 21.110 ms
game/solve/random_few_10000 vs oink-zlk: 6.982 ms / 2.756 ms
game/solve/random_few_10000 vs oink-ppp: 6.982 ms / 1.456 ms
game/solve/random_few_50000 vs spot: 77.026 ms / 679.939 ms
game/solve/random_few_50000 vs oink-zlk: 77.026 ms / 20.713 ms
game/solve/random_few_50000 vs oink-ppp: 77.026 ms / 16.001 ms
game/solve/random_many_10000 vs spot: 6.311 ms / 29.880 ms
game/solve/random_many_10000 vs oink-zlk: 6.311 ms / 3.042 ms
game/solve/random_many_10000 vs oink-ppp: 6.311 ms / 1.643 ms
game/solve/random_many_50000 vs spot: 82.897 ms / 779.680 ms
game/solve/random_many_50000 vs oink-zlk: 82.897 ms / 51.280 ms
game/solve/random_many_50000 vs oink-ppp: 82.897 ms / 35.202 ms
game/solve/ladder_10000 vs spot: 2.751 ms / 10.168 ms
game/solve/ladder_10000 vs oink-zlk: 2.751 ms / 0.335 ms
game/solve/ladder_10000 vs oink-ppp: 2.751 ms / 0.328 ms
game/solve/ladder_50000 vs spot: 18.927 ms / 77.590 ms
game/solve/ladder_50000 vs oink-zlk: 18.927 ms / 2.472 ms
game/solve/ladder_50000 vs oink-ppp: 18.927 ms / 2.122 ms
== head: 266 benchmarks, 16 wall-clock measurements, 74 comparisons