automata benchmarks

History · Versus Spot and Oink

main · 91ab2741

CsrTs: real compressed sparse rows, nondeterministic, any alphabet (#166)

status done · queued 2026-11-11 18:41 · started 2026-11-11 18:41 · finished 2026-11-11 18:59

· · rustc 1.95.0 (59807616e 2026-04-14) (built from a source tarball) · valgrind-3.26.0 · pinned to cpu 1

Results

benchmarkinstructionsest. cyclesL1 hitsLL hitsRAM hitsheap bytesallocations
dfa::dfa::accepts_many_words::scrambled_1000319 438399 881393 3861 2221100
dfa::dfa::complement::scrambled_1000318 874512 713495 0333 37523139 10419
dfa::dfa::complement::scrambled_40001 247 6492 024 5981 938 85316 99522556 89623
dfa::dfa::equivalent::deep_difference_10052 83182 61975 4994519716 19625
dfa::dfa::equivalent::deep_difference_400177 852277 913255 5681 85137465 36433
dfa::dfa::equivalent::different_1008 34914 14811 4887751 80412
dfa::dfa::equivalent::different_4008 39014 35311 44378721 80412
dfa::dfa::equivalent::minimized_10031 31246 50844 273104497 75617
dfa::dfa::equivalent::minimized_400103 504153 074147 6597265131 58023
dfa::dfa::give_word::empty_10001 795 2342 450 0352 422 4804 91685152 8643 738
dfa::dfa::give_word::empty_40006 603 9839 194 1339 091 42820 06568679 80815 828
dfa::dfa::give_word::far_1000157 743223 352218 8275485128 176394
dfa::dfa::give_word::far_4000909 2251 280 3471 258 3373 93367148 4642 233
dfa::dfa::intersection::scrambled_1002 716 0483 995 2953 817 11512 0113 375428 08842
dfa::dfa::intersection::scrambled_20010 809 51116 277 75415 140 77496 86718 6471 708 12049
dfa::dfa::intersection_is_empty::empty_10050 65979 03372 5333318115 70020
dfa::dfa::intersection_is_empty::empty_20092 418145 704132 52961928831 58023
dfa::dfa::is_empty_language::empty_10001 795 2342 450 0692 422 4794 91686152 8643 738
dfa::dfa::is_empty_language::empty_40006 603 9839 194 1679 091 42720 06569679 80815 828
dfa::dfa::minimize::minimal_10003 298 9794 979 3904 604 64039 7335 0311 135 4922 214
dfa::dfa::minimize::minimal_400012 876 08919 609 36617 952 606204 56818 1124 543 7248 254
dfa::dfa::minimize::redundant_10002 174 5513 263 2243 028 27925 8843 015760 7801 201
dfa::dfa::minimize::redundant_40008 406 75412 753 39211 696 777143 6129 6733 044 9004 241
dfa::dfa::minimize::scrambled_10003 502 2565 309 2954 877 65551 0345 0421 116 0522 217
dfa::dfa::minimize::scrambled_400013 587 25820 978 31618 872 886294 02218 1524 470 1408 261
dfa::dfa::minimize_moore_with_edge_colors::scrambled_10003 500 3175 293 0704 866 64550 0895 0281 116 0522 217
dfa::dfa::minimize_moore_with_edge_colors::scrambled_400013 571 89120 909 24618 808 236300 57917 0894 470 1408 261
dfa::dfa::union::scrambled_1002 716 0483 995 2953 817 11512 0113 375428 08842
dfa::dfa::union::scrambled_20010 809 51116 277 75415 140 77496 86718 6471 708 12049
dpa::dpa::accepted_word::chain_100114 774174 706162 98149328––
dpa::dpa::accepted_word::chain_400391 265589 285555 0652 966554––
dpa::dpa::accepted_word::scrambled_100145 660218 267206 02740344––
dpa::dpa::accepted_word::scrambled_400499 945740 793707 7732 817541––
dpa::dpa::accepted_word_fixtures::ltl495 442716 693690 4182 301422––
dpa::dpa::accepted_word_fixtures::synthetic1 255 4201 812 0101 739 54511 994357––
dpa::dpa::accepted_word_fixtures_by_atoms::ltl1 620 5282 381 5392 324 0347 686545––
dpa::dpa::accepted_word_fixtures_by_atoms::synthetic2 741 0794 051 4473 966 99213 118539––
dpa::dpa::collect_dpa::scrambled_1000258 437421 129402 0243 8003––
dpa::dpa::collect_dpa::scrambled_40001 004 7821 689 4741 565 92916 3931 188––
dpa::dpa::complement::chain_10003 806 7135 721 3765 221 07155 1636 414––
dpa::dpa::complement::chain_400015 498 94023 452 20421 213 694262 01326 527––
dpa::dpa::complement::scrambled_10004 640 2806 874 1466 331 34662 6546 558––
dpa::dpa::complement::scrambled_400018 790 50128 240 81825 579 138344 18326 879––
dpa::dpa::complement_fixtures::ltl1 318 7631 878 7601 840 2454 917398––
dpa::dpa::complement_fixtures::many_aps12 439 58217 914 57317 431 46886 5761 435––
dpa::dpa::complement_fixtures::synthetic9 051 76712 766 90812 268 22896 768424––
dpa::dpa::complement_without_normalizing::scrambled_1000542 899786 566757 8765 15783––
dpa::dpa::complement_without_normalizing::scrambled_40002 135 2353 133 7692 979 55421 9671 268––
dpa::dpa::equivalent_fixtures::many_aps18 698 03427 049 36826 383 813120 4131 814––
dpa::dpa::hoa_parse::ltl5 102 0067 691 0577 405 67235 1743 129––
dpa::dpa::hoa_parse::many_aps80 066 844132 197 944115 010 044289 526449 722––
dpa::dpa::hoa_parse::synthetic23 303 31935 261 72033 885 920175 15114 287––
dpa::dpa::hoa_print::ltl14 450 35231 507 49819 336 5684 619347 081––
dpa::dpa::hoa_print::many_aps17 708 49225 446 10125 324 14121 998342––
dpa::dpa::hoa_print::synthetic21 972 18342 573 14130 248 79622 149348 960––
dpa::dpa::included_fixtures::many_aps15 907 78323 120 48722 585 43794 7041 758––
dpa::dpa::included_in::complement_100260 739388 057371 327847357––
dpa::dpa::included_in::complement_2584 611131 538119 14891341––
dpa::dpa::included_in::complement_50146 080219 891206 691316332––
dpa::dpa::included_in::normalized_100524 060764 122738 4121 425531––
dpa::dpa::included_in::normalized_25177 925261 499249 499181317––
dpa::dpa::included_in::normalized_50302 723440 103425 463429357––
dpa::dpa::language_equivalent::complement_100268 661397 785380 485863371––
dpa::dpa::language_equivalent::complement_2586 600134 357121 47791355––
dpa::dpa::language_equivalent::complement_50149 885224 791211 036329346––
dpa::dpa::language_equivalent::normalized_100963 9591 387 4291 353 4242 237652––
dpa::dpa::language_equivalent::normalized_25315 935456 811442 156264381––
dpa::dpa::language_equivalent::normalized_50564 774808 312791 022553415––
dpa::dpa::map_edge_colors_collect::scrambled_1000263 002419 699399 6593 87519––
dpa::dpa::map_edge_colors_collect::scrambled_40001 021 3471 681 8541 553 85917 1711 204––
dpa::dpa::normalized::chain_10003 513 1885 338 9404 851 58052 9246 364––
dpa::dpa::normalized::chain_400014 326 41521 930 53619 737 761253 21626 477––
dpa::dpa::normalized::scrambled_10004 346 5726 491 3835 961 57360 4066 508––
dpa::dpa::normalized::scrambled_400017 617 79326 718 74724 102 942335 35826 829––
dpa::dpa::normalized::scrambled_k4_100025 249 99935 408 76233 257 547379 8297 202––
dpa::dpa::normalized_fixtures::ltl1 272 9301 821 5511 786 2964 692337––
dpa::dpa::normalized_fixtures::many_aps12 275 18317 722 04417 254 45483 8231 385––
dpa::dpa::normalized_fixtures::synthetic8 843 73312 508 69512 039 79091 093384––
dpa::dpa::prefix_partition::chain_163 574 1785 457 3985 390 1782 2861 594––
dpa::dpa::prefix_partition::chain_8916 1071 409 3611 379 801508772––
dpa::dpa::prefix_partition::scrambled_163 999 6986 155 5426 070 4223 7661 894––
dpa::dpa::prefix_partition::scrambled_81 086 5101 669 8811 639 181666782––
dpa::dpa::rejected_word::chain_100885 2311 281 9871 241 3972 833755––
dpa::dpa::rejected_word::chain_4003 441 5735 130 4584 832 06847 9811 671––
dpa::dpa::rejected_word::scrambled_100235 103360 734329 644506816––
dpa::dpa::rejected_word::scrambled_400887 0761 355 2841 244 35910 2361 707––
dpa::dpa::sccs::chain_1000511 023774 862699 6576 8301 173––
dpa::dpa::sccs::chain_40001 971 6303 022 6552 696 94029 8915 036––
dpa::dpa::sccs::scrambled_1000576 822871 308795 3438 221996––
dpa::dpa::sccs::scrambled_40002 295 9453 543 7473 164 88746 2744 214––
dpa::dpa::witness_colors::chain_1004 471 3416 365 9886 245 9039 0092 144––
dpa::dpa::witness_colors::chain_40043 697 42462 813 10461 212 849194 56917 926––
dpa::dpa::witness_colors::scrambled_1003 895 1885 531 9555 442 5458 7611 303––
dpa::dpa::witness_colors::scrambled_40029 949 98942 974 33842 154 343145 7362 609––
dpa::dpa::witness_inequivalence::complement_100268 661397 823380 483864372––
dpa::dpa::witness_inequivalence::complement_2586 600134 395121 47592356––
dpa::dpa::witness_inequivalence::complement_50149 885224 821211 036328347––
dpa::dpa::witness_not_subset_of::complement_100260 739388 101371 331841359––
dpa::dpa::witness_not_subset_of::complement_2584 611131 590119 15087343––
dpa::dpa::witness_not_subset_of::complement_50146 080219 955206 690315334––
dpa::dpa::witness_not_subset_of::normalized_100524 060764 174738 4141 421533––
dpa::dpa::witness_not_subset_of::normalized_25177 925261 563249 498180319––
dpa::dpa::witness_not_subset_of::normalized_50302 723440 143425 468422359––
dpa::dpa::witnesses_fixtures::ltl1 970 3492 739 9072 683 4075 903771––
dpa::dpa::witnesses_fixtures::synthetic15 821 11722 786 08322 257 723100 205781––
game::game::tangle_learning::few_priorities_10002 598 0813 917 5333 570 78335 9674 769745 4001 292
game::game::tangle_learning::few_priorities_40007 882 20712 361 12310 759 633194 36117 9912 407 4163 266
game::game::tangle_learning::ladder_10001 169 4601 811 0711 611 13112 2753 959377 868118
game::game::tangle_learning::ladder_40004 381 8167 035 9965 976 831110 24914 5121 511 668225
game::game::tangle_learning::many_priorities_10001 963 2522 956 1722 676 22725 5184 353566 440848
game::game::tangle_learning::many_priorities_40006 594 92110 279 4938 905 703165 65615 5862 187 3441 620
game::game::tangle_learning_min_odd::few_priorities_10001 500 2132 281 4772 044 14726 2703 028525 072189
game::game::tangle_learning_min_odd::few_priorities_40007 799 99212 020 31810 520 538215 73212 0322 560 0641 933
game::game::zielonka::few_priorities_10001 107 8761 656 0561 469 42112 4213 558356 67695
game::game::zielonka::few_priorities_40003 498 8975 565 1974 670 22281 12113 9821 278 56484
game::game::zielonka::ladder_10004 708 2536 180 8325 779 89749 8214 338549 716211
game::game::zielonka::ladder_400018 161 86723 979 39022 178 395245 35016 4072 200 404247
game::game::zielonka::many_priorities_10003 821 9635 020 0384 685 58338 3734 074841 196367
game::game::zielonka::many_priorities_400012 101 96916 234 07514 860 860175 38314 1802 808 868310
game::game::zielonka_min_odd::few_priorities_10002 040 9082 803 9652 583 83023 5522 925512 012195
game::game::zielonka_min_odd::few_priorities_40006 089 9658 778 7117 810 441130 9278 9611 705 700158
learning::learning::lstar_mealy::counting_161 753 1652 378 4362 341 356437997154 1842 615
learning::learning::lstar_mealy::counting_8446 458624 608598 86813671640 280852
learning::learning::sprout_parity::inf_ab_1001 146 4771 639 5581 582 4031 9741 351200 3201 868
learning::learning::sprout_parity::inf_ab_4002 353 1283 351 3643 241 5198 8581 873404 9443 645
nondeterministic::nondeterministic::fixture_accepts::ltl731 4871 132 4881 084 6185 934520––
nondeterministic::nondeterministic::nba_accepts::nba_400122 996 323179 087 545168 150 6851 663 54174 833––
nondeterministic::nondeterministic::nba_witness::nba_400017 383 33326 028 58225 014 717138 5209 179––
nondeterministic::nondeterministic::nca_accepts::nca_400254 927 534371 570 174345 867 3894 569 35781 600––
nondeterministic::nondeterministic::nca_witness::nca_400021 904 94133 241 97731 087 897325 66215 022––
nondeterministic::nondeterministic::nma_witness::nma_100025 383 66637 526 70735 512 952370 0054 678––
nondeterministic::nondeterministic::npa_accepts::empty_npa_400458 821 236670 096 369622 425 0298 911 40888 980––
nondeterministic::nondeterministic::npa_accepts::npa_10029 740 29842 889 01541 005 525287 74212 708––
nondeterministic::nondeterministic::npa_accepts::npa_400122 955 922179 060 750168 131 1201 662 01874 844––
nondeterministic::nondeterministic::npa_witness::empty_npa_10009 427 73713 978 64113 247 746123 1773 286––
nondeterministic::nondeterministic::npa_witness::empty_npa_400038 265 19257 449 42653 485 951657 34319 336––
nondeterministic::nondeterministic::npa_witness::npa_10002 979 0724 478 0894 313 26420 7781 741––
nondeterministic::nondeterministic::npa_witness::npa_400023 816 12935 551 28134 340 106178 0109 175––
nondeterministic::nondeterministic::nra_witness::nra_400021 455 58332 709 83330 163 393337 21424 582––
runs::runs::dfa_accepts_many_words::count_10004 053 1825 005 9024 981 6024 7901000
runs::runs::dfa_accepts_one_word::len_1003 5614 8524 45238600
runs::runs::dfa_accepts_one_word::len_10000276 039339 481335 781698600
runs::runs::dfa_accepts_one_word::len_1000002 750 7003 376 7793 344 6646 3251400
runs::runs::dfa_reached_state::count_10003 442 1804 272 8314 248 6014 790800
runs::runs::dpa_accepts_lassos::long8 225 40510 785 64410 761 1093 122255724 664500
runs::runs::dpa_accepts_lassos::short376 801502 346497 0864019326 236472
runs::runs::dpa_recurrent_edge_colors::long15 221 63619 508 09619 456 3765 948628922 1442 090
runs::runs::dpa_recurrent_edge_colors::short1 058 6241 435 4481 424 3031 242141168 6841 708
runs::runs::dpa_recurrent_states::long13 767 74019 256 81019 225 7453 784347788 104860
runs::runs::dpa_recurrent_states::short596 810814 523807 32355112781 996782
runs::runs::dpa_visited_sequences::long8 707 28312 043 61912 009 3094 5453312 190 780850
runs::runs::dpa_visited_sequences::short350 418483 834479 1845036129 986545
runs::runs::fixture_accepts_lassos::ltl7 833 94711 053 58211 022 4075 54998––
runs::runs::fixture_accepts_lassos::synthetic4 309 0576 188 5146 136 8599 624101––
scc::scc::scc_queries::small_cycles_1000855 9461 334 6741 174 0249 4363 242261 132559
scc::scc::scc_queries::small_cycles_40003 348 7615 268 9204 585 40050 89112 2591 045 3962 071
scc::scc::tarjan::back_edges_1000493 252795 135684 5705 4672 378170 27245
scc::scc::tarjan::back_edges_40001 914 6603 101 3902 656 01026 4478 947680 63253
scc::scc::tarjan::binary_tree_1000551 319848 324771 9344 0781 600105 76039
scc::scc::tarjan::binary_tree_40002 212 9413 410 0143 104 54421 1385 708419 51245
scc::scc::tarjan::fragmented_1000626 713981 364858 1849 6562 140146 084121
scc::scc::tarjan::fragmented_40002 413 5153 877 5493 297 11462 8457 606569 852167
scc::scc::tarjan::grid_1024575 645926 563802 9037 7432 427171 16045
scc::scc::tarjan::grid_40962 299 2053 824 7593 187 82963 6729 102684 18453
scc::scc::tarjan::path_10001 735 1372 678 4602 380 25523 2065 205455 2242 064
scc::scc::tarjan::path_40006 859 74810 607 2659 411 22098 32020 1271 821 5368 076
scc::scc::tarjan::ring_1000493 252798 771683 6616 3762 378170 27245
scc::scc::tarjan::ring_40001 914 6603 118 6382 651 69830 7598 947680 63253
scc::scc::tarjan::scrambled_1000559 051892 093780 1286 9862 201170 27245
scc::scc::tarjan::scrambled_40002 238 9753 618 6543 124 91441 8738 125680 63253
scc::scc::tarjan::small_cycles_1000854 9281 333 2991 172 7399 4463 238261 132559
scc::scc::tarjan::small_cycles_40003 344 7435 261 1294 581 02450 24312 2541 045 3962 071
scc::scc::tarjan_from::fragmented_1000232 901373 990324 8802 5491 03974 98564
scc::scc::tarjan_from::fragmented_4000872 5191 416 6221 221 48215 9563 296299 62578
scc::scc::tarjan_from::path_10001 724 7402 664 4852 369 14023 0615 144468 9932 091
scc::scc::tarjan_from::path_40006 763 89010 521 5729 291 232106 76119 9011 876 7538 109
scc::scc::tarjan_from::ring_1000482 362790 400672 0606 2662 486184 04172
scc::scc::tarjan_from::scrambled_1000548 161882 310768 8806 5232 309184 04172
storage::storage::add_edges_dense::k2_50001 390 8122 084 7262 039 5364 936586180 20037
storage::storage::add_edges_dense::k8_50004 311 4276 496 0496 309 52417 8312 782671 72840
storage::storage::add_edges_linked::k2_50001 252 4512 165 0622 003 04214 2182 5981 113 96026
storage::storage::add_edges_linked::k8_50006 144 06310 329 6249 400 50975 75515 7243 866 49628
storage::storage::clone_csr::k2_1000022 53776 08542 5756 61812210 0124
storage::storage::clone_dense::k2_1000015 04281 07225 7322 2481 260110 0084
storage::storage::clone_linked::k2_10000380 760911 493539 73312 4658 841680 0083
storage::storage::complete_dense::partial_5000230 702348 722343 45788524561
storage::storage::complete_linked::partial_50001 183 1291 690 4331 602 9686 9441 50765 59215
storage::storage::edges_from_all_states_csr::k2_200034 03838 32838 04329400
storage::storage::edges_from_all_states_csr::k8_200034 03838 62837 968104400
storage::storage::edges_from_all_states_dense::k2_200092 047101 25799 847233700
storage::storage::edges_from_all_states_dense::k8_2000128 047140 511135 0411 038800
storage::storage::edges_from_all_states_linked::k2_200070 03388 75577 9252 131500
storage::storage::edges_from_all_states_linked::k8_2000154 033205 731168 6817 375500
storage::storage::minimize_csr::redundant_20004 240 6456 376 5295 902 60960 6034 8831 505 4202 212
storage::storage::minimize_csr::redundant_800016 584 07325 284 39723 034 757275 48824 9206 023 3008 250
storage::storage::minimize_dense::redundant_20004 434 8236 603 9816 149 50658 0444 6931 522 1562 221
storage::storage::minimize_dense::redundant_800017 408 18526 418 89924 122 739278 89125 7636 090 3408 261
storage::storage::minimize_linked::redundant_20004 305 9576 557 5316 009 16161 8506 8321 522 1562 221
storage::storage::minimize_linked::redundant_800016 930 01726 057 75223 610 767302 70026 6716 090 3408 261
storage::storage::overwrite_edges_dense::k2_50002 170 8433 296 7813 236 6217 909589180 20037
storage::storage::overwrite_edges_linked::k2_50003 410 5536 005 6495 286 67944 75114 1492 031 46427
storage::storage::predecessors_of_all_states_csr::k2_2000251 025342 244316 5394 9592648 0083
storage::storage::predecessors_of_all_states_dense::k2_2000232 309311 709302 1591 6164232 0083
storage::storage::predecessors_of_all_states_linked::k2_200070 03393 65776 6923 365400
storage::storage::product_successor_csr::n20048 08258 50758 10738600
storage::storage::product_successor_dense::n20065 08876 56776 12240700
storage::storage::product_successor_linked::n20046 54060 13056 995585600
storage::storage::reachable_states_csr::k2_8000988 7091 471 2871 410 82711 8962817 36811
storage::storage::reachable_states_dense::k2_80001 173 5071 692 0121 646 9628 8352517 36811
storage::storage::reachable_states_linked::k2_8000980 0941 509 0631 410 25819 5163517 36811
storage::storage::remove_state_dense::k2_10000924 1621 347 8261 173 24621 5111 915176 0001 003
storage::storage::remove_state_linked::k2_10000997 0191 358 2131 319 7637 4173980 0003 000
storage::storage::run_many_words_csr::count_1000504 041624 024614 1441 934600
storage::storage::run_many_words_dense::count_1000512 042601 929592 1641 911600
storage::storage::run_many_words_linked::count_1000481 040611 107601 1121 957600
storage::storage::run_one_word_csr::len_1000002 450 0423 025 3952 993 7956 285500
storage::storage::run_one_word_dense::len_1000002 600 0423 025 3452 993 8006 274500
storage::storage::run_one_word_linked::len_1000002 350 0392 975 4282 943 7736 296500
storage::storage::successor_all_pairs_csr::k2_200096 205117 081113 641646681
storage::storage::successor_all_pairs_csr::k8_2000510 259627 149616 1992 1486321
storage::storage::successor_all_pairs_dense::k2_2000150 205169 478168 048237781
storage::storage::successor_all_pairs_dense::k8_2000778 259880 782875 2971 0487321
storage::storage::successor_all_pairs_linked::k2_2000110 202141 070130 1352 138781
storage::storage::successor_all_pairs_linked::k8_2000752 256948 110910 9507 3837321
ts::ts::add_edges_in_order::k2_50001 062 4461 774 9721 613 03714 2152 5961 113 96026
ts::ts::add_edges_in_order::k8_50005 384 0588 769 4627 840 52275 73415 7223 866 49628
ts::ts::add_edges_in_reverse::k2_50001 062 4461 774 1801 613 23514 0172 5961 113 96026
ts::ts::builder_with_edges::k2_500079 938 18295 907 04793 436 287450 5776 2251 739 53245
ts::ts::canonical_naming::scrambled_15003 034 5224 445 1694 255 45421 8712 296463 8085 846
ts::ts::canonical_naming::scrambled_500938 2661 350 7581 307 5833 175780134 3361 831
ts::ts::clone::scrambled_10000381 079912 268540 13812 4838 849680 0083
ts::ts::clone::scrambled_400001 521 3163 857 6812 156 70642 57642 5172 720 0083
ts::ts::collect_dts::scrambled_100002 934 9255 185 5844 626 49937 16910 6642 228 06427
ts::ts::collect_dts::scrambled_4000011 679 37220 762 63218 409 072135 30047 9168 912 73631
ts::ts::determinize::nth_from_end_104 305 6806 339 2425 978 94722 0937 138651 7206 013
ts::ts::determinize::nth_from_end_1219 177 58828 064 70026 556 09598 79128 9902 732 60026 423
ts::ts::determinize::nth_from_end_8980 4401 446 5991 364 2292 9921 926156 5521 396
ts::ts::determinize_propositional::aps_6_n_64 088 2486 427 4595 941 67920 03011 0181 332 8964 208
ts::ts::determinize_propositional::aps_7_n_64 295 5876 716 4106 179 15525 32711 7321 197 9301 967
ts::ts::determinize_propositional::aps_8_n_625 052 41338 429 95535 513 095168 37059 2865 240 45613 678
ts::ts::edges_from_all_states::scrambled_200074 64693 97882 7382 1571300
ts::ts::edges_from_all_states::scrambled_k8_2000158 578210 859173 3947 4021300
ts::ts::erase_state_colors_collect::scrambled_100001 846 8553 177 5972 618 44737 17510 6652 228 06427
ts::ts::into_dfa::scrambled_100003 054 9835 375 9204 816 59037 18310 6692 228 06427
ts::ts::map_edge_colors_collect::scrambled_100001 928 4683 407 5042 789 15938 45112 1742 490 19227
ts::ts::map_edge_colors_collect::scrambled_400007 655 04713 651 45611 061 741140 33553 9449 961 29631
ts::ts::matching_product_successor_index::n20047 60262 59858 4235973400
ts::ts::minimal_representative_of_all_states::scrambled_1002 002 0682 856 1222 852 7671394347 5044 679
ts::ts::minimal_representative_of_all_states::scrambled_30016 088 40423 069 25423 054 0892 354973 116 78438 196
ts::ts::minimal_representatives::scrambled_20003 248 7964 585 5324 538 7278 92063322 9447 709
ts::ts::minimal_representatives::scrambled_800014 321 19120 180 45319 965 68842 575541 464 89632 693
ts::ts::predecessors_of_all_states::scrambled_200074 64698 85081 5053 3921100
ts::ts::predecessors_of_all_states::scrambled_8000296 578397 297321 77715 0201200
ts::ts::reachable_states::scrambled_2000243 901374 295353 7303 609724 3369
ts::ts::reachable_states::scrambled_8000964 4801 493 8621 394 77719 5234217 36811
ts::ts::remove_edges_from_matching::scrambled_10000446 232589 810565 6554 6562564 0002 000
ts::ts::remove_edges_from_matching::scrambled_40000746 232941 310890 28010 0312564 0002 000
ts::ts::remove_state::scrambled_10000997 0191 358 3191 319 7597 4184280 0003 000
ts::ts::remove_state::scrambled_400001 357 0191 772 7831 703 64313 5344280 0003 000
ts::ts::scc_minimal_representatives::small_cycles_100210 321304 069289 5694440844 272309
ts::ts::scc_minimal_representatives::small_cycles_4001 192 0861 679 3711 629 9313 889857354 1481 325
ts::ts::subset_construction::nth_from_end_109 439 13313 173 43512 681 49545 2797 5871 073 5209 919
ts::ts::subset_construction::nth_from_end_1242 061 90058 466 83756 225 962218 20432 8534 518 97643 857
ts::ts::subset_construction::nth_from_end_82 040 0112 862 5042 756 7546 0512 157257 1042 276
ts::ts::successor_index_all_pairs::scrambled_2000110 814142 232130 9522 1581481
ts::ts::successor_index_all_pairs::scrambled_k8_2000752 800949 211911 6667 40415321

Wall-clock times

Criterion means of a single run. They depend on the machine and are not compared; use the instruction counts above for that.

benchmarkhead
collect/clone/1000004.41 ms (4.34 ms … 4.48 ms)
collect/clone/100000039.45 ms (39.07 ms … 39.83 ms)
collect/collect_owned/1000008.25 ms (8.18 ms … 8.33 ms)
collect/collect_owned/100000078.18 ms (77.10 ms … 79.25 ms)
collect/collect_ref/1000004.25 ms (4.16 ms … 4.35 ms)
collect/collect_ref/100000075.26 ms (74.42 ms … 76.11 ms)
collect/map_edge_colors_collect_owned/1000006.46 ms (6.32 ms … 6.61 ms)
collect/map_edge_colors_collect_owned/100000072.09 ms (71.47 ms … 72.71 ms)
wallclock/dfa/accepts/word_100000925.17 µs (900.65 µs … 949.70 µs)
wallclock/dfa/minimize/redundant_40001.36 ms (1.36 ms … 1.36 ms)
wallclock/dpa/complement/scrambled_40003.39 ms (3.35 ms … 3.44 ms)
wallclock/dpa/normalized/scrambled_40003.09 ms (3.08 ms … 3.09 ms)
wallclock/hoa/parse/ltl885.70 µs (868.96 µs … 902.45 µs)
wallclock/nfa/determinize/nth_from_end_123.42 ms (3.40 ms … 3.44 ms)
wallclock/nfa/subset_construction/nth_from_end_128.49 ms (8.38 ms … 8.61 ms)
wallclock/scc/tarjan/scrambled_100002.20 ms (2.17 ms … 2.22 ms)

Versus Spot and Oink

dpa/equivalence

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot125.00 µs119.82 µs1.04×– / –yes
hard/l-nba-2spot55.80 µs36.54 µs1.53×– / –yes
hard/m-nba-3spot236.72 µs135.28 µs1.75×– / –yes
ltl/and-gf-3spot9.12 µs12.36 µs0.74×– / –yes
ltl/eh-patterns-0spot21.84 µs20.08 µs1.09×– / –yes
ltl/gf-equiv-3spot20.42 µs34.22 µs0.60×– / –yes
ltl/gf-implies-3spot10.68 µs19.56 µs0.55×– / –yes
ltl/gh-q-3spot50.84 µs28.70 µs1.77×– / –yes
ltl/gh-r-3spot21.20 µs69.42 µs0.31×– / –yes
ltl/go-theta-3spot11.20 µs21.44 µs0.52×– / –yes
ltl/lily-patterns-1spot12.10 µs16.12 µs0.75×– / –yes
ltl/ms-phi-r-1spot10.48 µs39.02 µs0.27×– / –yes
ltl/ms-phi-s-1spot10.40 µs40.66 µs0.26×– / –yes
ltl/rv-counter-3spot27.80 µs13.04 µs2.13×– / –yes
ltl/sb-patterns-0spot12.70 µs14.12 µs0.90×– / –yes
ltl/sejk-k-2spot13.60 µs59.24 µs0.23×– / –yes
ltl/tv-uu-3spot41.30 µs37.56 µs1.10×– / –yes
ltl/u-left-3spot10.64 µs8.92 µs1.19×– / –yes
many_aps/and-gf-12spot160.20 µs65.08 µs2.46×– / –yes
many_aps/gf-equiv-12spot130.48 µs92.36 µs1.41×– / –yes
many_aps/r-right-10spot65.08 µs25.36 µs2.57×– / –yes
many_aps/random-n16-a8-p4-s1spot262.48 µs129.02 µs2.03×– / –yes
many_aps/random-n24-a12-p4-s1spot1.34 ms461.30 µs2.91×– / –yes
many_aps/random-n32-a16-p4-s1spot3.75 ms1.17 ms3.21×– / –yes
random/n128-a2-p6-s1spot330.50 µs205.96 µs1.60×– / –yes
random/n16-a1-p3-s1spot21.54 µs15.74 µs1.37×– / –yes
random/n256-a2-p8-s1spot892.41 µs469.72 µs1.90×– / –yes
random/n64-a2-p4-s1spot141.88 µs97.86 µs1.45×– / –yes

dpa/inclusion

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot71.28 µs61.08 µs1.17×– / –yes
hard/l-nba-2spot34.84 µs22.76 µs1.53×– / –yes
hard/m-nba-3spot148.20 µs81.78 µs1.81×– / –yes
ltl/and-gf-3spot5.80 µs8.12 µs0.71×– / –yes
ltl/eh-patterns-0spot14.42 µs17.96 µs0.80×– / –yes
ltl/gf-equiv-3spot11.58 µs18.96 µs0.61×– / –yes
ltl/gf-implies-3spot6.24 µs11.48 µs0.54×– / –yes
ltl/gh-q-3spot33.02 µs27.68 µs1.19×– / –yes
ltl/gh-r-3spot12.44 µs35.28 µs0.35×– / –yes
ltl/go-theta-3spot6.94 µs15.96 µs0.43×– / –yes
ltl/lily-patterns-1spot8.12 µs13.90 µs0.58×– / –yes
ltl/ms-phi-r-1spot6.12 µs19.36 µs0.32×– / –yes
ltl/ms-phi-s-1spot6.08 µs25.36 µs0.24×– / –yes
ltl/rv-counter-3spot17.92 µs14.22 µs1.26×– / –yes
ltl/sb-patterns-0spot7.72 µs12.08 µs0.64×– / –yes
ltl/sejk-k-2spot7.86 µs28.50 µs0.28×– / –yes
ltl/tv-uu-3spot28.34 µs28.78 µs0.98×– / –yes
ltl/u-left-3spot6.76 µs8.72 µs0.78×– / –yes
many_aps/and-gf-12spot122.86 µs51.82 µs2.37×– / –yes
many_aps/gf-equiv-12spot74.28 µs51.26 µs1.45×– / –yes
many_aps/r-right-10spot46.76 µs23.44 µs1.99×– / –yes
many_aps/random-n16-a8-p4-s1spot213.88 µs95.08 µs2.25×– / –yes
many_aps/random-n24-a12-p4-s1spot1.01 ms377.18 µs2.69×– / –yes
many_aps/random-n32-a16-p4-s1spot3.39 ms1.06 ms3.20×– / –yes
random/n128-a2-p6-s1spot201.46 µs123.84 µs1.63×– / –yes
random/n16-a1-p3-s1spot13.12 µs10.28 µs1.28×– / –yes
random/n256-a2-p8-s1spot508.26 µs293.02 µs1.73×– / –yes
random/n64-a2-p4-s1spot85.62 µs62.36 µs1.37×– / –yes

game/solve

inputreferenceoursreferenceours / referencesize (ours / reference)same result
ladder_10000oink-ppp2.53 ms303.00 µs8.36×10000 / 10000yes
ladder_10000oink-zlk2.53 ms307.00 µs8.25×10000 / 10000yes
ladder_10000spot2.53 ms9.54 ms0.27×10000 / 10000yes
ladder_50000oink-ppp19.49 ms2.09 ms9.33×50000 / 50000yes
ladder_50000oink-zlk19.49 ms2.12 ms9.21×50000 / 50000yes
ladder_50000spot19.49 ms72.52 ms0.27×50000 / 50000yes
random_few_10000oink-ppp6.74 ms1.39 ms4.84×10000 / 10000yes
random_few_10000oink-zlk6.74 ms2.66 ms2.53×10000 / 10000yes
random_few_10000spot6.74 ms20.43 ms0.33×10000 / 10000yes
random_few_50000oink-ppp72.46 ms15.43 ms4.70×50000 / 50000yes
random_few_50000oink-zlk72.46 ms18.40 ms3.94×50000 / 50000yes
random_few_50000spot72.46 ms651.40 ms0.11×50000 / 50000yes
random_many_10000oink-ppp6.48 ms1.54 ms4.20×10000 / 10000yes
random_many_10000oink-zlk6.48 ms2.60 ms2.49×10000 / 10000yes
random_many_10000spot6.48 ms29.96 ms0.22×10000 / 10000yes
random_many_50000oink-ppp65.65 ms26.19 ms2.51×50000 / 50000yes
random_many_50000oink-zlk65.65 ms37.03 ms1.77×50000 / 50000yes
random_many_50000spot65.65 ms643.46 ms0.10×50000 / 50000yes

Log

== head 91ab2741d67d80a15ea1c1cabf1ec70b87abaf59: building ["scc", "ts", "runs", "dfa", "dpa", "storage", "nondeterministic", "learning", "game"] ["wallclock"]
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
    Finished `bench` profile [optimized] target(s) in 4m 07s
  Executable benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-c46c79a89c1c2941)
  Executable benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-0b05141996aaf468)
  Executable benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-fc166083591c2309)
  Executable benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-72c8d077fb858ea9)
  Executable benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-e95514e1aa52b38c)
  Executable benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-3b37aa113a4e75ae)
  Executable benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-b88e81c5af8ec1fc)
  Executable benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-fffb4b0cac4d81af)
  Executable benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-bb6ee25ff0b06f0f)
  Executable benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-3b599e26071f9e69)
== head: running scc
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-b88e81c5af8ec1fc)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running ts
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-bb6ee25ff0b06f0f)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running runs
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-3b37aa113a4e75ae)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dfa
    Finished `bench` profile [optimized] target(s) in 0.18s
     Running benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-c46c79a89c1c2941)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dpa
    Finished `bench` profile [optimized] target(s) in 0.19s
     Running benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-0b05141996aaf468)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running storage
    Finished `bench` profile [optimized] target(s) in 0.19s
     Running benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-fffb4b0cac4d81af)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running nondeterministic
    Finished `bench` profile [optimized] target(s) in 0.19s
     Running benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-e95514e1aa52b38c)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running learning
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-72c8d077fb858ea9)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running game
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-fc166083591c2309)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running wallclock (wall clock)
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-3b599e26071f9e69)
Gnuplot not found, using plotters backend
Benchmarking wallclock/scc/tarjan/scrambled_10000
Benchmarking wallclock/scc/tarjan/scrambled_10000: Analyzing
Benchmarking wallclock/dpa/normalized/scrambled_4000
Benchmarking wallclock/dpa/normalized/scrambled_4000: Analyzing
Benchmarking wallclock/dpa/complement/scrambled_4000
Benchmarking wallclock/dpa/complement/scrambled_4000: Analyzing
Benchmarking wallclock/dfa/minimize/redundant_4000
Benchmarking wallclock/dfa/minimize/redundant_4000: Analyzing
Benchmarking wallclock/dfa/accepts/word_100000
Benchmarking wallclock/dfa/accepts/word_100000: Analyzing
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12: Analyzing
Benchmarking wallclock/nfa/determinize/nth_from_end_12
Benchmarking wallclock/nfa/determinize/nth_from_end_12: Analyzing
Benchmarking wallclock/hoa/parse/ltl
Benchmarking wallclock/hoa/parse/ltl: Analyzing
Benchmarking collect/clone/100000
Benchmarking collect/clone/100000: Analyzing
Benchmarking collect/collect_owned/100000
Benchmarking collect/collect_owned/100000: Analyzing
Benchmarking collect/collect_ref/100000
Benchmarking collect/collect_ref/100000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/100000
Benchmarking collect/map_edge_colors_collect_owned/100000: Analyzing
Benchmarking collect/clone/1000000
Benchmarking collect/clone/1000000: Analyzing
Benchmarking collect/collect_owned/1000000
Benchmarking collect/collect_owned/1000000: Analyzing
Benchmarking collect/collect_ref/1000000
Benchmarking collect/collect_ref/1000000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/1000000
Benchmarking collect/map_edge_colors_collect_owned/1000000: Analyzing
== head: building the Python bindings
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support
🐍 Found CPython 3.13 at /nix/store/vilynzly1pyhawy04ljmrp7n9ffr24qy-python3-3.13.17/bin/python3
📡 Using build options features from pyproject.toml
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
   Compiling automata-py v0.1.0 (/var/lib/automata-bench/work/src-head/python)
    Finished `release` profile [optimized] target(s) in 7m 25s
📦 Built wheel for abi3 Python ≥ 3.11 to /var/lib/automata-bench/work/versus-head/wheels/automatars-0.1.0-cp311-abi3-manylinux_2_34_aarch64.whl
== head: running benches/versus/compare.py (Spot, Oink)
dpa/equivalence/hard/l-dsa-3 vs spot: 0.125 ms / 0.120 ms
dpa/equivalence/hard/l-nba-2 vs spot: 0.056 ms / 0.037 ms
dpa/equivalence/hard/m-nba-3 vs spot: 0.237 ms / 0.135 ms
dpa/equivalence/ltl/and-gf-3 vs spot: 0.009 ms / 0.012 ms
dpa/equivalence/ltl/eh-patterns-0 vs spot: 0.022 ms / 0.020 ms
dpa/equivalence/ltl/gf-equiv-3 vs spot: 0.020 ms / 0.034 ms
dpa/equivalence/ltl/gf-implies-3 vs spot: 0.011 ms / 0.020 ms
dpa/equivalence/ltl/gh-q-3 vs spot: 0.051 ms / 0.029 ms
dpa/equivalence/ltl/gh-r-3 vs spot: 0.021 ms / 0.069 ms
dpa/equivalence/ltl/go-theta-3 vs spot: 0.011 ms / 0.021 ms
dpa/equivalence/ltl/lily-patterns-1 vs spot: 0.012 ms / 0.016 ms
dpa/equivalence/ltl/ms-phi-r-1 vs spot: 0.010 ms / 0.039 ms
dpa/equivalence/ltl/ms-phi-s-1 vs spot: 0.010 ms / 0.041 ms
dpa/equivalence/ltl/rv-counter-3 vs spot: 0.028 ms / 0.013 ms
dpa/equivalence/ltl/sb-patterns-0 vs spot: 0.013 ms / 0.014 ms
dpa/equivalence/ltl/sejk-k-2 vs spot: 0.014 ms / 0.059 ms
dpa/equivalence/ltl/tv-uu-3 vs spot: 0.041 ms / 0.038 ms
dpa/equivalence/ltl/u-left-3 vs spot: 0.011 ms / 0.009 ms
dpa/equivalence/many_aps/and-gf-12 vs spot: 0.160 ms / 0.065 ms
dpa/equivalence/many_aps/gf-equiv-12 vs spot: 0.130 ms / 0.092 ms
dpa/equivalence/many_aps/r-right-10 vs spot: 0.065 ms / 0.025 ms
dpa/equivalence/many_aps/random-n16-a8-p4-s1 vs spot: 0.262 ms / 0.129 ms
dpa/equivalence/many_aps/random-n24-a12-p4-s1 vs spot: 1.342 ms / 0.461 ms
dpa/equivalence/many_aps/random-n32-a16-p4-s1 vs spot: 3.747 ms / 1.169 ms
dpa/equivalence/random/n128-a2-p6-s1 vs spot: 0.331 ms / 0.206 ms
dpa/equivalence/random/n16-a1-p3-s1 vs spot: 0.022 ms / 0.016 ms
dpa/equivalence/random/n256-a2-p8-s1 vs spot: 0.892 ms / 0.470 ms
dpa/equivalence/random/n64-a2-p4-s1 vs spot: 0.142 ms / 0.098 ms
dpa/inclusion/hard/l-dsa-3 vs spot: 0.071 ms / 0.061 ms
dpa/inclusion/hard/l-nba-2 vs spot: 0.035 ms / 0.023 ms
dpa/inclusion/hard/m-nba-3 vs spot: 0.148 ms / 0.082 ms
dpa/inclusion/ltl/and-gf-3 vs spot: 0.006 ms / 0.008 ms
dpa/inclusion/ltl/eh-patterns-0 vs spot: 0.014 ms / 0.018 ms
dpa/inclusion/ltl/gf-equiv-3 vs spot: 0.012 ms / 0.019 ms
dpa/inclusion/ltl/gf-implies-3 vs spot: 0.006 ms / 0.011 ms
dpa/inclusion/ltl/gh-q-3 vs spot: 0.033 ms / 0.028 ms
dpa/inclusion/ltl/gh-r-3 vs spot: 0.012 ms / 0.035 ms
dpa/inclusion/ltl/go-theta-3 vs spot: 0.007 ms / 0.016 ms
dpa/inclusion/ltl/lily-patterns-1 vs spot: 0.008 ms / 0.014 ms
dpa/inclusion/ltl/ms-phi-r-1 vs spot: 0.006 ms / 0.019 ms
dpa/inclusion/ltl/ms-phi-s-1 vs spot: 0.006 ms / 0.025 ms
dpa/inclusion/ltl/rv-counter-3 vs spot: 0.018 ms / 0.014 ms
dpa/inclusion/ltl/sb-patterns-0 vs spot: 0.008 ms / 0.012 ms
dpa/inclusion/ltl/sejk-k-2 vs spot: 0.008 ms / 0.029 ms
dpa/inclusion/ltl/tv-uu-3 vs spot: 0.028 ms / 0.029 ms
dpa/inclusion/ltl/u-left-3 vs spot: 0.007 ms / 0.009 ms
dpa/inclusion/many_aps/and-gf-12 vs spot: 0.123 ms / 0.052 ms
dpa/inclusion/many_aps/gf-equiv-12 vs spot: 0.074 ms / 0.051 ms
dpa/inclusion/many_aps/r-right-10 vs spot: 0.047 ms / 0.023 ms
dpa/inclusion/many_aps/random-n16-a8-p4-s1 vs spot: 0.214 ms / 0.095 ms
dpa/inclusion/many_aps/random-n24-a12-p4-s1 vs spot: 1.013 ms / 0.377 ms
dpa/inclusion/many_aps/random-n32-a16-p4-s1 vs spot: 3.394 ms / 1.062 ms
dpa/inclusion/random/n128-a2-p6-s1 vs spot: 0.201 ms / 0.124 ms
dpa/inclusion/random/n16-a1-p3-s1 vs spot: 0.013 ms / 0.010 ms
dpa/inclusion/random/n256-a2-p8-s1 vs spot: 0.508 ms / 0.293 ms
dpa/inclusion/random/n64-a2-p4-s1 vs spot: 0.086 ms / 0.062 ms
game/solve/random_few_10000 vs spot: 6.740 ms / 20.428 ms
game/solve/random_few_10000 vs oink-zlk: 6.740 ms / 2.664 ms
game/solve/random_few_10000 vs oink-ppp: 6.740 ms / 1.393 ms
game/solve/random_few_50000 vs spot: 72.462 ms / 651.401 ms
game/solve/random_few_50000 vs oink-zlk: 72.462 ms / 18.399 ms
game/solve/random_few_50000 vs oink-ppp: 72.462 ms / 15.428 ms
game/solve/random_many_10000 vs spot: 6.479 ms / 29.964 ms
game/solve/random_many_10000 vs oink-zlk: 6.479 ms / 2.604 ms
game/solve/random_many_10000 vs oink-ppp: 6.479 ms / 1.542 ms
game/solve/random_many_50000 vs spot: 65.653 ms / 643.459 ms
game/solve/random_many_50000 vs oink-zlk: 65.653 ms / 37.026 ms
game/solve/random_many_50000 vs oink-ppp: 65.653 ms / 26.191 ms
game/solve/ladder_10000 vs spot: 2.532 ms / 9.542 ms
game/solve/ladder_10000 vs oink-zlk: 2.532 ms / 0.307 ms
game/solve/ladder_10000 vs oink-ppp: 2.532 ms / 0.303 ms
game/solve/ladder_50000 vs spot: 19.486 ms / 72.521 ms
game/solve/ladder_50000 vs oink-zlk: 19.486 ms / 2.116 ms
game/solve/ladder_50000 vs oink-ppp: 19.486 ms / 2.087 ms
== head: 266 benchmarks, 16 wall-clock measurements, 74 comparisons