automata benchmarks

History · Versus Spot and Oink

PR #161 · c80c10bc

PropAlphabet: oxidd BDDs instead of biodivine-lib-bdd (#156, PR C)

status done · queued 2026-11-10 12:30 · started 2026-11-10 12:30 · finished 2026-11-10 12:42

· · rustc 1.95.0 (59807616e 2026-04-14) (built from a source tarball) · valgrind-3.26.0 · pinned to cpu 1

Against merge base 1bf383b2 — 8 regressions, 28 improvements, 223 unchanged, 0 new

benchmarkbaseheadchangeheap bytes
dpa::dpa::complement_fixtures::synthetic9 444 8999 666 554+2.35 %
dpa::dpa::normalized_fixtures::synthetic9 128 7779 351 540+2.44 %
dpa::dpa::sccs::chain_1000487 826500 886+2.68 %
dpa::dpa::sccs::chain_40001 876 5391 928 613+2.78 %
dpa::dpa::sccs::scrambled_1000554 783567 843+2.35 %
dpa::dpa::sccs::scrambled_40002 205 4092 257 483+2.36 %
runs::runs::dpa_recurrent_edge_colors::long14 635 82615 116 266+3.28 %+0.00 %
runs::runs::dpa_recurrent_edge_colors::short1 098 1071 189 790+8.35 %+0.00 %
dfa::dfa::intersection::scrambled_1003 719 7273 319 709-10.75 %+0.00 %
dfa::dfa::intersection::scrambled_20014 824 89113 224 873-10.79 %+0.00 %
dfa::dfa::union::scrambled_1003 719 7123 319 709-10.75 %+0.00 %
dfa::dfa::union::scrambled_20014 824 87613 224 873-10.79 %+0.00 %
dpa::dpa::collect_dpa::scrambled_1000301 647290 651-3.65 %
dpa::dpa::collect_dpa::scrambled_40001 177 0401 133 044-3.74 %
dpa::dpa::complement_fixtures::many_aps128 792 34213 922 089-89.19 %
dpa::dpa::equivalent_fixtures::many_aps94 848 71821 636 042-77.19 %
dpa::dpa::hoa_parse::ltl10 979 6345 507 598-49.84 %
dpa::dpa::hoa_parse::many_aps270 019 54288 883 063-67.08 %
dpa::dpa::hoa_parse::synthetic40 738 32925 090 683-38.41 %
dpa::dpa::hoa_print::ltl49 107 21619 037 260-61.23 %
dpa::dpa::hoa_print::many_aps753 714 86019 662 969-97.39 %
dpa::dpa::hoa_print::synthetic143 208 85927 383 653-80.88 %
dpa::dpa::included_fixtures::many_aps92 011 61218 595 189-79.79 %
dpa::dpa::map_edge_colors_collect::scrambled_1000307 253289 296-5.84 %
dpa::dpa::map_edge_colors_collect::scrambled_40001 197 6461 125 691-6.01 %
dpa::dpa::normalized_fixtures::many_aps129 810 44913 662 040-89.48 %
dpa::dpa::witness_colors::chain_1004 743 8784 635 276-2.29 %
dpa::dpa::witness_colors::chain_40046 530 82545 080 177-3.12 %
dpa::dpa::witness_colors::scrambled_1004 128 7224 026 603-2.47 %
dpa::dpa::witness_colors::scrambled_40031 854 84630 619 065-3.88 %
dpa::dpa::witnesses_fixtures::synthetic19 746 17718 835 736-4.61 %
nondeterministic::nondeterministic::fixture_accepts::ltl822 089762 545-7.24 %
runs::runs::dpa_recurrent_states::long11 749 19310 502 988-10.61 %+0.00 %
runs::runs::dpa_recurrent_states::short589 492572 037-2.96 %+0.00 %
runs::runs::fixture_accepts_lassos::ltl9 167 9188 766 271-4.38 %
runs::runs::fixture_accepts_lassos::synthetic4 648 8584 340 189-6.64 %
dfa::dfa::accepts_many_words::scrambled_1000343 072343 072+0.00 %+0.00 %
dfa::dfa::complement::scrambled_1000366 068366 068+0.00 %+0.00 %
dfa::dfa::complement::scrambled_40001 435 8731 435 873+0.00 %+0.00 %
dfa::dfa::equivalent::deep_difference_10061 66761 667+0.00 %+0.00 %
dfa::dfa::equivalent::deep_difference_400212 158212 158+0.00 %+0.00 %
dfa::dfa::equivalent::different_1009 0479 047+0.00 %+0.00 %
dfa::dfa::equivalent::different_4009 0889 088+0.00 %+0.00 %
dfa::dfa::equivalent::minimized_10035 84935 849+0.00 %+0.00 %
dfa::dfa::equivalent::minimized_400120 587120 587+0.00 %+0.00 %
dfa::dfa::give_word::empty_10001 872 6131 872 613+0.00 %+0.00 %
dfa::dfa::give_word::empty_40006 930 8236 930 823+0.00 %+0.00 %
dfa::dfa::give_word::far_1000164 342164 342+0.00 %+0.00 %
dfa::dfa::give_word::far_4000946 982946 982+0.00 %+0.00 %
dfa::dfa::intersection_is_empty::empty_10059 26659 266+0.00 %+0.00 %
dfa::dfa::intersection_is_empty::empty_200109 269109 269+0.00 %+0.00 %
dfa::dfa::is_empty_language::empty_10001 872 6151 872 615+0.00 %+0.00 %
dfa::dfa::is_empty_language::empty_40006 930 8256 930 825+0.00 %+0.00 %
dfa::dfa::minimize::minimal_10003 435 7343 450 924+0.44 %+0.00 %
dfa::dfa::minimize::minimal_400013 412 36813 472 602+0.45 %+0.00 %
dfa::dfa::minimize::redundant_10002 284 6412 292 807+0.36 %+0.00 %
dfa::dfa::minimize::redundant_40008 836 8968 869 106+0.36 %+0.00 %
dfa::dfa::minimize::scrambled_10003 647 9263 670 256+0.61 %+0.00 %
dfa::dfa::minimize::scrambled_400014 161 22714 250 469+0.63 %+0.00 %
dfa::dfa::minimize_moore_with_edge_colors::scrambled_10003 646 9363 669 252+0.61 %+0.00 %
dfa::dfa::minimize_moore_with_edge_colors::scrambled_400014 149 84514 239 073+0.63 %+0.00 %
dpa::dpa::accepted_word::chain_100112 530114 174+1.46 %
dpa::dpa::accepted_word::chain_400379 899385 785+1.55 %
dpa::dpa::accepted_word::scrambled_100144 738144 718-0.01 %
dpa::dpa::accepted_word::scrambled_400493 406492 612-0.16 %
dpa::dpa::complement::chain_10004 108 5844 101 157-0.18 %
dpa::dpa::complement::chain_400016 700 09316 670 178-0.18 %
dpa::dpa::complement::scrambled_10004 971 4534 960 165-0.23 %
dpa::dpa::complement::scrambled_400020 090 46920 044 505-0.23 %
dpa::dpa::complement_fixtures::ltl1 500 4321 481 799-1.24 %
dpa::dpa::complement_without_normalizing::scrambled_1000574 449565 457-1.57 %
dpa::dpa::complement_without_normalizing::scrambled_40002 259 8532 223 850-1.59 %
dpa::dpa::included_in::complement_100269 290269 594+0.11 %
dpa::dpa::included_in::complement_2587 71687 754+0.04 %
dpa::dpa::included_in::complement_50150 987151 045+0.04 %
dpa::dpa::included_in::normalized_100541 195542 475+0.24 %
dpa::dpa::included_in::normalized_25184 110184 636+0.29 %
dpa::dpa::included_in::normalized_50312 439313 247+0.26 %
dpa::dpa::language_equivalent::complement_100278 865279 169+0.11 %
dpa::dpa::language_equivalent::complement_2590 24190 279+0.04 %
dpa::dpa::language_equivalent::complement_50155 862155 920+0.04 %
dpa::dpa::language_equivalent::normalized_100989 905992 450+0.26 %
dpa::dpa::language_equivalent::normalized_25325 469326 481+0.31 %
dpa::dpa::language_equivalent::normalized_50579 273580 879+0.28 %
dpa::dpa::normalized::chain_10003 836 0303 828 603-0.19 %
dpa::dpa::normalized::chain_400015 611 53915 581 624-0.19 %
dpa::dpa::normalized::scrambled_10004 698 7014 687 413-0.24 %
dpa::dpa::normalized::scrambled_400019 001 71718 955 753-0.24 %
dpa::dpa::normalized::scrambled_k4_100025 720 47925 967 230+0.96 %
dpa::dpa::normalized_fixtures::ltl1 409 4551 417 207+0.55 %
dpa::dpa::prefix_partition::chain_163 310 4143 307 163-0.10 %
dpa::dpa::prefix_partition::chain_8853 868853 054-0.10 %
dpa::dpa::prefix_partition::scrambled_163 700 1153 688 055-0.33 %
dpa::dpa::prefix_partition::scrambled_81 019 2211 015 701-0.35 %
dpa::dpa::rejected_word::chain_100937 946922 706-1.62 %
dpa::dpa::rejected_word::chain_4003 643 6013 582 223-1.68 %
dpa::dpa::rejected_word::scrambled_100247 652244 006-1.47 %
dpa::dpa::rejected_word::scrambled_400934 686920 138-1.56 %
dpa::dpa::witness_inequivalence::complement_100278 864279 168+0.11 %
dpa::dpa::witness_inequivalence::complement_2590 24090 278+0.04 %
dpa::dpa::witness_inequivalence::complement_50155 861155 919+0.04 %
dpa::dpa::witness_not_subset_of::complement_100269 289269 593+0.11 %
dpa::dpa::witness_not_subset_of::complement_2587 71587 753+0.04 %
dpa::dpa::witness_not_subset_of::complement_50150 986151 044+0.04 %
dpa::dpa::witness_not_subset_of::normalized_100541 194542 474+0.24 %
dpa::dpa::witness_not_subset_of::normalized_25184 109184 635+0.29 %
dpa::dpa::witness_not_subset_of::normalized_50312 438313 246+0.26 %
dpa::dpa::witnesses_fixtures::ltl2 245 9062 217 400-1.27 %
game::game::tangle_learning::few_priorities_10002 681 1742 697 048+0.59 %+0.00 %
game::game::tangle_learning::few_priorities_40008 074 1268 120 676+0.58 %+0.00 %
game::game::tangle_learning::ladder_10001 178 8141 179 314+0.04 %+0.00 %
game::game::tangle_learning::ladder_40004 406 5834 408 134+0.04 %+0.00 %
game::game::tangle_learning::many_priorities_10002 009 4442 018 028+0.43 %+0.00 %
game::game::tangle_learning::many_priorities_40006 730 5626 752 835+0.33 %+0.00 %
game::game::tangle_learning_min_odd::few_priorities_10001 520 4011 521 615+0.08 %+0.00 %
game::game::tangle_learning_min_odd::few_priorities_40007 998 1518 017 700+0.24 %+0.00 %
game::game::zielonka::few_priorities_10001 126 2021 126 199-0.00 %+0.00 %
game::game::zielonka::few_priorities_40003 470 2573 470 254-0.00 %+0.00 %
game::game::zielonka::ladder_10005 134 2845 134 281-0.00 %+0.00 %
game::game::zielonka::ladder_400019 816 37619 816 373-0.00 %+0.00 %
game::game::zielonka::many_priorities_10004 103 4374 103 518+0.00 %+0.00 %
game::game::zielonka::many_priorities_400012 909 79112 909 837+0.00 %+0.00 %
game::game::zielonka_min_odd::few_priorities_10002 139 0592 139 077+0.00 %+0.00 %
game::game::zielonka_min_odd::few_priorities_40006 302 0646 302 075+0.00 %+0.00 %
learning::learning::lstar_mealy::counting_161 787 0911 817 760+1.72 %+0.00 %
learning::learning::lstar_mealy::counting_8478 063481 496+0.72 %+0.00 %
learning::learning::sprout_parity::inf_ab_1001 272 7751 270 840-0.15 %+0.00 %
learning::learning::sprout_parity::inf_ab_4002 614 1052 610 820-0.13 %+0.00 %
nondeterministic::nondeterministic::nba_accepts::nba_400125 268 484125 269 894+0.00 %
nondeterministic::nondeterministic::nba_witness::nba_400014 586 04414 689 961+0.71 %
nondeterministic::nondeterministic::nca_accepts::nca_400266 215 466266 229 440+0.01 %
nondeterministic::nondeterministic::nca_witness::nca_400019 590 60919 458 261-0.68 %
nondeterministic::nondeterministic::nma_witness::nma_100025 727 43025 744 469+0.07 %
nondeterministic::nondeterministic::npa_accepts::empty_npa_400492 867 078493 167 601+0.06 %
nondeterministic::nondeterministic::npa_accepts::npa_10030 410 45830 411 628+0.00 %
nondeterministic::nondeterministic::npa_accepts::npa_400125 278 162125 279 542+0.00 %
nondeterministic::nondeterministic::npa_witness::empty_npa_10009 115 0249 121 952+0.08 %
nondeterministic::nondeterministic::npa_witness::empty_npa_400036 996 51737 023 303+0.07 %
nondeterministic::nondeterministic::npa_witness::npa_10002 471 0532 479 531+0.34 %
nondeterministic::nondeterministic::npa_witness::npa_400020 162 80420 354 255+0.95 %
nondeterministic::nondeterministic::nra_witness::nra_400020 617 67920 639 567+0.11 %
runs::runs::dfa_accepts_many_words::count_10004 303 2134 303 213+0.00 %+0.00 %
runs::runs::dfa_accepts_one_word::len_1003 4923 492+0.00 %+0.00 %
runs::runs::dfa_accepts_one_word::len_10000266 070266 070+0.00 %+0.00 %
runs::runs::dfa_accepts_one_word::len_1000002 650 7312 650 731+0.00 %+0.00 %
runs::runs::dfa_reached_state::count_10003 570 2113 570 211+0.00 %+0.00 %
runs::runs::dpa_accepts_lassos::long9 150 7459 150 745+0.00 %+0.00 %
runs::runs::dpa_accepts_lassos::short406 490406 490+0.00 %+0.00 %
runs::runs::dpa_visited_sequences::long8 738 2638 738 263+0.00 %+0.00 %
runs::runs::dpa_visited_sequences::short370 678370 678+0.00 %+0.00 %
scc::scc::scc_queries::small_cycles_1000906 614906 676+0.01 %+0.00 %
scc::scc::scc_queries::small_cycles_40003 548 8433 548 919+0.00 %+0.00 %
scc::scc::tarjan::back_edges_1000512 037512 099+0.01 %+0.00 %
scc::scc::tarjan::back_edges_40001 987 5511 987 627+0.00 %+0.00 %
scc::scc::tarjan::binary_tree_1000571 155571 217+0.01 %+0.00 %
scc::scc::tarjan::binary_tree_40002 290 3832 290 459+0.00 %+0.00 %
scc::scc::tarjan::fragmented_1000658 467658 522+0.01 %+0.00 %
scc::scc::tarjan::fragmented_40002 534 0392 534 108+0.00 %+0.00 %
scc::scc::tarjan::grid_1024596 018596 080+0.01 %+0.00 %
scc::scc::tarjan::grid_40962 378 4082 378 484+0.00 %+0.00 %
scc::scc::tarjan::path_10001 929 0571 929 119+0.00 %+0.00 %
scc::scc::tarjan::path_40007 632 8327 632 908+0.00 %+0.00 %
scc::scc::tarjan::ring_1000512 037512 099+0.01 %+0.00 %
scc::scc::tarjan::ring_40001 987 5511 987 627+0.00 %+0.00 %
scc::scc::tarjan::scrambled_1000578 994579 056+0.01 %+0.00 %
scc::scc::tarjan::scrambled_40002 316 4212 316 497+0.00 %+0.00 %
scc::scc::tarjan::small_cycles_1000905 596905 658+0.01 %+0.00 %
scc::scc::tarjan::small_cycles_40003 544 8253 544 901+0.00 %+0.00 %
scc::scc::tarjan_from::fragmented_1000242 363242 530+0.07 %+0.00 %
scc::scc::tarjan_from::fragmented_4000907 553907 762+0.02 %+0.00 %
scc::scc::tarjan_from::path_10001 920 9701 921 158+0.01 %+0.00 %
scc::scc::tarjan_from::path_40007 545 3547 545 584+0.00 %+0.00 %
scc::scc::tarjan_from::ring_1000503 457503 645+0.04 %+0.00 %
scc::scc::tarjan_from::scrambled_1000570 414570 602+0.03 %+0.00 %
storage::storage::add_edges_dense::k2_50001 356 4011 356 401+0.00 %+0.00 %
storage::storage::add_edges_dense::k8_50004 247 0594 247 059+0.00 %+0.00 %
storage::storage::add_edges_linked::k2_50001 402 6701 402 670+0.00 %+0.00 %
storage::storage::add_edges_linked::k8_50006 669 2846 669 284+0.00 %+0.00 %
storage::storage::clone_csr::k2_1000030 59030 590+0.00 %+0.00 %
storage::storage::clone_dense::k2_1000015 11315 113+0.00 %+0.00 %
storage::storage::clone_linked::k2_10000370 822370 824+0.00 %+0.00 %
storage::storage::complete_dense::partial_5000155 627155 628+0.00 %+0.00 %
storage::storage::complete_linked::partial_50001 210 8541 210 854+0.00 %+0.00 %
storage::storage::edges_from_all_states_csr::k2_200072 04772 047+0.00 %+0.00 %
storage::storage::edges_from_all_states_csr::k8_2000108 047108 047+0.00 %+0.00 %
storage::storage::edges_from_all_states_dense::k2_2000102 043102 043+0.00 %+0.00 %
storage::storage::edges_from_all_states_dense::k8_2000140 043140 043+0.00 %+0.00 %
storage::storage::edges_from_all_states_linked::k2_200068 03368 033+0.00 %+0.00 %
storage::storage::edges_from_all_states_linked::k8_2000152 033152 033+0.00 %+0.00 %
storage::storage::minimize_csr::redundant_20004 617 5824 633 756+0.35 %+0.00 %
storage::storage::minimize_csr::redundant_800018 095 64318 159 861+0.35 %+0.00 %
storage::storage::minimize_dense::redundant_20004 660 7364 676 910+0.35 %+0.00 %
storage::storage::minimize_dense::redundant_800018 300 67718 364 895+0.35 %+0.00 %
storage::storage::minimize_linked::redundant_20004 488 8644 505 038+0.36 %+0.00 %
storage::storage::minimize_linked::redundant_800017 650 50317 714 721+0.36 %+0.00 %
storage::storage::overwrite_edges_dense::k2_50002 126 4332 126 431-0.00 %+0.00 %
storage::storage::overwrite_edges_linked::k2_50003 790 7763 790 775-0.00 %+0.00 %
storage::storage::predecessors_of_all_states_csr::k2_2000102 046102 046+0.00 %+0.00 %
storage::storage::predecessors_of_all_states_dense::k2_2000290 369290 369+0.00 %+0.00 %
storage::storage::predecessors_of_all_states_linked::k2_200068 03368 033+0.00 %+0.00 %
storage::storage::product_successor_csr::n20064 09664 096+0.00 %+0.00 %
storage::storage::product_successor_dense::n20073 08773 087+0.00 %+0.00 %
storage::storage::product_successor_linked::n20046 54746 547+0.00 %+0.00 %
storage::storage::reachable_states_csr::k2_80001 101 1151 101 115+0.00 %+0.00 %
storage::storage::reachable_states_dense::k2_80001 133 6991 133 699+0.00 %+0.00 %
storage::storage::reachable_states_linked::k2_8000924 279924 279+0.00 %+0.00 %
storage::storage::remove_state_dense::k2_100001 072 2121 072 212+0.00 %+0.00 %
storage::storage::remove_state_linked::k2_100001 094 0121 094 012+0.00 %+0.00 %
storage::storage::run_many_words_csr::count_1000551 036551 036+0.00 %+0.00 %
storage::storage::run_many_words_dense::count_1000558 044558 044+0.00 %+0.00 %
storage::storage::run_many_words_linked::count_1000481 044481 044+0.00 %+0.00 %
storage::storage::run_one_word_csr::len_1000002 600 0542 600 054+0.00 %+0.00 %
storage::storage::run_one_word_dense::len_1000002 800 0502 800 050+0.00 %+0.00 %
storage::storage::run_one_word_linked::len_1000002 350 0392 350 039+0.00 %+0.00 %
storage::storage::successor_all_pairs_csr::k2_2000122 242122 242+0.00 %+0.00 %
storage::storage::successor_all_pairs_csr::k8_2000702 302702 302+0.00 %+0.00 %
storage::storage::successor_all_pairs_dense::k2_2000162 237162 237+0.00 %+0.00 %
storage::storage::successor_all_pairs_dense::k8_2000814 297814 297+0.00 %+0.00 %
storage::storage::successor_all_pairs_linked::k2_2000108 236108 236+0.00 %+0.00 %
storage::storage::successor_all_pairs_linked::k8_2000750 296750 296+0.00 %+0.00 %
ts::ts::add_edges_in_order::k2_50001 402 6711 402 671+0.00 %+0.00 %
ts::ts::add_edges_in_order::k8_50006 669 2856 669 285+0.00 %+0.00 %
ts::ts::add_edges_in_reverse::k2_50001 402 6711 402 671+0.00 %+0.00 %
ts::ts::builder_with_edges::k2_500081 367 02381 358 433-0.01 %+0.00 %
ts::ts::canonical_naming::scrambled_15003 167 6083 167 608+0.00 %+0.00 %
ts::ts::canonical_naming::scrambled_500980 422980 422+0.00 %+0.00 %
ts::ts::clone::scrambled_10000371 163371 163+0.00 %+0.00 %
ts::ts::clone::scrambled_400001 481 4001 481 400+0.00 %+0.00 %
ts::ts::collect_dts::scrambled_100003 465 1493 465 149+0.00 %+0.00 %
ts::ts::collect_dts::scrambled_4000013 799 62813 799 628+0.00 %+0.00 %
ts::ts::determinize::nth_from_end_104 485 5244 485 524+0.00 %+0.00 %
ts::ts::determinize::nth_from_end_1219 876 08519 876 085+0.00 %+0.00 %
ts::ts::determinize::nth_from_end_81 027 3331 027 333+0.00 %+0.00 %
ts::ts::edges_from_all_states::scrambled_200072 66972 669+0.00 %+0.00 %
ts::ts::edges_from_all_states::scrambled_k8_2000156 601156 601+0.00 %+0.00 %
ts::ts::erase_state_colors_collect::scrambled_100003 007 1663 007 166+0.00 %+0.00 %
ts::ts::into_dfa::scrambled_100003 565 1653 565 165+0.00 %+0.00 %
ts::ts::map_edge_colors_collect::scrambled_100003 008 7683 008 768+0.00 %+0.00 %
ts::ts::map_edge_colors_collect::scrambled_4000011 975 39111 975 391+0.00 %+0.00 %
ts::ts::matching_product_successor_index::n20075 66275 662+0.00 %+0.00 %
ts::ts::minimal_representative_of_all_states::scrambled_1002 100 8142 100 814+0.00 %+0.00 %
ts::ts::minimal_representative_of_all_states::scrambled_30016 911 26416 911 264+0.00 %+0.00 %
ts::ts::minimal_representatives::scrambled_20003 384 0403 384 040+0.00 %+0.00 %
ts::ts::minimal_representatives::scrambled_800014 900 24914 900 249+0.00 %+0.00 %
ts::ts::predecessors_of_all_states::scrambled_200072 66972 669+0.00 %+0.00 %
ts::ts::predecessors_of_all_states::scrambled_8000288 601288 601+0.00 %+0.00 %
ts::ts::reachable_states::scrambled_2000290 091290 091+0.00 %+0.00 %
ts::ts::reachable_states::scrambled_80001 148 6961 148 696+0.00 %+0.00 %
ts::ts::remove_edges_from_matching::scrambled_10000566 234566 234+0.00 %+0.00 %
ts::ts::remove_edges_from_matching::scrambled_40000896 234896 234+0.00 %+0.00 %
ts::ts::remove_state::scrambled_100001 091 0121 091 012+0.00 %+0.00 %
ts::ts::remove_state::scrambled_400001 421 0121 421 012+0.00 %+0.00 %
ts::ts::scc_minimal_representatives::small_cycles_100219 944219 985+0.02 %+0.00 %
ts::ts::scc_minimal_representatives::small_cycles_4001 246 3521 246 407+0.00 %+0.00 %
ts::ts::subset_construction::nth_from_end_1010 216 38410 216 439+0.00 %+0.00 %
ts::ts::subset_construction::nth_from_end_1245 206 09445 206 163+0.00 %+0.00 %
ts::ts::subset_construction::nth_from_end_82 230 4652 230 506+0.00 %+0.00 %
ts::ts::successor_index_all_pairs::scrambled_2000108 863108 863+0.00 %+0.00 %
ts::ts::successor_index_all_pairs::scrambled_k8_2000750 855750 855+0.00 %+0.00 %

Wall-clock times

Criterion means of a single run. They depend on the machine and are not compared; use the instruction counts above for that.

benchmarkbasehead
collect/clone/1000003.97 ms (3.95 ms … 3.99 ms)4.04 ms (3.98 ms … 4.11 ms)
collect/clone/100000040.52 ms (39.91 ms … 41.14 ms)41.58 ms (41.12 ms … 42.05 ms)
collect/collect_owned/1000007.46 ms (7.43 ms … 7.49 ms)8.68 ms (8.65 ms … 8.71 ms)
collect/collect_owned/100000090.62 ms (88.64 ms … 92.59 ms)86.90 ms (85.60 ms … 88.20 ms)
collect/collect_ref/1000005.44 ms (5.41 ms … 5.48 ms)4.87 ms (4.81 ms … 4.93 ms)
collect/collect_ref/100000090.63 ms (88.75 ms … 92.51 ms)83.69 ms (82.59 ms … 84.78 ms)
collect/map_edge_colors_collect_owned/1000007.37 ms (7.36 ms … 7.37 ms)7.69 ms (7.48 ms … 7.91 ms)
collect/map_edge_colors_collect_owned/100000084.39 ms (83.67 ms … 85.11 ms)81.12 ms (80.10 ms … 82.14 ms)
wallclock/dfa/accepts/word_100000937.29 µs (937.18 µs … 937.40 µs)922.82 µs (904.41 µs … 941.22 µs)
wallclock/dfa/minimize/redundant_40001.41 ms (1.41 ms … 1.41 ms)1.43 ms (1.42 ms … 1.44 ms)
wallclock/dpa/complement/scrambled_40003.60 ms (3.58 ms … 3.63 ms)3.73 ms (3.72 ms … 3.73 ms)
wallclock/dpa/normalized/scrambled_40003.41 ms (3.41 ms … 3.42 ms)3.11 ms (3.10 ms … 3.12 ms)
wallclock/hoa/parse/ltl1.75 ms (1.74 ms … 1.76 ms)878.48 µs (876.53 µs … 880.43 µs)
wallclock/nfa/determinize/nth_from_end_123.43 ms (3.39 ms … 3.46 ms)3.19 ms (3.18 ms … 3.19 ms)
wallclock/nfa/subset_construction/nth_from_end_128.85 ms (8.75 ms … 8.95 ms)7.55 ms (7.49 ms … 7.62 ms)
wallclock/scc/tarjan/scrambled_100002.58 ms (2.53 ms … 2.64 ms)2.26 ms (2.24 ms … 2.28 ms)

Versus Spot and Oink

dpa/equivalence

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot142.10 µs115.32 µs1.23×– / –yes
hard/l-nba-2spot66.10 µs36.90 µs1.79×– / –yes
hard/m-nba-3spot271.10 µs138.06 µs1.96×– / –yes
ltl/and-gf-3spot10.40 µs12.00 µs0.87×– / –yes
ltl/eh-patterns-0spot26.96 µs20.52 µs1.31×– / –yes
ltl/gf-equiv-3spot24.72 µs34.60 µs0.71×– / –yes
ltl/gf-implies-3spot13.04 µs19.62 µs0.66×– / –yes
ltl/gh-q-3spot60.28 µs29.80 µs2.02×– / –yes
ltl/gh-r-3spot26.44 µs69.76 µs0.38×– / –yes
ltl/go-theta-3spot12.88 µs23.62 µs0.55×– / –yes
ltl/lily-patterns-1spot13.96 µs16.70 µs0.84×– / –yes
ltl/ms-phi-r-1spot12.32 µs38.54 µs0.32×– / –yes
ltl/ms-phi-s-1spot12.24 µs39.24 µs0.31×– / –yes
ltl/rv-counter-3spot30.78 µs13.58 µs2.27×– / –yes
ltl/sb-patterns-0spot14.76 µs14.08 µs1.05×– / –yes
ltl/sejk-k-2spot16.10 µs58.18 µs0.28×– / –yes
ltl/tv-uu-3spot46.84 µs37.96 µs1.23×– / –yes
ltl/u-left-3spot12.04 µs8.64 µs1.39×– / –yes
many_aps/and-gf-12spot179.40 µs66.00 µs2.72×– / –yes
many_aps/gf-equiv-12spot149.06 µs92.36 µs1.61×– / –yes
many_aps/r-right-10spot74.56 µs26.16 µs2.85×– / –yes
many_aps/random-n16-a8-p4-s1spot309.28 µs131.14 µs2.36×– / –yes
many_aps/random-n24-a12-p4-s1spot1.71 ms464.57 µs3.68×– / –yes
many_aps/random-n32-a16-p4-s1spot5.22 ms1.25 ms4.17×– / –yes
random/n128-a2-p6-s1spot365.14 µs212.50 µs1.72×– / –yes
random/n16-a1-p3-s1spot24.24 µs15.68 µs1.55×– / –yes
random/n256-a2-p8-s1spot988.29 µs489.85 µs2.02×– / –yes
random/n64-a2-p4-s1spot156.12 µs100.06 µs1.56×– / –yes

dpa/inclusion

inputreferenceoursreferenceours / referencesize (ours / reference)same result
hard/l-dsa-3spot78.86 µs61.62 µs1.28×– / –yes
hard/l-nba-2spot39.20 µs23.54 µs1.67×– / –yes
hard/m-nba-3spot163.52 µs85.96 µs1.90×– / –yes
ltl/and-gf-3spot6.76 µs8.24 µs0.82×– / –yes
ltl/eh-patterns-0spot17.52 µs17.64 µs0.99×– / –yes
ltl/gf-equiv-3spot14.02 µs19.10 µs0.73×– / –yes
ltl/gf-implies-3spot7.28 µs11.54 µs0.63×– / –yes
ltl/gh-q-3spot38.42 µs28.98 µs1.33×– / –yes
ltl/gh-r-3spot14.44 µs34.40 µs0.42×– / –yes
ltl/go-theta-3spot8.36 µs15.92 µs0.53×– / –yes
ltl/lily-patterns-1spot9.46 µs13.52 µs0.70×– / –yes
ltl/ms-phi-r-1spot7.36 µs19.28 µs0.38×– / –yes
ltl/ms-phi-s-1spot7.52 µs25.60 µs0.29×– / –yes
ltl/rv-counter-3spot20.00 µs14.64 µs1.37×– / –yes
ltl/sb-patterns-0spot9.40 µs12.56 µs0.75×– / –yes
ltl/sejk-k-2spot9.36 µs28.88 µs0.32×– / –yes
ltl/tv-uu-3spot32.38 µs31.10 µs1.04×– / –yes
ltl/u-left-3spot8.48 µs9.48 µs0.89×– / –yes
many_aps/and-gf-12spot149.90 µs54.30 µs2.76×– / –yes
many_aps/gf-equiv-12spot84.42 µs52.54 µs1.61×– / –yes
many_aps/r-right-10spot54.52 µs24.80 µs2.20×– / –yes
many_aps/random-n16-a8-p4-s1spot266.60 µs134.18 µs1.99×– / –yes
many_aps/random-n24-a12-p4-s1spot1.58 ms375.78 µs4.20×– / –yes
many_aps/random-n32-a16-p4-s1spot4.49 ms1.10 ms4.07×– / –yes
random/n128-a2-p6-s1spot215.60 µs128.58 µs1.68×– / –yes
random/n16-a1-p3-s1spot15.06 µs10.44 µs1.44×– / –yes
random/n256-a2-p8-s1spot560.35 µs303.14 µs1.85×– / –yes
random/n64-a2-p4-s1spot92.74 µs63.00 µs1.47×– / –yes

game/solve

inputreferenceoursreferenceours / referencesize (ours / reference)same result
ladder_10000oink-ppp2.94 ms331.00 µs8.90×10000 / 10000yes
ladder_10000oink-zlk2.94 ms328.00 µs8.98×10000 / 10000yes
ladder_10000spot2.94 ms10.21 ms0.29×10000 / 10000yes
ladder_50000oink-ppp22.04 ms2.15 ms10.26×50000 / 50000yes
ladder_50000oink-zlk22.04 ms2.18 ms10.10×50000 / 50000yes
ladder_50000spot22.04 ms77.16 ms0.29×50000 / 50000yes
random_few_10000oink-ppp7.69 ms1.46 ms5.26×10000 / 10000yes
random_few_10000oink-zlk7.69 ms2.69 ms2.86×10000 / 10000yes
random_few_10000spot7.69 ms21.72 ms0.35×10000 / 10000yes
random_few_50000oink-ppp80.26 ms16.44 ms4.88×50000 / 50000yes
random_few_50000oink-zlk80.26 ms19.88 ms4.04×50000 / 50000yes
random_few_50000spot80.26 ms686.55 ms0.12×50000 / 50000yes
random_many_10000oink-ppp7.34 ms1.72 ms4.27×10000 / 10000yes
random_many_10000oink-zlk7.34 ms3.03 ms2.42×10000 / 10000yes
random_many_10000spot7.34 ms30.17 ms0.24×10000 / 10000yes
random_many_50000oink-ppp82.49 ms41.43 ms1.99×50000 / 50000yes
random_many_50000oink-zlk82.49 ms44.98 ms1.83×50000 / 50000yes
random_many_50000spot82.49 ms716.79 ms0.12×50000 / 50000yes

Log

  Executable benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-f164d05108dbce73)
  Executable benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-85a152aa046a29b4)
  Executable benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-1b22c641fd9d2d89)
  Executable benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-f95e3d4a8c506941)
  Executable benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-4948f6cc6a43d201)
  Executable benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-93fb383f608d3bab)
== head: running scc
    Finished `bench` profile [optimized] target(s) in 0.38s
     Running benches/scc.rs (/var/lib/automata-bench/work/target-head/release/deps/scc-1b22c641fd9d2d89)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running ts
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/ts.rs (/var/lib/automata-bench/work/target-head/release/deps/ts-4948f6cc6a43d201)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running runs
    Finished `bench` profile [optimized] target(s) in 0.20s
     Running benches/runs.rs (/var/lib/automata-bench/work/target-head/release/deps/runs-85a152aa046a29b4)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dfa
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/dfa.rs (/var/lib/automata-bench/work/target-head/release/deps/dfa-a84ba41f760e70ed)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running dpa
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/dpa.rs (/var/lib/automata-bench/work/target-head/release/deps/dpa-23bb829f074058ad)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running storage
    Finished `bench` profile [optimized] target(s) in 0.23s
     Running benches/storage.rs (/var/lib/automata-bench/work/target-head/release/deps/storage-f95e3d4a8c506941)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running nondeterministic
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/nondeterministic.rs (/var/lib/automata-bench/work/target-head/release/deps/nondeterministic-f164d05108dbce73)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running learning
    Finished `bench` profile [optimized] target(s) in 0.20s
     Running benches/learning.rs (/var/lib/automata-bench/work/target-head/release/deps/learning-ee53c00379271c59)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running game
    Finished `bench` profile [optimized] target(s) in 0.22s
     Running benches/game.rs (/var/lib/automata-bench/work/target-head/release/deps/game-14422f732ec85029)
gungraun_runner: Warn : The RUST_LOG environment variable to set the log level changed to GUNGRAUN_LOG
== head: running wallclock (wall clock)
    Finished `bench` profile [optimized] target(s) in 0.23s
     Running benches/wallclock.rs (/var/lib/automata-bench/work/target-head/release/deps/wallclock-93fb383f608d3bab)
Gnuplot not found, using plotters backend
Benchmarking wallclock/scc/tarjan/scrambled_10000
Benchmarking wallclock/scc/tarjan/scrambled_10000: Analyzing
Benchmarking wallclock/dpa/normalized/scrambled_4000
Benchmarking wallclock/dpa/normalized/scrambled_4000: Analyzing
Benchmarking wallclock/dpa/complement/scrambled_4000
Benchmarking wallclock/dpa/complement/scrambled_4000: Analyzing
Benchmarking wallclock/dfa/minimize/redundant_4000
Benchmarking wallclock/dfa/minimize/redundant_4000: Analyzing
Benchmarking wallclock/dfa/accepts/word_100000
Benchmarking wallclock/dfa/accepts/word_100000: Analyzing
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12
Benchmarking wallclock/nfa/subset_construction/nth_from_end_12: Analyzing
Benchmarking wallclock/nfa/determinize/nth_from_end_12
Benchmarking wallclock/nfa/determinize/nth_from_end_12: Analyzing
Benchmarking wallclock/hoa/parse/ltl
Benchmarking wallclock/hoa/parse/ltl: Analyzing
Benchmarking collect/clone/100000
Benchmarking collect/clone/100000: Analyzing
Benchmarking collect/collect_owned/100000
Benchmarking collect/collect_owned/100000: Analyzing
Benchmarking collect/collect_ref/100000
Benchmarking collect/collect_ref/100000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/100000
Benchmarking collect/map_edge_colors_collect_owned/100000: Analyzing
Benchmarking collect/clone/1000000
Benchmarking collect/clone/1000000: Analyzing
Benchmarking collect/collect_owned/1000000
Benchmarking collect/collect_owned/1000000: Analyzing
Benchmarking collect/collect_ref/1000000
Benchmarking collect/collect_ref/1000000: Analyzing
Benchmarking collect/map_edge_colors_collect_owned/1000000
Benchmarking collect/map_edge_colors_collect_owned/1000000: Analyzing
== head: building the Python bindings
 Downloading crates ...
  Downloaded redox_syscall v0.5.18
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support
🐍 Found CPython 3.13 at /nix/store/vilynzly1pyhawy04ljmrp7n9ffr24qy-python3-3.13.17/bin/python3
📡 Using build options features from pyproject.toml
   Compiling dashu-int v0.6.2
   Compiling num-modular v0.6.5
   Compiling crossbeam-utils v0.8.23
   Compiling proc-macro-error-attr v1.0.4
   Compiling static_assertions v1.1.0
   Compiling crossbeam-epoch v0.9.21
   Compiling proc-macro-error v1.0.4
   Compiling nanorand v0.8.0
   Compiling crossbeam-deque v0.8.8
   Compiling parking_lot_core v0.9.12
   Compiling litrs v1.0.1
   Compiling rayon-core v1.13.0
   Compiling scopeguard v1.2.0
   Compiling lock_api v0.4.14
   Compiling sptr v0.3.2
   Compiling hugealloc v0.1.1
   Compiling parking_lot v0.12.5
   Compiling oxidd-derive v0.13.0
   Compiling document-features v0.2.12
   Compiling dashu-base v0.6.1
   Compiling num-integer v0.1.47
   Compiling linear-hashtbl v0.1.4
   Compiling derive-where v1.7.0
   Compiling rustc-hash v1.1.0
   Compiling ndarray v0.17.2
   Compiling hoars v0.5.0
   Compiling rayon v1.12.0
   Compiling oxidd-core v0.13.0
   Compiling oxidd-dump v0.8.0
   Compiling oxidd-cache v0.13.0
   Compiling oxidd-rules-bdd v0.13.0
   Compiling oxidd-manager-index v0.13.1
   Compiling oxidd v0.13.0
   Compiling automata v0.2.1 (/var/lib/automata-bench/work/src-head)
   Compiling numpy v0.27.1
   Compiling pyo3-stub-gen v0.23.2
   Compiling automata-py v0.1.0 (/var/lib/automata-bench/work/src-head/python)
    Finished `release` profile [optimized] target(s) in 2m 21s
📦 Built wheel for abi3 Python ≥ 3.11 to /var/lib/automata-bench/work/versus-head/wheels/automatars-0.1.0-cp311-abi3-manylinux_2_34_aarch64.whl
== head: running benches/versus/compare.py (Spot, Oink)
dpa/equivalence/hard/l-dsa-3 vs spot: 0.142 ms / 0.115 ms
dpa/equivalence/hard/l-nba-2 vs spot: 0.066 ms / 0.037 ms
dpa/equivalence/hard/m-nba-3 vs spot: 0.271 ms / 0.138 ms
dpa/equivalence/ltl/and-gf-3 vs spot: 0.010 ms / 0.012 ms
dpa/equivalence/ltl/eh-patterns-0 vs spot: 0.027 ms / 0.021 ms
dpa/equivalence/ltl/gf-equiv-3 vs spot: 0.025 ms / 0.035 ms
dpa/equivalence/ltl/gf-implies-3 vs spot: 0.013 ms / 0.020 ms
dpa/equivalence/ltl/gh-q-3 vs spot: 0.060 ms / 0.030 ms
dpa/equivalence/ltl/gh-r-3 vs spot: 0.026 ms / 0.070 ms
dpa/equivalence/ltl/go-theta-3 vs spot: 0.013 ms / 0.024 ms
dpa/equivalence/ltl/lily-patterns-1 vs spot: 0.014 ms / 0.017 ms
dpa/equivalence/ltl/ms-phi-r-1 vs spot: 0.012 ms / 0.039 ms
dpa/equivalence/ltl/ms-phi-s-1 vs spot: 0.012 ms / 0.039 ms
dpa/equivalence/ltl/rv-counter-3 vs spot: 0.031 ms / 0.014 ms
dpa/equivalence/ltl/sb-patterns-0 vs spot: 0.015 ms / 0.014 ms
dpa/equivalence/ltl/sejk-k-2 vs spot: 0.016 ms / 0.058 ms
dpa/equivalence/ltl/tv-uu-3 vs spot: 0.047 ms / 0.038 ms
dpa/equivalence/ltl/u-left-3 vs spot: 0.012 ms / 0.009 ms
dpa/equivalence/many_aps/and-gf-12 vs spot: 0.179 ms / 0.066 ms
dpa/equivalence/many_aps/gf-equiv-12 vs spot: 0.149 ms / 0.092 ms
dpa/equivalence/many_aps/r-right-10 vs spot: 0.075 ms / 0.026 ms
dpa/equivalence/many_aps/random-n16-a8-p4-s1 vs spot: 0.309 ms / 0.131 ms
dpa/equivalence/many_aps/random-n24-a12-p4-s1 vs spot: 1.709 ms / 0.465 ms
dpa/equivalence/many_aps/random-n32-a16-p4-s1 vs spot: 5.223 ms / 1.253 ms
dpa/equivalence/random/n128-a2-p6-s1 vs spot: 0.365 ms / 0.213 ms
dpa/equivalence/random/n16-a1-p3-s1 vs spot: 0.024 ms / 0.016 ms
dpa/equivalence/random/n256-a2-p8-s1 vs spot: 0.988 ms / 0.490 ms
dpa/equivalence/random/n64-a2-p4-s1 vs spot: 0.156 ms / 0.100 ms
dpa/inclusion/hard/l-dsa-3 vs spot: 0.079 ms / 0.062 ms
dpa/inclusion/hard/l-nba-2 vs spot: 0.039 ms / 0.024 ms
dpa/inclusion/hard/m-nba-3 vs spot: 0.164 ms / 0.086 ms
dpa/inclusion/ltl/and-gf-3 vs spot: 0.007 ms / 0.008 ms
dpa/inclusion/ltl/eh-patterns-0 vs spot: 0.018 ms / 0.018 ms
dpa/inclusion/ltl/gf-equiv-3 vs spot: 0.014 ms / 0.019 ms
dpa/inclusion/ltl/gf-implies-3 vs spot: 0.007 ms / 0.012 ms
dpa/inclusion/ltl/gh-q-3 vs spot: 0.038 ms / 0.029 ms
dpa/inclusion/ltl/gh-r-3 vs spot: 0.014 ms / 0.034 ms
dpa/inclusion/ltl/go-theta-3 vs spot: 0.008 ms / 0.016 ms
dpa/inclusion/ltl/lily-patterns-1 vs spot: 0.009 ms / 0.014 ms
dpa/inclusion/ltl/ms-phi-r-1 vs spot: 0.007 ms / 0.019 ms
dpa/inclusion/ltl/ms-phi-s-1 vs spot: 0.008 ms / 0.026 ms
dpa/inclusion/ltl/rv-counter-3 vs spot: 0.020 ms / 0.015 ms
dpa/inclusion/ltl/sb-patterns-0 vs spot: 0.009 ms / 0.013 ms
dpa/inclusion/ltl/sejk-k-2 vs spot: 0.009 ms / 0.029 ms
dpa/inclusion/ltl/tv-uu-3 vs spot: 0.032 ms / 0.031 ms
dpa/inclusion/ltl/u-left-3 vs spot: 0.008 ms / 0.009 ms
dpa/inclusion/many_aps/and-gf-12 vs spot: 0.150 ms / 0.054 ms
dpa/inclusion/many_aps/gf-equiv-12 vs spot: 0.084 ms / 0.053 ms
dpa/inclusion/many_aps/r-right-10 vs spot: 0.055 ms / 0.025 ms
dpa/inclusion/many_aps/random-n16-a8-p4-s1 vs spot: 0.267 ms / 0.134 ms
dpa/inclusion/many_aps/random-n24-a12-p4-s1 vs spot: 1.578 ms / 0.376 ms
dpa/inclusion/many_aps/random-n32-a16-p4-s1 vs spot: 4.488 ms / 1.103 ms
dpa/inclusion/random/n128-a2-p6-s1 vs spot: 0.216 ms / 0.129 ms
dpa/inclusion/random/n16-a1-p3-s1 vs spot: 0.015 ms / 0.010 ms
dpa/inclusion/random/n256-a2-p8-s1 vs spot: 0.560 ms / 0.303 ms
dpa/inclusion/random/n64-a2-p4-s1 vs spot: 0.093 ms / 0.063 ms
game/solve/random_few_10000 vs spot: 7.693 ms / 21.725 ms
game/solve/random_few_10000 vs oink-zlk: 7.693 ms / 2.686 ms
game/solve/random_few_10000 vs oink-ppp: 7.693 ms / 1.462 ms
game/solve/random_few_50000 vs spot: 80.261 ms / 686.555 ms
game/solve/random_few_50000 vs oink-zlk: 80.261 ms / 19.877 ms
game/solve/random_few_50000 vs oink-ppp: 80.261 ms / 16.440 ms
game/solve/random_many_10000 vs spot: 7.338 ms / 30.174 ms
game/solve/random_many_10000 vs oink-zlk: 7.338 ms / 3.034 ms
game/solve/random_many_10000 vs oink-ppp: 7.338 ms / 1.717 ms
game/solve/random_many_50000 vs spot: 82.487 ms / 716.789 ms
game/solve/random_many_50000 vs oink-zlk: 82.487 ms / 44.978 ms
game/solve/random_many_50000 vs oink-ppp: 82.487 ms / 41.426 ms
game/solve/ladder_10000 vs spot: 2.945 ms / 10.207 ms
game/solve/ladder_10000 vs oink-zlk: 2.945 ms / 0.328 ms
game/solve/ladder_10000 vs oink-ppp: 2.945 ms / 0.331 ms
game/solve/ladder_50000 vs spot: 22.038 ms / 77.160 ms
game/solve/ladder_50000 vs oink-zlk: 22.038 ms / 2.182 ms
game/solve/ladder_50000 vs oink-ppp: 22.038 ms / 2.148 ms
== head: 259 benchmarks, 16 wall-clock measurements, 74 comparisons