update benchmark

Author: neuecc

ReadMe.md

@@ -4,7 +4,7 @@
 
 ConsoleAppFramework v5 is Zero Dependency, Zero Overhead, Zero Reflection, Zero Allocation, AOT Safe CLI Framework powered by C# Source Generator; achieves exceptionally high performance, fastest start-up time(with NativeAOT) and minimal binary size. Leveraging the latest features of .NET 8 and C# 13 ([IncrementalGenerator](https://github.com/dotnet/roslyn/blob/main/docs/features/incremental-generators.md), [managed function pointer](https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/proposals/csharp-9.0/function-pointers#function-pointers-1), [params arrays and default values lambda expression](https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/operators/lambda-expressions#input-parameters-of-a-lambda-expression), [`ISpanParsable<T>`](https://learn.microsoft.com/en-us/dotnet/api/system.ispanparsable-1), [`PosixSignalRegistration`](https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.posixsignalregistration), etc.), this library ensures maximum performance while maintaining flexibility and extensibility.
 
-![image](https://github.com/Cysharp/ConsoleAppFramework/assets/46207/db4bf599-9fe0-4ce4-801f-0003f44d5628)
+![image](https://github.com/user-attachments/assets/4a10fd2b-fa56-467f-8d7f-e47bee634753)
 > Set `RunStrategy=ColdStart WarmupCount=0` to calculate the cold start benchmark, which is suitable for CLI application.
 
 The magical performance is achieved by statically generating everything and parsing inline. Let's take a look at a minimal example:
@@ -746,8 +746,6 @@ While there are some standards for command-line arguments, such as UNIX tools an
 
 For example, specifications that change behavior based on `-x` and `-X` or allow bundling `-f -d -x` as `-fdx` are not easy to understand and also take time to parse. The poor performance of System.CommandLine may be influenced by its adherence to complex grammar. Therefore, ConsoleAppFramework prioritizes performance and clear rules. It uses lower-kebab-case as the basis while allowing case-insensitive matching. It does not support ambiguous grammar that cannot be processed in a single pass or takes time to parse.
 
-[System.CommandLine seems to be aiming for a new direction in .NET 9 and .NET 10](https://github.com/dotnet/command-line-api/issues/2338), but from a performance perspective, it will never surpass ConsoleAppFramework.
-
 CancellationToken(Gracefully Shutdown) and Timeout
 ---
 In ConsoleAppFramework, when you pass a `CancellationToken` as an argument, it can be used to check for interruption commands (SIGINT/SIGTERM/SIGKILL - Ctrl+C) rather than being treated as a parameter. For handling this, ConsoleAppFramework performs special code generation when a `CancellationToken` is included in the parameters.

sandbox/CliFrameworkBenchmark/Benchmark.cs

@@ -37,10 +37,16 @@ public ValueTask<int> ExecuteWithCliFx()
         return new CliApplicationBuilder().AddCommand(typeof(CliFxCommand)).Build().RunAsync(Arguments);
     }
 
-    [Benchmark(Description = "System.CommandLine")]
+    [Benchmark(Description = "System.CommandLine v2")]
     public int ExecuteWithSystemCommandLine()
     {
-        return new SystemCommandLineCommand().Execute(Arguments);
+        return SystemCommandLineCommand.ParseInvoke(Arguments);
+    }
+
+    [Benchmark(Description = "System.CommandLine v2(InvokeAsync)")]
+    public Task<int> ExecuteWithSystemCommandLineAsync()
+    {
+        return SystemCommandLineCommand.ParseInvokeAsync(Arguments);
     }
 
     //[Benchmark(Description = "McMaster.Extensions.CommandLineUtils")]
@@ -69,9 +75,17 @@ public int ExecuteWithSystemCommandLine()
     //}
 
     [Benchmark(Description = "ConsoleAppFramework v5", Baseline = true)]
-    public unsafe void ExecuteConsoleAppFramework()
+    public void ExecuteConsoleAppFramework()
+    {
+        ConsoleApp.Run(Arguments, ConsoleAppFrameworkCommand.Execute);
+    }
+
+    [Benchmark(Description = "ConsoleAppFramework v5(app with CancellationToken)")]
+    public Task ExecuteConsoleAppFramework2()
     {
-        ConsoleApp.Run(Arguments, &ConsoleAppFrameworkCommandWithCancellationToken.Execute);
+        var app = ConsoleApp.Create();
+        app.Add("", ConsoleAppFrameworkCommand.ExecuteWithCancellationToken);
+        return app.RunAsync(Arguments);
     }
 
     // for alpha testing

sandbox/CliFrameworkBenchmark/Commands/ConsoleAppFrameworkCommand.cs

@@ -29,19 +29,16 @@ public static void Execute(string? str, int intOption, bool boolOption)
     {
 
     }
-}
 
-public class ConsoleAppFrameworkCommandWithCancellationToken
-{
     /// <summary>
     /// 
     /// </summary>
     /// <param name="str">-s</param>
     /// <param name="intOption">-i</param>
     /// <param name="boolOption">-b</param>
-    public static void Execute(string? str, int intOption, bool boolOption, CancellationToken cancellationToken)
+    public static Task ExecuteWithCancellationToken(string? str, int intOption, bool boolOption, CancellationToken cancellationToken)
     {
-
+        return Task.CompletedTask;
     }
 }
 

sandbox/CliFrameworkBenchmark/Commands/SystemCommandLineCommand.cs

@@ -6,7 +6,7 @@ namespace Cocona.Benchmark.External.Commands;
 
 public class SystemCommandLineCommand
 {
-    public int Execute(string[] args)
+    public static int ParseInvoke(string[] args)
     {
         var stringOption = new Option<string>("--str", "-s");
         var intOption = new Option<int>("--int", "-i");
@@ -23,4 +23,23 @@ public int Execute(string[] args)
 
         return command.Parse(args).Invoke();
     }
+
+    public static Task<int> ParseInvokeAsync(string[] args)
+    {
+        var stringOption = new Option<string>("--str", "-s");
+        var intOption = new Option<int>("--int", "-i");
+        var boolOption = new Option<bool>("--bool", "-b");
+
+        var command = new RootCommand { stringOption, intOption, boolOption };
+
+        command.SetAction((parseResult, cancellationToken) =>
+        {
+            _ = parseResult.GetValue(stringOption);
+            _ = parseResult.GetValue(intOption);
+            _ = parseResult.GetValue(boolOption);
+            return Task.CompletedTask;
+        });
+
+        return command.Parse(args).InvokeAsync();
+    }
 }

sandbox/CliFrameworkBenchmark/Program.cs

@@ -18,18 +18,20 @@ static void Main(string[] args)
     {
         var config = DefaultConfig.Instance
                                   .WithSummaryStyle(SummaryStyle.Default
-                                  .WithTimeUnit(TimeUnit.Millisecond));
+                                  .WithTimeUnit(TimeUnit.Millisecond))
+                                  .HideColumns(BenchmarkDotNet.Columns.Column.Error)
+                                  ;
 
         config.AddDiagnoser(MemoryDiagnoser.Default);
-        config.AddDiagnoser(new ThreadingDiagnoser(new ThreadingDiagnoserConfig(displayLockContentionWhenZero: false, displayCompletedWorkItemCountWhenZero: false)));
+        // config.AddDiagnoser(new ThreadingDiagnoser(new ThreadingDiagnoserConfig(displayLockContentionWhenZero: false, displayCompletedWorkItemCountWhenZero: false)));
 
         config.AddJob(Job.Default
                          .WithStrategy(RunStrategy.ColdStart)
                          .WithLaunchCount(1)
                          .WithWarmupCount(0)
                          .WithIterationCount(1)
                          .WithInvocationCount(1)
-                         .WithToolchain(CsProjCoreToolchain.NetCoreApp80)
+                         .WithToolchain(CsProjCoreToolchain.NetCoreApp10_0) // .NET 10
                          .DontEnforcePowerPlan());
 
         BenchmarkRunner.Run<Benchmark>(config, args);