A benchmark is a measurement or a set of measurements related to the performance of a piece of code in an application.
To work with BenchmarkDotNet you must install the BenchmarkDotNet package for benchmarking, see this webpage.
1 2 3 4 5 6 | [mythcat@desk ~]$ cd CSharpProjects/ [mythcat@desk CSharpProjects]$ cd HelloWorld/ [mythcat@desk HelloWorld]$ dotnet add package BenchmarkDotNet --version 0.13.0 Determining projects to restore... ... log : Restored /home/mythcat/CSharpProjects/HelloWorld/HelloWorld.csproj (in 47.99 sec). |
The default source code of HelloWord is this:
1 2 3 4 5 6 7 8 9 10 11 12 13 | using System; namespace HelloWorld { class Program { static void Main(string[] args) { Console.WriteLine("Hello World!"); } } } |
Let write methods for benchmarking, see the final source code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | using System; using System.Collections.Generic; using BenchmarkDotNet.Attributes; using BenchmarkDotNet.Running; namespace HelloWorld { [MemoryDiagnoser] public class MemoryBenchmarkerDemo { int NumberOfItems = 100000; [Benchmark] public string ConcatStringsUsingGenericList() { var list = new List(NumberOfItems); for (int i = 0; i < NumberOfItems; i++) { list.Add("Hello World!" + i); } return list.ToString(); } } class Program { static void Main(string[] args) { var summary = BenchmarkRunner.Run(); Console.WriteLine("Hello World!"); } } } |
Let’s test it:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 | [mythcat@desk HelloWorld]$ dotnet run -p HelloWorld.csproj -c Release // Validating benchmarks: // ***** BenchmarkRunner: Start ***** // ***** Found 1 benchmark(s) in total ***** // ***** Building 1 exe(s) in Parallel: Start ***** // start dotnet restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in /home/mythcat/CSharpProjects/HelloWorld/bin/Release/netcoreapp3.1/ 2a9ff64c-fd5d-4385-ab86-e92ae9d97514 // command took 2.12s and exited with 0 // start dotnet build -c Release --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in /home/mythcat/CSharpProjects/HelloWorld/bin/Release/netcoreapp3.1/ 2a9ff64c-fd5d-4385-ab86-e92ae9d97514 // command took 4.14s and exited with 0 // ***** Done, took 00:00:06 (6.78 sec) ***** // Found 1 benchmarks: // MemoryBenchmarkerDemo.ConcatStringsUsingGenericList: DefaultJob // ************************** // Benchmark: MemoryBenchmarkerDemo.ConcatStringsUsingGenericList: DefaultJob // *** Execute *** // Launch: 1 / 1 // Execute: dotnet "2a9ff64c-fd5d-4385-ab86-e92ae9d97514.dll" --benchmarkName "HelloWorld.MemoryBenchmarkerDemo.ConcatStringsUsingGenericList" --job "Default" --benchmarkId 0 in /home/mythcat/CSharpProjects/HelloWorld/bin/Release/netcoreapp3.1/ 2a9ff64c-fd5d-4385-ab86-e92ae9d97514/bin/Release/netcoreapp3.1 Failed to set up high priority. Make sure you have the right permissions. Message: Permission denied // BeforeAnythingElse // Benchmark Process Environment Information: // Runtime=.NET Core 3.1.15 (CoreCLR 4.700.21.21202, CoreFX 4.700.21.21402), X64 RyuJIT // GC=Concurrent Workstation // Job: DefaultJob OverheadJitting 1: 1 op, 724515.00 ns, 724.5150 us/op WorkloadJitting 1: 1 op, 34830191.00 ns, 34.8302 ms/op WorkloadPilot 1: 2 op, 64947361.00 ns, 32.4737 ms/op WorkloadPilot 2: 3 op, 161391817.00 ns, 53.7973 ms/op WorkloadPilot 3: 4 op, 132641035.00 ns, 33.1603 ms/op WorkloadPilot 4: 5 op, 157075089.00 ns, 31.4150 ms/op WorkloadPilot 5: 6 op, 182932013.00 ns, 30.4887 ms/op WorkloadPilot 6: 7 op, 253514941.00 ns, 36.2164 ms/op WorkloadPilot 7: 8 op, 291452294.00 ns, 36.4315 ms/op WorkloadPilot 8: 9 op, 348057030.00 ns, 38.6730 ms/op WorkloadPilot 9: 10 op, 361127393.00 ns, 36.1127 ms/op WorkloadPilot 10: 11 op, 407032892.00 ns, 37.0030 ms/op WorkloadPilot 11: 12 op, 452471480.00 ns, 37.7060 ms/op WorkloadPilot 12: 13 op, 554754183.00 ns, 42.6734 ms/op WorkloadWarmup 1: 13 op, 537008087.00 ns, 41.3083 ms/op WorkloadWarmup 2: 13 op, 457423103.00 ns, 35.1864 ms/op WorkloadWarmup 3: 13 op, 537614510.00 ns, 41.3550 ms/op WorkloadWarmup 4: 13 op, 547335433.00 ns, 42.1027 ms/op WorkloadWarmup 5: 13 op, 543365392.00 ns, 41.7973 ms/op WorkloadWarmup 6: 13 op, 470312080.00 ns, 36.1779 ms/op WorkloadWarmup 7: 13 op, 474495467.00 ns, 36.4997 ms/op WorkloadWarmup 8: 13 op, 466255110.00 ns, 35.8658 ms/op // BeforeActualRun WorkloadActual 1: 13 op, 638447602.00 ns, 49.1114 ms/op WorkloadActual 2: 13 op, 461682390.00 ns, 35.5140 ms/op WorkloadActual 3: 13 op, 464323732.00 ns, 35.7172 ms/op WorkloadActual 4: 13 op, 464812800.00 ns, 35.7548 ms/op WorkloadActual 5: 13 op, 539852407.00 ns, 41.5271 ms/op WorkloadActual 6: 13 op, 456135113.00 ns, 35.0873 ms/op WorkloadActual 7: 13 op, 505113832.00 ns, 38.8549 ms/op ... WorkloadActual 96: 13 op, 460897296.00 ns, 35.4536 ms/op WorkloadActual 97: 13 op, 540030521.00 ns, 41.5408 ms/op WorkloadActual 98: 13 op, 464892330.00 ns, 35.7609 ms/op WorkloadActual 99: 13 op, 467877398.00 ns, 35.9906 ms/op WorkloadActual 100: 13 op, 544304107.00 ns, 41.8695 ms/op // AfterActualRun WorkloadResult 1: 13 op, 638447602.00 ns, 49.1114 ms/op WorkloadResult 2: 13 op, 461682390.00 ns, 35.5140 ms/op WorkloadResult 3: 13 op, 464323732.00 ns, 35.7172 ms/op WorkloadResult 4: 13 op, 464812800.00 ns, 35.7548 ms/op WorkloadResult 5: 13 op, 539852407.00 ns, 41.5271 ms/op WorkloadResult 6: 13 op, 456135113.00 ns, 35.0873 ms/op WorkloadResult 7: 13 op, 505113832.00 ns, 38.8549 ms/op ... WorkloadResult 94: 13 op, 464892330.00 ns, 35.7609 ms/op WorkloadResult 95: 13 op, 467877398.00 ns, 35.9906 ms/op WorkloadResult 96: 13 op, 544304107.00 ns, 41.8695 ms/op GC: 21 9 4 124799072 13 Threading: 2 0 13 // AfterAll // Benchmark Process 11144 has exited with code 0 Mean = 39.531 ms, StdErr = 0.327 ms (0.83%), N = 96, StdDev = 3.208 ms Min = 35.087 ms, Q1 = 36.190 ms, Median = 39.830 ms, Q3 = 41.849 ms, Max = 49.111 ms IQR = 5.659 ms, LowerFence = 27.702 ms, UpperFence = 50.337 ms ConfidenceInterval = [38.419 ms; 40.643 ms] (CI 99.9%), Margin = 1.112 ms (2.81% of Mean) Skewness = 0.35, Kurtosis = 2.5, MValue = 3.87 // ***** BenchmarkRunner: Finish ***** // * Export * BenchmarkDotNet.Artifacts/results/HelloWorld.MemoryBenchmarkerDemo-report.csv BenchmarkDotNet.Artifacts/results/HelloWorld.MemoryBenchmarkerDemo-report-github.md BenchmarkDotNet.Artifacts/results/HelloWorld.MemoryBenchmarkerDemo-report.html // * Detailed results * MemoryBenchmarkerDemo.ConcatStringsUsingGenericList: DefaultJob Runtime = .NET Core 3.1.15 (CoreCLR 4.700.21.21202, CoreFX 4.700.21.21402), X64 RyuJIT; GC = Concurrent Workstation Mean = 39.531 ms, StdErr = 0.327 ms (0.83%), N = 96, StdDev = 3.208 ms Min = 35.087 ms, Q1 = 36.190 ms, Median = 39.830 ms, Q3 = 41.849 ms, Max = 49.111 ms IQR = 5.659 ms, LowerFence = 27.702 ms, UpperFence = 50.337 ms ConfidenceInterval = [38.419 ms; 40.643 ms] (CI 99.9%), Margin = 1.112 ms (2.81% of Mean) Skewness = 0.35, Kurtosis = 2.5, MValue = 3.87 -------------------- Histogram -------------------- [35.033 ms ; 36.872 ms) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ [36.872 ms ; 39.110 ms) | @@@@@@@@@@@@@ [39.110 ms ; 41.182 ms) | @@ [41.182 ms ; 43.021 ms) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ [43.021 ms ; 44.379 ms) | @@@@@ [44.379 ms ; 46.000 ms) | @ [46.000 ms ; 48.010 ms) | [48.010 ms ; 50.031 ms) | @@ --------------------------------------------------- // * Summary * BenchmarkDotNet=v0.13.0, OS=fedora 34 Intel Celeron CPU G1620 2.70GHz, 1 CPU, 2 logical and 2 physical cores .NET SDK=5.0.203 [Host] : .NET Core 3.1.15 (CoreCLR 4.700.21.21202, CoreFX 4.700.21.21402), X64 RyuJIT DefaultJob : .NET Core 3.1.15 (CoreCLR 4.700.21.21202, CoreFX 4.700.21.21402), X64 RyuJIT | Method | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated | |------------------------------ |---------:|---------:|---------:|----------:|---------:| ---------:|----------:| | ConcatStringsUsingGenericList | 39.53 ms | 1.112 ms | 3.208 ms | 1615.3846 | 692.3077 | 307.6923 | 9 MB | // * Warnings * MultimodalDistribution MemoryBenchmarkerDemo.ConcatStringsUsingGenericList: Default -> It seems that the distribution is bimodal (mValue = 3.87) // * Hints * Outliers MemoryBenchmarkerDemo.ConcatStringsUsingGenericList: Default -> 4 outliers were removed (51.93 ms..60.24 ms) // * Legends * Mean : Arithmetic mean of all measurements Error : Half of 99.9% confidence interval StdDev : Standard deviation of all measurements Gen 0 : GC Generation 0 collects per 1000 operations Gen 1 : GC Generation 1 collects per 1000 operations Gen 2 : GC Generation 2 collects per 1000 operations Allocated : Allocated memory per single operation (managed only, inclusive, 1KB = 1024B) 1 ms : 1 Millisecond (0.001 sec) // * Diagnostic Output - MemoryDiagnoser * // ***** BenchmarkRunner: End ***** // ** Remained 0 benchmark(s) to run ** Run time: 00:01:02 (62.14 sec), executed benchmarks: 1 Global total time: 00:01:08 (68.94 sec), executed benchmarks: 1 // * Artifacts cleanup * Hello World! |
You can see information related to the performance of the application.