add benchmark section to docs

Author: mileslucas

docs/Project.toml

@@ -1,10 +1,13 @@
 [deps]
 AbstractMCMC = "80f14c24-f653-4e6a-9b94-39d6b0f70001"
+CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
+DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
 Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
 LogExpFunctions = "2ab3a3ac-af41-5b50-aa03-7779005ae688"
 MCMCChains = "c7f686f2-ff18-58e9-bc7b-31028e88f75d"
 Measurements = "eff96d63-e80a-5855-80a2-b1b0885c5ab7"
+Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
 StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
 StatsFuns = "4c63d2b9-4356-54db-8cca-17b64c39e42c"
 StatsPlots = "f3b207a7-027a-5e70-b257-86293d7955fd"

docs/make.jl

@@ -17,6 +17,7 @@ makedocs(
             "Correlated Gaussian" => "examples/correlated.md",
             "Eggbox" => "examples/eggbox.md",
         ],
+        "Benchmarks" => "benchmarks.md",
         "API/Reference" => "api.md"
     ],
     format = Documenter.HTML(prettyurls = get(ENV, "CI", nothing) == "true"),

docs/src/benchmarks.md

@@ -0,0 +1,83 @@
+# Benchmarks
+
+The following benchmarks show the performance of NestedSamplers.jl. As with any statistical inference package, the likelihood function will often dominate the runtime. This is important to consider when comparing packages across different languages- in general a custom Julia likelihood function may be faster than the same code written in Python/numpy. As an example, compare the relative timings of these two simple Guassian likelihoods
+
+```julia
+using BenchmarkTools
+using PyCall
+
+# julia version
+gauss_loglike(X) = sum(x -> exp(-0.5 * x^2) / sqrt(2π), X)
+
+# python version
+py"""
+import numpy as np
+def gauss_loglike(X):
+    return np.sum(np.exp(-0.5 * X ** 2) / np.sqrt(2 * np.pi))
+"""
+gauss_loglike_py = py"gauss_loglike"
+xs = randn(100)
+```
+
+```julia
+@btime gauss_loglike($xs)
+```
+
+```
+  611.971 ns (0 allocations: 0 bytes)
+26.813747896467206
+```
+
+```julia
+
+@btime gauss_loglike_py($xs)
+```
+
+```
+  13.129 μs (6 allocations: 240 bytes)
+26.81374789646721
+```
+
+In certain cases, you can use language interop tools (like [PyCall.jl](https://github.com/JuliaPy/PyCall.jl)) to use Julia likelihoods with Python libraries.
+
+## Setup and system information
+
+The benchmark code can be found in the [`bench`](https://github.com/TuringLang/NestedSamplers.jl/blob/main/bench/) folder. The system information at the time these benchmarks were ran is
+
+```julia
+julia> versioninfo()
+Julia Version 1.7.1
+Commit ac5cc99908* (2021-12-22 19:35 UTC)
+Platform Info:
+  OS: macOS (x86_64-apple-darwin20.5.0)
+  CPU: Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz
+  WORD_SIZE: 64
+  LIBM: libopenlibm
+  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)
+Environment:
+  JULIA_NUM_THREADS = 1
+```
+
+## Highly-correlated multivariate Guassian
+
+This benchmark uses [`Models.CorrelatedGaussian`](@ref) and simply measures the time it takes to fully sample down to `dlogz=0.01`. This benchmark is exactly the same as the benchmark detailed in the [JAXNS paper](https://ui.adsabs.harvard.edu/abs/2020arXiv201215286A/abstract).
+
+### Timing
+
+```@example sample-benchmark
+using CSV, DataFrames, Plots # hide
+benchdir = joinpath(dirname(pathof(NestedSamplers)), "..", "bench") # hide
+results = DataFrame(CSV.File(joinpath(benchdir, "sampling_results.csv"))) # hide
+plot(results.D, results.t, label="NestedSamplers.jl", marker=:o, yscale=:log10 # hide
+    ylabel="runtime (s)", xlabel="prior dimension", leg=:topleft) # hide
+```
+
+### Accuracy
+
+The following shows the Bayesian evidence estmiate as compared to the true value
+
+```@example sample-benchmark
+plot(results.D, results.dlnZ, yerr=results.lnZstd, label="NestedSamplers.jl", # hide
+    marker=:o, ylabel="ΔlnZ", xlabel="prior dimension", leg=:topleft) # hide
+hlines([0.0], c=:black, ls=:dash, alpha=0.7, label="") # hide
+```