Implement recipe map lookup time benchmarking command #2880
Conversation
ALongStringOfNumbers
left a comment
ALongStringOfNumbers
left a comment
There was a problem hiding this comment.
Running the command in game gave me a warning that 6 checks failed validation.
|
|
||
| @Override | ||
| public String getName() { | ||
| return "lookup"; |
There was a problem hiding this comment.
I feel like instead of lookup this should be run or something. Because we are not really looking up stuff about recipes, rather we are running sampling on recipes.
There was a problem hiding this comment.
This command is more dev-facing, and is a subcommand of "benchmarking". If there are more benchmarks in the future, I think the simplest name to differentiate this one from the others is "lookup", though I could also see "recipe-lookup"
src/main/java/gregtech/common/command/benchmark/CommandBenchmarkLookup.java
Show resolved
Hide resolved
src/main/java/gregtech/common/command/benchmark/CommandBenchmarkLookup.java
Outdated
Show resolved
Hide resolved
src/main/java/gregtech/common/command/benchmark/CommandBenchmarkLookup.java
Outdated
Show resolved
Hide resolved
| // run this matching while timing to penalize returning too many possible matches | ||
| for (Recipe r : out) { | ||
| if (r.matches(false, items, fluids)) { | ||
| break; |
There was a problem hiding this comment.
Shouldn't a match in this case also increment the failure count? Since we have already filtered out the list of items and fluids to not match, if now we match, that should be a failure?
There was a problem hiding this comment.
No, because our construction of inputs is only supposed to guarantee not matching within the sample. We could still match within the recipe map outside the sample, and that is expected behavior.
src/main/java/gregtech/common/command/benchmark/CommandBenchmarkLookup.java
Show resolved
Hide resolved
| if (r.matches(false, items, fluids)) { | ||
| break; | ||
| } | ||
| } |
There was a problem hiding this comment.
I am a bit confused why we have this code block for all three trials, and it is always exactly the same. In the first place, I don't really understand what it is doing, but for the three different cases, why is it always break if there is one match, when the different cases are no match, three matches, and exactly one match.
There was a problem hiding this comment.
This bit is more in the context of comparing with competitor recipe searches that return multiple possible matching recipes. This is to prevent, say, a recipe search that returns everything in the recipemap to iterate through from appearing as super time efficient.
|
|
||
| // one match exact trial | ||
| items.clear(); | ||
| fluids.clear(); |
There was a problem hiding this comment.
Should we also clear these before the three match trial? Right now it is only cleared before this trial
There was a problem hiding this comment.
The three match trial is only trying to guarantee three or more matches, and a lot of excess inputs, so for what the three match test is trying to test, there's no real point to clearing.
| new TextComponentTranslation("gregtech.command.benchmark.lookup.failures", failureCount.get()) | ||
| .setStyle(new Style().setColor(TextFormatting.RED))); | ||
| try (FileWriter writer = new FileWriter("benchmark-lookup-results.csv")) { | ||
| writer.append("Recipe Map,Trial Type,Minimum,Q1,Median,Q3,Maximum\n"); |
There was a problem hiding this comment.
Any chance we can make this fancy and have evenly spaced entries, with separators like | between the entries? If to hard, feel free to ignore.
There was a problem hiding this comment.
It's meant to be a CSV that can be imported into anything that can handle CSVs (e.g. Google Sheets) for data analysis, not human readable. The log provides human readable outputs.
What
Adds a new command that benchmarks how long it takes to do recipe lookups for every recipe map under different conditions.
Implementation Details
Takes a sample of 10 recipes from each recipe map, then queries the recipe map in ways that should match none of the recipes in the sample, match 3 or more recipes in the sample, and match exactly one recipe in the sample (and in the recipe map.)
Times a configurable number of these operations defaulting to 100, computes the five characteristic numbers, then outputs them to the log file and a csv file.
Outcome
Allows for understanding how much lookup time a given recipe map generally costs, as well as enabling sensible comparison to any alternative lookup scheme.
Additional Information
Benchmarking lookup is a subcommand of "benchmark", leaving open the option for different benchmarks to be added in the future.