Added new scan executor metric #5986
Conversation
keith-turner
left a comment
keith-turner
left a comment
There was a problem hiding this comment.
Would be nice to add an IT for this. Could look at the ITs that use the class TestStatsDSink for examples of how to create an IT that reads metrics.
Looked around for an IT that exercises scan executors and did not see one, its a purely performance feature and hard to test. Can look at these docs for info on setting up scan executors. Need to do something like the following in the IT.
- Before the cluster starts set some
tserver.scan.executors.<executor name>.threads=<num>props in the site config to create different named scan executors. - Create a few tables and set them to have different scan executors by setting the
table.scan.dispatcher.opts.executor=<executor name>props. - Run a scan against a table w/ an iterator that will throw an exception. Check the metrics for errors for the executor the table was configured to use.
In a way this test will also help us verify that scan dispatching is working correctly and sending things to the expected executor on the server side.
core/src/main/java/org/apache/accumulo/core/metrics/MetricsProducer.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/accumulo/core/metrics/MetricsProducer.java
Show resolved
Hide resolved
server/tserver/src/main/java/org/apache/accumulo/tserver/scan/NextBatchTask.java
Outdated
Show resolved
Hide resolved
server/tserver/src/main/java/org/apache/accumulo/tserver/scan/NextBatchTask.java
Show resolved
Hide resolved
|
Not a change for this PR, but its interesting how this PR is tagging a metric w/ the scan executor name. Could potentially tag all scan metrics w/ the executor name. |
|
Be sure to run the MetricsIT and other Metrics-related ITs. |
| } | ||
|
|
||
| log.info("Waiting for metrics to be published..."); | ||
| Thread.sleep(15_000); |
There was a problem hiding this comment.
This could make this test sporadically fail as it's assuming that all of the metrics will be published by this time. We have run into this before and the approach that we have taken with the other tests is to continue to get lines from the sink until we have seen the expected metrics. See MetricsIT for an example of this. You might want to perform the scan, check the metrics, then perform the batch scan and check those metrics.
|
|
||
| // Create table with BadIterator configured for scan scope | ||
| NewTableConfiguration ntc = new NewTableConfiguration(); | ||
| IteratorSetting badIterSetting = new IteratorSetting(50, "bad", BadIterator.class); |
There was a problem hiding this comment.
Might want to use the ErrorThrowingIterator instead. This would allow you to insert M entries into the table and set the ErrorThrowingIterator to throw an exception N times. Then in the test you can confirm that you see N in the metrics and get M-N entries back in the scan.
3 participants
Closes issue #5057
Introduced a new metric to track the number of exceptions that occur within the scan executor. Allows for users to monitor exceptions thrown from the iterator stack, instead of relying on parsing logs.