[WIP] Add JFR scrubbing before profile upload#10577
Draft
[WIP] Add JFR scrubbing before profile upload#10577
Conversation
322c7a1 to
51136f3
Compare
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 56 metrics, 15 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.60.0-SNAPSHOT~46507cd649, baseline=1.60.0-SNAPSHOT~5418feb23a
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.091 s) : 0, 1091331
Total [baseline] (11.302 s) : 0, 11301726
Agent [candidate] (1.1 s) : 0, 1099783
Total [candidate] (11.16 s) : 0, 11159570
section appsec
Agent [baseline] (1.273 s) : 0, 1272837
Total [baseline] (11.249 s) : 0, 11248571
Agent [candidate] (1.264 s) : 0, 1264498
Total [candidate] (11.262 s) : 0, 11261653
section iast
Agent [baseline] (1.265 s) : 0, 1265052
Total [baseline] (11.53 s) : 0, 11530408
Agent [candidate] (1.264 s) : 0, 1263800
Total [candidate] (11.436 s) : 0, 11435951
section profiling
Agent [baseline] (1.217 s) : 0, 1217187
Total [baseline] (11.282 s) : 0, 11282074
Agent [candidate] (1.227 s) : 0, 1227371
Total [candidate] (11.313 s) : 0, 11313043
gantt
title petclinic - break down per module: candidate=1.60.0-SNAPSHOT~46507cd649, baseline=1.60.0-SNAPSHOT~5418feb23a
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.237 ms) : 0, 1237
crashtracking [candidate] (1.263 ms) : 0, 1263
BytebuddyAgent [baseline] (641.645 ms) : 0, 641645
BytebuddyAgent [candidate] (645.473 ms) : 0, 645473
AgentMeter [baseline] (29.665 ms) : 0, 29665
AgentMeter [candidate] (29.829 ms) : 0, 29829
GlobalTracer [baseline] (264.627 ms) : 0, 264627
GlobalTracer [candidate] (267.446 ms) : 0, 267446
AppSec [baseline] (34.082 ms) : 0, 34082
AppSec [candidate] (34.337 ms) : 0, 34337
Debugger [baseline] (67.719 ms) : 0, 67719
Debugger [candidate] (67.874 ms) : 0, 67874
Remote Config [baseline] (649.154 µs) : 0, 649
Remote Config [candidate] (656.412 µs) : 0, 656
Telemetry [baseline] (11.248 ms) : 0, 11248
Telemetry [candidate] (11.322 ms) : 0, 11322
Flare Poller [baseline] (3.879 ms) : 0, 3879
Flare Poller [candidate] (4.789 ms) : 0, 4789
section appsec
crashtracking [baseline] (1.25 ms) : 0, 1250
crashtracking [candidate] (1.228 ms) : 0, 1228
BytebuddyAgent [baseline] (674.171 ms) : 0, 674171
BytebuddyAgent [candidate] (669.537 ms) : 0, 669537
AgentMeter [baseline] (12.405 ms) : 0, 12405
AgentMeter [candidate] (12.288 ms) : 0, 12288
GlobalTracer [baseline] (265.5 ms) : 0, 265500
GlobalTracer [candidate] (263.995 ms) : 0, 263995
AppSec [baseline] (172.542 ms) : 0, 172542
AppSec [candidate] (172.261 ms) : 0, 172261
Debugger [baseline] (69.538 ms) : 0, 69538
Debugger [candidate] (68.602 ms) : 0, 68602
Remote Config [baseline] (686.536 µs) : 0, 687
Remote Config [candidate] (683.486 µs) : 0, 683
Telemetry [baseline] (9.894 ms) : 0, 9894
Telemetry [candidate] (9.561 ms) : 0, 9561
Flare Poller [baseline] (3.86 ms) : 0, 3860
Flare Poller [candidate] (3.649 ms) : 0, 3649
IAST [baseline] (26.477 ms) : 0, 26477
IAST [candidate] (26.318 ms) : 0, 26318
section iast
crashtracking [baseline] (1.232 ms) : 0, 1232
crashtracking [candidate] (1.23 ms) : 0, 1230
BytebuddyAgent [baseline] (812.801 ms) : 0, 812801
BytebuddyAgent [candidate] (813.602 ms) : 0, 813602
AgentMeter [baseline] (11.728 ms) : 0, 11728
AgentMeter [candidate] (11.64 ms) : 0, 11640
GlobalTracer [baseline] (256.575 ms) : 0, 256575
GlobalTracer [candidate] (256.304 ms) : 0, 256304
AppSec [baseline] (34.393 ms) : 0, 34393
AppSec [candidate] (35.148 ms) : 0, 35148
Debugger [baseline] (70.086 ms) : 0, 70086
Debugger [candidate] (68.33 ms) : 0, 68330
Remote Config [baseline] (575.98 µs) : 0, 576
Remote Config [candidate] (563.131 µs) : 0, 563
Telemetry [baseline] (9.05 ms) : 0, 9050
Telemetry [candidate] (8.832 ms) : 0, 8832
Flare Poller [baseline] (3.687 ms) : 0, 3687
Flare Poller [candidate] (3.508 ms) : 0, 3508
IAST [baseline] (28.524 ms) : 0, 28524
IAST [candidate] (28.292 ms) : 0, 28292
section profiling
crashtracking [baseline] (1.247 ms) : 0, 1247
crashtracking [candidate] (1.25 ms) : 0, 1250
BytebuddyAgent [baseline] (695.92 ms) : 0, 695920
BytebuddyAgent [candidate] (702.639 ms) : 0, 702639
AgentMeter [baseline] (8.812 ms) : 0, 8812
AgentMeter [candidate] (8.873 ms) : 0, 8873
GlobalTracer [baseline] (223.59 ms) : 0, 223590
GlobalTracer [candidate] (225.161 ms) : 0, 225161
AppSec [baseline] (33.937 ms) : 0, 33937
AppSec [candidate] (34.181 ms) : 0, 34181
Debugger [baseline] (68.722 ms) : 0, 68722
Debugger [candidate] (69.909 ms) : 0, 69909
Remote Config [baseline] (666.274 µs) : 0, 666
Remote Config [candidate] (662.024 µs) : 0, 662
Telemetry [baseline] (9.242 ms) : 0, 9242
Telemetry [candidate] (9.217 ms) : 0, 9217
Flare Poller [baseline] (4.695 ms) : 0, 4695
Flare Poller [candidate] (3.834 ms) : 0, 3834
ProfilingAgent [baseline] (98.994 ms) : 0, 98994
ProfilingAgent [candidate] (99.626 ms) : 0, 99626
Profiling [baseline] (99.586 ms) : 0, 99586
Profiling [candidate] (100.221 ms) : 0, 100221
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.60.0-SNAPSHOT~46507cd649, baseline=1.60.0-SNAPSHOT~5418feb23a
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.095 s) : 0, 1094905
Total [baseline] (9.315 s) : 0, 9315056
Agent [candidate] (1.097 s) : 0, 1096534
Total [candidate] (9.303 s) : 0, 9302729
section iast
Agent [baseline] (1.266 s) : 0, 1265853
Total [baseline] (9.991 s) : 0, 9991448
Agent [candidate] (1.257 s) : 0, 1256818
Total [candidate] (9.981 s) : 0, 9980969
gantt
title insecure-bank - break down per module: candidate=1.60.0-SNAPSHOT~46507cd649, baseline=1.60.0-SNAPSHOT~5418feb23a
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.243 ms) : 0, 1243
crashtracking [candidate] (1.242 ms) : 0, 1242
BytebuddyAgent [baseline] (642.966 ms) : 0, 642966
BytebuddyAgent [candidate] (644.363 ms) : 0, 644363
AgentMeter [baseline] (29.753 ms) : 0, 29753
AgentMeter [candidate] (29.849 ms) : 0, 29849
GlobalTracer [baseline] (265.773 ms) : 0, 265773
GlobalTracer [candidate] (265.748 ms) : 0, 265748
AppSec [baseline] (34.226 ms) : 0, 34226
AppSec [candidate] (34.22 ms) : 0, 34220
Debugger [baseline] (65.396 ms) : 0, 65396
Debugger [candidate] (67.793 ms) : 0, 67793
Remote Config [baseline] (662.385 µs) : 0, 662
Remote Config [candidate] (655.096 µs) : 0, 655
Telemetry [baseline] (11.025 ms) : 0, 11025
Telemetry [candidate] (12.052 ms) : 0, 12052
Flare Poller [baseline] (7.163 ms) : 0, 7163
Flare Poller [candidate] (3.861 ms) : 0, 3861
section iast
crashtracking [baseline] (1.249 ms) : 0, 1249
crashtracking [candidate] (1.232 ms) : 0, 1232
BytebuddyAgent [baseline] (817.678 ms) : 0, 817678
BytebuddyAgent [candidate] (809.482 ms) : 0, 809482
AgentMeter [baseline] (11.733 ms) : 0, 11733
AgentMeter [candidate] (11.626 ms) : 0, 11626
GlobalTracer [baseline] (255.692 ms) : 0, 255692
GlobalTracer [candidate] (255.406 ms) : 0, 255406
AppSec [baseline] (32.54 ms) : 0, 32540
AppSec [candidate] (32.613 ms) : 0, 32613
Debugger [baseline] (69.532 ms) : 0, 69532
Debugger [candidate] (69.331 ms) : 0, 69331
Remote Config [baseline] (579.359 µs) : 0, 579
Remote Config [candidate] (557.821 µs) : 0, 558
Telemetry [baseline] (8.822 ms) : 0, 8822
Telemetry [candidate] (8.751 ms) : 0, 8751
Flare Poller [baseline] (3.552 ms) : 0, 3552
Flare Poller [candidate] (3.514 ms) : 0, 3514
IAST [baseline] (27.896 ms) : 0, 27896
IAST [candidate] (28.012 ms) : 0, 28012
LoadParameters
See matching parameters
SummaryFound 2 performance improvements and 3 performance regressions! Performance is the same for 13 metrics, 18 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~46507cd649, baseline=1.60.0-SNAPSHOT~5418feb23a
dateFormat X
axisFormat %s
section baseline
no_agent (17.673 ms) : 17494, 17852
. : milestone, 17673,
appsec (18.776 ms) : 18587, 18964
. : milestone, 18776,
code_origins (19.17 ms) : 18975, 19365
. : milestone, 19170,
iast (17.605 ms) : 17430, 17781
. : milestone, 17605,
profiling (19.581 ms) : 19384, 19777
. : milestone, 19581,
tracing (17.607 ms) : 17432, 17782
. : milestone, 17607,
section candidate
no_agent (19.276 ms) : 19076, 19475
. : milestone, 19276,
appsec (18.316 ms) : 18130, 18502
. : milestone, 18316,
code_origins (17.951 ms) : 17773, 18129
. : milestone, 17951,
iast (17.577 ms) : 17406, 17749
. : milestone, 17577,
profiling (19.011 ms) : 18824, 19199
. : milestone, 19011,
tracing (18.794 ms) : 18604, 18985
. : milestone, 18794,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~46507cd649, baseline=1.60.0-SNAPSHOT~5418feb23a
dateFormat X
axisFormat %s
section baseline
no_agent (1.167 ms) : 1156, 1178
. : milestone, 1167,
iast (3.236 ms) : 3188, 3284
. : milestone, 3236,
iast_FULL (5.862 ms) : 5802, 5921
. : milestone, 5862,
iast_GLOBAL (3.543 ms) : 3482, 3604
. : milestone, 3543,
profiling (2.152 ms) : 2132, 2173
. : milestone, 2152,
tracing (1.774 ms) : 1759, 1789
. : milestone, 1774,
section candidate
no_agent (1.166 ms) : 1155, 1178
. : milestone, 1166,
iast (3.188 ms) : 3148, 3227
. : milestone, 3188,
iast_FULL (5.963 ms) : 5902, 6024
. : milestone, 5963,
iast_GLOBAL (3.32 ms) : 3273, 3367
. : milestone, 3320,
profiling (2.089 ms) : 2070, 2107
. : milestone, 2089,
tracing (1.849 ms) : 1832, 1865
. : milestone, 1849,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~46507cd649, baseline=1.60.0-SNAPSHOT~5418feb23a
dateFormat X
axisFormat %s
section baseline
no_agent (1.47 ms) : 1459, 1482
. : milestone, 1470,
appsec (3.735 ms) : 3519, 3951
. : milestone, 3735,
iast (2.257 ms) : 2188, 2326
. : milestone, 2257,
iast_GLOBAL (2.287 ms) : 2219, 2356
. : milestone, 2287,
profiling (2.083 ms) : 2028, 2137
. : milestone, 2083,
tracing (2.049 ms) : 1996, 2102
. : milestone, 2049,
section candidate
no_agent (1.476 ms) : 1464, 1487
. : milestone, 1476,
appsec (3.707 ms) : 3491, 3922
. : milestone, 3707,
iast (2.251 ms) : 2182, 2320
. : milestone, 2251,
iast_GLOBAL (2.287 ms) : 2218, 2355
. : milestone, 2287,
profiling (2.067 ms) : 2013, 2121
. : milestone, 2067,
tracing (2.067 ms) : 2014, 2121
. : milestone, 2067,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~46507cd649, baseline=1.60.0-SNAPSHOT~5418feb23a
dateFormat X
axisFormat %s
section baseline
no_agent (15.296 s) : 15296000, 15296000
. : milestone, 15296000,
appsec (14.836 s) : 14836000, 14836000
. : milestone, 14836000,
iast (18.215 s) : 18215000, 18215000
. : milestone, 18215000,
iast_GLOBAL (17.66 s) : 17660000, 17660000
. : milestone, 17660000,
profiling (14.831 s) : 14831000, 14831000
. : milestone, 14831000,
tracing (14.827 s) : 14827000, 14827000
. : milestone, 14827000,
section candidate
no_agent (15.554 s) : 15554000, 15554000
. : milestone, 15554000,
appsec (14.605 s) : 14605000, 14605000
. : milestone, 14605000,
iast (18.266 s) : 18266000, 18266000
. : milestone, 18266000,
iast_GLOBAL (17.635 s) : 17635000, 17635000
. : milestone, 17635000,
profiling (14.734 s) : 14734000, 14734000
. : milestone, 14734000,
tracing (14.906 s) : 14906000, 14906000
. : milestone, 14906000,
|
0efec0b to
6d2f36d
Compare
bd39939 to
46507cd
Compare
The JVM only eagerly resolves superclass and interfaces during defineClass. Field types, method parameter/return types, and declared exceptions are resolved lazily. Marking them as REQUIRES created false dependency cycles that broke topological sort ordering when injecting large helper batches (2000+ classes). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scrub sensitive fields (system properties, JVM arguments, environment variables, process command lines) from JFR recordings before upload. - Add profiling-scrubber module wrapping jafar-tools Scrubber - Wire ScrubRecordingDataListener decorator into ProfilingAgent - Add RecordingData.getPath() to avoid stream materialization for file-backed recordings (ddprof) - Add config: dd.profiling.scrub.enabled, dd.profiling.scrub.fail-open, dd.profiling.scrub.event-type-excludes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Guard ThrowableInstanceAdvice during native-image build to prevent JFR event class initialization errors - Enable profiling scrubber in native image build args - Add smoke test verifying JFR files with system property events are produced (scrubbing assertion deferred until jafar handles SubstrateVM JFR chunk format) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
46507cd to
52dde93
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Adds configurable scrubbing of sensitive fields in JFR profiling recordings before upload. Sensitive data (system properties, JVM arguments, environment variables, command lines) is replaced with redacted values.
Motivation
Prevents accidental exposure of sensitive information (credentials, API keys, internal paths) in profiling data uploaded to Datadog backend.
Additional Notes
This is a proof of concept. All string values from the defined attributes will be scrubbed, there is no support for regex and similar matching. It is disabled by default and present only to be able to infer the extra overhead it would expose on a real-life system after enabling.
If the concept is validated, we will add proper value matching and enable scrubbing by default in a later PR.
profiling-scrubber module (new)
JfrScrubberwraps jafar-toolsScrubber(Java 8 compatible)DefaultScrubDefinitiondefines fields to scrub per event type:jdk.InitialSystemProperty/value,jdk.JVMInformation/jvmArguments,jdk.InitialEnvironmentVariable/value,jdk.SystemProcess/commandLinedd.profiling.scrub.event-type-excludesPipeline wiring
ScrubRecordingDataListenerdecoratesRecordingDataListener, scrubbing JFR data before delegating to the upload listenerRecordingData.getPath()(new) to avoid stream materialization for file-backed recordings (ddprof)dd.profiling.scrub.fail-open) passes unscrubbed data on scrubber errors, with null-safety guard to prevent passing already-released dataConfiguration
dd.profiling.scrub.enabled(default: false) — master switchdd.profiling.scrub.fail-open(default: false) — upload unscrubbed data on scrubber failuredd.profiling.scrub.event-type-excludes— comma-separated event types to skipNative image support
ThrowableInstanceAdviceduring native-image build to prevent JFR event class initialization errorsHelperScanner.visitField()andvisitMethod()changed fromREQUIREStoUSES.This fix was necessary to support the ~2000 jafar helper classes injected transitively via
VMRuntimeModule.injectHelperDependencies(). The previous behavior marked field types and method parameter/return types as load-time dependencies (REQUIRES), which created false dependency cycles. WhenremoveCycles()broke these, subclasses could be positioned before their superclass in the topological sort, causingNoClassDefFoundErrorduring batch class injection.The fix is correct per the JVM spec:
defineClassonly eagerly resolves superclass and interfaces (handled byvisit()). Field types, method parameter/return types, and declared exceptions are resolved lazily. However, this change affects all instrumentations that useinjectHelperDependencies(), not just profiling. If any instrumentation relies on the old (incorrect) ordering where field/method types were treated as load-time dependencies, this change could surface latent issues.Requesting review from someone familiar with
HelperScanner/HelperInjectorinternals to verify no regressions.Testing
JfrScrubberTest(scrubbing correctness, exclusion, no-op),ScrubRecordingDataListenerTest(decorator behavior, fail-open, error paths)JFRBasedProfilingIntegrationTest— verifies end-to-end scrubbing ofjdk.InitialSystemPropertyandjdk.JVMInformationin a real profiling sessionSpringBootNativeInstrumentationTest— verifies native image builds and runs with scrubbing enabledContributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: PROF-13548
Note: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.