Improve custom metrics documentation with clearer explanations #3789

AmberAlston · 2026-01-26T19:19:22Z

Summary

This PR enhances the custom metrics documentation to better help application developers understand how to implement and use custom metrics effectively.

Preview: https://deploy-preview-3789--replicated-docs.netlify.app/vendor/custom-metrics
Original page: https://docs.replicated.com/vendor/custom-metrics#overview

Changes

1. Added "How Custom Metrics Work" Section

Explains the change-detection system with concrete examples
Shows when events are created vs when they're not
Helps developers understand why they might not see events for repeated identical values

2. Added "Supported Data Types" Section

Clear list of supported types: numbers, strings, booleans, null
Examples showing valid and invalid payloads
Addresses confusion about nested objects/arrays

3. Improved "POST vs PATCH" Section

Clearer explanation with side-by-side comparison
Better examples showing state transitions
Explicit guidance on when to use each method

4. Added "Best Practices" Section

Sending frequency recommendations
Change detection efficiency tips
Method selection guidance
Metric naming conventions

5. Added "Troubleshooting" Section

"I'm not seeing any events" - explains change detection
"My metrics aren't showing up" - common setup issues
"I see duplicate events" - debugging value consistency

Motivation

These improvements address common confusion points identified through codebase analysis:

The system stores every submission but only creates events on value changes
This two-tier approach (storage vs events) wasn't clearly explained
POST vs PATCH behavior was confusing
Developers needed clearer guidance on implementation

Related Research

Based on internal codebase analysis documented in vandoor/research_custom_metrics_system.md

🤖 Generated with Claude Code

This commit enhances the custom metrics documentation to better help application developers understand how to implement and use custom metrics: - Add "How Custom Metrics Work" section explaining change detection behavior with concrete examples showing when events are created - Add "Supported Data Types" section with clear examples of valid and invalid data types (numbers, strings, booleans, null) - Improve "POST vs PATCH" section with clearer examples and explicit guidance on when to use each method - Add "Best Practices" section covering sending frequency, change detection efficiency, method selection, and naming conventions - Add "Troubleshooting" section addressing common issues like missing events, metrics not appearing, and duplicate events These changes address common confusion points identified through codebase analysis and make the documentation more actionable for developers instrumenting their applications. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

netlify · 2026-01-26T19:19:27Z

✅ Deploy Preview for replicated-docs ready!

Name	Link
🔨 Latest commit	`41ad381`
🔍 Latest deploy log	https://app.netlify.com/projects/replicated-docs/deploys/6977f90e6160330008b2cc1b
😎 Deploy Preview	https://deploy-preview-3789--replicated-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

netlify · 2026-01-26T19:19:30Z

✅ Deploy Preview for replicated-docs-upgrade ready!

Name	Link
🔨 Latest commit	`41ad381`
🔍 Latest deploy log	https://app.netlify.com/projects/replicated-docs-upgrade/deploys/6977f90e7ae79c0008467db9
😎 Deploy Preview	https://deploy-preview-3789--replicated-docs-upgrade.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

AmberAlston · 2026-01-26T19:41:59Z

docs/vendor/custom-metrics.md


 [View a larger version of this image](/images/custom-metrics-flow.png)

+Custom metrics use a change-detection system to efficiently track meaningful change events from the previous value:


So this part is a bit redundant to the click-through link about how all events are calculated, and maybe it should move there. The main thing I wanted to incorporate was more clarity to the example. Vendors have been struggling to understand how the concept of "event" (which is essentially change in value) and the idea of a "metric" that's continuously reported and stored by VP

AmberAlston · 2026-01-26T19:42:29Z

docs/vendor/custom-metrics.md

+**Not supported**: Nested objects or arrays

-For example, if a component of your application sends the following via the `POST` method:
+**Example:**


I'd like to keep this part because i have to look this up myself all the time.

AmberAlston · 2026-01-26T19:43:18Z

docs/vendor/custom-metrics.md

 ## Export Custom Metrics

 You can use the Vendor API v3 `/app/{app_id}/events` endpoint to programatically access historical timeseries data containing instance level events, including any custom metrics that you have defined. For more information about the endpoint, see [Export Customer and Instance Data](/vendor/instance-data-export).
+


Best practices and Troubleshooting clearly add a lot more content to this page, but I thought the AI suggestion to add them were actually quite helpful and so I've left them in.

I shuffled the best practices a bit so that they each live with the relevant content, rather than being in a separate Best Practices heading. For example, info about naming custom metrics in under the new "Define custom metrics" heading, and info about frequency of sending metrics lives in a subheading under "Send Custom Metrics". The other best practices were a little redundant (eg, restating the use cases for post vs patch which is already covered in "PATCH vs POST" section)

For the troubleshooting info, I reformatted so that they follow our style guide. I also made these edits:

Removed "I'm not seeing any events", for which the stated cause was "Events are only created when values change." It seemed like it wasn't really providing troubleshooting info so much as restating the info about how the Vendor Portal generates events. Also wasn't sure it was fair to state definitively that the cause for this is "Events are only created when values change."

Updated the copy to add more context and make sure everything's a full sentence

paigecalvert

thanks @AmberAlston lgtm! details on my edits in comments

paigecalvert · 2026-01-26T23:38:10Z

docs/vendor/custom-metrics.md

 ## Export Custom Metrics

 You can use the Vendor API v3 `/app/{app_id}/events` endpoint to programatically access historical timeseries data containing instance level events, including any custom metrics that you have defined. For more information about the endpoint, see [Export Customer and Instance Data](/vendor/instance-data-export).
+


I shuffled the best practices a bit so that they each live with the relevant content, rather than being in a separate Best Practices heading. For example, info about naming custom metrics in under the new "Define custom metrics" heading, and info about frequency of sending metrics lives in a subheading under "Send Custom Metrics". The other best practices were a little redundant (eg, restating the use cases for post vs patch which is already covered in "PATCH vs POST" section)

For the troubleshooting info, I reformatted so that they follow our style guide. I also made these edits:

Removed "I'm not seeing any events", for which the stated cause was "Events are only created when values change." It seemed like it wasn't really providing troubleshooting info so much as restating the info about how the Vendor Portal generates events. Also wasn't sure it was fair to state definitively that the cause for this is "Events are only created when values change."

Updated the copy to add more context and make sure everything's a full sentence

paigecalvert · 2026-01-26T23:38:36Z

docs/vendor/custom-metrics.md

+- Ensure that the application instance is using the Replicated SDK version 1.0.0-beta.12 or later. If your application is using the `PATCH` and `DELETE` methods, version 1.0.0-beta.23 or later of the SDK is required.
+- Verify that your payload only contains scalar values (no nested objects or arrays), and is valid JSON with proper content type headers
+- Test network connectivity by confirming that the application can reach `http://replicated:3000`
+- Check that your application is using the intended method (PATCH or POST). If a metric payload is sent using the POST method, any existing metrics that are not included in the payload are removed from the instance summary. For more information, see [PATCH vs POST](#patch-vs-post).


Check that your application is using the intended method (PATCH or POST). If a metric payload is sent using the POST method, any existing metrics that are not included in the payload are removed from the instance summary. For more information, see PATCH vs POST.

Added this one. Could remove if it seems unlikely

paigecalvert · 2026-01-26T23:40:38Z

docs/vendor/custom-metrics.md

+To troubleshoot this issue:
+- Review your metric collection logic to ensure consistent value calculation
+- Check for floating-point precision issues with numeric values
+- Verify that boolean values are consistently true or false and are not truthy or falsy conversions


I left in this "Duplicate events for unchanged values" troubleshooting item, but wasn't sure how likely it was. Floating-point precision issues and truthy or falsy conversions seemed like very low level/rare issues

paigecalvert · 2026-01-26T23:41:29Z

docs/vendor/instance-insights-event-data.mdx

+10:10 AM → Send {"users": 150, "region": "us-west"}
+           Creates one event: users changed to 150
+           No event for region because the value is unchanged
+```


I did end up moving this example over to the linked page about how the vendor portal generates events, since that info is not specific to custom metrics

paigecalvert · 2026-01-26T23:43:12Z

docs/vendor/custom-metrics.md

+* The SDK API accepts only JSON scalar values for metrics. Any requests containing nested objects or arrays are rejected.

-## Configure Custom Metrics
+## Define Custom Metrics in JSON Payload


Updated the h2s on this page so that we have Define Custom Metrics and Send Custom Metrics. This gives us space to talk about requirements for the json payload itself and how to name custom metrics (ie, "define custom metrics") as well as space to talk about patch v post and how often to send payloads to the sdk (ie, "send custom metrics")

paigecalvert · 2026-01-26T23:44:25Z

docs/vendor/custom-metrics.md

+    "active_users": 150,              // Number
+    "cpu_usage_percent": 75.5,        // Number
+    "sso_enabled": true,               // Boolean
+    "deployment_region": "us-east-1", // String


Note: I removed the one item in this example that was showing an unsupported format (array). It felt like it would be easy for people to just skim this example and think that the array was supported

AmberAlston · 2026-01-27T16:55:04Z

Excellent edits, thanks for the collaboration @paigecalvert

replicated-ci added type::docs Improvements or additions to documentation type::feature labels Jan 26, 2026

Revise initial draft

76d8e31

AmberAlston commented Jan 26, 2026

View reviewed changes

AmberAlston marked this pull request as ready for review January 26, 2026 19:44

AmberAlston requested a review from a team as a code owner January 26, 2026 19:44

docs edits

41ad381

paigecalvert approved these changes Jan 26, 2026

View reviewed changes

AmberAlston merged commit 27bd0cf into main Jan 27, 2026
4 checks passed

AmberAlston deleted the improve-custom-metrics-documentation branch January 27, 2026 16:55


		[View a larger version of this image](/images/custom-metrics-flow.png)

		Custom metrics use a change-detection system to efficiently track meaningful change events from the previous value:

		## Export Custom Metrics

		You can use the Vendor API v3 `/app/{app_id}/events` endpoint to programatically access historical timeseries data containing instance level events, including any custom metrics that you have defined. For more information about the endpoint, see [Export Customer and Instance Data](/vendor/instance-data-export).

Improve custom metrics documentation with clearer explanations #3789

Improve custom metrics documentation with clearer explanations #3789

Uh oh!

Conversation

AmberAlston commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

1. Added "How Custom Metrics Work" Section

2. Added "Supported Data Types" Section

3. Improved "POST vs PATCH" Section

4. Added "Best Practices" Section

5. Added "Troubleshooting" Section

Motivation

Related Research

Uh oh!

netlify bot commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for replicated-docs ready!

Uh oh!

netlify bot commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for replicated-docs-upgrade ready!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paigecalvert left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmberAlston commented Jan 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AmberAlston commented Jan 26, 2026 •

edited

Loading

netlify bot commented Jan 26, 2026 •

edited

Loading

netlify bot commented Jan 26, 2026 •

edited

Loading