From 9c8f97c49f96b5f55e64c48978c9d085211543b7 Mon Sep 17 00:00:00 2001 From: John Simons Date: Fri, 23 May 2025 12:18:50 +1000 Subject: [PATCH 1/3] Document the reasons why servicecontrol.throughput queue exists --- docs/throughput-collection.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 docs/throughput-collection.md diff --git a/docs/throughput-collection.md b/docs/throughput-collection.md new file mode 100644 index 0000000000..46e82f7568 --- /dev/null +++ b/docs/throughput-collection.md @@ -0,0 +1,27 @@ +From version 5.4.0 we have introduce a new feature to collect usage data as part of the Platform. + +Usage data is collected from 3 different sources: +- Audited messages +- Monitoring metrics +- Directly from the broker + +The error instance is the orchestrator for all the collection from all these different endpoints. + +Audited messages usage is collected by querying the audit instances every so often. +Broker collection is also collected by querying the borker directly every 24 hours. +The monitoring metrics collection is collected by the metrics intance sending a direct message to a well known queue that by default is named "servicecontrol.throughput". + + +### Why is the "servicecontrol.throughput" queue hardcoded? +The TF that initially created this new feature decided that by hardcoding the queue name, it would be simpler for the customers to get started because they don't need to configure anything up front. +This was decided based on simplicity of upgrade vs maintanability. + +### Why is the "servicecontrol.throughput" not a sub queue of the error instance? +Because this would mean customer would have to configure the sending endpoint (monitoring instance) with the name of this queue, which we wanted to avoid. + +#### But we have SCMU, can't that make sure the queue names match? +SCMU is a Windows only tool, as well as, this would only work if they are installing all instances on the same machine. +Same reasons why, we do not configure the remote audit instances. + +### Can the "servicecontrol.throughput" queue be renamed? +Yes, see https://docs.particular.net/servicecontrol/monitoring-instances/configuration#usage-reporting From 9ca63295e6761d1b4f68a3ad51b01dc58cc223b8 Mon Sep 17 00:00:00 2001 From: John Simons Date: Fri, 23 May 2025 15:23:20 +1000 Subject: [PATCH 2/3] Apply suggestions from code review Co-authored-by: Jo Palac --- docs/throughput-collection.md | 44 ++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/docs/throughput-collection.md b/docs/throughput-collection.md index 46e82f7568..3f8bbc5b2a 100644 --- a/docs/throughput-collection.md +++ b/docs/throughput-collection.md @@ -1,27 +1,39 @@ -From version 5.4.0 we have introduce a new feature to collect usage data as part of the Platform. +From version 5.4.0 a new `Licensing Component` feature was introduced to [collect usage data](https://docs.particular.net/servicepulse/usage) from within the Particular Platform. Usage data is collected from 3 different sources: -- Audited messages -- Monitoring metrics +- Audit instance(s) +- Monitoring instance - Directly from the broker -The error instance is the orchestrator for all the collection from all these different endpoints. +The Error instance is the orchestrator for collecting usage data from all the different sources. -Audited messages usage is collected by querying the audit instances every so often. -Broker collection is also collected by querying the borker directly every 24 hours. -The monitoring metrics collection is collected by the metrics intance sending a direct message to a well known queue that by default is named "servicecontrol.throughput". +The Audit instance(s) is/are queried once a day to obtain usage data. The initial query will grab all available historic data. +The broker is queried once a day to obtain usage data. Depending on the broker, the initial query will grab the last 30 days of data. +The Monitoring instance uses its metrics calculations to send usage data to the Error instance every 5 minutes to a pre-defined satellite queue with the default name of `servicecontrol.throughput`. -### Why is the "servicecontrol.throughput" queue hardcoded? -The TF that initially created this new feature decided that by hardcoding the queue name, it would be simpler for the customers to get started because they don't need to configure anything up front. -This was decided based on simplicity of upgrade vs maintanability. +### Why is the "servicecontrol.throughput" queue not a sub queue of the Error instance? -### Why is the "servicecontrol.throughput" not a sub queue of the error instance? -Because this would mean customer would have to configure the sending endpoint (monitoring instance) with the name of this queue, which we wanted to avoid. +The usage collection queue needs to be known to the Monitoring instance. -#### But we have SCMU, can't that make sure the queue names match? -SCMU is a Windows only tool, as well as, this would only work if they are installing all instances on the same machine. -Same reasons why, we do not configure the remote audit instances. +At the time of creating this feature it was decided that having a queue name that is not dependent on the name of the Error instance means less setup for majority of customers since the feature would "just work" out of the box. + +If the queue name was based on the Error instance name (i.e. "ErrorInstanceQueueName.throughput") then **every ** customer would have to make updates to their Monitoring instance config to set the correct queue name. + +The decision favoured simplicity of upgrade over existing ServiceControl queue name conventions, keeping in line with the tech lead preferences for [software that "just works"](https://github.com/Particular/Strategy/blob/master/tech-lead-preferences/it-just-works.md#it-just-works) and [convenience](https://github.com/Particular/Strategy/blob/master/tech-lead-preferences/usability.md#convenience). + + +#### Why isn't SCMU used to ensure the names match? + +SCMU is a Windows only tool, plus the install of the Monitoring instance is separate to that of the Error instance. +Additionally, the Monitoring instance can be installed on a different server to the Error instance. +For similar reasons we do not configure the remote audit instances in SCMU. ### Can the "servicecontrol.throughput" queue be renamed? -Yes, see https://docs.particular.net/servicecontrol/monitoring-instances/configuration#usage-reporting + +Yes, the queue name can be changed via: + +- the [LicensingComponent/ServiceControlThroughputDataQueue](https://docs.particular.net/servicecontrol/servicecontrol-instances/configuration#usage-reporting-when-using-servicecontrol-licensingcomponentservicecontrolthroughputdataqueue) setting on the Error instance +- the [Monitoring/ServiceControlThroughputDataQueue](https://docs.particular.net/servicecontrol/monitoring-instances/configuration#usage-reporting-monitoringservicecontrolthroughputdataqueue) on the Monitoring instance + +These two settings must match for the usage reporting from Monitoring to work correctly. From 677c21d2c693e66caa360ba9410f073a418b0adb Mon Sep 17 00:00:00 2001 From: John Simons Date: Fri, 23 May 2025 15:23:45 +1000 Subject: [PATCH 3/3] Fixed formatting --- docs/throughput-collection.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/throughput-collection.md b/docs/throughput-collection.md index 3f8bbc5b2a..246102de74 100644 --- a/docs/throughput-collection.md +++ b/docs/throughput-collection.md @@ -18,7 +18,7 @@ The usage collection queue needs to be known to the Monitoring instance. At the time of creating this feature it was decided that having a queue name that is not dependent on the name of the Error instance means less setup for majority of customers since the feature would "just work" out of the box. -If the queue name was based on the Error instance name (i.e. "ErrorInstanceQueueName.throughput") then **every ** customer would have to make updates to their Monitoring instance config to set the correct queue name. +If the queue name was based on the Error instance name (i.e. "ErrorInstanceQueueName.throughput") then **every** customer would have to make updates to their Monitoring instance config to set the correct queue name. The decision favoured simplicity of upgrade over existing ServiceControl queue name conventions, keeping in line with the tech lead preferences for [software that "just works"](https://github.com/Particular/Strategy/blob/master/tech-lead-preferences/it-just-works.md#it-just-works) and [convenience](https://github.com/Particular/Strategy/blob/master/tech-lead-preferences/usability.md#convenience).