Skip to content

Commit 2dcc658

Browse files
piochelepiotrestherk15
piochelepiotr
and
estherk15
authored
DSMON-1152: Add Kafka actions check (#21951)
* Add Kafka actions * fix tests * update ci * simplify tests * add e2e test * sync ci * mark unit tests * update * update * run ddev validate ci * remove emojis from readme * validate models * update labels * update labels * Update kafka_actions/README.md Co-authored-by: Esther Kim <esther.kim@datadoghq.com> * Update kafka_actions/assets/configuration/spec.yaml Co-authored-by: Esther Kim <esther.kim@datadoghq.com> * Update kafka_actions/assets/configuration/spec.yaml Co-authored-by: Esther Kim <esther.kim@datadoghq.com> * Update kafka_actions/README.md Co-authored-by: Esther Kim <esther.kim@datadoghq.com> * Update kafka_actions/README.md Co-authored-by: Esther Kim <esther.kim@datadoghq.com> * Update kafka_actions/README.md Co-authored-by: Esther Kim <esther.kim@datadoghq.com> * Update kafka_actions/README.md Co-authored-by: Esther Kim <esther.kim@datadoghq.com> * Update kafka_actions/README.md Co-authored-by: Esther Kim <esther.kim@datadoghq.com> * Update kafka_actions/README.md Co-authored-by: Esther Kim <esther.kim@datadoghq.com> * Update kafka_actions/README.md Co-authored-by: Esther Kim <esther.kim@datadoghq.com> * Update kafka_actions/README.md Co-authored-by: Esther Kim <esther.kim@datadoghq.com> * Update kafka_actions/README.md Co-authored-by: Esther Kim <esther.kim@datadoghq.com> * Update kafka_actions/README.md Co-authored-by: Esther Kim <esther.kim@datadoghq.com> * update readme * validate models * Update kafka_actions/README.md Co-authored-by: Esther Kim <esther.kim@datadoghq.com> * address PR comments * Comment that only one action should be configured * Add escaped string support * format --------- Co-authored-by: Esther Kim <esther.kim@datadoghq.com>
1 parent a58ded8 commit 2dcc658

34 files changed

+5178
-6
lines changed

‎.codecov.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -334,6 +334,10 @@ coverage:
334334
target: 75
335335
flags:
336336
- istio
337+
Kafka_Actions:
338+
target: 75
339+
flags:
340+
- kafka_actions
337341
Kafka_Consumer:
338342
target: 75
339343
flags:
@@ -1264,6 +1268,11 @@ flags:
12641268
paths:
12651269
- istio/datadog_checks/istio
12661270
- istio/tests
1271+
kafka_actions:
1272+
carryforward: true
1273+
paths:
1274+
- kafka_actions/datadog_checks/kafka_actions
1275+
- kafka_actions/tests
12671276
kafka_consumer:
12681277
carryforward: true
12691278
paths:

‎.github/CODEOWNERS‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,11 @@ datadog_checks_base/tests/**/test_db_statements.py @DataDog/database-monitoring
171171
/mongo/*.md @DataDog/database-monitoring @DataDog/agent-integrations @DataDog/documentation
172172
/mongo/manifest.json @DataDog/database-monitoring @DataDog/agent-integrations @DataDog/documentation
173173

174+
# Data Streams Monitoring
175+
/kafka_actions/ @DataDog/data-streams-monitoring @DataDog/agent-integrations
176+
/kafka_actions/*.md @DataDog/data-streams-monitoring @DataDog/agent-integrations @DataDog/documentation
177+
/kafka_actions/manifest.json @DataDog/data-streams-monitoring @DataDog/agent-integrations @DataDog/documentation
178+
174179
# APM Integrations
175180
/langchain/ @DataDog/ml-observability @DataDog/agent-integrations @DataDog/documentation
176181
/openai/ @DataDog/ml-observability @DataDog/agent-integrations @DataDog/documentation

‎.github/workflows/config/labeler.yml‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,8 @@ integration/azure_active_directory:
8989
- azure_active_directory/**/*
9090
integration/azure_iot_edge:
9191
- azure_iot_edge/**/*
92+
integration/barracuda_secure_edge:
93+
- barracuda_secure_edge/**/*
9294
integration/bentoml:
9395
- bentoml/**/*
9496
integration/beyondtrust_identity_security_insights:
@@ -157,6 +159,8 @@ integration/cloud_foundry_api:
157159
- cloud_foundry_api/**/*
158160
integration/cloudera:
159161
- cloudera/**/*
162+
integration/cloudgen_firewall:
163+
- cloudgen_firewall/**/*
160164
integration/cockroachdb:
161165
- cockroachdb/**/*
162166
integration/cofense_triage:
@@ -363,12 +367,14 @@ integration/juniper_srx_firewall:
363367
- juniper_srx_firewall/**/*
364368
integration/kafka:
365369
- kafka/**/*
370+
integration/kafka_actions:
371+
- kafka_actions/**/*
366372
integration/kafka_consumer:
367373
- kafka_consumer/**/*
368-
integration/karpenter:
369-
- karpenter/**/*
370374
integration/kandji:
371375
- kandji/**/*
376+
integration/karpenter:
377+
- karpenter/**/*
372378
integration/keda:
373379
- keda/**/*
374380
integration/keeper:
@@ -787,10 +793,6 @@ integration/zerofox_cloud_platform:
787793
- zerofox_cloud_platform/**/*
788794
integration/zk:
789795
- zk/**/*
790-
integration/barracuda_secure_edge:
791-
- barracuda_secure_edge/**/*
792-
integration/cloudgen_firewall:
793-
- cloudgen_firewall/**/*
794796
integration/zscaler_private_access:
795797
- zscaler_private_access/**/*
796798
qa/skip-qa:

‎.github/workflows/test-all.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1883,6 +1883,26 @@ jobs:
18831883
minimum-base-package: ${{ inputs.minimum-base-package }}
18841884
pytest-args: ${{ inputs.pytest-args }}
18851885
secrets: inherit
1886+
jf06de9b:
1887+
uses: ./.github/workflows/test-target.yml
1888+
with:
1889+
job-name: Kafka Actions
1890+
target: kafka_actions
1891+
platform: linux
1892+
runner: '["ubuntu-22.04"]'
1893+
repo: "${{ inputs.repo }}"
1894+
context: ${{ inputs.context }}
1895+
python-version: "${{ inputs.python-version }}"
1896+
latest: ${{ inputs.latest }}
1897+
agent-image: "${{ inputs.agent-image }}"
1898+
agent-image-py2: "${{ inputs.agent-image-py2 }}"
1899+
agent-image-windows: "${{ inputs.agent-image-windows }}"
1900+
agent-image-windows-py2: "${{ inputs.agent-image-windows-py2 }}"
1901+
test-py2: ${{ inputs.test-py2 }}
1902+
test-py3: ${{ inputs.test-py3 }}
1903+
minimum-base-package: ${{ inputs.minimum-base-package }}
1904+
pytest-args: ${{ inputs.pytest-args }}
1905+
secrets: inherit
18861906
jeaad214:
18871907
uses: ./.github/workflows/test-target.yml
18881908
with:

‎kafka_actions/CHANGELOG.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# CHANGELOG - Kafka Actions
2+
3+
<!-- towncrier release notes start -->
4+

‎kafka_actions/README.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Kafka Actions
2+
3+
<div class="alert alert-danger">
4+
This integration is in Preview. To access and use this feature, reach out to the Data Streams Monitoring team at Datadog.
5+
</div>
6+
7+
## Overview
8+
9+
The Kafka Actions integration enables one-time administrative and operational actions on Kafka clusters through the Datadog Agent.
10+
11+
<div class="alert alert-warning">
12+
This integration is exclusively triggered through [Remote Configuration][4] and should never be scheduled manually in your Agent configuration. It is not meant to run continuously like traditional checks.
13+
</div>
14+
15+
### Supported Actions
16+
17+
| Action | Description |
18+
|--------|-------------|
19+
| `read_messages` | Read and filter messages with jq-style expressions, supporting JSON, string, BSON, Avro, and Protobuf formats |
20+
| `produce_message` | Produce messages to topics with base64-encoded payloads and headers |
21+
| `create_topic` | Create topics with custom partitions, replication factor, and configurations |
22+
| `update_topic_config` | Update topic configurations and partition counts |
23+
| `delete_topic` | Delete topics |
24+
| `delete_consumer_group` | Delete consumer groups |
25+
| `update_consumer_group_offsets` | Reset consumer group offsets to specific positions |
26+
27+
## Setup
28+
29+
### Installation
30+
31+
The Kafka Actions integration is included in the [Datadog Agent][1] package (version 7.0+).
32+
33+
### Configuration
34+
35+
<div class="alert alert-warning">
36+
<strong>Do not configure this check manually</strong>. This integration is triggered exclusively through [Remote Configuration][4] from the Datadog UI.
37+
</div>
38+
39+
Actions are configured and triggered from the Datadog platform, with results visible in the [Events Explorer][3].
40+
Each instance should configure only **one** action. Multiple actions in the same instance are not supported.
41+
42+
## Data Collected
43+
44+
### Events
45+
46+
- **Action Events** (`kafka_action_success` / `kafka_action_error`): Emitted when an action completes
47+
- **Message Events** (`kafka_message`): Emitted for each message retrieved by `read_messages`
48+
49+
### Metrics
50+
51+
This integration does not collect metrics.
52+
53+
### Service Checks
54+
55+
This integration does not include service checks.
56+
57+
## Troubleshooting
58+
59+
### Actions not executing
60+
1. Confirm the Agent version meets minimum requirements
61+
2. Verify [Remote Configuration][4] is enabled on your Agent.
62+
3. Check Agent logs for errors.
63+
64+
### Message deserialization failures
65+
Ensure the correct format is specified (`json`, `string`, `bson`, `avro`, `protobuf`) and provide schemas for Avro/Protobuf.
66+
67+
Need help? Contact [Datadog support][2] or reach out to the **Data Streams Monitoring team** at Datadog for questions about this integration.
68+
69+
[1]: https://app.datadoghq.com/account/settings/agent/latest
70+
[2]: https://docs.datadoghq.com/help/
71+
[3]: https://app.datadoghq.com/event/explorer
72+
[4]: https://docs.datadoghq.com/remote_configuration/

0 commit comments

Comments
 (0)