Skip to content

[release-6.2]LOG-8292:Add support for configuring the maximum message length (in bytes) after merging partial log lines in the kubernetes_logs source #3168

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vparfonov wants to merge 1 commit into
base: release-6.2
Choose a base branch
from
Open

[release-6.2]LOG-8292:Add support for configuring the maximum message length (in bytes) after merging partial log lines in the kubernetes_logs source #3168

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ test-functional-benchmarker-vector: bin/functional-benchmarker
@out=$$(RELATED_IMAGE_VECTOR=$(IMAGE_LOGGING_VECTOR) bin/functional-benchmarker --image=$(IMAGE_LOGGING_VECTOR) --artifact-dir=/tmp/benchmark-test-vector 2>&1); if [ "$$?" != "0" ] ; then echo "$$out"; exit 1; fi

.PHONY: test-unit
test-unit: test-forwarder-generator test-unit-api
test-unit: test-unit-api
RELATED_IMAGE_VECTOR=$(IMAGE_LOGGING_VECTOR) \
RELATED_IMAGE_LOG_FILE_METRIC_EXPORTER=$(IMAGE_LOGFILEMETRICEXPORTER) \
go test -coverprofile=test.cov -race ./api/... ./internal/... `go list ./test/... | grep -Ev 'test/(e2e|functional|framework|client|helpers)'`
Expand Down
23 changes: 23 additions & 0 deletions api/observability/v1/input_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ limitations under the License.
package v1

import (
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

Expand Down Expand Up @@ -96,6 +97,12 @@ type ContainerInputTuningSpec struct {
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Per-Container Rate Limit"
RateLimitPerContainer *LimitSpec `json:"rateLimitPerContainer,omitempty"`

// MaxMessageSize The maximum message length in bytes that a single log event can be when all
// partial log lines are merged. Messages exceeding this limit are dropped.
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Max Message Size"
MaxMessageSize *resource.Quantity `json:"maxMessageSize,omitempty"`
}

// ApplicationSource defines the type of ApplicationSource log source to use.
Expand Down Expand Up @@ -184,6 +191,16 @@ var (
}
)

// InfrastructureInputTuningSpec is the infrastructure input tuning spec, for now available only for container sources
type InfrastructureInputTuningSpec struct {

// Container is the input tuning spec for container sources
//
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Input Tuning"
Container *ContainerInputTuningSpec `json:"container,omitempty"`
}

// Infrastructure enables infrastructure logs.
// Sources of these logs:
// * container workloads deployed to namespaces: default, kube*, openshift*
Expand All @@ -195,6 +212,12 @@ type Infrastructure struct {
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Log Sources"
Sources []InfrastructureSource `json:"sources,omitempty"`

// Tuning is the infrastructure input tuning spec, for now available only for container sources
//
// +kubebuilder:validation:Optional
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Container Input Tuning"
Tuning *InfrastructureInputTuningSpec `json:"tuning,omitempty"`
}

// AuditSource defines which type of audit log source is used.
Expand Down
30 changes: 30 additions & 0 deletions api/observability/v1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

31 changes: 30 additions & 1 deletion bundle/manifests/cluster-logging.clusterserviceversion.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ metadata:
categories: OpenShift Optional, Logging & Tracing
certified: "false"
containerImage: quay.io/openshift-logging/cluster-logging-operator:latest
createdAt: "2025-07-29T15:25:15Z"
createdAt: "2025-12-12T17:14:22Z"
description: The Red Hat OpenShift Logging Operator for OCP provides a means for
configuring and managing log collection and forwarding.
features.operators.openshift.io/cnf: "false"
Expand Down Expand Up @@ -357,6 +357,11 @@ spec:
sources
displayName: Input Tuning
path: inputs[0].application.tuning
- description: |-
MaxMessageSize The maximum message length in bytes that a single log event can be when all
partial log lines are merged. Messages exceeding this limit are dropped.
displayName: Max Message Size
path: inputs[0].application.tuning.maxMessageSize
- description: |-
RateLimitPerContainer is the limit applied to each container
by this input. This limit is applied per collector deployment.
Expand Down Expand Up @@ -385,6 +390,30 @@ spec:
This field is optional and omission results in the collection of all infrastructure sources.
displayName: Log Sources
path: inputs[0].infrastructure.sources
- description: Tuning is the infrastructure input tuning spec, for now available
only for container sources
displayName: Container Input Tuning
path: inputs[0].infrastructure.tuning
- description: Container is the input tuning spec for container sources
displayName: Input Tuning
path: inputs[0].infrastructure.tuning.container
- description: |-
MaxMessageSize The maximum message length in bytes that a single log event can be when all
partial log lines are merged. Messages exceeding this limit are dropped.
displayName: Max Message Size
path: inputs[0].infrastructure.tuning.container.maxMessageSize
- description: |-
RateLimitPerContainer is the limit applied to each container
by this input. This limit is applied per collector deployment.
displayName: Per-Container Rate Limit
path: inputs[0].infrastructure.tuning.container.rateLimitPerContainer
- description: |-
MaxRecordsPerSecond is the maximum number of log records
allowed per input/output in a pipeline
displayName: Max Records Per Second
path: inputs[0].infrastructure.tuning.container.rateLimitPerContainer.maxRecordsPerSecond
x-descriptors:
- urn:alm:descriptor:com.tectonic.ui:number
- description: Name used to refer to the input of a `pipeline`.
displayName: Input Name
path: inputs[0].name
Expand Down
44 changes: 44 additions & 0 deletions bundle/manifests/observability.openshift.io_clusterlogforwarders.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -629,6 +629,15 @@ spec:
description: Tuning is the container input tuning spec for
this container sources
properties:
maxMessageSize:
anyOf:
- type: integer
- type: string
description: |-
MaxMessageSize The maximum message length in bytes that a single log event can be when all
partial log lines are merged. Messages exceeding this limit are dropped.
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
rateLimitPerContainer:
description: |-
RateLimitPerContainer is the limit applied to each container
Expand Down Expand Up @@ -680,6 +689,41 @@ spec:
- node
type: string
type: array
tuning:
description: Tuning is the infrastructure input tuning spec,
for now available only for container sources
properties:
container:
description: Container is the input tuning spec for
container sources
properties:
maxMessageSize:
anyOf:
- type: integer
- type: string
description: |-
MaxMessageSize The maximum message length in bytes that a single log event can be when all
partial log lines are merged. Messages exceeding this limit are dropped.
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
rateLimitPerContainer:
description: |-
RateLimitPerContainer is the limit applied to each container
by this input. This limit is applied per collector deployment.
properties:
maxRecordsPerSecond:
description: |-
MaxRecordsPerSecond is the maximum number of log records
allowed per input/output in a pipeline
exclusiveMinimum: true
format: int64
minimum: 0
type: integer
required:
- maxRecordsPerSecond
type: object
type: object
type: object
type: object
name:
description: Name used to refer to the input of a `pipeline`.
Expand Down
44 changes: 44 additions & 0 deletions config/crd/bases/observability.openshift.io_clusterlogforwarders.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -629,6 +629,15 @@ spec:
description: Tuning is the container input tuning spec for
this container sources
properties:
maxMessageSize:
anyOf:
- type: integer
- type: string
description: |-
MaxMessageSize The maximum message length in bytes that a single log event can be when all
partial log lines are merged. Messages exceeding this limit are dropped.
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
rateLimitPerContainer:
description: |-
RateLimitPerContainer is the limit applied to each container
Expand Down Expand Up @@ -680,6 +689,41 @@ spec:
- node
type: string
type: array
tuning:
description: Tuning is the infrastructure input tuning spec,
for now available only for container sources
properties:
container:
description: Container is the input tuning spec for
container sources
properties:
maxMessageSize:
anyOf:
- type: integer
- type: string
description: |-
MaxMessageSize The maximum message length in bytes that a single log event can be when all
partial log lines are merged. Messages exceeding this limit are dropped.
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
rateLimitPerContainer:
description: |-
RateLimitPerContainer is the limit applied to each container
by this input. This limit is applied per collector deployment.
properties:
maxRecordsPerSecond:
description: |-
MaxRecordsPerSecond is the maximum number of log records
allowed per input/output in a pipeline
exclusiveMinimum: true
format: int64
minimum: 0
type: integer
required:
- maxRecordsPerSecond
type: object
type: object
type: object
type: object
name:
description: Name used to refer to the input of a `pipeline`.
Expand Down
29 changes: 29 additions & 0 deletions config/manifests/bases/cluster-logging.clusterserviceversion.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,11 @@ spec:
sources
displayName: Input Tuning
path: inputs[0].application.tuning
- description: |-
MaxMessageSize The maximum message length in bytes that a single log event can be when all
partial log lines are merged. Messages exceeding this limit are dropped.
displayName: Max Message Size
path: inputs[0].application.tuning.maxMessageSize
- description: |-
RateLimitPerContainer is the limit applied to each container
by this input. This limit is applied per collector deployment.
Expand Down Expand Up @@ -308,6 +313,30 @@ spec:
This field is optional and omission results in the collection of all infrastructure sources.
displayName: Log Sources
path: inputs[0].infrastructure.sources
- description: Tuning is the infrastructure input tuning spec, for now available
only for container sources
displayName: Container Input Tuning
path: inputs[0].infrastructure.tuning
- description: Container is the input tuning spec for container sources
displayName: Input Tuning
path: inputs[0].infrastructure.tuning.container
- description: |-
MaxMessageSize The maximum message length in bytes that a single log event can be when all
partial log lines are merged. Messages exceeding this limit are dropped.
displayName: Max Message Size
path: inputs[0].infrastructure.tuning.container.maxMessageSize
- description: |-
RateLimitPerContainer is the limit applied to each container
by this input. This limit is applied per collector deployment.
displayName: Per-Container Rate Limit
path: inputs[0].infrastructure.tuning.container.rateLimitPerContainer
- description: |-
MaxRecordsPerSecond is the maximum number of log records
allowed per input/output in a pipeline
displayName: Max Records Per Second
path: inputs[0].infrastructure.tuning.container.rateLimitPerContainer.maxRecordsPerSecond
x-descriptors:
- urn:alm:descriptor:com.tectonic.ui:number
- description: Name used to refer to the input of a `pipeline`.
displayName: Input Name
path: inputs[0].name
Expand Down
Loading