Standard label to identify "system" vs "user" namespaces

[danwinship]

Enhancement Description

• One-line enhancement description (can be used as a release note): Standard label to identify "system" vs "user" namespaces

• Kubernetes Enhancement Proposal:

• Discussion Link: discussion below

• PRs by stage and milestone:

  • Alpha - v1.xx
    • KEP (k/enhancements) update PR(s):
    • Code (k/k) update PR(s):
    • Docs (k/website) update PR(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

---

The idea is that real-world use of ClusterNetworkPolicy will likely require a way to distinguish "system" namespaces (eg, default and kube-system in a basic vanilla k8s cluster) from "user" namespaces, so that admins can write policies that apply to all user workloads without worrying that they will accidentally break the cluster infrastructure. There may be a use case for more than just "system" vs "user".

For example, our own documentation on "Securing a Cluster" suggests restricting access to cloud metadata APIs. But components like the cloud provider and cloud-specific ingress/gateway controllers may need access to it (and in some clouds, the upstream DNS server has the same IP as the metadata API, so CoreDNS would also need access to it). So administrators would want to be able to write a policy that prevents "user workload" namespaces from accessing the metadata IP without blocking "cluster infrastructure" namespaces. And ideally, we would like to be able to provide an example of such a ClusterNetworkPolicy in the doc:

apiVersion: policy.networking.k8s.io/v1alpha2
kind: ClusterNetworkPolicy
metadata:
  name: block-user-namespaces-from-metadata-api
spec:
  tier: Admin
  priority: 100 # adjust as appropriate
  subject:
    namespaces:
      matchExpressions:
      - # FIXME: fill this part in
  egress:
  - action: "Deny"
    to:
    - networks:
      # This is the IP used by GCP, AWS, and Azure, but other clouds may
      # use different IPs...
      - 169.254.169.254/32

But without a distribution-independent way of distinguishing "user workload" and "cluster infrastructure" components, we can't fill in the actual matchExpressions.

/sig network architecture
/help

[k8s-ci-robot]

@danwinship:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

Enhancement Description

• One-line enhancement description (can be used as a release note): Standard label to identify "system" vs "user" namespaces
• Kubernetes Enhancement Proposal:
• Discussion Link: None online...

The idea is that real-world use of ClusterNetworkPolicy will likely require a way to distinguish "system" namespaces (eg, default and kube-system in a basic vanilla k8s cluster) from "user" namespaces, so that admins can write policies that apply to all user workloads without worrying that they will accidentally break the cluster infrastructure. There may be a use case for more than just "system" vs "user".

/sig network architecture
/help

• PRs by stage and milestone:
• Alpha - v1.xx
  • KEP (k/enhancements) update PR(s):
  • Code (k/k) update PR(s):
  • Docs (k/website) update PR(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[danwinship]

cc @tssurya @npinaeva

[aojea]

some references https://github.com/kubernetes/kubernetes/blob/4bed36e03e7bd699b089d33da6f7d7c9ef9eb661/pkg/controlplane/apiserver/options/options.go#L131 , however, each distro may have different "system namespaces"

[bowei]

I'm assuming you would need to prevent arbitrary assignment of the system label as well? Would the system label be baked in (e.g. only kube-system and default) or flexible?

[danwinship]

With the default RBAC, only cluster-admins can modify Namespace labels. (Namespace admins can't modify their own Namespaces.)

[aditigupta96]

Hi, I've been studying this issue to understand the full scope.

It seems the core problem is that relying on hardcoded strings (like kube-system) to identify system components is brittle. We need a standard, reserved label (e.g., kubernetes.io/system-namespace) to safely target these namespaces for things like NetworkPolicies.

Based on the discussion and the references, here is how I understand the approach:

1. Standardization: We define a reserved label key in the API definitions.
2. Bootstrap Automation: We modify the kube-apiserver bootstrap logic. When the cluster starts, it must ensure the default system namespaces (kube-system, kube-public, kube-node-lease) automatically get this label.
3. Security & Flexibility: Per @danwinship 's point about RBAC, we don't need admission control to prevent spoofing, as standard users cannot modify Namespace labels. Cluster Admins can still manually apply this label to custom infrastructure if they choose but unauthorized users cannot.

Does this breakdown sound correct?

[SergeyKanzhelev]

is the desire to extend the notion of "system" to other things beyond ClusterNetworkPolicy? Should this label be named somethow networking-specific? If opting out of some networking policies requires to mark the namespace as "system", will it break some other logic that may also require distinguishing of "system" from "non-system"?

[danwinship]

It seems the core problem is that relying on hardcoded strings (like kube-system) to identify system components is brittle.

It's not just brittle; it's non-portable. Different clusters may be running other system components in other namespaces. (In OpenShift there are lots of system namespaces.)

We need a standard, reserved label (e.g., kubernetes.io/system-namespace)

Part of the problem is deciding whether "system" vs "user" is the entire distinction that needs to be made, or if further distinctions would be useful (in which case "system-namespace" would not be the right name for the label). For example, there might be a three-way distinction between "core components of the cluster", "add-on components", and "user workloads". (For example, a Gateway API implementation might fall into the second category rather than the first in some clusters.) (OTOH, it may be that there aren't any situations where you'd really care about the distinction between the first and second groups.)

to safely target these namespaces for things like NetworkPolicies.

ClusterNetworkPolicies. NetworkPolicies only affect the namespace that they are deployed into.

[aditigupta96]

Thanks for the correction on ClusterNetworkPolicy as standard NPs are namespace-scoped anyway.

Regarding "System vs. User" vs. "Core vs. Add-on":
By portability do you mean that users on OpenShift or GKE shouldn't have to rewrite their policies because the system namespaces have different names?

To your point about the three-way distinction, do we actually want to force users to distinguish between "Core" (kube-system) and "Add-ons" (Gateway API/monitoring)?
From a policy perspective, both usually represent "Cluster Infrastructure" that should be excluded from strict User-land policies. If we split them into distinct labels now, we might force admins to write complex selectors (e.g.,exclude: [core, addons]) rather than a simple broad exclusion.

Perhaps we can solve this by using a value-based label design (e.g., kubernetes.io/namespace-role: system vs kubernetes.io/namespace-role: addon)? That allows us to start with a unified System definition to solve the portability issue now but allows for granular segmentation later if a concrete use-case arises.

[danwinship]

is the desire to extend the notion of "system" to other things beyond ClusterNetworkPolicy? Should this label be named somethow networking-specific?

The only use case for it (currently) is networking, but the intention was that the meaning is not networking-specific.

If opting out of some networking policies requires to mark the namespace as "system"

No, that's not the intent. The intent is that the label actually, correctly distinguishes between "namespaces that contain user workloads" and "namespaces that don't contain user workloads", so you can write policies that apply to user workloads but not system components. If the label doesn't carve the world up in exactly the way you need, you could match on additional labels, or add additional rules.

So for example, you might have a policy that says "Deny all from user namespaces to 169.254.169.254" to enforce a blanket rule that user workloads can't talk to the cloud metadata API (without worrying that you're also accidentally blocking the cloud-provider, ingress controller, etc, from reaching it). But then if it turns out there was one user job that needed to be able to access it, you could just have a higher-precedence "Accept from namespace X to 169.254.169.254" rule.