Kubernetes network policy: a safe default-deny rollout

A Kubernetes network policy is an allow-list at layers 3 and 4 that decides which pods can reach which other pods and ports. Kubernetes applies none of it by default: every pod can talk to every other pod, in every namespace, and out to anything the node can route to. Moving a namespace to default deny closes that gap, and this guide covers how to do it without taking your own services down.

Two failure modes bring teams here, and they are the same worry from opposite ends. One is whether near-zero-CVE base images actually reduce risk. The other is a service that pulled a barely maintained dependency and had it running before review saw it. Both are about what gets into the cluster. A Kubernetes network policy governs what a workload can reach once it is already there, which is the half you still control after the image has landed.

The stakes are concrete. A compromised container in your monitoring namespace can open a connection to your validator's signing sidecar, and nothing in the cluster will object.

What NetworkPolicy controls, and what it does not

A NetworkPolicy is an allow-list at layers 3 and 4. It selects pods by label and describes which sources may reach them and which destinations they may reach, by pod selector, namespace selector, IP block and port.

Four things it does not do cause most of the confusion:

An unenforced manifest is false confidence

kubectl apply on a NetworkPolicy succeeds whether or not anything in the cluster implements it. The object is stored, kubectl get netpol lists it, and traffic carries on exactly as before. Flannel in its default configuration does not enforce NetworkPolicy. Several managed offerings ship a CNI where enforcement is an option you have to switch on.

So verify before you write a single rule. Apply a deny-all to a scratch namespace with two test pods, then try to curl one from the other. If it connects, nothing is enforcing and everything you build on top is decoration.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: netpol-test
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

An empty podSelector selects every pod in the namespace. With both policy types listed and no rules underneath, nothing is permitted in either direction.

Start from observed flows, not from a diagram

The architecture diagram is wrong. It is not wrong because anyone was careless, it is wrong because it was drawn before the sidecar, the backup job and the third-party agent were added.

Before writing policy, record what the namespace actually talks to over a period long enough to include the slow stuff. A week catches nightly batch jobs, certificate renewals and the weekly report nobody remembers owning. Cilium's Hubble, or your CNI's equivalent flow logging, will give you source pod, destination and port.

Sort the result into three buckets. Flows you can name and must keep. Flows you can name and want gone. Flows nobody can identify, which are the reason you are doing this at all.

Deny ingress first, then egress

Split the rollout. Ingress and egress fail in different ways and it is much easier to debug one at a time.

Ingress is the forgiving half. Apply a default-deny for ingress, then add explicit allows for the callers you identified. If you miss one, that caller breaks loudly and obviously, and the fix is a rule.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: chain-workloads
spec:
  podSelector: {}
  policyTypes:
    - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-gateway-to-rpc
  namespace: chain-workloads
spec:
  podSelector:
    matchLabels:
      app: rpc-endpoint
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: gateway
          podSelector:
            matchLabels:
              app: edge-proxy
      ports:
        - protocol: TCP
          port: 8545

Note the shape of that from entry. The namespace selector and pod selector sit in one list item, so both must match. Put them in separate list items and you have written "any pod in the gateway namespace, or any pod anywhere with this label", which is far wider than intended. This is a common mistake in hand-written policy.

Egress is where outages come from, so do it second and do it with DNS handled first.

DNS and the platform exceptions

The moment egress is denied, name resolution stops, and every symptom you see afterwards will look like something else. Failing readiness probes, timeouts to services that are plainly running, and clients that hang rather than error.

Allow DNS to CoreDNS explicitly, on both UDP and TCP port 53. TCP matters for responses too large for a UDP datagram.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: chain-workloads
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Then work through the rest of the platform before you touch application rules. Metrics scraping is an ingress allow from your monitoring namespace to the metrics port. Log shipping is usually egress from every pod to a collector. Admission webhooks are ingress to the webhook service from the API server, and that traffic often arrives from a node IP rather than a pod, so it needs an ipBlock. Certificate issuance needs egress to the ACME endpoint, which is external.

Get those wrong and the breakage will be attributed to the application for a day and a half.

Stateful workloads have dependencies that outlive a pod

Chain workloads make this harder than a stateless web tier, in a specific way.

An RPC node holds long-lived peer connections that it initiates outbound to arbitrary internet addresses on non-obvious ports. You cannot enumerate peers, so egress for that pod is a CIDR range and a port range, not a list of hosts. Narrow it by excluding your own internal ranges from the allowed block rather than trying to list the world.

A validator has a signing path that should be the most restricted thing in the cluster. That is one client, one destination, one port, and no reason for anything else in the namespace to reach it.

An oracle node reads from upstream data providers over TLS and writes transactions through an RPC endpoint. Both directions are external and both should be explicit.

Anything with a quorum, and etcd is the obvious case, needs peer-to-peer allows among its own members. Write those as a pod selector matching the set itself, so the rule survives rescheduling.

The general rule is that stateful workloads reconnect. A policy that a pod tolerates while running can still break it on restart, when it tries to re-establish everything at once. Test by deleting the pod, not just by watching it.

Test from both sides, keep a rollback

Policy is an intersection. Traffic is permitted only if the source's egress rules allow it out and the destination's ingress rules allow it in. Testing one direction proves nothing.

Run each check as a pair. From a pod in the source namespace, attempt the connection. From the destination, confirm arrival. A nc -zv and a look at the receiving service's own connection log will do.

Also test what should fail. A policy that allows everything you tested and everything you did not is the outcome you are trying to avoid, and it looks identical to success until an incident.

Rollback is one delete away, which is the good news. Keep the policies for a namespace in their own directory, apply them as a unit, and make sure whoever is on call knows that removing them restores the previous behaviour immediately. Do not bury network policy inside an application chart where deleting it means redeploying the application.

Roll out namespace by namespace, starting with one that is important enough to be watched and not so important that a mistake is a customer incident.

Failure modes worth knowing before you meet them

Nothing is enforced. Covered above, and the reason to test the CNI first.

Ingress allow written as two list items. Widens the rule to an "or". Read every from and to block for this specific shape.

DNS forgotten, or allowed on UDP only. Intermittent failures on larger responses, which is worse than a clean break because it looks random.

Policies applied but the pods are not selected. Labels drift. kubectl get pods -l <selector> against the same selector the policy uses is a five second check that saves an hour.

Host-network pods assumed covered. They are not, and they are often the agents with the most access.

Probes blocked. The kubelet reaches the pod from the node, so probe traffic can need an ipBlock for the node CIDR rather than a pod selector.

When the standard API is not enough

Standard NetworkPolicy is namespaced, allow-only and stops at layer 4. Some requirements sit outside that.

If you need to permit egress to a hostname whose IP changes, the standard API cannot express it. If you need a rule that applies cluster-wide rather than per namespace, or an explicit deny that overrides a broad allow, or filtering on HTTP method and path, those are CNI extensions rather than portable Kubernetes objects. Cilium provides all of them.

Use them where the requirement is real, and know the trade. Those objects do not move to another CNI, so keep the portable baseline in standard NetworkPolicy and treat the extensions as a documented layer on top.

We run dedicated Kubernetes for chain workloads with Cilium eBPF networking in place of kube-proxy, and default deny is the starting position for a namespace rather than a hardening step added later. More on how that cluster is built is in dedicated Kubernetes for web3 before enterprise.

If you would rather not run this yourself

The rollout above is a week of flow logging and a careful sequence, and it has to be repeated for every namespace you care about. That is reasonable work for a platform team that already owns its cluster. It is a poor use of a small engineering team whose actual product is somewhere else.

We build and operate dedicated Kubernetes for chain workloads, where default deny is the starting position for a namespace rather than a hardening step added later. If you are weighing that against doing it in-house, the LP MKS overview sets out how those clusters are put together and what we take on.

Sources

• Kubernetes network policies
• Kubernetes multi-tenancy
• Cilium deny policies

Frequently asked questions

What does a Kubernetes NetworkPolicy actually control?

It is an allow-list at layers 3 and 4. It selects pods by label and describes which sources may reach them and which destinations they may reach, by pod selector, namespace selector, IP block and port. It cannot filter by hostname or URL path, and it does not apply to pods running with hostNetwork set to true.

Why does my NetworkPolicy appear to have no effect?

Most likely nothing in the cluster is enforcing it. The API accepts and stores a NetworkPolicy whether or not the CNI implements it, so kubectl get netpol lists the object while traffic carries on unchanged. Flannel in its default configuration does not enforce NetworkPolicy, and several managed offerings require enforcement to be switched on. Test with a deny-all in a scratch namespace before writing real rules.

Should default-deny be applied to ingress or egress first?

Ingress first. Ingress fails loudly and locally, so a missed caller breaks in an obvious way and the fix is one rule. Egress is where outages come from, because denying it stops name resolution and every later symptom looks like something else.

What breaks first when egress is denied in Kubernetes?

DNS. Once egress is denied, pods cannot reach CoreDNS, and the failures present as failing readiness probes, timeouts to services that are plainly running, and clients that hang rather than error. Allow DNS to CoreDNS explicitly on both UDP and TCP port 53, because TCP carries responses too large for a single UDP datagram.

When is standard Kubernetes NetworkPolicy not enough?

When you need egress to a hostname whose IP changes, a rule that applies cluster-wide rather than per namespace, an explicit deny that overrides a broader allow, or filtering on HTTP method and path. Those are CNI extensions rather than portable Kubernetes objects, so keep a portable baseline in standard NetworkPolicy and treat the extensions as a documented layer on top.