Runtime Security (Falco)

Key Takeaways for AI & Readers

• Dynamic Threat Detection: Runtime security focuses on monitoring container behavior during execution to detect and respond to threats that static analysis (image scanning, admission control) cannot catch. It answers the question: "What is this container actually doing right now?"
• System Call Interception (Falco): Falco uses kernel-level mechanisms (eBPF probes or a kernel module) to intercept and analyze every system call, providing deep visibility into container activity without modifying application code or container images.
• Rule-Based Anomaly Detection: Falco identifies suspicious behavior based on YAML rules that define conditions, outputs, and priorities. Rules can detect shell execution in containers, writes to sensitive directories, unexpected network connections, and privilege escalation attempts.
• Automated Response: Falco integrates with Falco Sidekick to enable automated responses, including sending alerts to Slack, PagerDuty, or SIEM systems, and triggering Kubernetes actions such as pod termination or network isolation of compromised workloads.
• Defense in Depth: Runtime security complements admission-time controls and image scanning. Seccomp profiles, AppArmor, and SELinux provide kernel-level enforcement to restrict what containers can do, while Falco detects when those boundaries are probed or violated.

Static scanning (checking images for CVEs before deployment) is a necessary first step, but it is not sufficient. A container that passed all scans at build time can still be compromised at runtime through zero-day exploits, supply chain attacks, or misconfigured permissions. You need to know what your containers are doing while they are running. Falco is the CNCF graduated project that has become the industry standard for Kubernetes runtime security.

1. Detecting Intrusion in Real Time

Runtime security tools observe every action taken by every process in every container. When a process deviates from expected behavior, an alert fires immediately. Try triggering a shell in the simulation below to see how Falco detects the system call.

Target Pod

📦

Simulate an attacker or developer opening a shell.

🔴 Falco Event Stream

Waiting for syscall events...

Falco monitors kernel system calls in real-time. It uses rules to identify abnormal behavior that standard log tools might miss.

The key insight is that legitimate application containers have predictable behavior. A web server reads configuration files, listens on a port, and serves responses. If that same container suddenly spawns a shell process, opens /etc/shadow, or initiates an outbound connection to an unknown IP, something is wrong.

2. Falco Architecture

Falco operates at the kernel level, sitting between the application and the operating system to observe every system call made by every process.

Data Collection Layer

Falco captures system calls using one of two mechanisms:

• eBPF Probe (recommended): A modern, safe approach that loads a small eBPF program into the kernel. No kernel module compilation required, and it works with most modern Linux distributions (kernel 4.14+). The eBPF probe cannot crash the kernel and is the preferred approach for production.
• Kernel Module: The legacy approach that compiles a kernel module matching your exact kernel version. Provides marginally better performance but requires kernel headers and introduces the risk of kernel instability. Being phased out in favor of eBPF.

Rule Engine

The captured system calls are fed into Falco's rule engine, which evaluates them against a set of YAML-defined rules. Each rule specifies a condition (a filter expression over system call fields), an output format, and a priority level.

Alerting Pipeline

When a rule matches, Falco emits an alert. Alerts can be sent to stdout, files, gRPC endpoints, or HTTP endpoints. In production, Falco Sidekick receives these alerts and fans them out to dozens of downstream systems.

3. Falco Rules: Syntax and Examples

Falco rules use a powerful filter syntax based on system call fields. Rules are organized into three constructs: rules (the detection logic), macros (reusable condition fragments), and lists (reusable sets of values).

# Falco rule: Detect a shell spawned inside any container
- rule: Terminal shell in container
  desc: >
    A shell (bash, sh, zsh) was started inside a container.
    This is almost never legitimate in production.
  condition: >
    spawned_process
    and container
    and shell_procs
    and proc.tty != 0
  output: >
    Shell spawned in container
    (user=%user.name container=%container.name
    shell=%proc.name parent=%proc.pname
    cmdline=%proc.cmdline image=%container.image.repository)
  priority: WARNING
  tags: [shell, mitre_execution]

# Macro: reusable condition fragment
- macro: spawned_process
  condition: evt.type = execve and evt.dir = <

# List: set of shell binary names
- list: shell_procs
  items: [bash, sh, zsh, csh, ksh, dash]

# Detect sensitive file access inside containers
- rule: Read sensitive file in container
  desc: Detects attempts to read sensitive files like /etc/shadow
  condition: >
    open_read
    and container
    and sensitive_files
    and not trusted_containers
  output: >
    Sensitive file opened for reading
    (user=%user.name file=%fd.name container=%container.name
    image=%container.image.repository)
  priority: CRITICAL
  tags: [filesystem, mitre_credential_access]

- macro: open_read
  condition: evt.type in (open, openat) and evt.is_open_read = true

- list: sensitive_files
  items: [/etc/shadow, /etc/passwd, /etc/pki, /root/.ssh]

- macro: trusted_containers
  condition: container.image.repository in (my-registry.io/auth-service)

# Detect unexpected outbound network connections
- rule: Unexpected outbound connection
  desc: A container made a network connection to a non-approved destination
  condition: >
    outbound
    and container
    and not (fd.sip in (allowed_outbound_ips))
    and not known_network_containers
  output: >
    Unexpected outbound connection
    (container=%container.name image=%container.image.repository
    connection=%fd.name dest=%fd.sip)
  priority: NOTICE
  tags: [network, mitre_command_and_control]

4. Response with Falco Sidekick

Falco only detects. To respond, you deploy Falco Sidekick, a companion service that receives Falco alerts and routes them to over 50 output destinations.

Common Response Patterns

1. Alerting: Send alerts to Slack, PagerDuty, OpsGenie, or email for human review.
2. SIEM Integration: Forward events to Elasticsearch, Splunk, or Loki for correlation with other security data.
3. Automated Remediation: Trigger a Kubernetes Job or AWS Lambda function to delete the compromised pod, cordon the node, or apply a NetworkPolicy to isolate the workload.
4. Forensics: Write alert data to object storage (S3) for long-term retention and incident investigation.

# Falco Sidekick Helm values for multi-destination alerting
config:
  slack:
    webhookurl: "https://hooks.slack.com/services/T00/B00/xxxxx"
    minimumpriority: "warning"         # Only alert on WARNING and above
  pagerduty:
    routingkey: "your-pagerduty-key"
    minimumpriority: "critical"        # Page on-call for CRITICAL only
  elasticsearch:
    hostport: "https://elasticsearch.monitoring:9200"
    index: "falco-alerts"
    minimumpriority: "notice"          # Index everything NOTICE and above
  kubernetesclient:
    enabled: true                      # Enable pod deletion response
    namespace: "falco"
    kubeconfig: ""                     # Uses in-cluster config

5. Tetragon: eBPF-Based Security Observability

Tetragon, a CNCF project from Isovalent (the Cilium team), takes a different approach to runtime security. Rather than rule-based detection, Tetragon uses eBPF to attach security-relevant hooks at multiple kernel layers, providing both observability and enforcement.

Key differences from Falco:

• Enforcement, not just detection: Tetragon can block system calls in real time using eBPF, killing processes or denying operations before they complete. Falco only detects and alerts.
• TracingPolicy CRDs: Security policies are defined as Kubernetes Custom Resources, making them manageable through GitOps.
• Lower overhead: Because Tetragon filters events in the kernel before they reach user space, it generates less data and uses fewer resources for high-volume workloads.

# Tetragon TracingPolicy: block binary execution in sensitive containers
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: block-shell-in-production
spec:
  kprobes:
    - call: "sys_execve"
      syscall: true
      args:
        - index: 0
          type: "string"
      selectors:
        - matchArgs:
            - index: 0
              operator: "In"
              values:
                - "/bin/bash"
                - "/bin/sh"
          matchNamespaces:
            - namespace: production
          matchActions:
            - action: Sigkill          # Kill the process immediately

6. Security Profiles: Seccomp, AppArmor, and SELinux

While Falco and Tetragon detect or respond to threats, security profiles prevent them at the kernel level by restricting what system calls and resources a container can access.

Seccomp (Secure Computing Mode)

Seccomp profiles define a whitelist or blacklist of allowed system calls for a container. Kubernetes supports seccomp profiles natively through the pod security context.

# Pod with a custom seccomp profile that blocks dangerous syscalls
apiVersion: v1
kind: Pod
metadata:
  name: secure-app
spec:
  securityContext:
    seccompProfile:
      type: Localhost
      localhostProfile: profiles/restricted.json  # Stored on the node
  containers:
    - name: app
      image: my-app:v1.2.0
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true

The Security Profiles Operator (SPO) simplifies seccomp management by allowing you to define profiles as Kubernetes Custom Resources and automatically distributing them to nodes. SPO can also record the system calls an application makes during normal operation and generate a least-privilege seccomp profile.

AppArmor

AppArmor provides mandatory access control using profiles that restrict file access, network operations, and capabilities. Kubernetes supports AppArmor annotations on pods, and starting in Kubernetes 1.30, AppArmor profiles can be set directly in the securityContext field.

SELinux

SELinux uses labels and policies to control access between processes and files. It is commonly used on RHEL-based distributions. SELinux labels can be set on pods via the seLinuxOptions field in the security context.

7. Runtime vs. Admission-Time Security

Understanding the distinction between these two layers is critical for building defense in depth:

Aspect	Admission-Time	Runtime
When	Before the pod is created	While the pod is running
Tools	OPA/Gatekeeper, Kyverno	Falco, Tetragon
What it catches	Policy violations in YAML	Actual malicious behavior
Examples	Block privileged pods, require labels	Detect shell execution, file access
Limitation	Cannot detect runtime exploits	Cannot prevent pod creation

A well-secured cluster uses both: admission controllers prevent known-bad configurations from being deployed, and runtime security detects unknown threats that bypass static checks.

8. CIS Benchmarks for Runtime Security

The Center for Internet Security (CIS) Kubernetes Benchmark includes runtime-relevant controls:

• 4.2.1: Ensure that the kubelet --anonymous-auth argument is set to false.
• 5.1.1: Ensure that the cluster-admin role is used sparingly.
• 5.7.1: Create administrative boundaries between resources using namespaces.
• 5.7.2: Ensure that seccomp profiles are applied to all pods.

Tools like kube-bench automate CIS benchmark checks. Run them regularly and combine findings with Falco alerts for comprehensive security posture assessment.

Common Pitfalls

1. Alert fatigue: Deploying Falco with default rules in a large cluster generates thousands of alerts daily. Tune rules aggressively: whitelist known-good behaviors, suppress noisy containers, and set appropriate priority thresholds.
2. Missing container metadata: If Falco cannot resolve container names or images, alerts become useless. Ensure the container runtime socket is mounted correctly and that Falco has access to the CRI endpoint.
3. Kernel compatibility: eBPF probes require kernel 4.14+ with BTF (BPF Type Format) support. Older kernels may need the kernel module, which introduces maintenance overhead.
4. Seccomp profiles that break applications: An overly restrictive seccomp profile can crash your application. Always profile in "log" mode first (using the Security Profiles Operator's recording feature) before switching to "enforce."
5. Ignoring egress monitoring: Most teams focus on detecting shell access, but data exfiltration via unexpected outbound connections is equally dangerous. Always include network-based rules.
6. Running Falco without Sidekick: Falco logging to stdout is useless in production. Deploy Falco Sidekick from day one with at least one alerting destination configured.

What's Next?

• Deploy Falco using the official Helm chart and customize rules for your workload patterns.
• Evaluate Tetragon for workloads that need kernel-level enforcement, not just detection.
• Implement the Security Profiles Operator to generate and manage seccomp profiles as Kubernetes resources.
• Integrate Falco alerts with your SIEM platform for correlation with network and infrastructure events.
• Run kube-bench to assess your cluster against CIS benchmarks and remediate findings.
• Explore runtime class configurations to isolate sensitive workloads using gVisor or Kata Containers for additional runtime protection.