Falco is a Cloud Native Runtime Security tool designed to detect anomalous activity in your applications. You can use Falco to monitor runtime security of your Kubernetes applications and internal components.
The deployment of Falco in a Kubernetes cluster is managed through a Helm chart. This chart manages the lifecycle of Falco in a cluster by handling all the k8s objects needed by Falco to be seamlessly integrated in your environment. Based on the configuration in values.yaml file, the chart will render and install the required k8s objects. Keep in mind that Falco could be deployed in your cluster using a daemonset or a deployment. See next sections for more info.
Before installing Falco in a Kubernetes cluster, a user should check that the kernel version used in the nodes is supported by the community. Also, before reporting any issue with Falco (missing kernel image, CrashLoopBackOff and similar), make sure to read about the driver section and adjust your setup as required.
Before installing the chart, add the falcosecurity charts repository:
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo updateTo install the chart with the release name falco in namespace falco run:
helm install falco falcosecurity/falco --namespace falco --create-namespaceAfter a few minutes Falco instances should be running on all your nodes. The status of Falco pods can be inspected through kubectl:
kubectl get pods -n falco -o wideIf everything went smoothly, you should observe an output similar to the following, indicating that all Falco instances are up and running in you cluster:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
falco-57w7q 1/1 Running 0 3m12s 10.244.0.1 control-plane <none> <none>
falco-h4596 1/1 Running 0 3m12s 10.244.1.2 worker-node-1 <none> <none>
falco-kb55h 1/1 Running 0 3m12s 10.244.2.3 worker-node-2 <none> <none>The cluster in our example has three nodes, one control-plane node and two worker nodes. The default configuration in values.yaml of our helm chart deploys Falco using a daemonset. That's the reason why we have one Falco pod in each node.
Tip: List Falco release using
helm list -n falco, a release is a name used to track a specific deployment
Starting from Falco 0.31.0 the new plugin system is stable and production ready. The plugin system can be seen as the next step in the evolution of Falco. Historically, Falco monitored system events from the kernel trying to detect malicious behaviors on Linux systems. It also had the capability to process k8s Audit Logs to detect suspicious activities in Kubernetes clusters. Since Falco 0.32.0 all the related code to the k8s Audit Logs in Falco was removed and ported in a plugin. At the time being Falco supports different event sources coming from plugins or drivers (system events).
Note that a Falco instance can handle multiple event sources in parallel. you can deploy Falco leveraging drivers for syscalls events and at the same time loading plugins. A step by step guide on how to deploy Falco with multiple sources can be found here.
Falco needs a driver to analyze the system workload and pass security events to userspace. The supported drivers are:
The driver should be installed on the node where Falco is running. The kernel module (default option) and the eBPF probe are installed on the node through an init container (i.e. falco-driver-loader) that tries to build drivers to download a prebuilt driver or build it on-the-fly or as a fallback. The Modern eBPF probe doesn't require an init container because it is shipped directly into the Falco binary. However, the Modern eBPF probe requires recent BPF features
The kernel-crawler automatically discovers kernel versions and flavors. At the time being, it runs weekly. We have a site where users can check for the discovered kernel flavors and versions, example for Amazon Linux 2.
The discovery of a kernel version by the kernel-crawler does not imply that pre-built kernel modules and bpf probes are available. That is because once kernel-crawler has discovered new kernels versions, the drivers need to be built by jobs running on our Driver Build Grid infra. Please keep in mind that the building process is based on best effort. Users can check the existence of prebuilt modules at the following link.
If a prebuilt driver is not available for your distribution/kernel, users can build the modules by them self or install the kernel headers on the nodes, and the init container (falco-driver-loader) will try and build the module on the fly.
Falco needs kernel headers installed on the host as a prerequisite to build the driver on the fly correctly. You can find instructions for installing the kernel headers for your system under the Install section of the official documentation.
Plugins are used to extend Falco to support new data sources. The current plugin framework supports plugins with the following capabilities:
- Event sourcing capability;
- Field extraction capability;
Plugin capabilities are composable, we can have a single plugin with both capabilities. Or on the other hand, we can load two different plugins each with its capability, one plugin as a source of events and another as an extractor. A good example of this is the Kubernetes Audit Events and the Falcosecurity Json plugins. By deploying them both we have support for the K8s Audit Logs in Falco
Note that the driver is not required when using plugins.
gVisor is an application kernel, written in Go, that implements a substantial portion of the Linux system call interface. It provides an additional layer of isolation between running applications and the host operating system. For more information please consult the official docs. In version 0.32.1, Falco first introduced support for gVisor by leveraging the stream of system call information coming from gVisor.
Falco requires the version of runsc to be equal to or above 20220704.0. The following snippet shows the gVisor configuration variables found in values.yaml:
gvisor:
enabled: true
runsc:
path: /home/containerd/usr/local/sbin
root: /run/containerd/runsc
config: /run/containerd/runsc/config.tomlFalco uses the runsc binary to interact with sandboxed containers. The following variables need to be set:
runsc.path: absolute path of therunscbinary in the k8s nodes;runsc.root: absolute path of the root directory of therunsccontainer runtime. It is of vital importance for Falco sincerunscstores there the information of the workloads handled by it;runsc.config: absolute path of therunscconfiguration file, used by Falco to set its configuration and make awaregVisorof its presence.
If you want to know more how Falco uses those configuration paths please have a look at the falco.gvisor.initContainer helper in helpers.tpl.
A preset values.yaml file values-gvisor-gke.yaml is provided and can be used as it is to deploy Falco with gVisor support in a GKE cluster. It is also a good starting point for custom deployments.
If you use GKE with k8s version at least 1.24.4-gke.1800 or 1.25.0-gke.200 with gVisor sandboxed pods, you can install a Falco instance to monitor them with, e.g.:
helm install falco-gvisor falcosecurity/falco -f https://raw.githubusercontent.com/falcosecurity/charts/master/falco/values-gvisor-gke.yaml --namespace falco-gvisor --create-namespace
Note that the instance of Falco above will only monitor gVisor sandboxed workloads on gVisor-enabled node pools. If you also need to monitor regular workloads on regular node pools you can use the eBPF driver as usual:
helm install falco falcosecurity/falco --set driver.kind=ebpf --namespace falco --create-namespace
The two instances of Falco will operate independently and can be installed, uninstalled or configured as needed. If you were already monitoring your regular node pools with eBPF you don't need to reinstall it.
An exhaustive blog post about Falco and gVisor can be found on the Falco blog. If you need help on how to set gVisor in your environment please have a look at the gVisor official docs
Historically rules files and plugins used to be shipped inside the Falco docker image and/or inside the chart. Starting from version v0.3.0 of the chart, the falcoctl tool can be used to install/update rules files and plugins. When referring to such objects we will use the term artifact. For more info please check out the following proposal.
The default configuration of the chart for new installations is to use the falcoctl tool to handle artifacts. The chart will deploy two new containers along the Falco one:
falcoctl-artifact-installan init container that makes sure to install the configured artifacts before the Falco container starts;falcoctl-artifact-followa sidecar container that periodically checks for new artifacts (currently only falco-rules) and downloads them;
For more info on how to enable/disable and configure the falcoctl tool checkout the config values here and the upgrading notes
After the clarification of the different event sources and how they are consumed by Falco using the drivers and the plugins, now let us discuss how Falco is deployed in Kubernetes.
The chart deploys Falco using a daemonset or a deployment depending on the event sources.
When using the drivers, Falco is deployed as daemonset. By using a daemonset, k8s assures that a Falco instance will be running in each of our nodes even when we add new nodes to our cluster. So it is the perfect match when we need to monitor all the nodes in our cluster.
Kernel module
To run Falco with the kernel module you can use the default values of the helm chart:
driver:
enabled: true
kind: moduleeBPF probe
To run Falco with the eBPF probe you just need to set driver.kind=ebpf as shown in the following snippet:
driver:
enabled: true
kind: ebpfThere are other configurations related to the eBPF probe, for more info please check the values.yaml file. After you have made your changes to the configuration file you just need to run:
helm install falco falcosecurity/falco --namespace "your-custom-name-space" --create-namespacemodern eBPF probe
To run Falco with the modern eBPF probe you just need to set driver.kind=modern-bpf as shown in the following snippet:
driver:
enabled: true
kind: modern-bpfIn the scenario when Falco is used with plugins as data sources, then the best option is to deploy it as a k8s deployment. Plugins could be of two types, the ones that follow the push model or the pull model. A plugin that adopts the firs model expects to receive the data from a remote source in a given endpoint. They just expose and endpoint and wait for data to be posted, for example Kubernetes Audit Events expects the data to be sent by the k8s api server when configured in such way. On the other hand other plugins that abide by the pull model retrieves the data from a given remote service.
The following points explain why a k8s deployment is suitable when deploying Falco with plugins:
- need to be reachable when ingesting logs directly from remote services;
- need only one active replica, otherwise events will be sent/received to/from different Falco instances;
To uninstall a Falco release from your Kubernetes cluster always you helm. It will take care to remove all components deployed by the chart and clean up your environment. The following command will remove a release called falco in namespace falco;
helm uninstall falco --namespace falcoThere are many reasons why we would have to inspect the messages emitted by the Falco container. When deployed in Kubernetes the Falco logs can be inspected through:
kubectl logs -n falco falco-pod-namewhere falco-pods-name is the name of the Falco pod running in your cluster.
The command described above will just display the logs emitted by falco until the moment you run the command. The -f flag comes handy when we are doing live testing or debugging and we want to have the Falco logs as soon as they are emitted. The following command:
kubectl logs -f -n falco falco-pod-nameThe -f (--follow) flag follows the logs and live stream them to your terminal and it is really useful when you are debugging a new rule and want to make sure that the rule is triggered when some actions are performed in the system.
If we need to access logs of a previous Falco run we do that by adding the -p (--previous) flag:
kubectl logs -p -n falco falco-pod-nameA scenario when we need the -p (--previous) flag is when we have a restart of a Falco pod and want to check what went wrong.
By default in Falco the output is buffered. When live streaming logs we will notice delays between the logs output (rules triggering) and the event happening.
In order to enable the logs to be emitted without delays you need to set .Values.tty=true in values.yaml file.
Falco ships with a nice default ruleset. It is a good starting point but sooner or later, we are going to need to add custom rules which fit our needs.
So the question is: How can we load custom rules in our Falco deployment?
We are going to create a file that contains custom rules so that we can keep it in a Git repository.
cat custom-rules.yamlAnd the file looks like this one:
customRules:
rules-traefik.yaml: |-
- macro: traefik_consider_syscalls
condition: (evt.num < 0)
- macro: app_traefik
condition: container and container.image startswith "traefik"
# Restricting listening ports to selected set
- list: traefik_allowed_inbound_ports_tcp
items: [443, 80, 8080]
- rule: Unexpected inbound tcp connection traefik
desc: Detect inbound traffic to traefik using tcp on a port outside of expected set
condition: inbound and evt.rawres >= 0 and not fd.sport in (traefik_allowed_inbound_ports_tcp) and app_traefik
output: Inbound network connection to traefik on unexpected port (command=%proc.cmdline pid=%proc.pid connection=%fd.name sport=%fd.sport user=%user.name %container.info image=%container.image)
priority: NOTICE
# Restricting spawned processes to selected set
- list: traefik_allowed_processes
items: ["traefik"]
- rule: Unexpected spawned process traefik
desc: Detect a process started in a traefik container outside of an expected set
condition: spawned_process and not proc.name in (traefik_allowed_processes) and app_traefik
output: Unexpected process spawned in traefik container (command=%proc.cmdline pid=%proc.pid user=%user.name %container.info image=%container.image)
priority: NOTICESo next step is to use the custom-rules.yaml file for installing the Falco Helm chart.
helm install falco -f custom-rules.yaml falcosecurity/falcoAnd we will see in our logs something like:
Tue Jun 5 15:08:57 2018: Loading rules from file /etc/falco/rules.d/rules-traefik.yaml:And this means that our Falco installation has loaded the rules and is ready to help us.
The Kubernetes Audit Log is now supported via the built-in k8saudit plugin. It is entirely up to you to set up the webhook backend of the Kubernetes API server to forward the Audit Log event to the Falco listening port.
The following snippet shows how to deploy Falco with the k8saudit plugin:
# -- Disable the drivers since we want to deplouy only the k8saudit plugin.
driver:
enabled: false
# -- Disable the collectors, no syscall events to enrich with metadata.
collectors:
enabled: false
# -- Deploy Falco as a deployment. One instance of Falco is enough. Anyway the number of replicas is configurabale.
controller:
kind: deployment
deployment:
# -- Number of replicas when installing Falco using a deployment. Change it if you really know what you are doing.
# For more info check the section on Plugins in the README.md file.
replicas: 1
falcoctl:
artifact:
install:
# -- Enable the init container. We do not recommend installing (or following) plugins for security reasons since they are executable objects.
enabled: true
follow:
# -- Enable the sidecar container. We do not support it yet for plugins. It is used only for rules feed such as k8saudit-rules rules.
enabled: true
config:
artifact:
install:
# -- Do not resolve the depenencies for artifacts. By default is true, but for our use case we disable it.
resolveDeps: false
# -- List of artifacts to be installed by the falcoctl init container.
# Only rulesfiles, we do no recommend plugins for security reasonts since they are executable objects.
refs: [k8saudit-rules:0.5]
follow:
# -- List of artifacts to be followed by the falcoctl sidecar container.
# Only rulesfiles, we do no recommend plugins for security reasonts since they are executable objects.
refs: [k8saudit-rules:0.5]
services:
- name: k8saudit-webhook
type: NodePort
ports:
- port: 9765 # See plugin open_params
nodePort: 30007
protocol: TCP
falco:
rules_file:
- /etc/falco/k8s_audit_rules.yaml
- /etc/falco/rules.d
plugins:
- name: k8saudit
library_path: libk8saudit.so
init_config:
""
# maxEventBytes: 1048576
# sslCertificate: /etc/falco/falco.pem
open_params: "http://:9765/k8s-audit"
- name: json
library_path: libjson.so
init_config: ""
# Plugins that Falco will load. Note: the same plugins are installed by the falcoctl-artifact-install init container.
load_plugins: [k8saudit, json]
Here is the explanation of the above configuration:
- disable the drivers by setting
driver.enabled=false; - disable the collectors by setting
collectors.enabled=false; - deploy the Falco using a k8s deploment by setting
controller.kind=deployment; - makes our Falco instance reachable by the
k8s api-serverby configuring a service for it inservices; - enable the
falcoctl-artifact-installinit container; - configure
falcoctl-artifact-installto install the required plugins; - disable the
falcoctl-artifact-followsidecar container; - load the correct ruleset for our plugin in
falco.rulesFile; - configure the plugins to be loaded, in this case, the
k8sauditandjson; - and finally we add our plugins in the
load_pluginsto be loaded by Falco.
The configuration can be found in the values-k8saudit.yaml file ready to be used:
#make sure the falco namespace exists
helm install falco falcosecurity/falco --namespace falco -f ./values-k8saudit.yaml --create-namespaceAfter a few minutes a Falco instance should be running on your cluster. The status of Falco pod can be inspected through kubectl:
kubectl get pods -n falco -o wideIf everything went smoothly, you should observe an output similar to the following, indicating that the Falco instance is up and running:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
falco-64484d9579-qckms 1/1 Running 0 101s 10.244.2.2 worker-node-2 <none> <none>Furthermore you can check that Falco logs through kubectl logs
kubectl logs -n falco falco-64484d9579-qckmsIn the logs you should have something similar to the following, indcating that Falco has loaded the required plugins:
Fri Jul 8 16:07:24 2022: Falco version 0.32.0 (driver version 39ae7d40496793cf3d3e7890c9bbdc202263836b)
Fri Jul 8 16:07:24 2022: Falco initialized with configuration file /etc/falco/falco.yaml
Fri Jul 8 16:07:24 2022: Loading plugin (k8saudit) from file /usr/share/falco/plugins/libk8saudit.so
Fri Jul 8 16:07:24 2022: Loading plugin (json) from file /usr/share/falco/plugins/libjson.so
Fri Jul 8 16:07:24 2022: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Fri Jul 8 16:07:24 2022: Starting internal webserver, listening on port 8765Note that the support for the dynamic backend (also known as the AuditSink object) has been deprecated from Kubernetes and removed from this chart.
Using kops edit cluster, ensure these options are present, then run kops update cluster and kops rolling-update cluster:
spec:
kubeAPIServer:
auditLogMaxBackups: 1
auditLogMaxSize: 10
auditLogPath: /var/log/k8s-audit.log
auditPolicyFile: /srv/kubernetes/assets/audit-policy.yaml
auditWebhookBatchMaxWait: 5s
auditWebhookConfigFile: /srv/kubernetes/assets/webhook-config.yaml
fileAssets:
- content: |
# content of the webserver CA certificate
# remove this fileAsset and certificate-authority from webhook-config if using http
name: audit-ca.pem
roles:
- Master
- content: |
apiVersion: v1
kind: Config
clusters:
- name: falco
cluster:
# remove 'certificate-authority' when using 'http'
certificate-authority: /srv/kubernetes/assets/audit-ca.pem
server: https://localhost:32765/k8s-audit
contexts:
- context:
cluster: falco
user: ""
name: default-context
current-context: default-context
preferences: {}
users: []
name: webhook-config.yaml
roles:
- Master
- content: |
# ... paste audit-policy.yaml here ...
# https://raw.githubusercontent.com/falcosecurity/evolution/master/examples/k8s_audit_config/audit-policy.yaml
name: audit-policy.yaml
roles:
- MasterThe Falco gRPC server and the Falco gRPC Outputs APIs are not enabled by default. Moreover, Falco supports running a gRPC server with two main binding types:
- Over a local Unix socket with no authentication
- Over the network with mandatory mutual TLS authentication (mTLS)
Tip: Once gRPC is enabled, you can deploy falco-exporter to export metrics to Prometheus.
The preferred way to use the gRPC is over a Unix socket.
To install Falco with gRPC enabled over a unix socket, you have to:
helm install falco \
--set falco.grpc.enabled=true \
--set falco.grpc_output.enabled=true \
falcosecurity/falcoThe gRPC server over the network can only be used with mutual authentication between the clients and the server using TLS certificates. How to generate the certificates is documented here.
To install Falco with gRPC enabled over the network, you have to:
helm install falco \
--set falco.grpc.enabled=true \
--set falco.grpc_output.enabled=true \
--set falco.grpc.unixSocketPath="" \
--set-file certs.server.key=/path/to/server.key \
--set-file certs.server.crt=/path/to/server.crt \
--set-file certs.ca.crt=/path/to/ca.crt \
falcosecurity/falcoFalcosidekick can be installed with Falco by setting --set falcosidekick.enabled=true. This setting automatically configures all options of Falco for working with Falcosidekick.
All values for the configuration of Falcosidekick are available by prefixing them with falcosidekick.. The full list of available values is here.
For example, to enable the deployment of Falcosidekick-UI, add --set falcosidekick.enabled=true --set falcosidekick.webui.enabled=true.
If you use a Proxy in your cluster, the requests between Falco and Falcosidekick might be captured, use the full FQDN of Falcosidekick by using --set falcosidekick.fullfqdn=true to avoid that.
All the configurable parameters of the falco chart and their default values can be found here.