Abnormal Kubernetes Behavior using Splunk Infrastructure Monitoring
Kubernetes Process with Resource Ratio Anomalies: networkKubernetesNoneversion:9
The following analytic detects anomalous changes in resource utilization ratios for processes running on a Kubernetes node. It leverages process metrics collected via an OTEL collector and hostmetrics receiver, analyzed through Splunk Observability Cloud. The detection uses a lookup table containing average and standard deviation values for various resource ratios (e.g., CPU:memory, CPU:disk operations). Significant deviations from these baselines may indicate compromised processes, malicious activity, or misconfigurations. If confirmed malicious, this could signify a security breach, allowing attackers to manipulate workloads, potentially leading to data exfiltration or service disruption.
Kubernetes Anomalous Inbound Network Activity from Process: networkKubernetesNoneversion:9
The following analytic identifies anomalous inbound network traffic volumes from processes within containerized workloads. It leverages Network Performance Monitoring metrics collected via an OTEL collector and pulled from Splunk Observability Cloud. The detection compares recent metrics (tcp.bytes, tcp.new_sockets, tcp.packets, udp.bytes, udp.packets) over the last hour with the average over the past 30 days. This activity is significant as it may indicate unauthorized data reception, potential breaches, vulnerability exploitation, or malware propagation. If confirmed malicious, it could lead to command and control installation, data integrity damage, container escape, and further environment compromise.
Kubernetes newly seen TCP edge: networkKubernetesNoneversion:10
The following analytic identifies newly seen TCP communication between source and destination workload pairs within a Kubernetes cluster. It leverages Network Performance Monitoring metrics collected via an OTEL collector and pulled from Splunk Observability Cloud. The detection compares network activity over the last hour with the past 30 days to spot new inter-workload communications. This is significant as new connections can indicate changes in application behavior or potential security threats. If malicious, unauthorized connections could lead to data breaches, privilege escalation, lateral movement, or disruption of critical services, compromising the application's integrity, availability, and confidentiality.
Kubernetes Process with Anomalous Resource Utilisation: networkKubernetesNoneversion:9
The following analytic identifies high resource utilization anomalies in Kubernetes processes. It leverages process metrics from an OTEL collector and hostmetrics receiver, fetched via the Splunk Infrastructure Monitoring Add-on. The detection uses a lookup table with average and standard deviation values to spot anomalies. This activity is significant as high resource utilization can indicate security threats like cryptojacking, unauthorized data exfiltration, or compromised containers. If confirmed malicious, such anomalies can disrupt services, exhaust resources, increase costs, and allow attackers to evade detection or maintain access.
Kubernetes Process Running From New Path: networkKubernetesNoneversion:10
The following analytic identifies processes running from newly seen paths within a Kubernetes environment. It leverages process metrics collected via an OTEL collector and hostmetrics receiver, and data is pulled from Splunk Observability Cloud using the Splunk Infrastructure Monitoring Add-on. This detection compares processes observed in the last hour with those seen over the previous 30 days. This activity is significant as it may indicate unauthorized changes, compromised nodes, or the introduction of malicious software. If confirmed malicious, it could lead to unauthorized process execution, control over critical resources, data exfiltration, privilege escalation, or malware introduction within the Kubernetes cluster.
Kubernetes newly seen UDP edge: networkKubernetesNoneversion:10
The following analytic detects UDP communication between a newly seen source and destination workload pair within a Kubernetes cluster. It leverages Network Performance Monitoring metrics collected via an OTEL collector and pulled from Splunk Observability Cloud. This detection compares network activity over the last hour with the past 30 days to identify new inter-workload communication. Such changes in network behavior can indicate potential security threats or anomalies. If confirmed malicious, unauthorized connections may enable attackers to infiltrate the application ecosystem, leading to data breaches, privilege escalation, lateral movement, or disruption of critical services.
Kubernetes Anomalous Inbound Outbound Network IO: networkKubernetesNoneversion:9
The following analytic identifies high inbound or outbound network I/O anomalies in Kubernetes containers. It leverages process metrics from an OTEL collector and Kubelet Stats Receiver, along with data from Splunk Observability Cloud. A lookup table with average and standard deviation values for network I/O is used to detect anomalies persisting over a 1-hour period. This activity is significant as it may indicate data exfiltration, command and control communication, or unauthorized data transfers. If confirmed malicious, it could lead to data breaches, service outages, financial losses, and reputational damage.
Kubernetes Anomalous Traffic on Network Edge: networkKubernetesNoneversion:9
The following analytic identifies anomalous network traffic volumes between Kubernetes workloads or between a workload and external sources. It leverages Network Performance Monitoring metrics collected via an OTEL collector and pulled from Splunk Observability Cloud. The detection compares recent network metrics (tcp.bytes, tcp.new_sockets, tcp.packets, udp.bytes, udp.packets) over the last hour with the average over the past 30 days to identify significant deviations. This activity is significant as unexpected spikes may indicate unauthorized data transfers or lateral movement. If confirmed malicious, it could lead to data exfiltration or compromise of additional services, potentially resulting in data breaches.
Kubernetes Anomalous Inbound to Outbound Network IO Ratio: networkKubernetesNoneversion:9
The following analytic identifies significant changes in network communication behavior within Kubernetes containers by examining the inbound to outbound network IO ratios. It leverages process metrics from an OTEL collector and Kubelet Stats Receiver, along with data from Splunk Observability Cloud. Anomalies are detected using a lookup table containing average and standard deviation values for network IO, triggering an event if the anomaly persists for over an hour. This activity is significant as it may indicate data exfiltration, command and control communication, or compromised container behavior. If confirmed malicious, it could lead to data breaches, service outages, and unauthorized access within the Kubernetes cluster.
Kubernetes Previously Unseen Process: networkKubernetesNoneversion:10
The following analytic detects previously unseen processes within the Kubernetes environment on master or worker nodes. It leverages process metrics collected via an OTEL collector and hostmetrics receiver, and data is pulled from Splunk Observability Cloud. This detection compares processes observed in the last hour against those seen in the previous 30 days. Identifying new processes is crucial as they may indicate unauthorized activity or attempts to compromise the node. If confirmed malicious, these processes could lead to data exfiltration, privilege escalation, denial-of-service attacks, or the introduction of malware, posing significant risks to the Kubernetes cluster.
Kubernetes Shell Running on Worker Node with CPU Activity: networkKubernetesNoneversion:10
The following analytic identifies shell activity within the Kubernetes privilege scope on a worker node, specifically when shell processes are consuming CPU resources. It leverages process metrics from an OTEL collector hostmetrics receiver, pulled from Splunk Observability Cloud via the Splunk Infrastructure Monitoring Add-on, focusing on process.cpu.utilization and process.memory.utilization. This activity is significant as unauthorized shell processes can indicate a security threat, potentially compromising the node and the entire Kubernetes cluster. If confirmed malicious, attackers could gain full control over the host's resources, leading to data theft, service disruption, privilege escalation, and further attacks within the cluster.
Kubernetes Anomalous Outbound Network Activity from Process: networkKubernetesNoneversion:9
The following analytic identifies anomalously high outbound network activity from processes running within containerized workloads in a Kubernetes environment. It leverages Network Performance Monitoring metrics collected via an OTEL collector and pulled from Splunk Observability Cloud. The detection compares recent network metrics (tcp.bytes, tcp.new_sockets, tcp.packets, udp.bytes, udp.packets) over the last hour with the average metrics over the past 30 days. This activity is significant as it may indicate data exfiltration, process modification, or container compromise. If confirmed malicious, it could lead to unauthorized data exfiltration, communication with malicious entities, or further attacks within the containerized environment.
Kubernetes Shell Running on Worker Node: networkKubernetesNoneversion:10
The following analytic identifies shell activity within the Kubernetes privilege scope on a worker node. It leverages process metrics from an OTEL collector hostmetrics receiver, specifically process.cpu.utilization and process.memory.utilization, pulled from Splunk Observability Cloud. This activity is significant as unauthorized shell processes can indicate potential security threats, providing attackers an entry point to compromise the node and the entire Kubernetes cluster. If confirmed malicious, this activity could lead to data theft, service disruption, privilege escalation, lateral movement, and further attacks, severely compromising the cluster's security and integrity.
Kubernetes Previously Unseen Container Image Name: networkKubernetesNoneversion:10
The following analytic identifies the creation of containerized workloads using previously unseen images in a Kubernetes cluster. It leverages process metrics from an OTEL collector and Kubernetes cluster receiver, pulled from Splunk Observability Cloud. The detection compares container image names seen in the last hour with those from the previous 30 days. This activity is significant as unfamiliar container images may introduce vulnerabilities, malware, or misconfigurations, posing threats to the cluster's integrity. If confirmed malicious, compromised images can lead to data breaches, service disruptions, unauthorized access, and potential lateral movement within the cluster.
