Skip to main content

K8s Audit Log Workload Created

This alert occurs when Lacework detects a new workload, such as a Pod, Deployment, ReplicaSet, or StatefulSet, was created within the cluster.

Why this alert is important​

Detecting new workloads allows early identification of unauthorized or malicious deployments in the Kubernetes cluster. It helps prevent potential security breaches, such as introducing compromised containers or unauthorized access attempts.

Investigation​

Follow these steps to investigate the alert:

  1. Review the logs generated by the Kubernetes cluster to identify any relevant events or activities related to workload creation. Check for log entries that indicate the creation of pods, deployments, or other workload resources.
  2. Use the Kubernetes API server to gather information about the newly created workload. Retrieve details such as the workload name, namespace, labels, and associated resources. The API server provides a comprehensive view of the cluster's state and can help identify anomalies.
  3. Monitor Kubernetes cluster events to identify any specific events related to workload creation. Kubernetes events capture various actions and changes within the cluster, including creating new workloads. Look for events associated with the workload's namespace, pod, or deployment.
  4. Examine the cluster's configuration, including the deployed resources, deployments, pods, and associated services. Verify if the newly created workload aligns with the intended configurations and policies.
  5. Analyze the network traffic within the Kubernetes cluster to identify any suspicious communication patterns or unexpected connections originating from the new workload. This can help detect potential malicious activities or unauthorized access attempts.
  6. Check the cluster's Role-Based Access Control (RBAC) configuration to verify the permissions associated with the newly created workload. Ensure that the workload has the appropriate permissions and that no unauthorized access has been granted.
  7. Engage with the cluster administrators, developers, or other relevant stakeholders to gather additional information about the new workload. Discuss the workload's purpose, intended functionality, and expected behavior to gain insights into its creation.
  8. Compare the characteristics of the new workload against the known workloads in the cluster. Look for deviations in naming conventions, resource utilization, labels, or other attributes that may indicate a malicious or unauthorized workload.

Resolution​

Follow these steps to resolve the alert:

  1. Disable or remove the unauthorized workload from your Kubernetes cluster. This can be done by deleting the associated pods, deployments, or other workload resources. Take necessary actions to ensure that the workload is no longer active and any associated permissions or access granted by the workload are revoked.
  2. Determine the source of the unauthorized workload creation. It could be due to a compromised account, misconfigured permissions, or other security vulnerabilities. Investigate the root cause to prevent similar incidents in the future. This may involve reviewing access logs, examining audit trails, or further analyzing the cluster's security posture.
  3. Review the Role-Based Access Control (RBAC) configuration of your Kubernetes cluster. Ensure that the RBAC policies are properly configured and that only authorized users or service accounts have permission to create workloads. Adjust the RBAC settings as needed to align with the principle of least privilege.
  4. Implement additional security measures to prevent unauthorized workload creation. This can include implementing multi-factor authentication (MFA) for user accounts, regularly updating and patching your Kubernetes cluster, conducting security audits and assessments, and implementing network security controls such as network policies and firewalls.
  5. Monitor your Kubernetes cluster for any further unauthorized workload creations or suspicious activities. Implement logging and monitoring solutions to detect and alert potential security breaches. Regularly review logs, monitor cluster events, and analyze network traffic to identify anomalies or unauthorized activities.
  6. Educate and train your cluster users on Kubernetes security best practices, including proper access control and reporting unauthorized activities. Encourage them to follow secure coding practices and provide clear workload creation and management guidelines.
  7. Implement a change management process that includes reviewing and approving any workload creations or modifications before they are applied to the production environment. This can help ensure all workloads are authorized and adhere to the security requirements.