Logging & Monitoring Strategy Guide

Resources for implementing a comprehensive logging and alerting strategy in an AWS environment. Included are customizable configuration items and packages, as well as guides to configure logging and alerting for AWS account activity, threat detection, configuration compliance, and service-specific logs. In addition to enabling logging and monitoring, resources for reviewing, analyzing and visualizing logs are also covered.

     

AWS Account Activity

Ensure governance, compliance, operational auditing, and risk auditing of AWS accounts by configuring AWS CloudTrail and AWS Config:

  • AWS CloudTrail provides event-history of an AWS account activity, including actions taken through the AWS Management Console, AWS SDKs, command-line tools, and other AWS services.
  • AWS Config tracks changes in configurations and relationships between AWS resources and provides detailed resource configuration histories. Config can also be used for overall configuration compliance using Config Rules.

CloudTrail and Config store their logs in S3 buckets. CloudTrail also provides an option to forward logs to a CloudWatch Log Group, which can allow for better search capability and notification creation using CloudWatch metrics and alarms. 

Enable AWS CloudTrail with CloudWatch Logs Integration
Configuration Item
Add to Stack
Configuration to enable AWS CloudTrail including configuration to stream CloudTrail events to CloudWatch Logs.
Enable AWS Config
Configuration Item
Add to Stack
Configuration to enable AWS Config including support configuration such as S3 Buckets and Iam Roles as required.

CloudWatch Metrics and Alarms can be defined to track and alert on critical CloudTrail events (if CloudTrail to CloudWatch forwarding is enabled).

Common CloudWatch Alarm Configuration Templates
Collection
A repository of common CloudWatch Alarm configurations

In addition to tracking account activity and configuration changes, it is recommended to track cost and usage. AWS Budgets allows tracking both costs and usage with notification capability based on actual or predicted thresholds

AWS Cost Budget with Notification
Configuration Item
Add to Stack
AWS Budgets provide the ability to set custom budgets that can alert when costs exceed (or are forecasted to exceed) the budgeted amount. A notification has been configured when the actual costs exceed 80% of the budget (Default is 1000 USD).

Network Activity

AWS provides capabilities to log network activity for resources deployed in VPCs using the following options:

  • VPC Flow Logs capture network flow information for IP traffic going to and from network interfaces in a VPC (includes source/destination IP address and ports, bytes transferred, firewall action, and more). Flow log data can be published to Amazon CloudWatch Logs or Amazon S3
  • VPC Traffic Mirroring creates a copy of the network traffic to/from a specific interface in a VPC and streams it to a specified destination for analysis
  • VPC DNS Logging monitor DNS queries in a VPC by configuring Route53 Resolver Query Logging
VPC Flow Logs
Configuration Item
Add to Stack
Enable VPC Flow Logs for an existing VPC, subnet or network interface. Flow Logs enables you to capture information about the IP traffic going to and from network interfaces in your VPC.
VPC Traffic Mirroring
Configuration Item
Add to Stack
Configuration to enable Traffic Mirroring from a network interface (ENI) of an Amazon EC2 instance, which can then be used for monitoring and security analysis. Traffic Mirroring supports filters and packet truncation so that only traffic of interest is monitored.
VPC DNS Query Logging (Route53 Resolver Query Logging)
Configuration Item
Add to Stack
Configuration to enable logging the DNS queries that originate in an Amazon VPC using the Route53 Resolver Query Logging feature. Query logs can be sent to CloudWatch logs, S3 Buckets, or Kinesis Data Firehose

Threat Detection

Amazon GuardDuty is a threat detection service that continuously monitors for malicious activity and unauthorized behavior by analyzing CloudTrail, VPC Flow Log and DNS Log activity in an AWS Account.

Amazon GuardDuty
Configuration Item
Add to Stack
Configuration to enable Amazon GuardDuty.
Alert on Amazon GuardDuty Findings with CloudWatch Events
Configuration Item
Add to Stack
A CloudWatch Event Rule that triggers on Amazon GuardDuty findings.
Configuration Package: Amazon GuardDuty with Alerting and Compliance Checks
Configuration Package
A configuration package to enable Amazon GuardDuty in an AWS account as well as email notifications for GuardDuty findings (using a CloudWatch Event Rule), and an AWS Config Rule to verify that GuardDuty is continuously enabled.

AWS WAF helps protect internet-facing applications and API endpoints. AWS WAF integrates with CloudFront, Load Balancers, and API Gateway to inspect (and optionally drop) traffic deemed malicious. Use the AWS Managed Rules package to get started or one of the partner-managed rule packages (e.g. F5, Imperva, Fortinet, etc.)

AWS WAF Configuration Templates
Collection
A collection of AWS Security controls for AWS WAF. Configuration items include templates to set up AWS Managed Rules for AWS WAF Rules in an AWS account to protect CloudFront, API Gateway and ALB resources.
How to Configure AWS WAF Comprehensive Logging to Store Logs in Amazon S3
External: Solution/Guide
Step-by-step instructions on how to enable AWS WAF logging using S3 as the destination

Compliance Monitoring

AWS provides several services to help monitor configuration and ensure compliance with security standards and best-practices: 

  • AWS Security Hub runs automated, continuous security checks based on industry standards and best practices, such as the Center for Internet Security (CIS) AWS Foundations Benchmark, AWS Foundational Security Best Practices and Payment Card Industry Data Security Standard (PCI DSS)
  • AWS Config Rules define conditions that describe the target ideal configuration. When resource configuration changes, AWS Config continuously tracks these changes and checks whether they violate the defined rules.
  • Amazon Macie is used to discover, monitor, and help you protect sensitive data in Amazon S3. Macie automates the discovery of sensitive data, such as personal identifying information (PII) and intellectual property. Macie also identifies overly permissive or unencrypted buckets across AWS accounts.  
  • IAM Access Analyzer helps identify resources in AWS accounts, such as Amazon S3 buckets or IAM roles, that are shared with an external entity, and alert when that happens. 
  • Trusted Advisor provides real-time guidance to help provision resources following AWS best practices including security checks. 
Configuration Package: AWS Security Hub with Alerting and Compliance Checks
Configuration Package
A configuration package to enable Amazon GuardDuty in an AWS account as well as email notifications for GuardDuty findings (using a CloudWatch Event Rule), and an AWS Config Rule to verify that GuardDuty is continuously enabled.
A Collection of AWS Config Compliance Rules
Collection
Repository of AWS Config rules examples - both AWS managed and custom Config rules.
Amazon Macie (S3 Security and Data Classification)
Configuration Item
Add to Stack
Configuration to enable Amazon Macie in an AWS Account. Amazon Macie is used to discover, monitor, and help protect sensitive data in Amazon S3 Buckets
IAM Access Analyzer
Configuration Item
Add to Stack
Configure Access Analyzer to discover which resources in an AWS account are shared with external principals outside of the account. Access Analyzer analyzes generates findings for supported resources in the region it was enabled, with the exception of IAM resources which generates findings in each region (as IAM is a global service)
Alert on AWS Security Hub Findings with CloudWatch Events
Configuration Item
Add to Stack
A CloudWatch Event Rule that triggers on AWS Security Hub findings.
Alert on IAM Access Analyzer Findings with CloudWatch Events
Configuration Item
Add to Stack
A CloudWatch Event Rule that triggers on IAM Access Analyzer findings.
Alert on Config Rule Compliance Changes with CloudWatch Events
Configuration Item
Add to Stack
A CloudWatch Event Rule that triggers on Config Rule Compliance Changes.
Alert on Trusted Advisor Findings with CloudWatch Events
Configuration Item
Add to Stack
A CloudWatch Event Rule that triggers on Trusted Advisor findings.

Service-Specific Access Logs

AWS also provides service-specific activity and access logs that can be turned on selectively for supported resources. It should be noted that there are additional charges associated with enabling logging for these services and depending on the usage, the costs might be high. Some of the common services: 

  • S3: For S3 buckets with important data, S3 Server Access Logs or CloudTrail Data Events for S3 can be enabled to monitor access to S3 buckets and files (including operations such as upload/download/delete).
  • Lambda: CloudTrail Data Events for Lambda can be enabled to monitor Lambda function activity in an AWS account.
  • Load Balancer Logs: ALB/NLB/Classic LB support access logging that can be enabled to monitor requests and traffic patterns through the load balancers. 
  • CloudFront Access Logs: Enable access logs for CloudFront to monitor requests for sites hosted behind CloudFront as well as traffic patterns. 
S3 Bucket with Server Access Logs Enabled
Configuration Item
Add to Stack
Configuration to create an S3 bucket with security configuration options including s3 block public access configuration, encryption, logging, and versioning
Configure S3 Data Event Logging with CloudTrail
Configuration Item
Add to Stack
Configuration to enable AWS CloudTrail in an AWS account for logging S3 Data Events. Data Events for Amazon S3 record object-level API activity (for example, GetObject, DeleteObject, and PutObject API operations)
Configure Lambda Data Event Logging with CloudTrail
Configuration Item
Add to Stack
Configuration to enable AWS CloudTrail in an AWS account for logging Lambda Data Events. Data Events for AWS Lambda record function execution activity (the Invoke API)
AWS Documentation: Configure Application Load Balancer (ALB) Access Logging
External: Solution/Guide
Configuration instructions to enable logging for Application Load Balancers which captures detailed information about requests sent to the load balancer.
AWS Documentation: Configure Network Load Balancer (NLB) Access Logging
External: Solution/Guide
Configuration instructions to enable logging for Network Load Balancers which captures detailed information about requests sent to the load balancer.
AWS Documentation: Configure Classic Load Balancer (ELB) Access Logging
External: Solution/Guide
Configuration instructions to enable logging for Classic Load Balancers which captures detailed information about requests sent to the load balancer.
AWS Documentation: Configure CloudFront Access Logs
External: Solution/Guide
Configuration instructions to enable logging for CloudFront distributions which captures detailed information about requests sent to the load balancer.

EC2 Logs

Using the CloudWatch Log Agent, metrics and logs can be collected from EC2 instances and forwarded to CloudWatch. This includes operating system logs (Windows or Linux) as well as logs for applications running on EC2 instances.

AWS Documentation: Collecting Metrics and Logs from Amazon EC2 Instances
External: Solution/Guide
Step-by-step instructions for installing and configuring the CloudWatch Logs Agent on Windows or Linux systems to collect logs and metrics.
Collect custom metrics from EC2 instances
External: Solution/Guide
This post covers how to enable custom detail monitoring and collect memory and disk metrics using AWS CloudWatch agent, which can be used to build custom CloudWatch dashboards.
CloudWatch Log Group
Configuration Item
Add to Stack
Configuration to create a CloudWatch Log Group with option for defining a log retention period.

Log Visualization and Analysis

In addition to logging services, AWS provides several options to review, analyze and visualize logs (or 3rd party tools such as Splunk or DataDog can be used):

  • CloudWatch Log Insights: For logs stored in CloudWatch, CloudWatch Insights provides an interface to quickly search through and visualize logs using a powerful and flexible langauge. Predefined sample queries are available for CloudTrail, VPC Flow logs, Lambda and more.
  • Amazon Detectivee makes it easy to analyze, investigate, and quickly identify the root cause of potential security issues or suspicious activities. Amazon Detective automatically collects log data from your AWS resources and uses machine learning, statistical analysis, and graph theory to build a linked set of data that enables you to easily conduct faster and more efficient security investigations.
  • Amazon Elasticsearch (and Kibana) which allows for powerful queries and visualizations of the log data, as well as alerting. There are several methods supported to ingest data into Amazon Elasticsearch including a native streaming option for logs stored in CloudWatch Logs.
  • Amazon Athena which is an interactive query service that makes it easy to analyze log data in Amazon S3 using standard SQL. For visualizations, Amazon QuickSight can be used for logs stored in S3.
Using Amazon CloudWatch Logs Insights to Analyze CloudTrail Logs
External: Solution/Guide
This blog post provides an overview of how to use CloudWatch Log Insights to analyze CloudTrail logs. The blog also demonstrates how to visualize and create custom dashboards based on the log data.
How-to Use Amazon Detective for Rapid Security Investigation and Analysis
External: Solution/Guide
This post provides an overview of Amazon Detective, how to enable it, and how to use it to investigate alerts in an AWS account
How-to Investigate VPC flows with Amazon Detective
External: Solution/Guide
Amazon Detective makes it easy to analyze, investigate, and quickly identify the root cause of potential security issues or suspicious activities. Detective automatically collects VPC flow logs from monitored AWS accounts, aggregates them by EC2 instance, and presents visual summaries and analytics about these network flows
Build a SIEM on Amazon Elasticsearch Service
External: Solution/Guide
SIEM on Amazon Elasticsearch Service (Amazon ES) is a solution that collects multiple types of logs from AWS multiple accounts, correlates and visualizes the logs to investigate security incidents
Visualizing Amazon GuardDuty findings with Amazon Elasticsearch
External: Solution/Guide
A solution for visualizing and searching Amazon GuardDuty findings using Amazon Elasticsearch and CloudWatch Events. The solution also includes real time email notifications using SNS for GuardDuty findings.
How to analyze AWS WAF logs using Amazon Elasticsearch Service
External: Solution/Guide
This blog post shows you how you can analyze AWS WAF logs using Amazon Elasticsearch Service (Amazon ES). It also shows how to find out in near-real time which AWS WAF rules get triggered, why, and by which request. Finally, it shows how to create a historical view of your web applications’ access trends for long-term analysis.
How to Visualize Multi-Account Amazon Inspector Findings with Amazon Elasticsearch Service
External: Solution/Guide
This solution automates sending Amazon Inspector findings directly to Amazon ES for visualization in Kibana. These visualizations can be used to build dashboards that security analysts can use for centralized monitoring. This solution also demonstrates how to store the findings from Amazon Inspector in an S3 bucket, which makes it easier for you to use those findings to create visualizations in your preferred security monitoring software.
How to Set Up Alerts in Amazon Elasticsearch Service
External: Solution/Guide
This blog post presents how Amazon ES alerting allows monitoring of critical data in log files for quick response when things start to go wrong. By identifying KPIs, setting thresholds, and distributing alerts to first responders, organizations can improve their response time for critical issues.
Centralizing Windows Logs with Amazon Elasticsearch Services
External: Solution/Guide
This This blog post provides step by step instructions for creating a near real-time event management solution running on AWS using Amazon Elasticsearch Services and Winlogbeat from Elastic as a Windows Agent, and how such a management solution can be used to analyze Windows Logs based Amazon EC2 instances without the need of expensive third-party products.
How to Query AWS Service Logs on S3 Using Amazon Athena
External: Solution/Guide
This post introduces a new open-source library that you can use to efficiently process various types of AWS service logs using AWS Glue. The following log sources are supported: Application Load Balancer, Classic Load Balancer, AWS CloudTrail, Amazon CloudFront, S3 Access, and Amazon VPC Flows
Enabling serverless security analytics using AWS WAF full logs, Amazon Athena, and Amazon QuickSight
External: Solution/Guide
This blog post shows how to analyze AWS Web Application Firewall (AWS WAF) logs and quickly build multiple dashboards, without booting up any servers, and by using Amazon QuickSight dashboards to help visualize web application security logs.
How to Analyze Amazon CloudFront access logs at scale with Amazon Athena
External: Solution/Guide
This blog post shows how you to restructure Amazon CloudFront access logs storage to optimize the cost and performance for queries, and use Amazon Athena to run queries against the logs.
How to Automatically Parse Route 53 Resolver Query Logs with Amazon Athena
External: Solution/Guide
This solution shows how to automatically parse Route 53 Resolver query logs. The example included shows how to mark any query for a domain matching a pre-defined list, and identify where it came from.

Other

How to Monitor Service Usage with CloudWatch Alarms and Service Quotas
External: Solution/Guide
This blog post illustrates how to use Service Quotas to simplify new AWS account setup and easily configure dynamic alarms to monitor service usage.
Central Logging Solution for Applications in Multi-Account Environments
External: Solution/Guide
The solution uses Amazon Kinesis Data Streams and a log destination to set up an endpoint in the logging account to receive streamed logs and uses Amazon Kinesis Data Firehose to deliver log data to the Amazon Simple Storage Solution (S3) bucket. Application accounts will subscribe to stream all (or part) of their Amazon CloudWatch logs to a defined destination in the logging account via subscription filters.
CloudFormation Drift Monitoring
Configuration Package
A configuration package to automatically monitor CloudFormation stack drift (When resources deployed through CloudFormation are manaully changed after), and optionally alert on these events.