Logging & Monitoring Strategy Guide

Ensure governance, compliance, operational auditing, and risk auditing of AWS accounts by configuring AWS CloudTrail and AWS Config:

• AWS CloudTrail provides event-history of an AWS account activity, including actions taken through the AWS Management Console, AWS SDKs, command-line tools, and other AWS services.
• AWS Config tracks changes in configurations and relationships between AWS resources and provides detailed resource configuration histories. Config can also be used for overall configuration compliance using Config Rules.

CloudTrail and Config store their logs in S3 buckets. CloudTrail also provides an option to forward logs to a CloudWatch Log Group, which can allow for better search capability and notification creation using CloudWatch metrics and alarms. 

CloudWatch Metrics and Alarms can be defined to track and alert on critical CloudTrail events (if CloudTrail to CloudWatch forwarding is enabled).

In addition to tracking account activity and configuration changes, it is recommended to track cost and usage. AWS Budgets allows tracking both costs and usage with notification capability based on actual or predicted thresholds

AWS provides capabilities to log network activity for resources deployed in VPCs using the following options:

• VPC Flow Logs capture network flow information for IP traffic going to and from network interfaces in a VPC (includes source/destination IP address and ports, bytes transferred, firewall action, and more). Flow log data can be published to Amazon CloudWatch Logs or Amazon S3
• VPC Traffic Mirroring creates a copy of the network traffic to/from a specific interface in a VPC and streams it to a specified destination for analysis
• VPC DNS Logging monitor DNS queries in a VPC by configuring Route53 Resolver Query Logging

Amazon GuardDuty is a threat detection service that continuously monitors for malicious activity and unauthorized behavior by analyzing CloudTrail, VPC Flow Log and DNS Log activity in an AWS Account.

AWS WAF helps protect internet-facing applications and API endpoints. AWS WAF integrates with CloudFront, Load Balancers, and API Gateway to inspect (and optionally drop) traffic deemed malicious. Use the AWS Managed Rules package to get started or one of the partner-managed rule packages (e.g. F5, Imperva, Fortinet, etc.)

AWS provides several services to help monitor configuration and ensure compliance with security standards and best-practices: 

• AWS Security Hub runs automated, continuous security checks based on industry standards and best practices, such as the Center for Internet Security (CIS) AWS Foundations Benchmark, AWS Foundational Security Best Practices and Payment Card Industry Data Security Standard (PCI DSS)
• AWS Config Rules define conditions that describe the target ideal configuration. When resource configuration changes, AWS Config continuously tracks these changes and checks whether they violate the defined rules.
• Amazon Macie is used to discover, monitor, and help you protect sensitive data in Amazon S3. Macie automates the discovery of sensitive data, such as personal identifying information (PII) and intellectual property. Macie also identifies overly permissive or unencrypted buckets across AWS accounts.  
• IAM Access Analyzer helps identify resources in AWS accounts, such as Amazon S3 buckets or IAM roles, that are shared with an external entity, and alert when that happens. 
• Trusted Advisor provides real-time guidance to help provision resources following AWS best practices including security checks. 

AWS also provides service-specific activity and access logs that can be turned on selectively for supported resources. It should be noted that there are additional charges associated with enabling logging for these services and depending on the usage, the costs might be high. Some of the common services: 

• S3: For S3 buckets with important data, S3 Server Access Logs or CloudTrail Data Events for S3 can be enabled to monitor access to S3 buckets and files (including operations such as upload/download/delete).
• Lambda: CloudTrail Data Events for Lambda can be enabled to monitor Lambda function activity in an AWS account.
• Load Balancer Logs: ALB/NLB/Classic LB support access logging that can be enabled to monitor requests and traffic patterns through the load balancers. 
• CloudFront Access Logs: Enable access logs for CloudFront to monitor requests for sites hosted behind CloudFront as well as traffic patterns. 

Using the CloudWatch Log Agent, metrics and logs can be collected from EC2 instances and forwarded to CloudWatch. This includes operating system logs (Windows or Linux) as well as logs for applications running on EC2 instances.

In addition to logging services, AWS provides several options to review, analyze and visualize logs (or 3rd party tools such as Splunk or DataDog can be used):

• CloudWatch Log Insights: For logs stored in CloudWatch, CloudWatch Insights provides an interface to quickly search through and visualize logs using a powerful and flexible langauge. Predefined sample queries are available for CloudTrail, VPC Flow logs, Lambda and more.
• Amazon Detectivee makes it easy to analyze, investigate, and quickly identify the root cause of potential security issues or suspicious activities. Amazon Detective automatically collects log data from your AWS resources and uses machine learning, statistical analysis, and graph theory to build a linked set of data that enables you to easily conduct faster and more efficient security investigations.
• Amazon Elasticsearch (and Kibana) which allows for powerful queries and visualizations of the log data, as well as alerting. There are several methods supported to ingest data into Amazon Elasticsearch including a native streaming option for logs stored in CloudWatch Logs.
• Amazon Athena which is an interactive query service that makes it easy to analyze log data in Amazon S3 using standard SQL. For visualizations, Amazon QuickSight can be used for logs stored in S3.