Mallory Mooney

In Part 1 of this series, we looked at how AWS WAF helps you monitor network traffic to AWS resources, as well as key metrics and logs for detecting WAF misconfigurations and malicious activity. In this post, we’ll walk through using AWS’s native tooling to query that data.

AWS provides built-in monitoring, logging, and auditing tools through Amazon CloudWatch and AWS CloudTrail. CloudWatch is AWS’s primary monitoring platform, which operates as a central location for collecting metrics and analyzing metric and log data. For AWS WAF, this means that you can use CloudWatch to monitor the metrics generated by your web ACLs, rules, rule groups, and labels, in addition to web ACL activity logs. CloudTrail is AWS’s audit logging service, which records data that’s necessary for adhering to governance and compliance requirements. You can access the data from both of these services via the AWS Management Console and AWS Command Line Interface (CLI).

In the next few sections, we’ll show how you can use these services to analyze AWS WAF metrics, review WAF activity and audit logs, and query this data using the AWS CLI.

Analyze AWS WAF metrics

For each web ACL, AWS WAF includes a built-in dashboard that enables you to get a high-level overview of activity. With it, you can view a breakdown of each of the rules and rule actions that were triggered by requests. As seen in the following screenshot, you can look at how many requests a web ACL blocked overall and which rules executed the action.

For more granular visibility into WAF activity, you can use CloudWatch to query metrics and logs based on your needs. CloudWatch divides metrics for each AWS service into namespaces, such as AWS/WAFV2, and then by dimensions. Dimensions are specific to the AWS service and enable you to filter metrics by specific service components in dashboards and the CLI. For example, AWS WAF dimensions include country, region, device, web ACL, rule group, and attack.

Using a specific dimension or label can help you visualize the right data, especially when a particular metric has a larger dataset. For example, as mentioned in Part 1, AWS’s managed Bot Control rule group will monitor requests for evidence of multiple categories of bots. If you need to review metrics for a specific type of bot, such as those that don’t match any other defined category, you can include that label (bot:category:miscellaneous in the following screenshot) in your metric query in order to instantly drill down to the relevant data.

Review AWS WAF activity and audit logs

AWS WAF activity logs provide detailed information about the traffic that’s flowing through your web ACLs. When enabled for a web ACL, you can send all activity logs to CloudWatch for analysis. You can also send logs to other logging destinations, like Amazon Simple Storage Service (S3) and Amazon Data Firehose, but we’ll focus on CloudWatch in this post.

Activity logs

Activity logs, like the example in the following snippet, capture all information associated with a request, including which rule group it matched and the resultant rule action. In this example, AWS WAF blocked a request because it matched on AWS’s SQL managed rule group.

{
    "timestamp": 1709575226408,
    "formatVersion": 1,
    "webaclId": "arn:aws:wafv2:us-west-1:123456789012:regional/webacl/waf-demo-webacl/c76656bd-ba5e-48ec-a8af-b6a92b2c4b7b",
    "terminatingRuleId": "Default_Action",
    "terminatingRuleType": "REGULAR",
    "action": "BLOCK",
    "terminatingRuleMatchDetails": [],
    "httpSourceName": "ALB",
    "httpSourceId": "123456789012-app/waf-demo-mm-alb/c4d5f7bbb20d775b",
    "ruleGroupList": [
        {
            "ruleGroupId": "AWS#AWSManagedRulesSQLiRuleSet",
        }
    ],
    "rateBasedRuleList": [],
    "nonTerminatingMatchingRules": [],
    "requestHeadersInserted": null,
    "responseCodeSent": null,
    "httpRequest": {
        "clientIp": "19.192.122.123",
        "country": "US",
        "headers": [...],
        "uri": "/",
        "args": "",
        "httpVersion": "HTTP/1.1",
        "httpMethod": "GET",
        "requestId": "1-65e60c3a-1cd7904810de94db3235e13d"
    }
}

CloudWatch Logs Insights can take this visibility a step further by helping you sift through the large volumes of data that a particular web ACL generates in order to surface trends in traffic. For example, you can quickly review all requests that matched the SQL rule statement by using the following query:

fields @timestamp, terminatingRuleId, action, terminatingRuleMatchDetails.0.conditionType as ConditionType, terminatingRuleMatchDetails.0.location as Location, terminatingRuleMatchDetails.0.matchedData.0 as MatchedData
| filter ConditionType in["SQL_INJECTION"]

The results of this query, as seen in the following screenshot, will include the timestamp of the request as well as details about the attempted SQL injection, such as the URL query string that the rule matched on (gifts).

Audit logs

For insights into any administrative activity for your web ACLs, you can look at your audit logs. Audit logs capture any admin-level action made through the AWS WAF API, which CloudTrail will log as an event. There are two ways you can review this activity in the Management Console: via CloudTrail Event history or a CloudTrail trail. We’ll focus on viewing events in CloudTrail’s dedicated console, but you can create trails that automatically send AWS WAF events to an Amazon S3 bucket. This option gives you the ability to store events for more than 90 days, which is the maximum number of days allowed in CloudTrail’s Event history.

Audit logs provide key information about the sources of a particular action. In the following example, CloudTrail logged a failed attempt to list all web ACLs for a particular region. The log includes details about the user that made the request (admin-team-1), the region (us-west-1), the generated error code (AccessDenied), and more.

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "principalId",
        "arn": "arn:aws:iam::123456789012:user/admin-team-1",
        "accountId": "123456789012",
        "accessKeyId":"accessKeyId",
        "userName": "admin-team-1"
    },
    "eventTime": "2023-12-08T18:25:50Z",
    "eventSource": "wafv2.amazonaws.com",
    "eventName": "ListWebACLs",
    "awsRegion": "us-west-1",
    "sourceIPAddress": "12.345.678.12",
    "userAgent": "",
    "errorCode": "AccessDenied",
    "errorMessage": "User: arn:aws:iam::123456789012:user/admin-team-1 is not authorized to perform: wafv2:ListWebACLs on resource: arn:aws:wafv2:us-west-1:123456789012:regional/webacl/*/* because no identity-based policy allows the wafv2:ListWebACLs action",
    "requestParameters": null,
    "responseElements": null,
    "requestID": "requestID",
    "eventID": "eventID",
    "readOnly": true,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "123456789012",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.2",
        "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "clientProvidedHostHeader": "wafv2.us-west-1.amazonaws.com"
    }
}

You can also use CloudTrail Insights to search for trends related to a particular event based on its name, source, or ID. For example, the following screenshot illustrates how you can search for the ListWebACLs event name from the previous example audit log to review how many times a user attempted to list web ACLs and whether they were successful.

This information can help you determine if you need to adjust permissions for the user who made the API call or investigate further for a potential security issue.

Query AWS WAF data using the AWS CLI

If you prefer to use a command line interface, the AWS CLI enables you to easily query AWS’s public APIs for information about your services, including metric data. Using the CLI is often easier than accessing the Management Console because it allows you to quickly run commands locally and build scripts that automate parts of cloud infrastructure management. For example, you can query a list of web ACLs using the aws waf list-web-acls command, which will give you the following output:

{
    "WebACLs": [
        {
            "WebACLId": "123asdfg-asfg-1e3g-3ddd-1234rtyu6789",
            "Name": "waf-acl-1"
        },
        {
            "WebACLId": "123abcd-asfg-1e3g-2aaa-2987rtyu6789",
            "Name": "waf-acl-2"
        }
    ]
}

To query metrics, you need to use the aws cloudwatch list-metrics command and include the appropriate namespace (AWS/WAFV2) and metric name. As an example, the following command lists all blocked requests for all available web ACLs in your default region:

 aws cloudwatch list-metrics --namespace AWS/WAFV2 --metric-name BlockedRequests

You can also filter this command further by adding the --dimensions option, which gives you more control over the web ACLs you’re querying. For example, if you want to look at all blocked requests in a region other than your default one, you can use the following command:

aws cloudwatch list-metrics --namespace AWS/WAFV2 --metric-name BlockedRequests --dimensions Name=Region,Value=us-west-1

The output from this command shows which rules were matched to block incoming requests for that region:

{
    "Metrics": [
        {
            "Namespace": "AWS/WAFV2",
            "MetricName": "BlockedRequests",
            "Dimensions": [
                {
                    "Name": "WebACL",
                    "Value": "waf-demo-webacl"
                },
                {
                    "Name": "Region",
                    "Value": "us-west-1"
                },
                {
                    "Name": "Rule",
                    "Value": "AWS-AWSManagedRulesSQLiRuleSet"
                }
            ]
        }
    ]
}

Monitor and manage your firewall infrastructure in one platform

Amazon CloudWatch offers a central location for monitoring AWS WAF activity. However, monitoring WAF activity and tuning their configurations to adapt to a constantly growing environment is challenging. And while perimeter-based WAFs can protect applications from external threats, they may leave gaps that affect applications at the service level. In Part 3 of this series, we’ll look at how Datadog’s AWS WAF integration enables you to easily monitor the key metrics and logs covered in Part 1 to get better visibility into your overall firewall infrastructure and enhance your perimeter-based security with our distributed, in-app WAF.