Moesif AWS API Gateway

Gain deep analytics from your AWS API Gateway access logs without any code changes or redeploy. This integration installs automatically via a CloudFormation template.

How it works

The integration works by Logging API calls to a Kinesis Data Firehose which sends access logs in batches to Moesif. Besides tracking typical API parameters like URI route and status code, the integration also tracks the user identity which makes them perfect for user behavior analytics tools like Moesif.

Diagram of Moesif AWS API Gateway logging architecture

Because your monitoring at the API gateway itself, you also gain visibility into rejected requests that never reach your underlying service such as AWS Lambda or EC2.

It’s important to note that Moesif only supports the use of the AWS Gateway with an API Type of REST API. Currently Moesif does not support Web Socket or HTTP APIs through AWS Gateway.

How to Install

1. Create a Kinesis Data Firehose

Click the below Launch Stack button to open CloudFormation which will automatically create a new Data Firehose configured with Moesif collection API as the destination.

Launch CloudFormation Stack

Ensure you checkbox I acknowledge that AWS CloudFormation might create IAM resources as shown below. Then click Create Stack.

AWS API Gateway Cloudformation create stack

If you experience errors or want to create the Firehose manually, follow these instructions

After the stack is created, search for “Firehose” in the AWS Console search bar and select Kinesis Data Firehose. You’ll need the newly created Firehose ARN for the next step. Make sure you are in the same region you deployed your stack to.

2. Enable API Gateway access logging

With the Firehose created, you need to configure API Gateway to send access logs to it.

  • Go to your AWS API Gateway instance within the AWS Console.
  • Select Stages on the left menu and then select the Logs/Tracing tab
  • Toggle on Enable Access Logging.
  • Add your Kinesis Firehose ARN created from Step 1 under Access Log Destination ARN.

3. Add the JSON log format

Under Log Format, add the below JSON snippet and ensure JSON format is selected. Moesif will safely ignore any extra keys.

It’s not possible to log the request and response body with this integration. If you require body analytics, choose a different integration such as a language-specific SDK

{
  "apiId": "$context.apiId",
  "requestId": "$context.requestId",
  "requestTime": "$context.requestTime",
  "protocol": "$context.protocol",
  "httpMethod": "$context.httpMethod",
  "resourcePath": "$context.path",
  "requestHostHeader": "$context.domainName",
  "requestUserAgentHeader": "$context.identity.userAgent",
  "ip": "$context.identity.sourceIp",
  "status": "$context.status",
  "responseLength": "$context.responseLength",
  "durationMs": "$context.responseLatency",
  "caller": "$context.identity.caller",
  "user": "$context.identity.user",
  "principalId": "$context.authorizer.principalId",
  "cognitoIdentityId": "$context.identity.cognitoIdentityId",
  "userArn": "$context.identity.userArn",
  "apiKey": "$context.identity.apiKey",
  "apiKeyId": "$context.identity.apiKeyId"
}

For more info surrounding available $context variables, check out the AWS docs

Click Save Changes and that’s it! API logs should start showing up after a few minutes.

Manual Firehose Creation

If you cannot use the CloudFormation template or having issues, you can manually create and configure a Kinesis Data Firehose using the HTTP destination.

Go to Amazon Kinesis within the AWS Console and select Create Delivery Stream.

The name of your delivery stream must start with amazon-apigateway-

  • For the source, select Direct PUT or other sources.
  • For the destination, select HTTP Endpoint and enter the following:
  • For HTTP endpoint URL, enter https://api.moesif.net/v1/partners/aws/kinesis
  • For Access key, enter your Moesif Application Id.
  • For Buffer interval, enter 60 seconds.
  • Enable GZIP compression.

Finish the creation wizard and make note of the Firehose ARN. Complete the rest of guide using your Firehose ARN. This should be in the same region where you created the Firehose.

Enabling User Tracking

The Moesif AWS API Gateway integration automatically identifies users by extracting it from the API gateway’s request context. This is handled automatically using the following priority and is dependent on the access control method used.

  Variable Description
1. $context.identity.user The principal identifier of the user. Supported for IAM authorization.
2. $context.identity.apiKeyId The API key ID associated with an API request that requires an API key.
3. $context.authorizer.principalId The principal user id associated returned from an API Gateway Lambda authorizer.
4. $context.identity.cognitoIdentityId The Amazon Cognito identity ID of the caller making the request.
5. $context.identity.caller The principal identifier of the caller that signed the request. Supported for IAM authorization.
6. $context.identity.userArn The Amazon Resource Name (ARN) of the effective user identified after authentication.

Customize User Identification

While automatic extraction should be sufficient for most applications, you can override this behavior if required. To do so, modify the Moesif field user in your JSON log format to refer to the appropriate AWS context variable.

For example if you want to use the source IP address as the user id, modify user in your JSON log format to look like below:

{
  ...
  "user": "$context.identity.sourceIp"
}

Please Note

This integration support HTTP API Gateway and REST API Gateway only. If using Websockets Gateway, we recommend using one of the language specific SDKs such as for lambda.

For more options, check out the AWS docs on the different $context variables that are available.

Updated: