Moesif AWS API Gateway

Gain deep analytics from your AWS API Gateway access logs without any code changes or redeploy. This integration installs automatically via a CloudFormation template.

How it works

The integration works by Logging API calls to a Kinesis Data Firehose which sends access logs in batches to Moesif. Besides tracking typical API parameters like URI route and status code, the integration also tracks the user identity which makes them perfect for user behavior analytics tools like Moesif.

Diagram of Moesif AWS API Gateway logging architecture

Because you are monitoring at the API gateway itself, you also gain visibility into rejected requests that never reach your underlying service such as AWS Lambda or EC2.

It’s important to note that Moesif only supports the use of the AWS Gateway with an API Type of REST API. Currently Moesif does not support Web Socket through AWS Gateway. Instead, use one of the SDKs.

How to Install

1. Create a Kinesis Data Firehose

Click the below Launch Stack button to open CloudFormation which will automatically create a new Data Firehose configured with Moesif collection API as the destination.

Launch CloudFormation Stack

Ensure you checkbox I acknowledge that AWS CloudFormation might create IAM resources as shown below. Then click Create Stack.

AWS API Gateway Cloudformation create stack

If you experience errors or want to create the Firehose manually, follow these instructions

After the stack is created, search for “Firehose” in the AWS Console search bar and select Kinesis Data Firehose. You’ll need the newly created Firehose ARN for the next step. Make sure you are in the same region you deployed your stack to.

2. Enable API Gateway access logging

With the Firehose created, you need to configure API Gateway to send access logs to it.

  • Go to your AWS API Gateway instance within the AWS Console.
  • Select Stages on the left menu and then select the Logs/Tracing tab
  • Toggle on Enable Access Logging.
  • Add your Kinesis Firehose ARN created from Step 1 under Access Log Destination ARN.

3. Add the JSON log format

Under Log Format, add the below JSON snippet and ensure JSON format is selected. Moesif will safely ignore any extra keys.

It’s not possible to log the request and response body with this integration. If you require body analytics, choose a different integration such as a Moesif SDK like moesif-aws-lambda

{
  "apiId": "$context.apiId",
  "requestId": "$context.requestId",
  "requestTime": "$context.requestTime",
  "protocol": "$context.protocol",
  "httpMethod": "$context.httpMethod",
  "resourcePath": "$context.path",
  "requestHostHeader": "$context.domainName",
  "requestUserAgentHeader": "$context.identity.userAgent",
  "ip": "$context.identity.sourceIp",
  "status": "$context.status",
  "responseLength": "$context.responseLength",
  "durationMs": "$context.responseLatency",
  "caller": "$context.identity.caller",
  "user": "$context.identity.user",
  "principalId": "$context.authorizer.principalId",
  "cognitoIdentityId": "$context.identity.cognitoIdentityId",
  "userArn": "$context.identity.userArn",
  "apiKey": "$context.identity.apiKey",
  "apiKeyId": "$context.identity.apiKeyId"
}

For more info surrounding available $context variables, check out the AWS docs

Click Save Changes and that’s it! API logs should start showing up after a few minutes.

Manual Firehose Creation

If you cannot use the CloudFormation template or having issues, you can manually create and configure a Kinesis Data Firehose using the HTTP destination.

Go to Amazon Kinesis within the AWS Console and select Create Delivery Stream.

The name of your delivery stream must start with amazon-apigateway-

  • For the source, select Direct PUT or other sources.
  • For the destination, select HTTP Endpoint and enter the following:
  • For HTTP endpoint URL, enter https://api.moesif.net/v1/partners/aws/kinesis
  • For Access key, enter your Moesif Application Id.
  • For Buffer interval, enter 60 seconds.
  • Enable GZIP compression.

Finish the creation wizard and make a note of the Firehose ARN. Complete the rest of the guide using your Firehose ARN. This should be in the same region where you created the Firehose.

Enabling User Tracking

The Moesif AWS API Gateway integration automatically identifies users by extracting them from the API gateway’s request context. This is handled automatically using the following priority and is dependent on the access control method used.

  Variable Description
1. $context.identity.user The principal identifier of the user. Supported for IAM authorization.
2. $context.identity.apiKeyId The API key ID associated with an API request that requires an API key.
3. $context.authorizer.principalId The principal user id associated returned from an API Gateway Lambda authorizer such as a custom one that extracts the subject from a JWT (Json Web Token).
4. $context.identity.cognitoIdentityId The Amazon Cognito identity ID of the caller making the request.
5. $context.identity.caller The principal identifier of the caller that signed the request. Supported for IAM authorization.
6. $context.identity.userArn The Amazon Resource Name (ARN) of the effective user identified after authentication.

Automatic Identification Example

For example, if you created a custom lambda authorizer which extracts the subject from a JWT (Json Web Token), then the context variable $context.authorizer.principalId is set by AWS. Moesif will then use this (assuming $context.identity.user and $context.identity.apiKeyId are not defined as these are higher priority). If the above priority doesn’t work, you can customize via instructions below.

Customize User Identification

While automatic extraction should be sufficient for most applications, you can override this behavior if required. To do so, modify the Moesif field user in your JSON log format to refer to the appropriate AWS context variable. Any context variable can be used as long as it’s a “string” type.

For example, if you want to use the source IP address as the user id, modify user in your JSON log format to look like below:

{
  ...
  "user": "$context.identity.sourceIp"
}

Enabling Company Tracking

The Moesif AWS API Gateway integration can also enable company tracking as well. This value can be added into Moesif in two distinct ways:

  • The first option uses the companyId variable within the gateways log format (similar to user tracking). To do so, modify the Moesif field companyId in your JSON log format to refer to the appropriate AWS context variable. For example, you may use $context.identity.apiKeyId or another unique identifier within the available context variables.
{
  // Rest of fields

  "companyId": "$context.identity.apiKeyId"
}
  • Secondly, for more flexibility, users can also use the Moesif API to link users to companies via the update users API. Specifics can be found in the docs, here.

Event Metadata

You can include custom context fields as Moesif metadata. The field metadata can be set to an arbitrary JSON object which contains your custom fields.

Below is an example including an error message.

{
  // Rest of fields

  "metadata": {"my_error_field": "$context.error.message" }
}

Limitations

There are certain limitations to be aware of below. You can use a Moesif SDK as a workaround for any of these limitations.

  1. This integrations supports “REST API Gateway” only. It does not support a “Websockets” or “HTTP” Gateway.
  2. Request and Response Body fields are not logged.
  3. Governance rules is not supported unless a Moesif SDK is also installed.

You can also mix-and-match. For example, if you require governance, you can enforce governance by installing a Moesif SDK like moesif-nodejs, while still maintain logging through this API Gateway integration.

For more options, check out the AWS docs on the different $context variables that are available.

Updated: