Monitoring Azure API Management with Moesif

API gateways and management platforms like Azure API Management (APIM) are an excellent way to keep your APIs in order. They provide features that every API needs, like API keys, quotas, and caching in a standardized fashion. Your APIs will then behave more homogeneously and can be managed in a central place.

They’re also the entry point to your cloud infrastructure and can help you abstract implementation from the interface. If you change your backend from VMs to serverless, but keep using the same API management platform, everything stays the same for your customers.

Using a central API management platform like APIM, it’s straightforward to add observability features to all of your APIs with Moesif API analytics. Since everything goes in and out through APIM, you only need to integrate with this one cloud service.

API Observability

API observability can provide your business and engineering teams with deep insights into how your APIs are used. API observability is leveraged by a variety of groups in your company, including:

  • Product teams to understand API usage and business value
  • Engineering teams to monitor and troubleshoot API issues
  • Security teams to detect and protect from API threats

Moesif API Analytics is an API observability solution that you can leverage to understand API usage better. Native integration with Azure API Management makes deployment just a few clicks away and doesn’t require any code change or restarts. API management platforms are the natural place to provide API observability to your various business and engineering teams as they are the entry to the rest of your infrastructure.

Solution Overview and Use Case

This solution adds API analytics to your APIs hosted behind Azure API Management. It works by forwarding structured API access logs from APIM to Moesif via Event Hubs and WebJobs. Deployment of the solution can be done in a few clicks using the included Azure resource template and doesn’t require any downtime.

Azure API-Management logging architecture diagram

Set Up

The integration works by adding an Azure Event Hub, receiving API access logs from your Azure API Management, and sending them to Moesif with an Azure WebJob. The logs contain user identity, making them perfect for user behavior analytics tools like Moesif.

1. Start Azure Resource Deployment

Click the button below to start a Custom deployment with the Moesif Azure Resource Template.

2. Configure Parameters

Within the Azure Template Deployment panel, set the following properties:

Azure template deploy panel

If you deploy the Moesif related resources before creating APIM, you must manually create the moesif-log-to-event-hub logger.

  1. Set Moesif Application Id to the one displayed after logging into your Moesif account. You can create a free one on Moesif’s website.
  2. Set Existing Api Mgmt Name to the name of your Azure APIM instance. If blank, you will need to create the APIM logger manually.
  3. Once done, click the Review+create button at the bottom and finish the template creation wizard.

Occasionally, Azure reports a failed deployment due to slow propagation of new DNS settings even though everything was deployed successfully. We recommend proceeding with the rest of the process. If you still have issues after the last step, view troubleshooting.

After the deployment finishes, your resource group should look like this:

Azure deployed resources

In this example, the first-demo-api APIM was already deployed before the Moesif integration.

3. Add XML Policy

Within the Azure portal, navigate to your existing Azure API Management instance. Then, add the below XML policies to all products or APIs that you want API logging enabled.

We recommend adding the XML policy globally for all APIs and use Moesif dynamic sampling if you want to create rules that selectively sample or suppress data collection. Rules are dynamically enabled based on specific customer behaviors, regex rules, and more.

Open your APIM, go to APIs -> APIs -> All APIs and click on the </> symbol under Inbound processing to open the policy editor. In the screenshot, the APIM manages two APIs. We want to track all of them with Moesif, so the best way to integrate is will All APIs.

Azure API-Management dashboard

Paste the following policy XML:

<policies>
    <inbound>
        <set-variable name="moesif-message-id" value="@(Guid.NewGuid())" />
        <log-to-eventhub logger-id="moesif-log-to-event-hub" partition-id="0">@{
var body = context.Request.Body?.As<string>(true);
var MAX_BODY_EH = 145000;
var origBodyLen = (null != body) ? body.Length : 0;
if (MAX_BODY_EH < origBodyLen){ body = body.Remove(MAX_BODY_EH); }
var headers = context.Request.Headers
    .Where(h => h.Key != "Ocp-Apim-Subscription-Key")
    .Select(h => string.Format("{0}: {1}", h.Key, String.Join(", ", h.Value).Replace("\"", "\\\""))).ToArray<string>();
var jwtToken = context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt();
var userId = (context.User != null && context.User.Id != null) ? context.User.Id : (jwtToken != null && jwtToken.Subject != null ? jwtToken.Subject : string.Empty);
var cru = new JObject();
if (context.User != null) {
  cru.Add("Email", context.User.Email);
  cru.Add("Id", context.User.Id);
  cru.Add("FirstName", context.User.FirstName);
  cru.Add("LastName", context.User.LastName);}
var crus = System.Convert.ToBase64String(Encoding.UTF8.GetBytes(cru.ToString()));
var requestBody = (body != null ? System.Convert.ToBase64String(Encoding.UTF8.GetBytes(body)) : string.Empty);
return new JObject(
  new JProperty("event_type", "request"),
  new JProperty("message-id", context.Variables["moesif-message-id"]),
  new JProperty("method", context.Request.Method),
  new JProperty("ip_address", context.Request.IpAddress),
  new JProperty("uri", context.Request.OriginalUrl.ToString()),
  new JProperty("user_id", userId),
  new JProperty("contextRequestUser", crus),
  new JProperty("company_id", ""),
  new JProperty("request_headers", string.Join(";;", headers)),
  new JProperty("request_body", requestBody),
  new JProperty("metadata", $@"")
  ).ToString();}</log-to-eventhub>
        <set-variable name="sent-moesif-request" value="@(true)" />
    </inbound>
    <backend>
        <forward-request follow-redirects="true" />
    </backend>
    <outbound>
        <choose>
            <when condition="@(context.Variables.ContainsKey("sent-moesif-request") && !context.Variables.ContainsKey("sent-moesif-response"))">
                <log-to-eventhub logger-id="moesif-log-to-event-hub" partition-id="1">@{
var body = context.Response.Body?.As<string>(true);
var MAX_BODY_EH = 145000;
var origBodyLen = (null != body) ? body.Length : 0;
if (MAX_BODY_EH < origBodyLen){ body = body.Remove(MAX_BODY_EH);}
var headers = context.Response.Headers.Select(h => string.Format("{0}: {1}", h.Key, String.Join(", ", h.Value).Replace("\"", "\\\""))).ToArray<string>();
var responseBody = (body != null ? System.Convert.ToBase64String(Encoding.UTF8.GetBytes(body)) : string.Empty);
return new JObject(
  new JProperty("event_type", "response"),
  new JProperty("orig_body_len", origBodyLen),
  new JProperty("message-id", context.Variables["moesif-message-id"]),
  new JProperty("status_code", context.Response.StatusCode),
  new JProperty("response_headers", string.Join(";;", headers)),
  new JProperty("response_body", responseBody)
  ).ToString();}</log-to-eventhub>
                <set-variable name="sent-moesif-response" value="@(true)" />
            </when>
        </choose>
    </outbound>
    <on-error>
        <choose>
            <when condition="@(context.Variables.ContainsKey("sent-moesif-request") && !context.Variables.ContainsKey("sent-moesif-response"))">
                <log-to-eventhub logger-id="moesif-log-to-event-hub" partition-id="1">@{
var body = context.Response.Body?.As<string>(true);
var MAX_BODY_EH = 145000;
var origBodyLen = (null != body) ? body.Length : 0;
if (MAX_BODY_EH < origBodyLen){ body = body.Remove(MAX_BODY_EH);}
var headers = context.Response.Headers.Select(h => string.Format("{0}: {1}", h.Key, String.Join(", ", h.Value).Replace("\"", "\\\""))).ToArray<string>();
var responseBody = (body != null ? System.Convert.ToBase64String(Encoding.UTF8.GetBytes(body)) : string.Empty);
return new JObject(
  new JProperty("event_type", "response"),
  new JProperty("orig_body_len", origBodyLen),
  new JProperty("message-id", context.Variables["moesif-message-id"]),
  new JProperty("status_code", context.Response.StatusCode),
  new JProperty("response_headers", string.Join(";;", headers)),
  new JProperty("response_body", responseBody)
  ).ToString();}</log-to-eventhub>
                <set-variable name="sent-moesif-response" value="@(true)" />
            </when>
        </choose>
    </on-error>
</policies>

That’s it. Once the XML is added to your APIs, the logs should start showing up in Moesif.

More info on editing APIM policies is available on the Azure docs.

4. Sending Requests

Now that everything is set up, you can send a few requests to your API endpoints and see how they appear in the Moesif Dashboard.

Moesif Dashboard

Monitor All API Traffic With Moesif

Learn More

Understand Customer API Usage

A key objective for API analytics is understanding who is using your APIs and how they use them. By default, Moesif ties your API calls back to a user identifier through parsing the request context with $context.authorizer.principalId or $context.identity.cognitoIdentityId so you can understand user behavior.

A critical report is understanding which customers are using your APIs the most. APIs logs traditionally don’t include customer demographic information. Still, an API analytics system like Moesif can join with other data sets containing customer attributes like user email or company domain. Users can be tracked with client integrations like moesif-browser-js or Segment using the same user id. Then we can bring up a usage report showing API traffic by company.

Moesif tracking by company

Troubleshoot API Issues

With high-cardinality, high-dimension API observability, you can slice and dice your API logs by any number of fields, including HTTP headers or response time. This makes it easy to troubleshoot issues without manual log search quickly. A core engineering metric for APIs is latency percentiles such as the 90th percentile. The best practice is to look at 90th percentile latency over the average. This practice helps uncover significant variations in your latency that low standards can mask. Your API users are looking for consistently low latency, not the lowest average, as spikes can wreak havoc in their services.

To do this, go to Events -> Time series and Select the P90 Latency Metric. It’s a good idea also to understand this broken down by route or service. To do so, add a group by “Request URI.” Moesif will automatically consolidate routes such that /items/1 and /items/2 will show up as /items/:id.

Find API Security Threats

Your security risk increases as you expose more APIs to the internet used by customers, partners, and single-page apps. Traditional mechanisms like browser fingerprinting and captchas don’t work, so you need to leverage advanced user behavior analytics to find suspicious users.

A standard API security threat is not limiting access to your proprietary data. A hacker can then download all this data via a pagination attack. A method to detect customers abusing your API this way is to look at the amount of data downloaded per customer. To create this metric, add a summation of response.headers.Content-Length and then group it by customer name:

Moesif Content-Length by company

Advanced User Behavior Analytics

You can leverage your integration beyond looking at API calls in isolation and stitch your entire customer journey together. This approach makes it easier to see funnel reports on “Time to First Hello World” and “Time to Value.”

Track user actions in your UI such as “Signed In” or “Viewed Docs” and start tracking user actions in your UI like “Signed In” or “Viewed Docs.” This makes it easier to slice and dice API usage by customer traffics. To do so, add the moesif-browser-js to your UI and call the tracking method:

moesif.track("Clicked Sign Up", {
  button_label: "Get Started",
  sign_up_method: "Google SSO",
})

Once done, the first thing you should do is generate a funnel report. In the below report, we created a funnel analysis composing of three steps.

  1. A customer is signing in to your web application (a user action).
  2. A single payment transaction via the API. Thus moving from step 1 to step 2 shows the conversion rate of sign-ups to the first API call.
  3. Over 100 payment transactions. For this example, we consider this the “aha” moment demonstrating customer value. Moving from step 2 to step 3 shows the drop-off of customers who made API calls to see real value.
Moesif funnel analysis

Conclusion

Setting up Moesif API analytics for all of your APIs managed by Azure API Management takes just a few clicks. In five minutes, you can get insights into all of your APIs.

Moesif API Analytics offers information about your APIs across your teams: from performance metrics and error rates for engineers, through to user behavior for product teams.

Understand How Customers Use Your APIs

Learn More
Powerful API Insights With Moesif

Powerful API Insights With Moesif

Learn More