How to add Moesif API Analytics and Monitoring to Kong Ingress Controller

Kong is a popular open-source API gateway to help manage your APIs. With Kong, you can handle authentication, rate limiting, data transformation, among other things from a centralized location even though you have multiple microservices. Kong is built on NGINX at it’s core, one of the most popular HTTP servers. Being open-source, Kong is very easy to deploy on-premises usually in just a few minutes without requiring the installation of many components other than a Postgres or Cassandra store.

Kong Ingress Controller implements authentication, transformations, and other functionalities across Kubernetes clusters with zero downtime. Kong Gateway connects Kubernetes clusters with services running across any environment or platform. By integrating with the Kubernetes Ingress Controller, Kong ties directly to the Kubernetes lifecycle. As applications are deployed and new services are created, Kong will automatically live configure itself to serve traffic to these services. Kong and the Kong Ingress Controller has a full management layer and API, live configuration of targets and upstreams, and a durable, scalable state storage using either Postgres or Cassandra that ensures every Kong instance is synced without delay or downtime.

Moesif provides a plugin for Kong that makes getting started with API observability in just a few minutes so you can stay focused on shipping features customers love rather than deal with the maintenance cost of building your own data infrastructure. With Moesif, you’re able to understand how your API is used and by what customers, identify which customers are running into integration issues, and monitor for endpoints that need optimization. In this article, we’ll talk about the insights Moesif gives you, and how to integrate it with Kong Ingress Controller. Then, we will talk about how to best leverage API observability using Moesif and Kong.

Metrics showing users with 400 errors

Overview

Moesif Kong plugin is an agent that collects metrics and sends to the Moesif collection network. This enables you to get a complete picture of your API usage even across different Kong instances and data center regions. Moesif provides deep insights for engineering teams to understand how their APIs are used and quickly troubleshoot complex issues. Because Moesif also tracks who is calling your API and how they are accessed, product-driven teams can understand the entire customer journey and where to invest more resources. With API observability, forward-thinking engineering leaders can empowers customer-facing teams with self-service analytics on activation funnels, retention reports, and more. Moesif also analyzes your API payloads for troubleshooting and business insights so you’re able to understand utilization of specific payload keys, etc.

Setting up the Kong Ingress Controller

Setup Kong for Kubernetes

Setting up Kong for Kubernetes using Minikube is as simple as:

kubectl apply -f https://bit.ly/k4k8s

Setup environment variables

You’ll setup an environment variable with the IP address at which Kong is accessible. This will be used to actually send requests into the Kubernetes cluster. You will create a tunnel for kong-proxy and get the url.

minikube service --url kong-proxy -n kong
export PROXY_IP=<URL from the above step>

You could test connectivity to Kong using - curl -i $PROXY_IP

Add the Moesif plugin

You’ll need to download the kong-moesif-plugin and navigate to kong/plugin directory to create a configMap using

kubectl create configmap kong-plugin-moesif --from-file=moesif -n kong

Please ensure that this is created in the same namespace as the one in which Kong is going to be installed.

Next, you’ll need to edit Kong’s Deployment to load Moesif plugin using

kubectl edit deployment.v1.apps/ingress-kong -n kong

Patch which needs to be applied to the existing deployment (truncated)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-kong
  namespace: kong
spec:
  template:
    spec:
      containers:
      - name: proxy
        env:
        - name: KONG_PLUGINS
          value: moesif
        - name: KONG_LUA_PACKAGE_PATH
          value: "/opt/?.lua;;"
        volumeMounts:
        - name: kong-plugin-moesif
          mountPath: /opt/kong/plugins/moesif
      volumes:
      - name: kong-plugin-moesif
        configMap:
          name: kong-plugin-moesif

Once you patch the deployment, you’ll need to rollout the Kong Ingress Controller using -

kubectl rollout status deployment.v1.apps/ingress-kong -n kong

Setup KongPlugin resource

To setup KongPlugin resource, you will need your Moesif Application Id. You can get one by signing up for a free Moesif account, then select Kong during the onboarding. Moesif recommends using of a single application Id for all Kong instances and data-center regions. This ensures you have a unified view of your API usage data regardless of physical topology. You can still break down by any number of attributes using Moesif’s high-cardinality, high-dimension analytics engine.

Moesif still recommends creating separate Application Ids for each environment such as Production, Staging, and Development to keep data isolated.

echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: moesif
disabled: false
config:
  application_id: Your Moesif Application Id
  debug: false
plugin: moesif
" | kubectl apply -f -

Create a new Ingress resource which uses Moesif plugin

You can annotate an Ingress or Service resource to instruct Kong on when to execute the plugin. You could configure plugins globally, on Service resource, or for a specific consumer.

echo "
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: demo-example-com
  annotations:
    konghq.com/plugins: moesif
    kubernetes.io/ingress.class: kong
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /bar
        backend:
          serviceName: echo
          servicePort: 80
" | kubectl apply -f -

Send a Request

Once you have the Moesif plugin installed, your API traffic should start showing up in the live event stream within Moesif. You could send traffic to the resource configured using

curl -i -H "Host: example.com" $PROXY_IP/bar/sample

Real-time API logs

How to use API observability

Engineering metrics

The first thing you’ll probably interested in is engineering metrics like how is my API performing. This type of metric can be pulled up by going to Events -> Time Series view within Moesif. Then you can select 90th percentile latency as the metric to plot. You can then group by the URI route to understand which endpoints have the worst performance. Here, you can filter your traffic by API attributes like route, verb, along with HTTP headers and body fields.

90th percentile latency by API endpoint

Handling sensitive data

If your application consists of sensitive data such as healthcare or financial data, you can become data compliant with one of two options:

1. Zero-knowledge security

Client-side encryption has the benefits of low-maintenance SaaS while still putting you in control of your data. Because you control and encrypt the data on-premises before being sent to Moesif, Moesif physically cannot access your data. This requires only running a small appliance within your infrastructure called secure proxy to handle encryption/decryption of your data on the fly.

2. Data masking

If you don’t want client-side encryption, you can also mask data directly using the plugin. This is handled via the config.request_body_masks and config.response_body_masks configurations option. For example, you could mask a field called password that might be present in the payload. Additionally if you want to remove logging request and response body all together, you could set the config.disable_capture_request_body or config.disable_capture_response_body configuration option to true.

Closing thoughts

In this way, Moesif plugin will capture API requests and responses and log to Moesif for deep API observability and real-time monitoring of your API traffic via Kong and Kong handling all the other services around the application.

Get Deep API Observability for Kong Ingress Controller for Kubernetes

Learn More
Get Deep API Observability for Kong Ingress Controller for Kubernetes

Get Deep API Observability for Kong Ingress Controller for Kubernetes

Learn More