Secure Proxy

With the Secure Proxy appliance, sensitive data can stay private to your organization via client-side encryption and customer-managed keys. You gain the privacy benefits of an on-premises installation without the cost of maintaining your own data infrastructure. It’s recommended if you’re in a regulated environment with sensitive data such as financial or healthcare and can help you meet any internal compliance requirements like HIPAA (more info on security and compliance).

For a case study, view How pVerify Deploys Moesif HIPAA-Compliant API Analytics to scale COVID testing.

How it works

The secure proxy is deployed as a Docker container in your infrastructure which handles encryption and decryption on the fly using your customer-managed keys. Because your master keys are never transmitted or stored on Moesif servers, its employees cannot access or inspect your API data.

How Moesif works with secure proxy

Secure proxy is available only on enterprise plans. Reach out to our sales team for help.

Keystore

The secure proxy has connectors for popular Cloud key stores and will automatically rotates data keys as needed. Currently, AWS Key Management Service (KMS) and AWS CloudHSM are supported. If you require an alternative, reach out to our success team.

Moesif always encrypts your data at rest and in motion using Moesif-managed keys even if Secure Proxy is not deployed. Secure Proxy is an additional privacy layer enabling you to have full control over encryption keys (also called customer-managed keys).

Features supported

The majority of Moesif features will function normally even with the secure proxy deployed including the dashboards, Management API, alerts, and embedded dashboards. Moesif built a customized analytics store which works on encrypted data enabling you to still gain the reports you expect from Moesif. There are some minor limitations which we recommend reviewing before deploying secure proxy.

Access

There are two different ways to deploy secure proxy depending on whether you plan on utilizing Moesif’s embedded dashboards feature:

  • Local network only: If you don’t plan to use Moesif’s embedded dashboards feature, you can deploy the secure proxy on an internal host that’s not exposed to the internet. However, the host must be accessible by an employee’s laptop when they log into the Moesif portal such as by being on a corporate VPN.

  • Exposed to the internet: If you plan on using Moesif’s embedded dashboards feature, the secure proxy must be accessible by the internet so that your customers can also decrypt their data on the fly. The recommended way to do so is to deploy NGINX with SSL in front of Moesif secure proxy with SSL. A Docker example with NGINX and Moesif Secure Proxy is available on GitHub. Once done, you can add a CNAME or A Record to your DNS settings like analytics.acmeinc.com. Employees can also access the Moesif portal on the go and do not need to be on a corporate VPN.

What’s encrypted

All sensitive data such your HTTP headers, request and response body, and event metadata is encrypted. Non-sensitive fields such as request time is not encrypted for proper analytics. More info on unencrypted data.

How to install

A working docker-compose.yml is available on GitHub with moesifproxy and NGINX configured with SSL termination using Let’s Encrypt.

1. Generate API Keys

You will need both your Collector Application Id and a Management API Key which can be found by logging into Moesif and going to API Keys from the menu. For Management API Key, ensure you generate with at least the following scopes:

  • create:events
  • read:events
  • create:encrypted_keys
  • read:encrypted_keys
  • read:workspaces
  • update:workspaces
  • read:virtual_eventtypes
  • update:virtual_eventtypes

2. Create AWS Key Management Service

Create a new AWS Key Management Service and generate a new key. You can also use an existing KMS instance.
Obtain your Customer Key Id for the next step. To find KeyId please follow the instructions herehttps://docs.aws.amazon.com/kms/latest/developerguide/find-cmk-id-arn.html..

3. Run the Docker image

Run the moesifproxy docker image which is available in Docker Hub. In order for the container to access your AWS KMS, there are two ways to authenticate. You can use your AWS access key directly or you can assume an IAM role which allows access to your AWS KMS. You can use the 2nd option if you inject the AWS credentials directly to a EC2 instance or via Kubernetes service accounts that moesifproxy is running on.

Option A: AWS KMS access with access key

In this opion, pass the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, as env variables for your AWS account you would like to use

docker run -it  -e MOESIF_MANAGEMENT_API_KEY="Your Moesif Management Key" \
 -e MOESIF_APPLICATION_ID="Your moesif application Id" \
 -e AWS_CUSTOMER_KEY_ID="Your customer managed key(CMK) in AWS KMS" \
 -e AWS_ACCESS_KEY_ID="Your AWS access Key Id" \
 -e AWS_SECRET_ACCESS_KEY="Your AWS secret access key" \
 -p 9500:9500 --name moesifproxy moesifproxy

Option B: AWS KMS access assuming an IAM role

For this option, the Secure Proxy will use any injected credentials to your EC2 or k8s resource and assume this role to gain auth access to AWS KMS. For Secure Proxy to assume the role AWS_SECURE_PROXY_ROLE_ARN, the base credentials should have been injected already. If not, use option 1.

Pass the IAM role AWS_SECURE_PROXY_ROLE_ARN as an environment variable. This role must have access to your AWS KMS instance.

docker run -it  -e MOESIF_MANAGEMENT_API_KEY="Your Moesif Management Key" \
 -e MOESIF_APPLICATION_ID="Your moesif application Id" \
 -e AWS_CUSTOMER_KEY_ID="Your customer managed key (CMK) in AWS KMS" \
 -e AWS_SECURE_PROXY_ROLE_ARN="IAM role that secure proxy should assume to access AWS KMS" \
 -e AWS_KMS_REGION="Your AWS KMS region such as 'us-west-2'" \
 -p 9500:9500 --name moesifproxy moesifproxy

4. Update your server integration

By default, Moesif server integrationss send data to https://api.moesif.net You must update your server integration’s to instead send data to your new secure proxy host, which can be done by updating the baseUri option Refer to your respective server integration docs or reach out to support.

The collector API is accessible under the /collector route of the secure proxy. Thus if your secure proxy is running at http://localhost:9500, you will need to update your server integration’s base url to http://localhost:9500/collector

An example for moesif-nodejs is below:

const moesifMiddleware = moesif({
  applicationId: process.env.MOESIF_APPLICATION_ID,
  baseUri: 'http://localhost:9500/collector',
});

5. Let Moesif know

Once both the secure proxy and your server integration is set up, email Moesif support with your the hostname for secure proxy. Moesif will make an update to your account to finalize installation.

Accessing the Moesif Management API

Like the collector API, you can access your data programmatically via the Moesif Management API while having it decrypted on the fly as long as you route your requests through the proxy under the /api endpoint. Keep in mind any scripts or access you make must have access to your local proxy instance.

For example instead of querying data in Moesif directly via https://api.moesif.com/v1/search/~/search/events, you can search data in Moesif like so:

curl -XPOST http://localhost:9500/api/search/~/search/events

Scaling Moesif proxy

For light load, you may be fine with a single Docker container running. If you expect a high volume of data collected or want to ensure high availability, you can run a pool of Secure Proxy instances behind a load balancer such as NGINX or HaProxy. Because secure proxy doesn’t maintain any persistent state, scaling Secure Proxy is simplified.

Different Secure Proxy setups

Each application in Moesif can be configured with a different secure proxy hostname. This can be helpful if you want different encryption keys for different environments. For example, you might want different encryption keys for your staging and your production environments. In this case, you would run two different Secure Proxy setups, one for each application.

Configuration options

MOESIF_APPLICATION_ID

Required Your Moesif Collector Application Id which can be found by logging into Moesif and going to API Keys from the top-right menu.

MOESIF_MANAGEMENT_API_KEY

Required Your Moesif Management API key. You can generate a new one by logging into Moesif and going to API Keys from the top-right menu. Ensure your API key is generated with at least the following scopes:

  • create:events
  • read:events
  • create:encrypted_keys
  • read:encrypted_keys
  • read:workspaces
  • update:workspaces
  • read:virtual_eventtypes
  • update:virtual_eventtypes

If you’re using additional functionality from the Moesif Management API, you may need additional scopes.

MOESIF_ENCRYPTION_ENABLED

Set to false to disable client-side encryption.

MOESIF_ENCRYPT_NUMERICS

Set to false if numbers are not sensitive and don’t require encryption. This enables numeric math in Moesif.

MOESIF_ENCRYPT_DATES

Set to false if dates are not sensitive and don’t require encryption. This enables date math in Moesif.

MOESIF_ENCRYPT_REQUEST_HEADERS

Set to false if request headers are not sensitive and don’t require encryption.

MOESIF_ENCRYPT_RESPONSE_HEADERS

Set to false if response headers are not sensitive and don’t require encryption.

MOESIF_ENCRYPT_METADATA

Set to false if metadata is not sensitive and don’t require encryption.

MOESIF_ENCRYPT_GRAPHQL_QUERY

Set to true if GraphQL query string is sensitive and requires encryption (uncommon). Moesif will encrypt the GraphQL query in request body. When encrypted, GraphQL analytics will have limited functionality. Default to false, where Moesif will encrypt all the fields in the request body except query field.

MOESIF_ENCRYPT_REQUEST_HEADERS_BLACKLIST

Set a string(comma separated) which can be a blacklist of fields that will bypass request headers encryption. For example - If MOESIF_ENCRYPT_REQUEST_HEADERS_BLACKLIST=Host is set, then Moesif will skip encrypting host at any level while creating an event. Please note that this configuration option is case insensitive.

MOESIF_ENCRYPT_RESPONSE_HEADERS_BLACKLIST

Set a string(comma separated) which can be a blacklist of fields that will bypass response headers encryption. For example - If MOESIF_ENCRYPT_RESPONSE_HEADERS_BLACKLIST=x-moesif-transaction-id is set, then Moesif will skip encrypting x-moesif-transaction-id at any level while creating an event. Please note that this configuration option is case insensitive.

MOESIF_ENCRYPT_METADATA_BLACKLIST

Set a string(comma separated) which can be a blacklist of fields that will bypass metadata encryption. For example - If MOESIF_ENCRYPT_METADATA_BLACKLIST=datacenter,sf is set, then Moesif will skip encrypting datacenter and sf at any level while creating an event. Please note that this configuration option is case insensitive.

ENABLE_DEBUG_LOGS

Set to true to print debug logs. This is helpful to debug internal issues with secure proxy.

AWS configuration config

The below options are used to configure AWS KMS.

AWS_CUSTOMER_KEY_ID

AWS_CUSTOMER_KEY_ID is the Customer managed key (CMK) in AWS KMS, To find KeyId follow the instructions here

AWS_KMS_REGION

AWS_KMS_REGION is a string representing aws region where AWS KMS is configured. Defaults to ‘us-west-2’

AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

These are access keys needed to access AWS KMS via api. More information about access keys here. Make sure that IAM user/role associated with the access keys has permissions to access AWS KMS.

AWS_SECURE_PROXY_ROLE_ARN

This is the ROLE ARN that secure proxy should assume to access AWS KMS

How to build

Moesif Proxy uses the revel webframework. To learn more about revel https://revel.github.io/

To build Docker image

docker build -t moesif/moesifproxy:latest

To push to Docker repo

docker login

docker push moesif/moesifproxy:latest

Limitations

  • If enabled for a pre-existing app, filtering and aggregations will not work on old data. We recommend creating a new app in Moesif for enabling secure proxy.
  • Numeric fields will behave like strings (i.e. no math supported) unless MOESIF_ENCRYPT_NUMERICS is set to false.
  • Date fields will behave like strings (i.e. no date math supported) unless MOESIF_ENCRYPT_DATES is set to false.
  • If using embedded templates (for customer-facing portal), the secure proxy must be accessible from the internet.
  • We recommend a subdomain like analytics.acmeinc.com that points to NGINX with SSL (See Docker example).
  • Alert rules with a group by will not have decrypted values in any notifications. A feature is on our roadmap for Q1 2022 to fix this.

Unencrypted data

There are certain parts of your API logs that bypass client-side encryption for proper analytics, including:

  • Request and response time (latency)
  • Route and verb
  • Response status code
  • user_id and company_id
  • Binary and non-JSON payloads such as Protobuf or Thrift
  • The schema/names of fields used within your API

Like most web analytics tools, the secure proxy does not encrypt data from client integrations like moesif-browser-js, which is used for tracking user attributes and actions like “Signed In” or “Purchased Plan”.

Other use cases

Besides client-side encryption and decryption, the secure proxy enables a variety of other use cases:

  • Reduce the number of outbound connections to Moesif’s APIs.
  • Relay requests from servers that don’t have direct access to the internet.
  • White label the Moesif Collector or Management API like analytics.acmeinc.com/api.

Updated: