Logging API Calls in Moesif from Logstash Using the HTTP Output Plugin
One of the main value propositions of the Moesif product is the fact that it works so well with a wide variety of tools. One such tool – Logstash – is an incredibly powerful solution for managing events and logs. Understanding how to pull Logstash logs into Moesif is a critical function for any business utilizing Logstash at scale.
Today, we’re going to look at how to log API calls from Logstash using the HTTP Output plugin for import into Moesif. Once you implement this solution, Moesif+ Logstash will become a powerful pair and an efficient way to track and analyze API usage.
Debug and Fix API Issues Quickly with Customer-Centric API Logging
14 day free trial. No credit card required.
Try for Free
How it Works
This method makes use of the Logstash HTTP Output Plugin. This plugin generates a generic endpoint that feeds all logged data to an external source. By defining the external source as the Moesif API Collector endpoint, you can feed the logs into Moesif as an external service. You will need to collect some basic information to get started – notably, you will need to ensure you have the authentication elements on the Moesif side (specifically the “X-Moesif-Application-Id” parameter) as well as the form and structure of the Logstash output.
The quality of what you get on Moesif’s side is going to rest heavily on how well-structured your data is on the Logstash side. Accordingly, make sure your codebase is clean and well-structured before you begin.
Step 1 - Create Your Moesif Account
Before you do anything, you should make sure that you have a Moesif account. To do this, simply head to the sign up page. Here, you will register using a few details:
- Organization Name
- Your current role at your organization
- What you want to achieve
Now that you have an account, you can move to prepping Logstash.
Step 2 - Prepare Logstash
On Logstash, you want to make sure that you have prepared your instance to properly push data to the HTTP Output plugin. Firstly, ensure that Logstash is accurately retrieving your API logs. This can be sourced from a variety of sources, but you must ensure that you are ultimately pushing your logs to a singular point.
One element you must consider at this stage is how you want your data filtered. Moesif benefits from having a lot of data, but if you want to limit the collected logs to only specific attributes, you can do this via a variety of Filter plugins. As an example, you can use the throttle filter to limit the amount of logging that is occurring, reducing noise for overly active systems:
filter {
throttle {
before_count => 3
after_count => 5
period => 3600
max_age => 7200
key => "%{host}%{message}"
add_tag => "throttled"
}
if "throttled" in [tags] {
drop { }
}
}
That being said, pre-filtering before the data exists Logstash will limit benefit you can get from integrating with Moesif, so it is better to not pre-filter in most cases.
Step 3 - Utilizing HTTP Output on Logstash
Output plugins are plugins that allow the logging result to be sent in a specified way. For those who are experienced with CLIs, this is essentially piping the output – in fact, stdout is one of the many outputs provided by Logstash!
The plugin we are more interested in right now, however, is HTTP Output. HTTP Output will allow you to send your logged events to any generic HTTP(S) endpoint. Within the configuration for this output, you will need to set the URL to the collector API endpoint (https://api.moseif.net/v1/events) along with any authentication or application ID data. From here, the HTTP Output plugin will POST the plugin content to the endpoint defined, and the Collector API will grab that data and store it for processing.
It’s important to note that you need to include some critical pieces of information on both sides of the call. On the Logstash side, your endpoint definition will need to include the application ID and other identification/authentication systems in order to properly log. You’ll also need to make sure that you have properly defined your API output, as the logs that will be grabbed by Moesif will depend heavily on the quality of the tagging and clarity on the Logstash side. In other words, if you want to understand anything, take the time to make everything well-defined, clear, and searchable.
Integrating into the Logstash Pipeline
In order to integrate Moesif into the Logstash pipeline, the following is required.
output {
http {
url => "https://api.moesif.net/v1/batch"
http_method => "post"
headers => {
"X-Moesif-Application-Id", "MOESIF_APPLICATAION_ID"
"Content-Type" => "application/json"
}
format => "json_batch"
http_compressioned => true
message => '[
{
"request": {
"time": "%{@timestamp}",
"uri": "%{[your_field_containing_uri]}",
"verb": "%{[your_field_containing_http_verb]}",
"headers": "%{[your_field_containing_request_headers]}",
"body": "%{[your_field_containing_request_body]}"
},
"response": {
"time": "%{your_field_containing_response_time}",
"status": "%{[your_field_containing_http_status]}",
"headers": "%{[your_field_containing_response_headers]}",
"body": "%{[your_field_containing_response_body]}"
},
"user_id": "%{[your_field_containing_user_id]}",
"metadata": {
"service": "%{your_field_containing_service_name}",
"host": "%{your_field_containing_host}",
"region": "%{your_field_containing_datacenter_region}"
}
}
]'
}
}
Let’s dive into the individual parts of this code. Firstly, we have the definition of the method for connecting to our endpoint as well as the specific headers needed to integrate:
output {
http {
url => "https://api.moesif.net/v1/batch"
http_method => "post"
headers => {
"X-Moesif-Application-Id", "MOESIF_APPLICATAION_ID"
"Content-Type" => "application/json"
}
format => "json_batch"
http_compressioned => true
The “url” part of this request is where we point to the API endpoint for batch logging. We define the HTTP method as “POST”, which allows us to send data to the server. By passing the MOESIF_APPLICATAION_ID through the “X-Moesif-Application-Id” parameter, we can define the specific application that is receiving the logs, and by using both the format and http_compressioned variables, we set the way these logs are sent to that application.
message => '[
{
"request": {
"time": "%{@timestamp}",
"uri": "%{[your_field_containing_uri]}",
"verb": "%{[your_field_containing_http_verb]}",
"headers": "%{[your_field_containing_request_headers]}",
"body": "%{[your_field_containing_request_body]}"
},
"response": {
"time": "%{your_field_containing_response_time}",
"status": "%{[your_field_containing_http_status]}",
"headers": "%{[your_field_containing_response_headers]}",
"body": "%{[your_field_containing_response_body]}"
},
"user_id": "%{[your_field_containing_user_id]}",
"metadata": {
"service": "%{your_field_containing_service_name}",
"host": "%{your_field_containing_host}",
"region": "%{your_field_containing_datacenter_region}"
}
}
]'
}
}
With our connection established, we can begin to set up the formatting of the request flow. With the rest of the variables set, we can append a timestamp, verb, body, metadata, pretty much any variable we would want to log that is supported by Logstash. It’s important to know that not all of this is required - in some environments, it may be beneficial to only log small amounts of data, and in those cases you can trim this down.
That being said, logging works best when you provide as much context as possible, and it’s advisable to be pretty liberal with what is being sent if you can manage it.
Step 4 - Process Data in Moesif
Now that you have your data in Moesif, you must ensure that data is being documented, tracked, and leveraged. This can take a few different forms, but you should at least cover the following areas:
- Ensure you are collecting timestamps or other time-based data to ensure you have context for logs over time outside of just the time where Moesif collected the information.
- Mirror your data structure on both Logstash and Moesif – if you are working on Client data, that data should be well-marked and categorized regardless of where you’re looking at it.
- Ensure you are not pushing authentication to a public endpoint – because you are transferring this information from one service to another, you should ensure basic security procedures are being followed to prevent any MITM attacks.
Conclusion
The method in this piece is just one potential solution for this use case. Using Logstash to push logs to Moesif can take a variety of forms – another route that may be easier depending on local setup is to simply deploy a server integration. If you are already using a compatible server and are running Logstash locally or pushing logs to a local instance, it may make sense to default to a server integration.
Debug and Fix API Issues Quickly with Customer-Centric API Logging
14 day free trial. No credit card required.
Try for Free
Related Articles
API Development
The Ultimate Guide to REST API Naming Convention: Best Practices for Clarity & Consistency
Discover the art of crafting perfect REST API endpoints, where consistency and simplicity transform complex web development into a seamless experience. Learn...
API Development
API for Dummies: Your Easy Guide to Understanding APIs
This beginner-friendly post explains the basics of API and how they create amazing experiences through the apps and services you daily use.
API Development
Step-by-Step Blueprint to Create RESTful APIs for Efficient Web Services
Learn from server setup to securing data, ensuring your APIs are scalable and efficient.Dive into the details for a comprehensive understanding of designing,...
API Development
The AI API Gold Rush: How to Turn Your Models into Revenue With Moesif
Monetize your AI model by transforming it into an AI API, creating new revenue streams by integrating your AI capabilities into other applications. This guid...