Bulk Export

Bulk export is an enterprise feature that provides direct access to your data stored in Moesif. With bulk export, you can quickly load hundreds of millions of events into data warehouses like Snowflake, Redshift, and BigQuery.

You can trigger a new bulk export job within the Moesif UI or through the Management API such as to setup recurring exports.

Bulk export is designed to move large amounts of your raw events for archival and data warehousing purposes. If you want to import a chart’s dataset into a tool like Microsoft Excel or Google Sheets, you may want to use the Export Chart button instead.

Supported Entities

Moesif supports bulk export on three types of entities in Moesif:

  • Events: Export your API calls and custom actions in Moesif with their associated fields like HTTP headers.
  • Users: Export your users stored in Moesif and their associated metadata.
  • Companies: Will export your companies stored in Moesif and their associated metadata.

Supported Export File Types

  • CSV: CSV files has some of the widest support for many of your favorite tools. This makes it ideal for exporting things like user and company data to your CRM and more.
  • JSON: JSON is ideal for parsing data quickly for scripting purposes. Many fields in Moesif are deeply nested which makes JSON a breeze.
  • Parquet: A Parquet file is column-oriented which can make analysis on a subset of columns faster vs loading the entire file in memory. Your schema is directly embedded in the file which can help SQL warehouse like tools which requires strict schema enforcement.

Export Schema

Data representing API calls can naturally be quite sparse and Moesif heavily partitions for performance (i.e. you may have certain HTTP headers set by only a few customers). To ensure exports are fast while reducing file size, Moesif will only include the columns with at least one defined value for the dataset.

If you’re using any automated ETL tooling, ensure it’s set up to handle schema evolution.

Some tools like BigQuery have very strict requirements around schema detection while incorrectly classifying data commonly. For these tools, it’s recommended to leverage a format like Parquet which can help overcome these challenges.

Exporting Events

To export events, select New from the left navigation bar and select Live Event Log in the presented modal. Then click the Export on the top left of the page and select Bulk Export.

Selecting Data to Export

The time it takes to export your data is directly proportional to both the number of events you are exporting and the number of columns selected.

If you want to trigger export jobs via API, contact your account manager or reach out to support for instructions.

Filtering Events

Bulk Exports adhere to any filters that are applied. Export all requests that meet certain conditions by first adding any filters you want applied before exporting and then click the Export button on the top left of the page and select Bulk Export.

Selecting Columns

If you don’t want all columns you can add some specific columns of interest before clicking the export. This can be done by clicking the Tables button above the live event log and selecting the Select Columns button.

Selecting Columns to Export

This will open a pop up enabling you to select your columns of interest.

Selecting Fields to Display

Once you have your event stream configured with the right criteria, click the Export button and select Bulk Export. In the modal that appears, specify the email address where the download link should be sent. Finally, select the file type and whether you want all columns or the previously selected columns.

Bulk Events Export

Exporting Users/Companies

Exporting users or companies is very similar and can be done using the Export button on the top right of the page:

Bulk export users and companies

In the modal that appears, specify the email address where the download link should be sent. Finally, select the file type and whether you want all columns or the previously selected columns.

Updated: