Bulk Export

Bulk Export is an enterprise feature that provides direct access to your data stored in Moesif. It allows you to efficiently retrieve hundreds of millions of event, user, and company data from Moesif and quickly load them into data warehouses like Snowflake, Redshift, and BigQuery.

You can use Bulk Export both from Moesif UI in the browser and Moesif Management API. Using the Management API, you can set up recurring bulk exports for different use cases—for example, a data pipeline that feeds your data in Moesif into a data warehouse like BigQuery.

Bulk Export is designed to move large amounts of your raw events for archival and data warehousing purposes. If you want to import a chart’s dataset into tools like Microsoft Excel and Google Sheets, download the chart instead for your Time Series and Segmentation metric types.

Supported Entities

Moesif supports bulk export on three types of entities:

Events

Exporting API calls and custom actions in Moesif with their associated fields like HTTP headers.

Users

Exporting users stored in Moesif and their associated metadata.

Companies

Exporting companies stored in Moesif and their associated metadata.

Supported Export File Types

Moesif supports the following file types for bulk export:

CSV

CSV files have some of the widest support for many of your favorite tools. This makes it ideal for exporting user and company data to your CRM and more.

JSON

JSON has become a standard for parsing data quickly for scripting purposes. Many fields in Moesif are deeply nested, making JSON an ideal choice.

Parquet

Parquet is an open source column-oriented data format that can make analysis on a subset of columns faster when compared to loading the entire file in memory. Your schema is directly embedded in the file. This can help SQL warehouse-like tools that require strict schema enforcement.

Export Schema

Data representing API calls can naturally become quite sparse—for example, you may have certain HTTP headers set by only a few customers. Moesif relies heavily on partitions for performance. Therefore, to ensure exports remain fast while reducing file size, Moesif only includes the columns with at least one defined value for the dataset.

If you use any automated ETL tooling, make sure to set up the tool to handle schema evolution. Some tools like BigQuery have very strict requirements around schema detection while incorrectly classifying data. For these tools, we recommend that you leverage a data format like Parquet to overcome the challenges.

Export Events

To export Live Event Log events, follow these steps:

1. Go to your Live Event Log workspace.
2. Select Export and then select Bulk Export.
3. Specify the email address where you want Moesif to send the export file.
4. Select the export file format.
5. Select the fields you want to include in the export.
6. Select Start Export.

The time it takes to export your data depends on the number of events you export and the number of fields you select.

If you want to trigger export jobs using Moesif Management API, contact your account manager or reach out to support for instructions.

Event Filters

Bulk exports adhere to any filters that you have applied. Make sure you apply the filters you require before exporting.

Fields Selection

If you don’t want to include all event fields in the bulk export, you can specify the fields you want in table view before selecting Bulk Export. Otherwise, select All Fields in the Bulk Events Export dialog.