Skip to content

Partial Export

Introduction

In our platform, when a file is manually exported by admin user, we don't have options to export specific fields. All the columns added by the crawler are automatically exported. Similarly, we don't have a feature to add extra fields in the crawler for future or debugging purposes. To address this, we are adding new features that will give admin users the ability to export specific columns and allow crawlers to add extra fields as meta fields.

The meta fields will be visible only to admin users, and the admin users can also choose to include these fields in the manual export.

Limitations

  • Only admin users are allowed to use partial export.
  • For meta fields, reports with dynamic pages are not supported.
  • Only data with type ROWS are supported. JSON type data are not supported.

How are meta fields stored ?

When a new crawler is written and pushed to Bitbucket, the Bitbucket pipeline will trigger an action in our bitbucket pipeline. The bitbucket will call our webhook and this will trigger a WebhookReceived event. The WebhookReceivedEventWorkflow will trigger a CrawlerWebhookReceivedHandler workflow. In this workflow, If the action is code_sync the workflow will parse the meta_fields from the crawler code and updates the META_FIELDS row in vt_extractor_properties in vortex database. The extractor's meta fields are set/unset when writing/updating the crawlers and are specified at the top of the crawler. The following format is used to specify the meta fields in the crawler.

When a crawler is run and data is exported for the first time, it will check if there are any meta fields from the vt_extractor_properties table. If there are any meta fields, it will save those meta fields and non-meta fields in the vt_extractor_history_properties table with key EXPORT_FIELDS for future use. When the export is initiated a second time, it will fetch both meta and non-meta fields from the vt_extractor_history_properties table, and the files are then exported with these fields. In this way, a client will always get the same columns even if the fields are changed in the future.

How Partial Export works ?

When an admin user exports a file, they will be shown a modal, as above, to select which fields are to be exported. The meta fields will be displayed in the Optional Columns section. The non-meta fields are selected by default. If the user doesn't change the selection, this will be regarded as a normal export. If the user adds or removes columns, then it will be marked as a partial export. The partial export and all activities related to it will be hidden from non-admin users.