Skip to content

Email Digest

Introduction

The Email Digest feature within the Grepsr platform is a sophisticated notification system designed to inform users about anomalies detected in our systems or data of the subscribed accounts(organization). This feature compiles and sends comprehensive summaries of anomalies and metrics directly to the subscribed users’ emails, based on their chosen time and frequency—daily, weekly, or monthly. This ensures timely awareness and action regarding any potential issues affecting data integrity or operational efficiency.

System Efficiency and Scalability

Previously, the system utilized a pod that polled the database every 10 seconds to identify the next scheduled run. While effective, this approach was not optimal in terms of computing resources and efficiency. To address these concerns and ensure a scalable solution, we have optimized the scheduling mechanism using Temporal Schedules. This enhancement aligns with our commitment to providing reliable, efficient, and timely notifications to our users.

Digest Content Details

The Email Digest provides a detailed overview of system health and data integrity, organized into the following sections:

Specific Alert Sections

  • Count Digest Data: Highlights data counts and fill rates, including significant changes with direct links to the application for more details.Any run is supposed to have issues if the data count difference percentage <= -10 or difference_percentage >= 30

  • Failed Runs Data: Lists instances of data retrieval or processing failures, with specifics and links to detailed information. A run is considered failed if the status of the run is failed or it consists of 0 data counts.

  • Long-Running Crawler Data: Identifies crawlers exceeding expected run times, indicating possible issues. A run is considered long running if the crawler execution time is more than its average execution time by.

  • Missed Runs Data: Details data runs that did not execute as scheduled, with direct links for further analysis. A scheduled run is considered failed if it doesn't start until its next scheduled time.

  • Delayed Runs Data: Reports on data runs starting later than scheduled. A scheduled run is considered delayed if it doesn’t run until 5 minutes of its scheduled time.

All these alerts are sent by the stats-service.

Temporal Schedules

Schedules are setup in temporal at grepsr-cron namespace for the digests setup by users.

The information about when the schedule ran and when the next runs are expected to run can also be found in temporal.