Event Processors (Temporal Workflows)

Introduction

The Event Processor is a critical backend process in Grepsr, an event-driven architecture system. In this documentation, we will explore the purpose, functionality, and benefits of the Event Processor. We will also discuss the transition from using a Node.js application and Apache Flink to the current implementation using Temporal.io, a microservice orchestration system.

Purpose

The Event Processor in Grepsr is responsible for handling various state changes within the system. These state changes are referred to as events and can include actions such as user sign-ups, crawler start/stop events, user requests for data exports in JSON format etc. By capturing and processing these events, the Event Processor enables the system to react and perform necessary actions based on these state changes.

Architecture Overview

The Grepsr architecture follows an event-driven approach, where events are emitted to an AWS Event Bridge whenever there is a state change within the system. The AWS Lambda function is then used to trigger a workflow in Temporal.io for event processing. Temporal.io is a microservice orchestration system that provides reliability and visibility for event handling.

Temporal.io Features and Benefits

Temporal.io offers numerous benefits over the previously used Node.js application and Apache Flink for event processing. Some of the notable features of Temporal.io include:

1. Reliability Temporal.io provides strong reliability guarantees, ensuring that events are processed consistently and reliably. It achieves this through the use of advanced workflow management and state tracking mechanisms. In the event of failures or system crashes, Temporal.io can recover and resume event processing from the exact point of failure, ensuring no data is lost or duplicated.

2. Visibility One of the key drawbacks of the previous event processing approaches was the lack of visibility into the event handling process. However, Temporal.io offers comprehensive visibility and observability features. It provides a graphical user interface (GUI) and an extensive set of monitoring and debugging tools, allowing developers and system administrators to monitor and trace events through the processing workflow. This enhanced visibility greatly simplifies troubleshooting and debugging of event processing issues.

3. Scalability Grepsr handles a large volume of events, and scalability is crucial for efficient event processing. Temporal.io excels in this aspect by offering horizontal scalability. It allows the system to handle an increasing number of events by automatically distributing the workload across multiple instances of event processors. This distributed processing capability enables Grepsr to scale seamlessly as the event load grows.

4. Workflow Orchestration Temporal.io acts as a powerful workflow orchestrator, providing a high-level abstraction for event processing. It allows developers to define complex event handling workflows using familiar programming languages, such as Go, Java, or Kotlin. This declarative approach simplifies the development and maintenance of event processing logic, making it easier to modify or extend the workflows as needed.

5. Event Scheduling and Retries Temporal.io supports event scheduling and retries out-of-the-box. It provides a convenient API for scheduling events at specific times or after certain intervals. Additionally, in the case of failures or errors during event processing, Temporal.io automatically retries the failed events based on configurable retry policies. This built-in functionality reduces the complexity of handling event scheduling and retry logic in the application code.

Conclusion The Event Processor plays a crucial role in the Grepsr system, enabling the handling of various state changes through event-driven architecture. With the transition to Temporal.io, the Event Processor has gained significant benefits in terms of reliability, visibility, scalability, workflow orchestration, and event scheduling. These features empower Grepsr to process events consistently and efficiently, ensuring a smooth and reliable experience for its users.

Registered Temporal Workflows

Name	Source of Event	Purpose of Event
CrawlerSummaryCalculatedEvent	eq-processor
ProjectCreatedEvent	phoenix-projects-service
UserProjectAddedEvent	phoenix-account-service
MetadataUpdatedEvent	phoenix-projects-service
ReportRunFailedEvent	phoenix-flink-crawler-events
TicketCommentedEvent	phoenix-webhooks	- Handled by TicketCommentedEventWorkflow in Temporal - Fetches the most recent comment from that specific ticket in JIRA. - Fetches watchers and participants user emails from that specific ticket in JIRA. - Fetches the project related to that JIRA ticket from platform. - Fetches platform users associated with that JIRA ticket. - Filters those project users on the basis of their subscription. - Segregates internal and external users on the basis of email domain. - Sends templated email to each of these users. - If the message is an internal note, sends templated email to internal users only. - Does not send templated email to the author of the comment.
CrawlerIncidentEvent	vortex-backend
ContactMessageRepliedEvent	phoenix-webhooks	- Handled by ContactMessageRepliedEventWorkflow in Temporal - Triggered when someone replies to the email sent by TicketCommmentedEvent - All flow similar to TicketCommentedEvent - Adds the replies as comment JIRA ticket as the person who initiated this event.
ContactFormSubmittedEvent	phoenix-webhooks
HistoryStatsAddedEvent	vortex-backend
ReportCreatedEvent	phoenix-webhooks, phoenix-api
ReportUpdatedEvent	phoenix-webhooks	- Fetches project information to which the report belongs to. - Fetches list of all reports associated with that project. - Updates algolia object with project’s uuid as ObjectID, does partial update with report list only
ReportCanceledEvent	phoenix-projects-service
phoenix-projects-service
ReportSchemaGeneratedEvent	phoenix-webhooks	Calls Report Service in projects-service to sync generated schema.
UserCreatedEvent
TicketUpdatedEvent		- Calls JSM Service rpc in support-service with ticket-key. - Calls JSM Service rpc in support-service to sync properties with hubspot integration.
LoginEmailDeliveredEvent
DataDeliveryFailedEvent
CrawlerRunStartedEvent
TicketCreatedEvent		Calls JSM Service rpc in support-service with ticket-key to sync properties from platform project to JIRA ticket and hubspot.
ReportRunStoppedEvent		Triggers pusher event to inform frontend for report-stop-progress bar.
ReportDeliveryInitiatedEvent
ProjectUpdatedEvent		Calls Report Service in projects-service to sync project object to algolia.
TicketTransitionedEvent		Calls JSM Service rpc in support-service to sync properties in hubspot.
UserTeamInvitedEvent
CustomerProjectTransferredEvent
PasswordResetRequestedEvent
InactiveUserDetectedEvent
MeetingScheduledEvent	Webooks	alls JSM Service in support-service to create new ticket in MC board in JIRA for lead-tracking purposes. - Triggered when users schedule a meeting from website.