Pheonix Webhooks¶

Introduction¶

Inbound webhooks are a powerful tool that allow businesses to receive real-time notifications and data from external sources. Grepsr offers a suite of inbound webhooks that can be integrated with various external and internal services, enabling us to streamline our workflows and improve our overall efficiency. By leveraging these webhooks, we can automate our data processing, integrate with third-party applications, and respond to customer inquiries in real-time. Additionally, these webhooks are highly customizable, allowing developers to tailor them to their specific needs and integrate them with a wide range of systems and applications.

Overall, the web hooks in repo phoenix-webhooks provide a powerful way for external as well as internal services to stay up to date with important changes within our system, enabling a wide range of integrations and workflows.

Repo: phoenix-webhooks
Deployment Branch: prd-stab

Available hooks¶

Web-hook	Source	Type	Remarks
hooks/crawler	Crawler Plugins	In House	Crawler Webhooks are an essential feature for automating web scraping and data extraction workflows. The purpose of Crawler Webhooks is to either sync schema or generate and update reports, depending on the needs of the user. When code gets pushed, a webhook is sent by drone. The webhook server then syncs all necessary details (like service code, report ID, etc. ) with the database. When syncing schema, a ReportSchemaGeneratedEvent event is emitted to AWS SNS. When syncing other details, a ReportCreatedEvent or ReportUpdatedEvent event is emitted to AWS SNS.
hooks/payment	Chargebee, Fastspring, Bluesnap	3rd Party	Payment Webhooks are an essential feature for processing online payments and automating payment-related workflows. The purpose of Payment Webhooks is to receive and process payment information from payment gateways such as Chargebee, Bluesnap, and FastSpring. When a payment is received, a PaymentReceivedEvent event is emitted to AWS SNS. This event contains information about the payment, such as the amount paid, the customer's information, and the transaction details.
hooks/sendgrid	Sendgrid	3rd Party	SendGrid offers a variety of webhooks that developers can use to receive real-time notifications and data about different events that occur within the SendGrid platform. Grepsr uses Inbound Parse Webhook from send grid which notifies you when a new inbound email has been parsed and provides you with the parsed data. The purpose of SendGrid webhooks is to emit an event to sns with event type SendgridInboundParsedEvent when an email is send via send grid.
hooks/contact_form	Grepsr Platform	In House	When a customer fills up contact us form via grepsr website, the data is send to contact_form web hook which then emits a sns event with event type ContactFormSubmittedEvent.
hooks/mailgun	Mailgun	3rd Party	Mailgun Webhooks are a feature that allows for automated communication between Mailgun and grepsr services. The purpose of Mailgun Webhook is to emit events to aws sns with event types MailgunInboundParsedEvent.

Temporal Migration :¶

Overview:¶

In August of 2023, our system underwent a significant architectural change regarding the processing of webhooks. We migrated our webhooks to Temporal for enhanced visibility and to improve the overall system efficiency.

Previous Workflow:¶

Prior to this migration, the webhooks executed business logic directly within themselves. This meant that the webhook not only had the responsibility of capturing the event but also the subsequent processing of the event data, making the entire process less modular and harder to monitor or troubleshoot.

Post-Migration Workflow:¶

With the migration to Temporal:

Event Relay: Instead of performing business logic within the webhook itself, now any hook that's received is relayed as an event to Temporal.
WebhookReceivedEvent Emission: Upon receipt of a webhook, a WebhookReceivedEvent is emitted. This event carries all the essential metadata associated with the received webhook.
Temporal Handling: Once Temporal receives the WebhookReceivedEvent, it's responsible for handling the business logic previously contained within the webhook. This division of responsibility ensures that webhooks are lean, simply acting as messengers, while Temporal handles the heavy lifting of processing.

Benefits:

Enhanced Visibility: With Temporal managing the events, there's better visibility into the event processing lifecycle, making it easier to monitor, audit, and troubleshoot issues.
Modularity: By separating the capture of webhooks and their processing, the system becomes more modular. This makes it easier to scale, maintain, and update either component without affecting the other.
Efficiency: Reducing the responsibility of webhooks to merely capturing and relaying events streamlines their functionality, possibly leading to faster event capture. Temporal, being designed for such processing tasks, can efficiently handle the subsequent business logic.

Recommendations for Developers:

Monitoring: Ensure you're monitoring the new flow in Temporal to catch and address any potential hitches early on.
Integration Testing: Post-migration, perform integration tests to confirm that the webhooks relay events to Temporal correctly, and Temporal processes these events as expected.
Documentation: Continuously update internal documentation to reflect this change, so that new team members or developers working on related components are aware of the new workflow.

In conclusion, this migration aims to improve the reliability, scalability, and visibility of our webhook event processing. All stakeholders are encouraged to familiarize themselves with this new architecture and provide feedback for continuous improvement.

WebhookReceivedEventWorkflow¶

The entry point to the temporal workflow of webhook is WebhookReceivedEventWorkflow.

The workflow is triggered when temporal receives a WebhookReceivedEvent in its events task queue. The workflow unpacks the webhook payload. The workflow then tries to match the endpoint from unpacked structured webhook payload to a map of endpoint-to-(handler/event).

MeetingScheduledEvent¶

Endpoint: /jira_schedule

The goal of this event is to create a ticket in JIRA.

The detailed documentation page for this specific event will be linked here later.

JsmWebhookReceivedHandler¶

Endpoint: /jsm

TicketCreatedEvent
TicketUpdatedEvent
TicketTransitionedEvent
TicketCommentedEvent
ontactMessageRepliedEvent
LeadClassifiedEvent

CrawlerWebhookReceivedHandler¶

Endpoint: /crawler

ReportSchemaGeneratedEvent¶

Action: schema_sync

The detailed documentation page for this specific event will be linked here later.

Code Sync¶

Action: code_sync

This webhook is triggered whenever a new commit is pushed to vortex plugin repositories as configured in the their bit-bucket pipeline as vortex-sync

PaymentWebhookReceivedHandle¶

Endpoint: /payment

chargebeePaymentWebhookHandler¶

Action: /chargebee

The goal of this handler is to emit PaymentReceivedEvent.

fastspringPaymentWebhookHandler¶

Action: /fastspring

The goal of this handler is to emit PaymentReceivedEvent.

ContactFormWebhookReceivedHandler¶

Endpoint: /contact_form

Whenever any “potential” client submits the contact-sales form, the ContactFormWebhookReceivedHandler is triggered. The goal of this handler is to emit ContactFormSubmittedEvent.

ContactFormSubmittedEvent¶

The event is emitted without any changes to webhook payload.

The detailed documentation page for this specific event will be linked here later.

MailgunWebhookReceivedHandler¶

Endpoint: /mailgun

This handler caters to any Mailgun webhook requests. The goal of this handler is to emit MailgunInboundParsedEvent.

MailgunInboundParsedEvent¶

The event is emitted regardless of the information on the payload. The purpose of the event is assumed to be only for logging purposes.

SendgridWebhookReceivedHandler¶

Endpoint: /sendgrid

This handler caters to any Sendgrid webhook requests. The goal of this handler is to emit SendgridInboundParsedEvent.

SendgridInboundParsedEvent¶

The event is emitted regardless of the information on the payload. The purpose of the event is assumed to be only for logging purposes.