Key Modules¶

The platform is organized by product feature area. Each area maps to a route segment under app/, a set of services/, the hooks/apiHooks/ that wrap them, and a backend gRPC service defined in proto/. This page is the map from "feature" to "where it lives".

flowchart TB
    Root["/projects (default landing)"]
    Root --> Proj[Projects]
    Root --> Rep[Reports]
    Proj --> DQ[Data Quality]
    Proj --> TF[Transformations]
    Proj --> DEL[Data Delivery]
    Rep --> SCH[Scheduling]
    Root --> RPT[Reporting]
    Root --> TEAM[Teams & Members]
    Root --> KEYS[API Keys]
    Root --> LOGS[Crawler Logs]
    Root --> NOTIF[Notifications]
    Root --> ADMIN[Admin]
    ADMIN --> ACC[Accounts]
    ADMIN --> BILL[Billing]
    ADMIN --> PROX[Proxies]
    ADMIN --> RBAC[Roles & Permissions]
    ADMIN --> SVC[Services]

Customer-Facing Modules¶

Projects¶

The organizing container for a customer's scraping work. Create, list, view, transfer ownership, and run bulk operations (stop / export / delete) across projects.

• Routes: app/projects, app/projects/[id], app/projects/create
• Services / proto: projects.service.ts, createProject.services.ts · projects.proto

Reports¶

A project contains reports, each tied to a service code (a customized crawler image). Run or schedule runs, view run history, and export datasets (CSV / JSON / Parquet).

• Routes: app/projects/[id]/reports, .../reports/[rid]
• Services / proto: allReportsService.service.ts, export.service.ts · customer-report.proto, export.proto

Data Quality¶

Author, test, and manage QA rules; profile datasets; surface anomalies; and request AI-suggested rules.

• Routes: app/quality
• Services / proto: dataQuality.service.ts, qualityrules.service.ts · data-quality.proto, data-profiler.proto

Data Transformations¶

Reshape raw crawled data via transformation rules, including AI-assisted ("generative") transformations, with a test-before-apply flow.

• Routes: app/projects/[id]/reports/[rid]/data-schema
• Services / proto: listTransformations.service.ts, generateTransformation.service.ts · transformations.proto, transformation_query.proto

Data Delivery¶

Configure how datasets are delivered: email, S3, Dropbox, Box, and SFTP/FTP, with format options.

• Routes: app/api/delivery
• Services / proto: dataDelivery.services.ts · delivery.proto
• See also: Integration Points

Reporting¶

Cross-project reporting, scheduling, automation, and sharing.

• Routes: app/reporting
• Proto: customer-report.proto

Teams & Members¶

Manage team membership and role assignment (Manager / Trusted / Viewer), and pending invites.

• Routes: app/teams
• Services / proto: teamss.services.ts · accounts.proto
• See also: User Invite workflow

API Keys¶

Generate, rotate, and revoke API keys for programmatic access to scraped data.

• Routes: app/api-keys
• Services: provisionApi.service.ts

Crawler Logs¶

Stream real-time, pod-level crawler execution logs and download them.

• Routes: app/projects/crawler-logs
• Services / proto / state: crawlerLogs.service.ts · crawler-logs-service.proto · store/crawlerLogsStore.ts
• Streamed over WebSocket — see Data Flow

Notifications & Messages¶

In-app notifications, email alerts, and digest subscriptions.

• Routes: app/notifications, app/messages
• Proto: notification.proto
• See also: Notification Subscription workflow

Internal / Admin Modules¶

The app/admin/* area is gated by the admin route guard (withAdminAuthentication) and authoritatively by Cerbos.

• Proto: accounts.proto, payments.proto, proxies.proto, stats.proto, support.proto

The Proto Service Catalogue¶

Each backend domain the platform talks to is defined as a .proto service in proto/ and compiled to a typed client in proto-types/. The current catalogue:

accounts · ai · auth · crawldata · crawldatakv · crawldataquery · crawler-logs-service · customer-report · data-access-service · data-profiler · data-quality · delivery · events · export · metadata · notification · payments · projects · proxies · schedulerv2 · stats · support · transformation_query · transformations