Pipelines & Workers

Analysis, data collection, notifications, and report generation run asynchronously through Redis-backed RQ workers. The HTTP API accepts and validates work; workers own execution and must communicate outcomes through the callback path rather than modifying Web App data directly.

Queue names, shared statuses, and delivery guarantees are cross-repository contracts. Use Cross-Repository Contracts as the canonical reference before changing them.

Pipeline Map

Analysis Pipeline

1. The API verifies the batch shape, each job's fields, requested processor, relative file path, and any required provider credential.
2. It returns an enqueue result for each accepted job and must not enqueue an existing RQ job twice.
3. A worker selects the processor from the requested gateway and model. Current documented paths include ml_server (Marsad and Tanbih), openrouter, and open_source.
4. The processor reads the requested input range, writes its repository-owned output as needed, and prepares progress or terminal-status information.
5. The worker enqueues callback delivery instead of contacting the Web App database.

to_row is exclusive. Invalid ranges, inaccessible files, malformed fields, and unavailable credentials must fail validation before work is enqueued. OpenRouter transient provider failures currently retry up to four times with exponential backoff; provider 400, 401, and 403 failures are not retryable. See the analysis flow for the end-to-end sequence.

Collection Pipeline

Collectors are platform-specific, but share a single lifecycle: validate the request, run outside the HTTP request, persist a CSV at the established collection path, and issue status callbacks. The documented routes cover Bluesky thread, timeline, and search collection; Reddit subreddit collection; Telegram channel messages; and YouTube comments.

A collector may deliver checkpointed when its partial CSV is usable. Treat this as a successful, labelled partial result—not as a silent failure or a substitute for completed. Preserve the jobId in filenames and logs so a worker result can always be correlated with the Web App record.

Report Pipeline

Report generation reads a pre-existing shared input and produces a PDF below shared-data/generated/analytics_reports/{fileMetaId}/. It is queued rather than generated in the request handler. Validate file_meta_id, the relative file path, output filename, and section selection before enqueueing; do not construct a path from untrusted absolute values. The report-generation flow describes the shared sequence.

Worker Operations and Recovery

Workers should be deployed by queue responsibility so a slow provider call or report cannot consume capacity intended for callback delivery. The current contract defines a two-hour RQ timeout and no general automatic retry policy; failed analysis work may be manually requeued from the failed-job registry. Any change to this policy must be idempotent: a retry may repeat file writes or callbacks, and neither should corrupt a result or overwrite a newer status.

When investigating a failure, correlate the Web App job ID, RQ job ID, pipeline name, and callback response. Check, in order:

1. Request validation and enqueue response.
2. Queue availability and the appropriate worker process.
3. Shared-data mount visibility and the relative input/output path.
4. Provider or platform response, with secrets redacted.
5. Callback delivery and the Web App's response.