Data Collection Flow

This sequence documents the platform-collection path used by Bluesky, Reddit, Telegram, and YouTube integrations. Each platform has its own Web App and Model Server route, but they follow the same job, queue, shared-file, and callback pattern.

Platform and Queue Boundary

• The Web App creates a DataCollectionJob with status pending, calls the relevant Model Server endpoint, and marks the job enqueued after an accepted response.
• The Model Server uses the data_collection_tasks Redis queue. The deployment's data-collection worker consumes this queue.
• The platform worker owns the external API call and writes the collection file as <job-id>.csv in the shared collected-data directory.
• The notification worker sends the collection status callback to the Web App. On completed or checkpointed, the Web App creates the user-owned Dataset and FileMeta records that make the file available through product workflows.

Implementation Variants

The platform request shape and collector are platform-specific: the current routes cover Bluesky thread, timeline, and search collection; Reddit subreddit collection; Telegram channel messages; and YouTube comments. Some collection paths can emit a checkpointed result in addition to completed or failure states.

Exact platform credentials, request schemas, callback authentication, statuses, and file-retention rules belong in Cross-Repository Contracts.