Cross-Repository Contracts
The shared HTTP, storage, queue, and security rules between the web app and model server.
Cross-Repository Contracts
This page records the rules that main/ and model-server/ must change together. Code, schemas, and root Compose files remain the source of truth.
Do not rename a route, payload field, status, queue, or shared-data directory in one repository only. Make compatible changes first, deploy both consumers, and then remove the old behavior.
HTTP Endpoints
The web app calls MODEL_API_URL, including its /api prefix. The model server calls MAIN_APP_API_BASE_URL, including the web app's /api prefix. Both directions use JSON.
Analysis requests
The web app enqueues analysis with POST /api/analyze/enqueue-batch and { jobs: [...] }. Every job includes its web-app job_id, analysis and model identifiers, gateway, relative file_path, text column, language, record range, and batch size. A custom prompt is optional; an OpenRouter job must also include credentials.openrouter_api_key.
to_row is exclusive. The model server rejects invalid fields, ranges, credentials, and inaccessible files with 422. A successful batch returns one enqueue result per job; an existing RQ job is not queued twice. The web app marks a job enqueued only after the request succeeds.
Collection and Telegram requests
For asynchronous collection, the web app creates the database record first and calls the platform route with /{userId}_{jobId}. It also includes job_id and dataset_name in the request body. The model server uses the web-app job ID to track the work and writes {jobId}.csv when it completes.
YouTube, Reddit, Bluesky, and Telegram routes return 422 for invalid input, 400 for duplicate jobs, and 500 for unhandled errors. Their successful response field is not yet consistent: Bluesky returns job_id; YouTube, Reddit, and Telegram return data. Preserve that distinction until both services deliberately standardize it.
Analytics and report requests
Analytics calls use a file_meta_id, a relative file_path, and optional date filters. Report generation additionally supplies output_filename and a section_choice from 1 to 6; the model server queues the report and returns 202 with an RQ job ID. Analytics validation and unsupported capabilities return 422.
Model-server callbacks to the web app
Workers enqueue callback delivery through the notifications queue instead of writing to the web-app database directly.
| Callback | Required payload | Web-app responsibility |
|---|---|---|
/api/analyze/marsad/{jobId} | status, processing counts | Update the analysis job and notify on completion. |
/api/analyze/tanbih/{jobId} | Incremental success and failure counts | Accumulate progress and derive the final status. |
/api/analyze/openrouter/{jobId} | Optional status, counts, message, and level | Update progress and retain status messages. |
/api/data-collection/jobs/{jobId} | status, totals | Update the collection job; publish its CSV when usable. |
/api/analytics/{fileMetaId}/generate-report/jobs/callback | Output filename and final status | Email the report download link on completion. |
Callback handlers return 400 for invalid input, 404 for an unknown job or file, and 500 for processing errors. The notification worker must log failed deliveries using stable IDs and redacted errors only.
Queue Ownership
Redis is the broker. The web app owns database records and user-visible state; the model server owns worker execution and emits callbacks.
| Queue | Work |
|---|---|
tasks | Unified analysis: Marsad, Tanbih, OpenRouter, and open-source models. |
data_collection_tasks | Platform collection and report generation. |
notifications | Model-server-to-web-app callback delivery. |
tanbih_tasks | A Compose worker exists, but current enqueue code does not produce work for this queue. |
Statuses, retries, and delivery
The shared status values are pending, enqueued, started, failed, completed, and checkpointed. checkpointed is a usable, partial collection result that the web app publishes as a labelled dataset.
RQ work has a two-hour timeout and no automatic retry configuration. Failed analysis jobs may be manually requeued from the failed-job registry. OpenRouter separately retries transient provider failures up to four times with exponential backoff; provider 400, 401, and 403 responses are not retried. Any retry-policy change must preserve idempotency and tolerate duplicate callbacks.
Shared Storage
Both Compose configurations mount ./main/shared-data into the web app, model server, and workers as /app/shared-data. Database fields and request payloads use forward-slash paths relative to the service root, beginning with shared-data/; never exchange absolute host or container paths.
| Location | Owner and purpose |
|---|---|
shared-data/ and configured dataset subdirectories | Web-app uploads and FileMeta records. |
shared-data/collected-data/{jobId}.csv | Model-server collection output; web app records it after callback. |
shared-data/output/ | Model-server intermediate outputs, named with _output or _failed. |
shared-data/generated/word-cloud/ | Model-server word-cloud output. |
shared-data/generated/analytics_reports/{fileMetaId}/ | Model-server PDF reports; web app lists and downloads them. |
All storage environment variables must resolve to these same locations in both services. The collection directory is collected-data; do not rely on an inconsistent collected-dataset fallback.
Lifecycle and cleanup
The web app cleans up newly uploaded files when a dataset create or update fails. There is no general model-output retention worker today. Before adding one, agree on retention, directory ownership, database-reference checks, and active-job exclusions. Never remove a file still referenced by a record or in-progress job.
Security Boundaries
Service authentication
MODEL_API_TOKEN/API_BEARER_TOKEN and MAIN_APP_API_KEY are separate, server-only service credentials. Rotate both consumers together and never expose them in browser variables, logs, fixtures, or documentation.
The model server sends MAIN_APP_API_KEY with callbacks, but the current web-app callback handlers do not validate it. The model server also leaves its current analysis and analytics blueprints outside its token-protection wrapper. Until both directions validate bearer tokens and return 401 for invalid credentials, these routes must remain private-network-only.
User authorization and downloads
A signed-in user must also be authorized for the target dataset or file. The report-download handler sanitizes the filename but currently looks up FileMeta by ID without an ownership/public-access check; add that check before treating report links as private.
Provider credentials and sensitive data
Keep provider keys, OAuth tokens, Telegram sessions, prompts, uploaded data, and callback bodies server-side. Forward an OpenRouter key only for its job and never persist or log it in the model server. Stored provider credentials need a mandatory, server-only encryption secret and a rotation plan. Never commit real credentials or sensitive generated data; rotate any that were committed.