The worker process, the BullMQ queues with their retry and dead-letter policies, the transactional outbox dispatcher, and every scheduled job with its cron in Africa/Algiers, idempotency, failure handling, monitoring, owner and release.
Everything asynchronous in the platform runs in the worker process (ADR-004) on BullMQ over Redis 7, fed by the transactional outbox (ADR-007). This page defines the queues, the dispatcher that turns committed events into jobs and live-stream messages, and every scheduled job. Status changes made by jobs are also listed on lifecycles and statuses; this page owns their names, schedules and behaviour. Built in SRV-08; each module's processors land with that module.
Library and connections. BullMQ with key prefix hoco on the environment's Redis. Producers share one connection; each worker uses its own blocking connection created with maxRetriesPerRequest: null, as BullMQ requires. Redis runs with appendonly yes, appendfsync everysec and maxmemory-policy noeviction.
Job names are <module>.<verb> in camelCase (orders.expireAssisted), the same name in code, logs, metrics and this page.
Payloads carry ids, never personal data: { "v": 1, "orderId": "…" }. A processor reloads what it needs, so a retried job always works on current data. The schema version v lets an old and a new worker coexist during a deploy.
Job ids deduplicate: evt:{eventId}:{handler} for event handlers, ntf:{deliveryId} for notification deliveries, doc:{type}:{entityId}:{version} for documents, exp:{exportJobId}, imp:{importJobId}:{phase}. Adding a job whose id already exists is a no-op while that job is retained.
Every processor is idempotent. At-least-once delivery is the contract: a job can run twice after a crash or a stalled lock. Effects are keyed (upsert on a natural key, unique sourceEventId, status guards in UPDATE … WHERE status = …) so a second run changes nothing.
Errors are classified. Transient failures (network, provider 5xx, timeouts, deadlocks) throw and are retried with backoff. Permanent ones (invalid FCM token, template missing, validation) throw BullMQ's UnrecoverableError, which skips the remaining attempts.
Time. Cron patterns are evaluated in Africa/Algiers (tz on every job scheduler). Algeria has no daylight-saving time, so the zone is a fixed UTC+1, but the zone is still passed explicitly; business days come from the shared helpers (ADR-011).
PDF rendering: invoices, delivery notes, credit notes, picking lists
2 (sandboxed processor)
3 · exponential from 30 s
24 h
dead-letter; alert (a missing invoice PDF is visible to resellers)
exports
ExportJob runs (PDF, XLSX, CSV)
2 (sandboxed)
2 · fixed 60 s
24 h
ExportJobFAILED; requester notified in-app
imports
ImportJob dry runs and applies
1 (sandboxed)
1 — re-run on request only
7 days
ImportJobFAILED with the row report
analytics
Dirty-day recomputation, daily rebuild, backfills
1
3 · exponential from 5 min
7 days
dead-letter; alert
cron
Ticks of the scheduled jobs below (thin processors that may enqueue bulk work elsewhere)
4
3 · exponential from 30 s
last 100 per scheduler
dead-letter; alert; the next tick catches up
dead-letter
Copies of jobs that exhausted their attempts
no worker
—
30 days
replayed by hand
Dead-letter policy. BullMQ has no built-in dead-letter queue. The worker's failed listener copies a job whose attemptsMade reached its attempts (or that threw UnrecoverableError) to dead-letter with the original queue, name, data, failure reason, stack and attempt count, then raises an alert. Queues whose failures already land on a business record (notify-*, exports, imports) do not copy to dead-letter: the record is the evidence, and notifications.reconcileDeliveries or the requester retries. Replay is explicit (runbook): the job is re-added to its original queue with a :replay-{n} job-id suffix.
Long and CPU-bound work (documents, exports, imports) runs in sandboxed processors — separate processes — so a 50 000-row export cannot block the event loop of the other workers or hold their locks. Those queues use a lockDuration of 120 s; the others keep BullMQ's default. Imports apply in chunks of 500 rows, each chunk committed with a progress marker on the ImportJob, so a restarted import resumes instead of starting over.
apps/server/src/worker.ts, started by docker/start.sh when APP=worker (same image as the api; the branch is added with SRV-08). Loads the environment through @hoco/env/server exactly like the api.
Boot
Connect Prisma and Redis → register the processors declared in every module's jobs.ts and the event handlers of every events.ts → upsert the job schedulers (idempotent: same id, same pattern) → run the catch-up of daily jobs → start the outbox dispatcher → expose /health and /metrics on WORKER_HEALTH_PORT (internal only, not routed by Traefik).
Migrations
Never. The api applies them before it starts; a worker started on an older schema fails its readiness check and is restarted by the platform.
Shutdown
On SIGTERM: stop the dispatcher after its current batch → close() every worker, which waits for active jobs → close Prisma and Redis → exit. Budget 25 s, inside a container stop grace period of 30 s. A job still running at the deadline is released and retried elsewhere (that is why processors are idempotent).
Stalled jobs
BullMQ's stalled-job check re-queues a job whose lock expired (process killed); maxStalledCount: 1, then it fails into the normal retry path.
Scaling
V1 runs one worker. More replicas are safe: BullMQ distributes jobs, the dispatcher is elected by a PostgreSQL advisory lock, job schedulers are idempotent, and scheduled jobs take a per-job Redis lock.
Ops UI
A queue dashboard (e.g. Bull Board) may be mounted on the internal port for super administrators through an SSH tunnel — never on a public route.
Services write OutboxEvent rows in the same transaction as the change (consistency); the dispatcher turns committed rows into live SSE messages and handler jobs. The outbox id is also the SSE event id, and reconnecting clients are replayed from the outbox table itself (ADR-008). Event ids come from a monotonic UUIDv7 generator, so ordering by id gives emission order, including several events of the same transaction.
Wake-up: LISTEN/NOTIFY with polling as the safety net. An AFTER INSERT … FOR EACH STATEMENT trigger on the outbox table calls pg_notify('outbox', ''). PostgreSQL delivers notifications only at commit, so a wake-up never precedes visibility. The dispatcher holds one dedicated pg connection with LISTEN outbox and also polls every second. Polling alone would add up to a second of latency to every live update; notify alone would miss events while the listening connection is down. Together: typically under 100 ms from commit to dispatch, and never stuck.
One active dispatcher. The dispatcher runs only while it holds pg_try_advisory_lock(<outbox-dispatcher key>); a second worker stays on standby and takes over when the lock is released.
Batch loop:
In a transaction, claim up to 100 events:
SELECT id, event_key, aggregate_type, aggregate_id, payload, attemptsFROM outbox_eventsWHERE dispatched_at IS NULL AND available_at <= now()ORDER BY idLIMIT 100FOR UPDATE SKIP LOCKED;
(The partial index outbox_events_pending on available_at WHERE dispatched_at IS NULL serves this query — OutboxEvent.)
For each event, in id order:
if it has an SSE projection, PUBLISH it on hoco:sse:admin and/or hoco:sse:user (with its recipients), carrying the outbox id as SSE id (SSE projection);
add one job per subscribed handler to events with job id evt:{eventId}:{handler}.
Set dispatched_at = now() on the batch and commit.
If the batch was full, loop at once; otherwise wait for a notification or the next poll.
Failure handling. If Redis refuses a publish or a job add, the transaction rolls back and the batch is retried: job ids deduplicate the jobs already added, and a message published twice costs the clients one refetch (they ignore an id they already applied). An event that keeps failing (10 attempts; attempts incremented, lastError stored, availableAt pushed back from 1 s to 5 minutes) is left undispatched with an alert, without blocking the events behind it.
Ordering per aggregate. Events are dispatched — and published to the SSE streams — in emission order. Handlers in the events queue run concurrently, and a retried handler can be overtaken by a later event of the same aggregate. Every handler is therefore written to be order-tolerant: it re-reads the aggregate's current state (the CRM lifecycle handler recomputes the status from facts) or compares aggregateVersion before projecting. No consumer depends on handler execution order.
Throughput and retention. A batch of 100 dispatches in a few milliseconds — far above HOCO's volume. Dispatched events are kept 30 days — the SSE resume window, debugging and replay (retention) — then purged by jobs.purgeDispatchedOutbox.
Handlers are declared next to the module that reacts:
Each job is a BullMQ job scheduler on the cron queue with a cron pattern and tz: "Africa/Algiers".
No overlap. A tick first takes the Redis lock cron:lock:{job} (TTL = the job's maximum duration); if the previous run still holds it, the tick is skipped and counted.
Catch-up. A successful run records cron:last:{job}. At worker boot, a daily job whose last success is older than its period runs once immediately. Jobs select work by state ("every planned tour dated today or earlier"), never by "since the last run", so a missed or repeated run converges.
Dead man's switch.cron_last_success_timestamp{job} is exported for every job; an alert fires when now − last success exceeds twice the period.
every 15 minutes · 10:00 and 16:00 · 09:30 · 10:30 · Sunday 09:00
see below
notifications, catalog, orders, analytics
V1 (change request 03)
Two jobs run outside cron. analytics.recomputeDirty is enqueued by event handlers and debounced (one run per dataset and day per minute). delegate.position is not a job at all: it is published directly by the api after a ping is stored.
Moves the day's tours PLANNED → TO_START so the start button appears (FLD-04, cahier §6.1 « disponible uniquement pour une tournée planifiée »); lists for the back-office the tours of the previous day never started or never closed.
Work
UPDATE tours WHERE status = 'PLANNED' AND date <= today(Africa/Algiers) → TO_START, one TourStatusChange (actor SYSTEM, trigger tours.activateToday) and one tour.to_start event each (silent sync hint to the delegate).
Idempotency
Status guard in the WHERE: a second run finds nothing. Tours created for today after 00:05 are created directly as TO_START. The server also accepts a start on a PLANNED tour dated today, so a late job never blocks a delegate (offline sync design).
Failure
Retried 3 times; the boot catch-up and the start tolerance cover a missed night.
Metrics, alerts
tours activated; last success older than 26 h → alert.
Assisted orders not confirmed by the reseller within orders.assistedConfirmationHours (72) expire (ORD-08); 24 h before expiry the reseller gets a reminder.
Work
Orders PENDING with assistedConfirmation = AWAITING_RESELLER and assistedExpiresAt <= now() → AssistedConfirmationEXPIRED, order CANCELLED (reason « Délai de confirmation expiré »), events assisted_order.expired and order.cancelled. Orders whose expiry falls within 24 h and not yet reminded → assisted_order.reminder_due, flag set.
Idempotency
Status guards; the reminder flag prevents a second reminder. A confirmation arriving after the deadline is refused by its own transaction (409 ASSISTED_ORDER_EXPIRED), so the 15-minute granularity never lets a late confirmation through.
Failure
Retried; the next tick catches up.
Metrics, alerts
expired and reminded counts per run; last success older than 1 h → alert.
No order is ever confirmed automatically (ORD-14), so a forgotten « commandes à confirmer » queue stops sales. Proposal: alert users holding orders.validate when the oldest SUBMITTED or PENDING order waits longer than orders.confirmationAlertMinutes (default 120) during working hours.
Work
Reads the queue age (the same query as admin.orders.toConfirm); emits at most one in-app notification per validator per hour.
Idempotency
Rate-limited by a Redis key per validator and hour.
Metrics
orders_to_confirm{status} and orders_to_confirm_oldest_seconds gauges, also on the dashboard.
Keeps promotion statuses in line with their dates: SCHEDULED → ACTIVE at startsAt, ACTIVE or PAUSED → ENDED at endsAt; emits promotion.started (targeted resellers notified) and promotion.ended.
Why every minute
Simpler and more robust than one delayed job per boundary that would need rescheduling on every edit. Prices never depend on the job: the cart engine evaluates startsAt/endsAt itself, so the job only drives displayed status and notifications. A cap reached at checkout ends the promotion inside the checkout transaction.
Idempotency
Status guards; one event per actual transition.
Metrics, alerts
transitions per run; last success older than 10 minutes → alert.
Publishes announcements whose scheduled time has come (NOT-07); emits announcement.published, whose handlers fan the push and inbox entries out in batches of 500 recipients.
Idempotency
Status guard (SCHEDULED → PUBLISHED); deliveries keyed by (announcement, user).
Rebuilds the daily aggregates behind dashboards and reports (ANA-01, ANA-03) for yesterday and the two days before — offline data may arrive up to 72 h late, so the last three days are recomputed every night and shown as provisional. Also runs the stock consistency check (levels equal to the sum of movements) and reports any drift.
Work
Delete-and-insert per (day, dataset) — sales, orders by status, returns and credit notes, field activity, CRM funnel, invitations — inside one transaction per day; enqueued on the analytics queue.
Idempotency
Recomputation from source tables; running twice gives the same rows. analytics.recomputeDirty uses the same code for the days marked dirty by events.
Failure
Retried 3 times from 5 minutes; dashboards keep the previous values.
Metrics, alerts
duration per dataset; drift count (any drift → alert to Anis); last success older than 26 h → alert.
Resellers ACTIVE (or ACCOUNT_CREATED without an order) with no confirmed order for crm.inactivityDays move to TO_FOLLOW_UP and get a follow-up task for their delegate (CRM-04, CRM-08).
Work
Per reseller: status change (automatic = true), CrmTask (source INACTIVITY), events reseller.status_changed and task.assigned; processed in pages of 200.
Idempotency
Only resellers whose status is still eligible are touched; one open inactivity task per reseller (unique on reseller + source + open).
Settings
crm.inactivityDays (default 60, range 15–365 — business rules).
Deletes uploads that never completed (pending for more than 24 h) and completed files never attached to a record after 7 days — object first, then row.
Session cleanup: deletes expired Session rows (Better Auth checks expiry at read but does not delete), expired Verification rows (OTP), and deactivates Device rows unseen for 180 days.
Implemented
Sessions expired more than 30 days ago and one-time codes expired more than 24 hours ago (BR-84, retention). Devices are not touched yet: this page says 180 days unseen → deactivated, the retention table says push tokens are deleted after 60 days without activity — to decide.
Anonymises the accounts closed by me.deleteAccount once their anonymiseAfter date has passed in Africa/Algiers (BR-85, SEC-6.3), up to 1 000 per run.
Work
User: name → « Compte supprimé », e-mail → deleted-{id}@anonymised.invalid, phone → a unique placeholder (resellers and delegates must keep one) or none (staff), avatar removed, credentials, sessions and devices deleted, anonymisedAt set. Reseller: owner name, WhatsApp and e-mail cleared, phone → placeholder, delivery-address recipients and phones replaced; store photos and documents deleted with their objects (BR-84), except a facade photo that is also a visit photo. Store name, legal name and tax identifiers stay with the orders and invoices kept by law (order address snapshots are untouched).
Export files older than exports.retentionDays (7): the file is deleted and the ExportJob moves SUCCEEDED → EXPIRED; the row stays for audit (lifecycles).
Backup verification ping (SEC-05, NFR-02): checks that the newest database dump in BACKUP_S3_BUCKET / BACKUP_S3_PREFIX is less than 26 h old and larger than the configured floor (the previous dump's size − 20 %), then pings HEARTBEAT_URL_BACKUP.
Failure
No ping → the external uptime monitor alerts (a dead worker is detected too); an explicit failure also alerts the team chat and e-mail.
Note
The dump itself is taken by Dokploy's scheduled database backup; restore drills are on deployment and operations.
One morning digest per assignee of follow-ups due today and overdue (task.due): push + inbox (« 3 relances aujourd'hui »), instead of one notification per task (CRM-08).
Idempotency
One task.due per (assignee, date) — event key deduplicated in the handler. Days without due tasks send nothing.
Settings
crm.followUpReminderTime (07:30) — changing it upserts the job scheduler.
Daily e-mail and inbox digest to stock managers of variants below their threshold or out of stock (STK-05). Real-time crossings already raise stock.low / stock.out in-app and on the back-office stream; the digest avoids an e-mail per crossing.
Proposal for ADR-016: tours still IN_PROGRESS at 21:00 move to SUSPENDED (reason AUTO_END_OF_DAY, actor SYSTEM) so location sampling never runs overnight when a delegate forgets to close. The delegate closes the tour the next time they are online (within field.reopenMaxDays).
Work
Status change, TourStatusChange, tour.suspended, silent sync hint so the app stops sampling at once.
Idempotency
Status guard. A suspension replayed from the device later is a no-op.
Metrics
tours auto-suspended (a high count means delegates do not close their day).
Notification retries beyond BullMQ's own: re-enqueues deliveries still QUEUED after 10 minutes (job lost on a Redis restart) and retryable failures with attempts left (e.g. provider down for longer than the job's backoff).
Idempotency
Job id ntf:{deliveryId} — never two live jobs for one delivery. A delivery is retried at most 3 times by the sweep, and OTP deliveries never (their codes have expired).
Metrics, alerts
re-enqueued deliveries; failure rate per channel above 10 % over 15 minutes → alert.
V2 (FIN-06, FIN-07): invoices UNPAID or PARTIALLY_PAID whose dueDate (Africa/Algiers) has passed move to OVERDUE; emits invoice.overdue. Payment terms and due dates arrive with detailed credit in V2; the job runs as a no-op in V1, where invoices carry no due date — unless the cadrage sets payment terms for V1.
V2.loyalty.expirePoints posts EXPIRE ledger entries for points older than loyalty.pointsValidityMonths (oldest first) and emits loyalty.points_expiring ahead of time; loyalty.recomputeLevels recomputes Bronze/Silver/Gold/VIP over the configured period and emits loyalty.level_changed. Both are keyed by (account, period) and recompute from the ledger.
V2.integrations.syncStock pulls stock from HOCO's stock system and posts the differences as ADJUSTMENT movements through the inventory service (STK-09); integrations.pollCarrierTracking polls carriers without webhooks and records delivery updates (DLV-05). Both are keyed by the external reference and skip unchanged values.
Added by HOCO's document 03 (smart notifications, SRV-40). Each job selects candidates by state, writes one OutboxEvent per (reseller, item) with a deterministic key, so a repeated run sends nothing twice, and leaves the caps, priority and quiet hours to the notification service.
Job
Schedule (Africa/Algiers)
Cron
Module
Selects
Release
smart.abandonedCarts
every 15 minutes
*/15 * * * *
notifications
Carts with lines, updatedAt older than smart.abandonedCartDelayHours, no reminder for this cart version, none in the last 24 h
V1
smart.newArrivalsDigest
daily 10:00 and 16:00
0 10,16 * * *
catalog
Products first published since the last run with announceAsNew
V1
smart.reorderReminders
daily 09:30
30 9 * * *
orders
Reseller × leaf sub-category with at least 3 orders in 180 days whose cadence + grace has elapsed
V1
smart.complementSuggestions
daily 10:30
30 10 * * *
catalog
Orders VALIDATED the previous day containing a product with ProductComplement rows
V1
smart.trendingWeekly
Sunday 09:00
0 9 * * 0
analytics
Top product by units on confirmed orders over 7 days, at least 5 buyers
V1
Event-driven (no cron): variant.restocked and variant.low_stock_reached come from stock movements in the inventory module, price.dropped comes from PriceChange, flash_sale.started from promotions.applySchedule, and commercial_push.sent and catalogue.published from support.publishScheduledAnnouncements or an immediate send.
cron_last_success_timestamp{job}, run duration, skipped ticks
last success older than twice the period
Worker liveness
/health on the internal port, heartbeat HEARTBEAT_URL_SCHEDULER pinged every 5 minutes by a tick
heartbeat missing for 15 minutes
Redis
memory, AOF status, connected clients
memory above 70 %
Every job log line carries jobName, jobId, attempt, queue and, for event handlers, eventKey and eventId, so a request id in the api can be followed through OutboxEvent.requestId to every job it caused.
Processors are unit-tested with the queue replaced by direct calls, and integration-tested against real PostgreSQL and Redis (the docker compose stack) for the outbox → dispatcher → handler path.
Idempotency: every processor test runs the job twice and asserts the second run changes nothing.
Time: scheduled jobs are tested with a frozen clock around the Africa/Algiers day boundary (22:59–01:01 UTC) and at month ends.
Failure: a test kills the worker mid-batch and asserts that every event is handled exactly once in effect, and that the dead-letter copy and alert fire after the last attempt.