Communication — System architecture

Clear boundary: logic (Platform/automation) decides what to send; dispatch ensures records exist; the send pipeline handles actual delivery. Automation handlers never call providers directly.

Overview

In one sentence: events and schedules from the left flow into NotificationScheduler; the worker pipeline (Scheduler → Generator → Distributor) renders and sends; webhook updates state back to OutgoingMessage.

Inbox Message: storage contract

UserMessage separates metadata/state from the body so lists stay fast and private content is not exposed:

Layer	Where it is stored	Main contents
Primary row	Azure Table UserMessage through CamelTableConnectionString	Identity, scope, source, preview, action, recipient state and ETag.
Private body	Azure Blob container message-content through CamelBlobConnectionString	HTML/text; the Table stores only MessageBodyBlobName.
Unread projection	Same UserMessage Table	Counters _InboxUnread and _InboxUnread_Account_{id} in the owner partition.
Retention projection	Same UserMessage Table	Monthly index _InboxUserMessageMaintenance_YYYYMM; it does not replace the primary row.
Realtime projection	Redis Stream aimy:{environment}:user-message-inbox:events	Invalidation metadata; not the Inbox data source.

Redis uses one Azure Managed Redis logical database (DB 0/default). Tux cache keys are isolated by environment with the tux:{environment}: prefix; the Inbox stream uses the separate key template above.

Real-world example: The Generator stores the body and then writes the UserMessage row. → The API returns only the preview in the list and loads the body after authorization in detail. → A Redis failure does not lose the row or body.

Realtime Inbox: SSE after persistence

Realtime is a delivery projection of UserMessage, not a place to store messages or authorize Inbox access.

• UserMessage is persisted first; Redis publication is best-effort and does not fail persistence.
• Each API instance reads the Redis Stream and fans out to its local SSE connections; each connection has a bounded buffer.
• Last-Event-ID supports replay within the retention window; outside the window, the client is asked to sync through REST.
• The event is invalidation-only: Tuke refreshes unread state and the first page when Inbox is open; detail/body still use REST.

SSE and SignalR in the Inbox context

Criterion	SSE in use	SignalR	Inbox assessment
Model	Server → client	Two-way, with hub methods	SSE is sufficient for one-way invalidation
Data	Small event, REST fetches real data	Events can be sent through a hub	No need to push the full message through a socket
Auth/scope	Bearer header, user from JWT sub	Hub auth, negotiate and user mapping	SSE introduces fewer new contracts
Reconnect	Last-Event-ID + Redis replay + REST sync	Auto reconnect, but REST sync is still needed	Both still need REST recovery
Operations	HTTP stream, heartbeat, proxy buffering	Hub route, client library, WebSocket/fallback, Azure SignalR	SSE is lighter for the current use case
Expansion	Not suited to chat/presence/typing	Suited to two-way communication	Choose SignalR only if Inbox changes character

Tux has SignalR infrastructure for other flows, but Inbox does not reuse that hub as a ready-made contract. Moving Inbox to SignalR would require a hub route, user mapping, client lifecycle and new reconnect/recovery testing; SignalR also does not replace Redis replay or REST revalidation.

Decision: keep SSE for Inbox. Consider SignalR only when two-way chat, typing/presence, realtime read receipts or one shared hub for multiple realtime features is required.

Real-world example: A new reminder creates a UserMessage. → Redis fans out inbox.changed to Tuke. → Tuke refreshes unread state and the first page through REST; no two-way SignalR channel is needed.

Choose send path by producer

Producer	Path
Manual / immediate entity message	NotificationJob → NotificationScheduler
Direct reminder	NotificationJob → NotificationScheduler → OutgoingMessage
UI-only reminder	NotificationJob → NotificationScheduler → UserMessage
Report subscription	AutomationScheduler → MessagingJob+Blob and/or NotificationJob → NotificationScheduler → OutgoingMessage
Messaging campaign	MessagingJob+Blob → NotificationScheduler → OutgoingMessage
Automation skipped/deactivated	No send; result is in AutomationScheduler.DispatchResultJson + support screens

Ensure contracts (IDeliveryDispatcher)

Every artifact is created through an Ensure* contract:

IDeliveryDispatcher
  EnsureAutomationJobAsync       EnsureNotificationJobAsync
  EnsureMessagingJobAsync        EnsureUserMessageAsync
  EnsureOutgoingMessageAsync     EnsureNotificationSchedulerAsync

Each method: find by deterministic key → check compatibility (InputHash/ContentHash + immutable fields) → create if missing → return stable coordinates. Compatible → idempotent success; immutable mismatch → conflict (stop sending, report support).

Key policy (short version)

Keys decide "where history opens" and "how duplicates are prevented".

NotificationJob

Manual / transactional:     PartitionKey = PrimaryEntity.Guid
                            RowKey       = {CreatedReverseTicks}_{NotificationJobGuid}
Direct automation reminder: PartitionKey = PrimaryEntity.Guid
                            RowKey       = AutomationRowKey
Report subscription:        PartitionKey = ReportSubscription.Guid
                            RowKey       = AutomationRowKey
Messaging campaign:         PartitionKey = {BusinessUnit.Guid}_Campaign
                            RowKey       = Campaign.Guid

PartitionKey is the entity where the user expects to open history — not the internal id of automation. RowKey = AutomationRowKey is the idempotency bridge from automation to notification. Event-derived notifications use NotificationEventKey (see Platform).

UserMessage

Canonical owner partitions:
  Account:        PartitionKey = Account_{AccountId}
  Employee:       PartitionKey = Employee_{EmployeeId}
  EnterpriseUser: PartitionKey = EnterpriseUser_{EnterpriseUserId}

Legacy read-only:
  Customer:       PartitionKey = Customer_{UserId}

New row keys are producer-deterministic; there is no single format for every producer. Current forms include inbox_{sourcePartitionKey}_{deterministicHash} and {reverseTicks}-{deterministicRowKey}; deployed legacy Incident rows also use {reverseTicks}. Customer_{UserId} is used only for compatibility reads/deduplication of old data, not as the target partition for new Campaign deliveries. CompatibilityHash protects immutable fields; a compatible retry does not change StatusId, ReadOn, ArchivedOn, SavedOn or Reply*.

Queries always start with portal identity → authorized owner scopes → a Table query per PartitionKey. Lists include only Unread/Read, with server-side category/BU/site/account filters; the default page size is 20 and the maximum is 100. Each scope's continuation token is wrapped in the opaque NextCursor; when merging scopes, only the selected scope advances. Detail is a point read by PartitionKey + RowKey; the body is read from Blob after authorization. State mutations use conditional ETag Replace.

Unread reads the counter projection and scans the bounded partition only when the counter does not exist. Retention deletes unsaved rows older than three months by CreatedOn; saved rows are exempt, and deletion is also conditional on ETag.

OutgoingMessage + send claim

Notification-generated: PartitionKey = NotificationJob.RowKey
Report/campaign:        PartitionKey = MessagingJob.Guid
RowKey = {RecipientKeyHash}_{DeliveryMethod}   (+ _{AttemptNo} only when intentionally a new attempt)

RecipientKeyHash is derived from stable recipient identity + normalized destination, not display text. One row / recipient / channel; Generator retry reuses the row.

Claim before provider call:

Created/RetryReady -> Sending  WHERE PartitionKey, RowKey, Status in (Created, RetryReady)

Rows already Sending | SentToProvider | Delivered | Opened | FailedFinal → duplicate queue message exits without calling provider. Provider success → SentToProvider + provider message id; webhook advances state. Only audited support-retry may reset to RetryReady.

NotificationScheduler dispatch guard

UNIQUE(ArtifactTypeId, ArtifactPartitionKey, ArtifactRowKey, NotificationEventTypeId)

EnsureNotificationSchedulerAsync returns the existing row when the guard matches. NotificationEventTypeId is the domain notification discriminator — mapped explicitly at the notification boundary, not assumed equal to the EntityEvent EventTypeId.

Partial failure handling

Found during retry	Behavior
NotificationJob exists, scheduler missing	Recreate scheduler through dispatch guard
UserMessage missing recipient row	Insert missing row with deterministic key
OutgoingMessage already exists	Reuse; provider call is decided by send claim
MessagingJob exists, blob incompatible	Stop, report conflict
Any key hash mismatch	Stop, report conflict

SMS usage ledger (billing)

• Each SMS accepted by a provider writes exactly one SmsUsage in SQL, append-only — aggregatable by BusinessUnit × time range (monthly billing) without scanning OutgoingMessage.
• The record carries BusinessUnitId, NotificationEventTypeId (assignable to Messaging Campaign / Incident / Onsite App SMS), SentOn (UTC at provider accept), MessageCount (=1), SmsCount.
• SmsCount = the number of segments the provider bills: provider-reported count → record that exact value; none → derive by encoding rules (GSM-7: 160/153; UCS-2: 70/67); always ≥ 1.
• Only recorded at provider accept: message not dispatched, dispatch rejected/thrown, short-circuit for missing mobile, simulated send not reaching the provider → not recorded, not billed.
• Idempotent per outgoing message: each record carries the identity of the outgoing message it bills; lease recovery / redelivery / duplicate queue item does not record a second time.
• Ledger write is a side effect, not a precondition: a failed persist does not change the delivery outcome, only logs an error with the outgoing message identity for reconciliation.

Real-world example: Twilio accepts a 200-character SMS for BU ABC (event IncidentUpdate). → The ledger records 1 record MessageCount=1, SmsCount=2 (UCS-2). → Monthly billing sums SmsCount by BusinessUnitId + SentOn; lease recovery of the same message does not add a second record.

Campaign: cancel, liveness, payload

• Cancel depends on scheduler dispatch state and reports the outcome honestly:
  • Before the scheduler row is claimed → deactivate schedule, no artifacts created.
  • During generation (row claimed) → stop creating remaining artifacts at the checkpoint; artifacts already on the provider delivery path stay; report partial cancel.
  • After generation completes → do not reset authoring state to draft; report conflict.
  • HTTP 200 = full cancel; HTTP 202 = cancel in progress; X-Campaign-Cancel-Outcome = machine-readable outcome code (old numeric response body kept).
  • A claimed row that is generating and then edited/rescheduled → reject, the running input stays.
• Liveness: generation refreshes the dispatch row's liveness signal at each checkpoint — long runs are not treated as stuck; a worker dying mid-way → the row becomes eligible for recovery after timeout.
• Payload API authenticates + authorizes by current business unit: resolve the campaign within the BU before reading payload storage; cannot resolve → not-found, no payload read. Topi uses this endpoint instead of fetching MessageJobBlob directly.
• Serial generation (P1): host config keeps concurrency = 1; a single Generator instance is a prerequisite for operation; raising above 1 breaks the invariant → dispatch-claim required first.

Real-world example: An operator cancels while generation is running, 300 emails already in the send path. → Cancel. → Generation stops at the next checkpoint; the 300 emails stay; response 202 — the operator knows what was sent, not told "stopped" while part already went out.

Next: concrete files & workers in Codebase flow.

Source-keyed delivery and Notification Settings

• SourceKey is resolved in the Generator after the source entity is in scope; unresolved sources use the current fallback. NotificationJob remains a work batch, not the source-history grain.
• Source-keyed trace is wired only for supported flows; there is no new SQL reverse index and no backfill or rewrite of historical partitions.
• Notification Settings group by recipient type and batch-toggle Site/BU contact copy. Delivery deduplicates addresses, and non-site-level notifications do not receive Site copy even if an old stored flag is true.

Real-world example: Site and BU contacts share ops@example.com. → Only one message is created. → Deduplicate after applying both copy flags and before writing OutgoingMessage.