Notifications Operational Workflows #

The primary workflows for the system can be divided into three (3) basic patterns which are repeated with minor variations across all operations.

Standard Sequence Diagram #

Find or read Operations #

All find operations are exposed through the API Management service only and all follow a similar pattern with the exception of specific Azure functions that may be executed.

1. Upon requests being received by the Gateway, they are immediately validated to ensure that the caller has a proper subscription that entitles them access.
2. Incoming request arguments are hashed along with the API method name and the redis cache is interrogated to determine if a copy of the request results have previously been cached within the expiration period (managed by configuration). Industry wide analysis of similar scenarios indicated that a single cached request in comparable scenarios can save approximately 20 additional database calls.
3. If a cached result is found, it is immediately returned and no other operations are performed.
4. If a cached item is not found, the read/fetch request is forwarded to the Azure function for the specific type of fetch (fetch by user, fetch by company, fetch by correlation, fetch a specific message by id).
5. If additional API requests arrive while the first request is being processed, additional instances of the appropriate Azure function are spun up automatically to process the additional requests (up to 2000 simultaneous functions can be done in this manner handling even very heavy loads).
6. The specific Azure function handling the read will validate the request and use the passed parameters (including limit and skip parameters for paging) to query the repository.
7. The Azure function interacts with the repository via parameterized stored procedures to return notification messages as defined in the request (also including paging data).
8. The DTO’s (Data Transfer Objects) returned from the database are converted into standard MessageDetail object(s) for return to the caller.
9. Results are passed through the API Gateway on the way back to the caller where the results are cached for future use.

Write Operations #

All write operations (create/add, update, and various delete methods) operate in a standard pattern as described below regardless of which method non-read method is executed.

1. Upon requests being received by the Gateway, they are immediately validated to ensure that the caller has a proper subscription that entitles them access.
2. The write request is forwarded to the appropriate Azure request handler (Azure function).
3. The request handler validates the request and, on success, publishes a brokered message on the NotifiationsChange topic (Azure ServiceBus).
4. The handler function immediately returns an acknowledgement, on success, to the caller via the API Gateway to free it up and minimize interaction time.
5. The Azure ServiceBus fans out brokered messages to the primary function handlers responsible for each task - based on a brokered message label and specific subscriptions for each one.
6. The brokered messages are validated and decomposed into DTO’s (Data Transfer Objects).
7. If any additional brokered messages arrive on the subscription beyond a specific thresh-hold (determined by configuration), additional function handlers can be automatically spun up to help drain the subscription queue.
8. The primary function handlers then interact with the repository via parameterized stored procedure call to complete the desired action defined within the API.
9. Brokered messages that either fail validation, cannot be processed, or are denied will be automatically sent to a dead letter queue for operational analysis. This dead-letter queue should be inspected regularly by operational support resources to ensure that errors are not building up. Dead-lettering could be configured to generate an alert to the support team to alert them to a growing number of errors.
10. If the repository is not available, brokered messages will automatically be retried at exponentially increasing intervals until either an absolute limit is reached or the messages expire (this provides the circuit breaker functionality and is completely configurable at the subscription level within Azure).
11. Upon completion of the request, the primary function handling the requests spins down and becomes free to accept another message.

Add Operation with a delivery provider specified #

In addition to the actions defined above (Write Operations), any add operation which specifies a delivery provider (email or sms) undergoes some additional steps before completing its workflow.

1. Add notifications with a delivery provider specified are inspected prior to exiting the function (step 11 above). If the message indicates either an SMS or Email provider, a copy of the message is then sent to a provider specific queue.
2. A ServiceBus Queue Event triggers a provider-specific send Azure function.
3. The Azure send function picks up the brokered message, validates that all required delivery values are present and valid.
4. The function transforms the brokered message into a highly specific set of parameters to send to the external service provider via https.
5. The external service provider will either acknowledge the request and provide a foreign key for message tracking or reject the request.
6. Failed requests will be re-queued and retried again until an absolute limit is reached or it is explicitly rejected by the external provider at which time it will be dead-lettered.
7. On successful submittal to the external provider, the message is updated in the repository with the returned foreign key along with the posted timestamp.

Provider Status Changes #

All defined external service providers being considered have the ability to call a URI with status messages when an individual message status changes - e.g. it was sent, delivered, opened/read, a link was clicked on, or it was deleted. It is necessary for audit reasons to not only ensure that messages/notifications are stored for review but to note when the user became aware of them. The status events provided via web-hooks is a positive way to handle this requirement.

1. When a message managed by an external service provider changes status, that provider will call a predefined web-hook (URI https endpoint).
2. The incoming message is validated quickly and on successful validation, it is placed on a status event queue for processing later.
3. The web-hook call is immediately acknowledged back to the provider to minimize impact, reduce costs, and maximize network utilization.
4. A ServiceBus Queue Event triggers a provider specific status event handler Azure function.
5. The event handler function validates the incoming status data (validating it comes from a trusted source and relates properly to a message in our system).
6. The event handler function creates a DTO (Data Transfer Object) to use in updating the repository specific to the event status type.
7. The event handler function calls the repository via parameterized stored procedures to force a status update using timestamps included in the provider status change message.
8. Failed requests will be re-queued and retried again until an absolute limit is reached or it is explicitly rejected by the repository at which time it will be dead-lettered.
9. Azure event handler function terminates or processes an additional request if one is available on the event queue.