Webhook delivery

Totara sends an HTTP POST to a webhooks endpoint and relies on the HTTP response code to decide whether a request was successful, bad, or requires resending.

Successful requests

Totara considers a webhook successful when the HTTP POST to the webhook responds with an HTTP code in the 200 range. Totara will update the time sent value of successful webhooks and they will cease to be sent again.

Bad requests

Totara webhooks considers a bad request when the webhook’s endpoint responds with an HTTP code in the 400 range. In this case, Totara will move the webhook payload to the dead letter queue (See Webhook dead lettering below). Once an item is moved to the dead letter queue, Totara will cease to try resending a request.

Unsuccessful requests

Any response that Totara webhooks receives with an HTTP code that is outside of 200 and 400, Totara will try and resend. For more details on retry attempts see the Webhook retries section below.

Webhook retries

Totara webhooks come with retries built in. We’ll try and deliver a webhook payload to the webhooks endpoint a maximum of 10 times. Each attempt, we’ll increase the time that the webhook sits on the queue before we attempt to resend (Backoff). The following table outlines the attempt and the associated wait time before we attempt to resend:

If a webhook continues to fail to be marked as successful (the webhook’s endpoint doesn’t respond with an HTTP response code in the 200 range), webhooks will move the payload to the dead letter queue. See Webhook dead lettering below for more information on this.

Webhook dead lettering

Totara has a standard queue that it reads through to send webhook payloads out. When a webhook fails to respond with a successful response after 10 attempts or responds with an HTTP response in the 400 range, the payload will be moved out into a dead letter queue.

The dead letter queue holds details on the webhook payload and allows an admin to investigate why the webhook failed to send. This could be something as simple as the webhooks endpoint is pointing at an incorrect URL.

Should an admin wish to retry a webhook payload that has been moved to the dead letter queue, they can via a CLI script: totara/webhook/cli/webhook_requeue_dead_letter.php.

This script accepts an ID flag, which corresponds to the item's ID in the dead letter queue table totara_webhook_dlq_item.id.

When this script is called, the item will be pushed back onto the standard webhook queue to be resent during the next cron run. However, the attempt will remain the same, so if an item has already used up 10 attempts and fails again, the item will end up right back on the dead letter queue. There are no restrictions on how many times a dead-lettered payload can be requeued.

Webhook payload lifetime

All requests to webhooks are held for 14 days after the webhook receives a successful response from the webhook’s endpoint. After that time, a scheduled job will run to purge the logs. Any items on the dead letter queue are held for 28 days after the payload was added to the dead letter queue. After 28 days, a separate scheduled job will run to purge the dead letter queue. These jobs can be configured with config settings and adjusted in the same way that any scheduled job can be.

Note: Event data may contain Personally identifiable information (PII) on users within the system.

The cleanup jobs above will handle removing this data after 14 days for the logs and 28 days for the dead letter queue.

If shorter timeframes are required to meet GDPR data deletion obligations these timeframes can be adjusted via the following config settings:

$CFG->forced_plugin_settings['totara_webhook'] = [

  'queue_purge_threshold' => (30  24  60 * 60) // given in number of seconds - default 30 days

  'dead_letter_purge_threshold' => (60  24  60 * 60) // given in number of seconds - default 60 days

]

Payload signing

Webhooks come with payload signing as a way to verify that an incoming request has come directly from Totara, and that the data hasn’t been tampered with. When you create a webhook, a secret will be automatically generated.

The secret can be toggled using the ‘Show/Hide’ button on the control. The secret, and the body of the request, is used when a webhook payload is sent to a webhooks endpoint. A Hash-based message authentication code (HMAC) using the sha256 algorithm will be created and added in the request header X-Totara-Signature.

Verifying the signature

You can compare a signature in PHP using the following code snippet:

$calculated_signature = hash_hmac(
    'sha256',
    $body,
    $secret
);
if (hash_equals($signature_to_verify, $calculated_signature)) {
    $message = "Signature verified";
}

CLI script to verify signature

Webhooks also comes with a CLI script for verifying the X-Totara-Signature. The script can be found at: server/totara/webhook/cli/webhook_verify_signature.php

To run the script, you simply provide the following flags:

E.g.

$ sudo -u www-data /usr/bin/php totara/webhook/cli/webhook_verify_signature.php --signatu

Note: Clicking ‘Rotate’ will replace the webhook's existing secret; any requests that are currently queued to send will use this new secret.

Performance considerations

Webhooks react to events happening in Totara. For this reason, there needs to be a balance around the number of webhooks that a site has versus the number of events a webhook is subscribed to. We recommend sticking to having either fewer webhooks subscribed to multiple events or multiple webhooks subscribed to fewer events.

Webhooks, by default, will set themselves to be scheduled so that they are queued and processed with the scheduled jobs. If you have a lot of webhooks set up with immediate send, they will interrupt the user when the event is triggered and could cause the page to feel sluggish. We recommend sticking with the scheduled setting.