Chapter 8: Webhooks

Overview

Webhooks provide real-time notifications about events in your Polysystems account. Instead of polling the API for status updates, webhooks push notifications to your server when events occur, enabling responsive, event-driven applications.

What are Webhooks?

Webhooks are HTTP callbacks that deliver event notifications to your specified URL. When an event occurs (like a payment confirmation or limit exceeded), the system sends an HTTP POST request to your webhook endpoint with event details.

Benefits of Webhooks

Real-Time: Instant notifications when events occur
Efficient: No need to poll the API repeatedly
Reliable: Automatic retry mechanism for failed deliveries
Secure: HMAC signature verification
Flexible: Subscribe to specific event types

Webhook Architecture

Event Occurs
    ↓
System Creates Event Payload
    ↓
System Signs Payload with HMAC
    ↓
System Sends POST to Webhook URL
    ↓
Your Server Receives Request
    ↓
Your Server Verifies Signature
    ↓
Your Server Processes Event
    ↓
Your Server Returns 2xx Status
    ↓
System Logs Successful Delivery

(If delivery fails, system retries with exponential backoff)

Webhook Events

Payment Events

Event Type	Description	Trigger
`payment.created`	New payment initiated	User creates x402 payment
`payment.confirming`	Transaction detected on blockchain	First blockchain confirmation
`payment.confirmed`	Payment fully confirmed	Required confirmations met
`payment.failed`	Payment failed	Transaction failed or invalid
`payment.expired`	Payment window expired	Expires_at timestamp passed

Balance Events

Event Type	Description	Trigger
`balance.updated`	Account balance changed	Credits added or deducted
`balance.low`	Balance below threshold	Balance falls below $10
`balance.depleted`	Balance reached zero	Balance = $0.00

Usage Events

Event Type	Description	Trigger
`usage.limit_exceeded`	Spending limit reached	Token hits daily/monthly limit
`usage.threshold`	Usage threshold crossed	Spending reaches 80% of limit
`usage.spike`	Unusual usage pattern	Anomaly detected

Security Events

Event Type	Description	Trigger
`anomaly.detected`	Suspicious activity detected	Anomaly detection triggered
`rate_limit.exceeded`	Rate limit exceeded	Too many requests
`key.revoked`	Access key revoked	Token revoked by user

Registering a Webhook

Create Webhook Endpoint

curl -X POST https://api.polysystems.ai/api/webhooks \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://yourdomain.com/webhooks/polysystems",
    "events": [
      "payment.confirmed",
      "balance.updated",
      "usage.limit_exceeded"
    ],
    "description": "Production webhook endpoint"
  }'

Request Parameters:

url (required): Your HTTPS webhook URL
events (required): Array of event types to subscribe to
description (optional): Human-readable description
secret (optional): Custom signing secret (auto-generated if omitted)

Response:

{
  "id": "webhook-123e4567-e89b-12d3-a456-426614174000",
  "url": "https://yourdomain.com/webhooks/polysystems",
  "secret": "whsec_a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6",
  "events": [
    "payment.confirmed",
    "balance.updated",
    "usage.limit_exceeded"
  ],
  "is_active": true,
  "description": "Production webhook endpoint",
  "created_at": "2024-01-15T12:00:00Z"
}

⚠️ Important: Save the secret securely! It’s used to verify webhook signatures.

URL Requirements

Must use HTTPS (HTTP not allowed in production)
Must be publicly accessible
Must return 2xx status code within 5 seconds
Should handle POST requests
Should be idempotent (handle duplicate deliveries)

Webhook Payload Structure

Standard Payload Format

{
  "event_id": "evt-123e4567-e89b-12d3-a456-426614174000",
  "event_type": "payment.confirmed",
  "timestamp": "2024-01-15T15:30:00Z",
  "data": {
    // Event-specific data
  },
  "signature": "sha256=a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6..."
}

Payload Fields

event_id: Unique identifier for this event
event_type: Type of event (see Event Types above)
timestamp: When the event occurred (ISO 8601)
data: Event-specific payload
signature: HMAC-SHA256 signature for verification

Event Payload Examples

Payment Confirmed

{
  "event_id": "evt-abc123",
  "event_type": "payment.confirmed",
  "timestamp": "2024-01-15T15:30:00Z",
  "data": {
    "payment_id": "pay-123",
    "user_id": "user-456",
    "amount_usd": 50.00,
    "currency": "ETH",
    "amount_crypto": 0.0275,
    "tx_hash": "0x1234567890abcdef...",
    "confirmations": 12,
    "previous_balance": 47.32,
    "new_balance": 97.32
  },
  "signature": "sha256=..."
}

Balance Updated

{
  "event_id": "evt-def456",
  "event_type": "balance.updated",
  "timestamp": "2024-01-15T16:45:00Z",
  "data": {
    "user_id": "user-456",
    "previous_balance": 97.32,
    "new_balance": 97.30,
    "change": -0.02,
    "reason": "api_request",
    "reference_id": "tx-789",
    "metadata": {
      "route": "/api/hub/agents/chat",
      "method": "POST"
    }
  },
  "signature": "sha256=..."
}

Usage Limit Exceeded

{
  "event_id": "evt-ghi789",
  "event_type": "usage.limit_exceeded",
  "timestamp": "2024-01-15T18:20:00Z",
  "data": {
    "user_id": "user-456",
    "access_key_id": "key-789",
    "limit_type": "daily",
    "limit": 10.00,
    "spent": 10.05,
    "exceeded_by": 0.05,
    "request_route": "/api/hub/agents/chat",
    "request_cost": 0.05
  },
  "signature": "sha256=..."
}

Verifying Webhook Signatures

Why Verify Signatures?

Signature verification ensures that:

Webhooks are actually from Polysystems
Payloads haven’t been tampered with
Your endpoint isn’t being spoofed

Signature Algorithm

signature = HMAC-SHA256(webhook_secret, payload_body)

Verification Examples

Python

import hmac
import hashlib
import json
 
def verify_webhook_signature(payload_body, signature_header, webhook_secret):
    """Verify webhook signature"""
    # Compute expected signature
    expected_signature = hmac.new(
        webhook_secret.encode('utf-8'),
        payload_body.encode('utf-8'),
        hashlib.sha256
    ).hexdigest()
    
    # Extract signature from header
    signature = signature_header.replace('sha256=', '')
    
    # Compare using constant-time comparison
    return hmac.compare_digest(signature, expected_signature)
 
# Flask example
from flask import Flask, request, jsonify
 
app = Flask(__name__)
WEBHOOK_SECRET = "whsec_your_secret_here"
 
@app.route('/webhooks/polysystems', methods=['POST'])
def handle_webhook():
    # Get payload and signature
    payload_body = request.get_data(as_text=True)
    signature = request.headers.get('X-Polysystems-Signature')
    
    # Verify signature
    if not verify_webhook_signature(payload_body, signature, WEBHOOK_SECRET):
        return jsonify({'error': 'Invalid signature'}), 401
    
    # Parse payload
    event = json.loads(payload_body)
    
    # Process event
    handle_event(event)
    
    return jsonify({'success': True}), 200
 
def handle_event(event):
    event_type = event['event_type']
    
    if event_type == 'payment.confirmed':
        handle_payment_confirmed(event['data'])
    elif event_type == 'balance.updated':
        handle_balance_updated(event['data'])
    elif event_type == 'usage.limit_exceeded':
        handle_limit_exceeded(event['data'])

Node.js

const crypto = require('crypto');
const express = require('express');
 
const app = express();
const WEBHOOK_SECRET = 'whsec_your_secret_here';
 
// Use raw body for signature verification
app.use('/webhooks/polysystems', express.raw({type: 'application/json'}));
 
function verifyWebhookSignature(payload, signature, secret) {
  const expectedSignature = crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');
  
  const sig = signature.replace('sha256=', '');
  
  return crypto.timingSafeEqual(
    Buffer.from(sig),
    Buffer.from(expectedSignature)
  );
}
 
app.post('/webhooks/polysystems', (req, res) => {
  const signature = req.headers['x-polysystems-signature'];
  const payload = req.body.toString();
  
  // Verify signature
  if (!verifyWebhookSignature(payload, signature, WEBHOOK_SECRET)) {
    return res.status(401).json({error: 'Invalid signature'});
  }
  
  // Parse event
  const event = JSON.parse(payload);
  
  // Process event
  handleEvent(event);
  
  res.status(200).json({success: true});
});
 
function handleEvent(event) {
  switch (event.event_type) {
    case 'payment.confirmed':
      handlePaymentConfirmed(event.data);
      break;
    case 'balance.updated':
      handleBalanceUpdated(event.data);
      break;
    case 'usage.limit_exceeded':
      handleLimitExceeded(event.data);
      break;
  }
}

Go

package main
 
import (
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "io/ioutil"
    "net/http"
    "strings"
)
 
const webhookSecret = "whsec_your_secret_here"
 
func verifyWebhookSignature(payload []byte, signature, secret string) bool {
    h := hmac.New(sha256.New, []byte(secret))
    h.Write(payload)
    expectedSignature := hex.EncodeToString(h.Sum(nil))
    
    sig := strings.TrimPrefix(signature, "sha256=")
    
    return hmac.Equal([]byte(sig), []byte(expectedSignature))
}
 
func webhookHandler(w http.ResponseWriter, r *http.Request) {
    // Read payload
    payload, err := ioutil.ReadAll(r.Body)
    if err != nil {
        http.Error(w, "Error reading payload", http.StatusBadRequest)
        return
    }
    
    // Get signature
    signature := r.Header.Get("X-Polysystems-Signature")
    
    // Verify signature
    if !verifyWebhookSignature(payload, signature, webhookSecret) {
        http.Error(w, "Invalid signature", http.StatusUnauthorized)
        return
    }
    
    // Parse event
    var event map[string]interface{}
    if err := json.Unmarshal(payload, &event); err != nil {
        http.Error(w, "Invalid JSON", http.StatusBadRequest)
        return
    }
    
    // Process event
    handleEvent(event)
    
    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(map[string]bool{"success": true})
}
 
func main() {
    http.HandleFunc("/webhooks/polysystems", webhookHandler)
    http.ListenAndServe(":8080", nil)
}

Managing Webhooks

List All Webhooks

curl -X GET https://api.polysystems.ai/api/webhooks \
  -H "Authorization: Bearer YOUR_JWT_TOKEN"

Response:

{
  "webhooks": [
    {
      "id": "webhook-123",
      "url": "https://yourdomain.com/webhooks/polysystems",
      "events": ["payment.confirmed", "balance.updated"],
      "is_active": true,
      "created_at": "2024-01-15T12:00:00Z",
      "last_delivered_at": "2024-01-15T18:30:00Z"
    }
  ]
}

Update Webhook

curl -X PUT https://api.polysystems.ai/api/webhooks/{webhook_id} \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "events": [
      "payment.confirmed",
      "balance.updated",
      "anomaly.detected"
    ],
    "is_active": true
  }'

Delete Webhook

curl -X DELETE https://api.polysystems.ai/api/webhooks/{webhook_id} \
  -H "Authorization: Bearer YOUR_JWT_TOKEN"

Delivery and Retry Logic

Delivery Behavior

Initial Attempt: System sends webhook immediately when event occurs
Timeout: Your endpoint must respond within 5 seconds
Success: 2xx status code = successful delivery
Failure: Non-2xx status or timeout = failed delivery

Retry Schedule

If delivery fails, the system retries with exponential backoff:

Attempt	Delay	Total Time
1	Immediate	0s
2	5 seconds	5s
3	25 seconds	30s
4	2 minutes	2.5m
5	10 minutes	12.5m
6	1 hour	1h 12.5m
7	6 hours	7h 12.5m
8	24 hours	31h 12.5m

After 8 failed attempts, delivery is abandoned.

Checking Delivery Status

curl -X GET https://api.polysystems.ai/api/webhooks/{webhook_id}/deliveries \
  -H "Authorization: Bearer YOUR_JWT_TOKEN"

Response:

{
  "deliveries": [
    {
      "id": "delivery-123",
      "event_id": "evt-abc",
      "event_type": "payment.confirmed",
      "url": "https://yourdomain.com/webhooks/polysystems",
      "status": "success",
      "response_status": 200,
      "response_body": "{\"success\":true}",
      "delivered_at": "2024-01-15T15:30:01Z",
      "retry_count": 0
    },
    {
      "id": "delivery-456",
      "event_id": "evt-def",
      "event_type": "balance.updated",
      "url": "https://yourdomain.com/webhooks/polysystems",
      "status": "failed",
      "response_status": 500,
      "error_message": "Connection timeout",
      "failed_at": "2024-01-15T16:00:00Z",
      "retry_count": 3
    }
  ]
}

Testing Webhooks

Local Testing with ngrok

# 1. Start your local server
python app.py  # Running on localhost:5000
 
# 2. Start ngrok tunnel
ngrok http 5000
 
# 3. Register webhook with ngrok URL
curl -X POST https://api.polysystems.ai/api/webhooks \
  -H "Authorization: Bearer $JWT_TOKEN" \
  -d '{
    "url": "https://abc123.ngrok.io/webhooks/polysystems",
    "events": ["payment.confirmed"]
  }'

Trigger Test Event

curl -X POST https://api.polysystems.ai/api/webhooks/{webhook_id}/test \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "event_type": "payment.confirmed"
  }'

This sends a test event to your webhook endpoint.

Manual Retry

Manually retry a failed delivery:

curl -X POST https://api.polysystems.ai/api/webhooks/deliveries/{delivery_id}/retry \
  -H "Authorization: Bearer YOUR_JWT_TOKEN"

Best Practices

1. Return 2xx Quickly

# ✅ Good: Return immediately, process async
@app.route('/webhooks/polysystems', methods=['POST'])
def handle_webhook():
    # Verify signature
    if not verify_signature(request):
        return jsonify({'error': 'Invalid signature'}), 401
    
    # Queue for background processing
    queue.enqueue(process_webhook, request.get_data())
    
    # Return immediately
    return jsonify({'success': True}), 200
 
# ❌ Bad: Long processing before returning
@app.route('/webhooks/polysystems', methods=['POST'])
def handle_webhook():
    event = request.get_json()
    
    # This could timeout!
    result = expensive_database_operation(event)
    send_email_notification(event)
    update_external_system(event)
    
    return jsonify({'success': True}), 200

2. Handle Idempotency

def process_webhook(event):
    event_id = event['event_id']
    
    # Check if already processed
    if redis.exists(f'processed:{event_id}'):
        print(f"Event {event_id} already processed, skipping")
        return
    
    # Process event
    handle_event(event)
    
    # Mark as processed (expire after 7 days)
    redis.setex(f'processed:{event_id}', 604800, '1')

3. Log Everything

import logging
 
logger = logging.getLogger(__name__)
 
@app.route('/webhooks/polysystems', methods=['POST'])
def handle_webhook():
    event = request.get_json()
    
    logger.info(f"Received webhook: {event['event_type']}", extra={
        'event_id': event['event_id'],
        'event_type': event['event_type'],
        'timestamp': event['timestamp']
    })
    
    try:
        process_webhook(event)
        logger.info(f"Successfully processed webhook: {event['event_id']}")
    except Exception as e:
        logger.error(f"Failed to process webhook: {event['event_id']}", exc_info=True)
    
    return jsonify({'success': True}), 200

4. Monitor Webhook Health

from datetime import datetime, timedelta
 
def check_webhook_health():
    """Alert if webhooks haven't been received"""
    last_webhook = get_last_webhook_time()
    
    if datetime.utcnow() - last_webhook > timedelta(hours=24):
        send_alert("No webhooks received in 24 hours")

5. Use Dead Letter Queue

MAX_RETRIES = 3
 
def process_webhook_with_retry(event, retry_count=0):
    try:
        process_webhook(event)
    except Exception as e:
        if retry_count < MAX_RETRIES:
            # Retry with exponential backoff
            delay = 2 ** retry_count
            queue.enqueue(
                process_webhook_with_retry,
                event,
                retry_count + 1,
                delay=delay
            )
        else:
            # Move to dead letter queue
            dead_letter_queue.enqueue(event)
            send_alert(f"Webhook processing failed after {MAX_RETRIES} retries")

Troubleshooting

Webhooks Not Being Received

Check:

Webhook URL is publicly accessible
URL uses HTTPS
Firewall allows incoming connections
Server is running and responding
Webhook is active (is_active: true)

Signature Verification Failing

Common Issues:

Using wrong secret
Modifying payload before verification
Character encoding issues
Using JSON-parsed body instead of raw body

Solution:

# ✅ Correct: Use raw body
payload_body = request.get_data(as_text=True)
verify_signature(payload_body, signature, secret)
 
# ❌ Wrong: Using parsed JSON
payload_json = request.get_json()
verify_signature(json.dumps(payload_json), signature, secret)

Delivery Failures

Check delivery logs:

curl -X GET https://api.polysystems.ai/api/webhooks/{webhook_id}/deliveries \
  -H "Authorization: Bearer $JWT_TOKEN"

Common causes:

Server timeout (>5 seconds)
Server returning non-2xx status
Network connectivity issues
SSL certificate problems

Summary

In this chapter, you learned:

✅ What webhooks are and why they’re useful
✅ Available event types and when they trigger
✅ How to register and configure webhooks
✅ Webhook payload structure and examples
✅ How to verify webhook signatures securely
✅ Managing webhooks (list, update, delete)
✅ Delivery and retry behavior
✅ Testing webhooks locally
✅ Best practices for webhook handling
✅ Troubleshooting common issues

Next Steps

Chapter 9: Rate Limiting - Understand rate limits
Chapter 10: Error Handling - Handle errors gracefully
Chapter 11: Code Examples - See complete integration examples

Routing Rate Limiting