> For the complete documentation index, see [llms.txt](https://docs.astralbeam.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.astralbeam.io/operators/monitoring.md).

# Monitoring

This guide covers setting up Prometheus and Grafana for bridge monitoring.

## Overview

```mermaid
flowchart LR
    subgraph Relayers
        R1[Relayer 1 :9080]
        R2[Relayer 2 :9080]
        R3[Relayer 3 :9080]
    end

    subgraph Monitoring
        Prometheus[Prometheus :9090]
        Grafana[Grafana :3000]
        Alertmanager[Alertmanager :9093]
    end

    R1 --> Prometheus
    R2 --> Prometheus
    R3 --> Prometheus
    Prometheus --> Grafana
    Prometheus --> Alertmanager
```

## Metrics Exposed

Each relayer exposes Prometheus metrics on port 9080.

### Bridge Metrics

| Metric                                | Type    | Description                      |
| ------------------------------------- | ------- | -------------------------------- |
| `bridge_events_detected_total`        | Counter | Events detected by type/chain    |
| `bridge_attestations_submitted_total` | Counter | Attestations submitted           |
| `bridge_transactions_total`           | Counter | Transactions by status           |
| `bridge_signatures_collected`         | Gauge   | Signatures per attestation       |
| `bridge_pending_transactions`         | Gauge   | Currently pending transactions   |
| `bridge_latest_block_processed`       | Gauge   | Latest processed block per chain |

### Independent Verification Metrics

| Metric                                               | Type    | Description                                                                                                                                                                                                             |
| ---------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bridge_casper2evm_verification_stalled_total`       | Counter | Events entering the parked slow-poll state — no public-class Casper source has confirmed the deploy within the fast retry window. Sustained growth means public node connectivity trouble (or a fabricated event feed). |
| `bridge_casper2evm_verification_stall_timeout_total` | Counter | Parked events dead-lettered after the stall timeout (\~6h) with reachable public sources never confirming the deploy — the event was almost certainly not real. Investigate immediately.                                |
| `bridge_casper2evm_verification_mismatch_total`      | Counter | Events rejected because public-source execution effects contradicted the claimed event. Any increment is a potential forgery attempt.                                                                                   |
| `bridge_casper2evm_parked_events`                    | Gauge   | Events currently parked awaiting public-source confirmation.                                                                                                                                                            |

### P2P Metrics

| Metric                        | Type    | Description                |
| ----------------------------- | ------- | -------------------------- |
| `p2p_connected_peers`         | Gauge   | Number of connected peers  |
| `p2p_messages_sent_total`     | Counter | Messages sent by topic     |
| `p2p_messages_received_total` | Counter | Messages received by topic |
| `p2p_connection_errors_total` | Counter | Connection errors          |

### Database Metrics

| Metric                      | Type      | Description        |
| --------------------------- | --------- | ------------------ |
| `db_query_duration_seconds` | Histogram | Query latency      |
| `db_connections_active`     | Gauge     | Active connections |
| `db_errors_total`           | Counter   | Database errors    |

## Prometheus Setup

### prometheus.yml

```yaml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

rule_files:
  - "/etc/prometheus/alert_rules.yml"

scrape_configs:
  - job_name: 'bridge-relayers'
    static_configs:
      - targets:
          - 'relayer-1:9080'
          - 'relayer-2:9080'
          - 'relayer-3:9080'
          - 'relayer-4:9080'
          - 'relayer-5:9080'
    relabel_configs:
      - source_labels: [__address__]
        regex: 'relayer-(\d+):.*'
        target_label: relayer_id
        replacement: '$1'

  - job_name: 'node-exporter'
    static_configs:
      - targets:
          - 'relayer-1:9100'
          - 'relayer-2:9100'
          - 'relayer-3:9100'
          - 'relayer-4:9100'
          - 'relayer-5:9100'
```

### Start Prometheus

```bash
# Docker
docker run -d \
  --name prometheus \
  -p 9090:9090 \
  -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

# Or with docker-compose
docker-compose up -d prometheus
```

## Alert Rules

### alert\_rules.yml

```yaml
groups:
  - name: bridge_alerts
    interval: 30s
    rules:
      # Relayer down
      - alert: RelayerDown
        expr: up{job="bridge-relayers"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Relayer {{ $labels.instance }} is down"
          description: "Relayer has been unreachable for 2 minutes"

      # High pending transactions
      - alert: HighPendingTransactions
        expr: bridge_pending_transactions > 50
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High number of pending transactions"
          description: "{{ $value }} transactions pending for over 5 minutes"

      # P2P connectivity issues
      - alert: LowPeerConnectivity
        expr: p2p_connected_peers < 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low P2P peer connectivity"
          description: "Only {{ $value }} peers connected"

      # Database errors
      - alert: DatabaseErrors
        expr: increase(db_errors_total[5m]) > 10
        labels:
          severity: critical
        annotations:
          summary: "Database errors detected"
          description: "{{ $value }} database errors in last 5 minutes"

      # Signature collection stuck
      - alert: SignatureCollectionStuck
        expr: bridge_signatures_collected < 3 and time() - bridge_attestation_created_timestamp > 600
        labels:
          severity: warning
        annotations:
          summary: "Signature collection taking too long"
          description: "Attestation has only {{ $value }} signatures after 10 minutes"

      # Block processing lag
      - alert: BlockProcessingLag
        expr: (bridge_chain_latest_block - bridge_latest_block_processed) > 100
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Block processing is lagging"
          description: "{{ $value }} blocks behind on {{ $labels.chain }}"
```

## Grafana Setup

### Install Grafana

```bash
# Docker
docker run -d \
  --name grafana \
  -p 3000:3000 \
  -v grafana-storage:/var/lib/grafana \
  grafana/grafana

# Default login: admin / admin
```

### Add Prometheus Data Source

1. Go to Configuration → Data Sources
2. Add data source → Prometheus
3. URL: `http://prometheus:9090`
4. Save & Test

### Import Dashboards

Pre-built dashboards are available at `/monitoring/grafana-dashboard.json`.

1. Go to Dashboards → Import
2. Upload JSON file or paste content
3. Select Prometheus data source
4. Import

## Dashboard Panels

### Bridge Overview

Key panels for the overview dashboard:

| Panel              | Query                                                                                 | Description             |
| ------------------ | ------------------------------------------------------------------------------------- | ----------------------- |
| Total Transactions | `sum(bridge_transactions_total)`                                                      | All-time transactions   |
| Success Rate       | `sum(bridge_transactions_total{status="completed"}) / sum(bridge_transactions_total)` | Completion percentage   |
| Pending Queue      | `bridge_pending_transactions`                                                         | Current queue depth     |
| 24h Volume         | `increase(bridge_transactions_total[24h])`                                            | Daily transaction count |

### Per-Chain Metrics

```promql
# Transactions per chain
sum by (chain_id) (bridge_events_detected_total)

# Latest block per chain
bridge_latest_block_processed

# Block lag per chain
bridge_chain_latest_block - bridge_latest_block_processed
```

### Relayer Health

```promql
# Relayer status (up/down)
up{job="bridge-relayers"}

# Attestations per relayer
sum by (relayer_id) (bridge_attestations_submitted_total)

# P2P messages per relayer
sum by (relayer_id) (rate(p2p_messages_sent_total[5m]))
```

## Alertmanager Configuration

### alertmanager.yml

```yaml
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'
    - match:
        severity: warning
      receiver: 'slack'

receivers:
  - name: 'default'
    email_configs:
      - to: 'ops@example.com'
        from: 'alerts@example.com'
        smarthost: 'smtp.example.com:587'

  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/XXX/YYY/ZZZ'
        channel: '#bridge-alerts'
        send_resolved: true

  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: 'your-pagerduty-key'
```

## Health Check Endpoint

The bridge exposes a health check endpoint:

```bash
curl http://localhost:9080/health
```

Response:

```json
{
  "status": "healthy",
  "components": {
    "database": "connected",
    "ethereum": "synced",
    "casper": "synced",
    "p2p": "connected"
  },
  "uptime": 86400
}
```

## Recommended Alerts

### Critical

* Relayer down for >2 minutes
* Database connection failed
* All P2P peers disconnected
* Attestation verification failures
* `SUBMISSION_PERMANENT_FAILURE` in logs — an event was parked after a verified deterministic on-chain rejection (e.g. unregistered token). No relayer will retry it. Requires operator action: fix the root cause (usually an admin registration/configuration transaction), then restart relayers to un-park.
* `VerificationMismatch` — public Casper node execution effects contradicted a claimed event (potential forgery attempt). Backed by `bridge_casper2evm_verification_mismatch_total`.
* `VerificationStallTimeout` — an event was dead-lettered after \~6h with no public-source confirmation (fabricated event or sustained public-node outage). Backed by `bridge_casper2evm_verification_stall_timeout_total`.

### Warning

* High pending transaction queue (>50)
* Low P2P connectivity (<3 peers)
* Block processing lag (>100 blocks)
* High error rate (>5 errors/minute)

### Info

* New relayer joined network
* Configuration reloaded
* Maintenance mode enabled

## Stranded-Transfer Recovery Runbook

When a user's source-side action succeeded (tokens locked on EVM / burned or locked on Casper) but the destination leg fails **permanently**, funds are not lost — the source chain still holds them — but the transfer is stranded until an operator acts. This is the documented recovery path.

### 1. Detect

* `SUBMISSION_PERMANENT_FAILURE` alert (see Critical alerts above) identifies the parked event, its failure classification, and the decoded error name (e.g. `User error: 60024 [TokenNotFound]`, `TokenNotRegistered(address)`).
* Cross-check with `bridge-cli admin get-contract-status` (EVM) and `bridge-cli admin casper-get-status` (Casper) for pause state, fee config, and registry health.

### 2. Classify and fix the root cause

| Decoded error                                            | Root cause                                                                                         | Fix (admin action)                                                                                    |
| -------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `60024 TokenNotFound` / `60038 NativeTokenNotRegistered` | missing token mapping                                                                              | `admin deploy-token` / `admin casper-register-token-mapping` / `admin register-token`                 |
| `60041 DecimalMismatch`                                  | mapping registered with wrong decimals                                                             | `admin casper-update-native-token-mapping` / `admin update-token-mapping` to the corrected deployment |
| `60046 AttestedAssetMismatch`                            | live mapping diverged from signed attestation                                                      | repair the mapping to match the attested asset, NOT vice versa                                        |
| `60031 ReceiptsRootMismatch`                             | EthClient state inconsistent with attestation                                                      | investigate relayer set / resubmit the correct header                                                 |
| `TokenNotRegistered` (EVM)                               | EVM-side registry gap                                                                              | `admin register-token`                                                                                |
| Corrupted Casper fee config (`64648` on fee reads)       | storage layout consolidation residue                                                               | `admin set-casper-fee`-equivalent repair via `repair_fee_config` (ceiling-checked)                    |
| `60051 NonceOverflow`                                    | nonce lane exhausted (2^32 transfers)                                                              | contract-level intervention — escalate to engineering; do NOT retry                                   |
| `60053 TokenBalanceDecreased`                            | registered CEP-18 reported a lower contract balance after `transfer_from` (hostile/rebasing token) | investigate & deregister the token — do NOT retry                                                     |
| `60054 LockedTotalOverflow`                              | total-locked accumulator would overflow U256 (only reachable with a hostile registered token)      | investigate & deregister the token — do NOT retry                                                     |

### 3. Un-park and replay

Parked state is **in-memory**: after the root-cause fix lands on-chain, restart the relayers (`bridge-cli stop && bridge-cli start` per host, rolling). On restart the event re-enters normal processing and replays cleanly — replay protection guarantees an already-processed leg is a no-op (`EventAlreadyProcessed` / `60026` is the expected duplicate signal, absorbed automatically).

### 4. Verify closure

* Destination-side balance reflects the transfer (`admin get-locked-balance` / CEP-18 balance query).
* The event no longer appears in `SUBMISSION_PERMANENT_FAILURE` logs after two full polling cycles.
* If the transfer STILL fails deterministically after a correct fix, escalate with the full decoded error and the attestation payload — do not loop restarts.

> **Design note:** the bridge never marks a failed submission as processed (audit-verified — cleanup only happens after confirmed success or an on-chain-confirmed duplicate). The park mechanism halts gas-burning retries of deterministically doomed payloads; it never discards them.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.astralbeam.io/operators/monitoring.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.