Skip to Content
Bsa Partner IntegrationBSA Arbitration Oracle Handbook

BSA Arbitration Oracle Handbook

This document provides a description of the operations needed to run and maintain an Arbitration Oracle (AO) for the Lombard BSA product.

A key property of enclave applications like the Arbitration Oracle is that they have no persistence across different restarts. This document walks through all the procedure needed to preserve and restore the required state, e.g. their Bitcoin identity.

Deployment

The aim of the deployment phase is to have three replicas of the Arbitration Oracle components. The three replicated setups are referred as:

  • Primary — the instance in sync with the production system and actually performing all the AO tasks, whose decisions are forwarded to the respective target chains (Bitcoin).
  • Secondary — the instance in sync with the production system used as a failover in case the Primary is detected as unavailable
  • Backup — the instance disconnected from the network after the first initialization to use in case of key recovery if both Primary and Secondary are unavailable

Each instance of an AO is supposed to be deployed in a different AWS region for redundancy purposes.

An instance of an AO comprises an AWS EC2 instance with the following component:

  1. A Nitro Enclave executing the AO TEE logic from https://github.com/lombard-finance/bsa-arbitration-oracle/releases 
  2. VSock proxies to provide RPC connection and log forwarding capabilities to the Nitro Enclave application (default tool available in AWS EC2 images)
  3. The AO CLI tool from https://github.com/lombard-finance/bsa-arbitration-oracle 
  4. The AO controller application which implements indexing and transaction forwarding logic from https://github.com/lombard-finance/bsa-arbitration-controller 

An AWS Cloud Formation description of the setup is provided at https://github.com/lombard-finance/arbiter-tee-configuration . It can be executed as-is by setting the requested parameters, or used as a description of the expected setup to let it fit the partner infrastructure.

The following guide assumes deployment happened via such Cloud Formation template.

Lombard Resources

Lombard provides by making them publicly accessible, or upon request by the partner the following resources:

  • Trusted Checkpoint — A payload signed by the Security Consortium which certifies as trusted by the AO in the Nitro Enclave a checkpoint to synchronize the Ethereum state from. Their trustworthiness can be verified on the following portals or via a Beacon Chain RPC:
  • KMS Backup Authorization — A payload signed by a Lombard owned key which authorizes the AO in a Nitro Enclave to create an AO Key backup encrypting it using a AWS KMS key.
  • Arbiter Version Expiration — A payload signed by a Lombard owned key which authorizes a certain version of the AO in a Nitro Enclave to run as legitimate BSA Arbitration Oracle for a finite time period. These payloads are stored on the Smart Account Registry on Ethereum which works as data availability layer, so no manual input to any component is needed.

Initialization

The initialization operation is made by an AO operator, Lombard partner, creating an AO instance for the first time when they first configure the BSA product.

  1. Create an AO deployment according to the Deployment section
  2. Ask Lombard for a Trusted Checkpoint of Ethereum Beacon chain
    1. Verify on a third party portal or RPC the checkpoint is an actual one
  3. Key generation (Backup). Access the instance assigned the Backup role:
    1. Execute ao-cli init-ethereum command
      1. --checkpoint-lombard the Trusted Checkpoint file provided by Lombard Security Consortium
      2. -c the Ethereum Beacon Chain RPC
      3. -e the Ethereum execution RPC
      4. --btc-network and --network are set to mainnet by default. Set to signet and sepolia for testnet.
      5. Current checkpoint is displayed after synchronization
    2. Execute the ao-cli init command
      1. The AO public key is displayed after generation
      2. Three files are generated and stored
        1. init_receipt.json a shareable file attesting correct key generation
        2. init_response.json attestation including the whole TEE response for integration with other components
    3. Provide init_receipt.json to Lombard to demonstrate the correct key generation
  4. Key replication (Primary)
    1. Access the instance assigned the Primary role:
      1. Execute the ao-cli restore-peer-request command
      2. An attested request file is saved locally
    2. Access the instance assigned the Backup role:
      1. Copy the attested request file
      2. Execute ao-cli restore-peer-respond command
      3. An attested response file is saved locally
    3. Access the instance assigned the Primary role:
      1. Copy the attested response file
      2. Execute ao-cli restore-peer-respond command
      3. The AO public key is displayed and it signals the replication process completion.
  5. Key replication (Secondary)
    1. Repeat key replication steps from point 4 to replicate key on the Secondary instance
  6. Proceed with the AO Key Backup section to:
    1. produce a persistent backup of the Arbitration Oracle Bitcoin key
    2. isolate the Backup instance from the network

AO Key Backup

The AO operator can produce a persistent key backup via a Lombard provisioned KMS Backup Authorization file. Such a backup is stored by the AO operator but usable under Lombard signed authorization.

  1. Collect the KMS Backup Authorization file from Lombard
  2. Access the instance assigned the Backup role
    1. Copy the KMS Backup Authorization
    2. Make sure the vsock-proxy to the AWS KMS region of the backup key is running and take not about its vsock port.
    3. Execute ao-cli backup-kms command
      1. --kms-key-authorization <KMS_KEY_AUTHORIZATION> Path to the Lombard provisioned KMS Backup Authorization
      2. --kms-proxy-port <KMS_PROXY_PORT> port of the vsock-proxy pointing to the AWS KMS region
    4. Store the encrypted key backup in a safe long-term storage
    5. Provide the create-backup receipt response to Lombard to demonstrate the correct backup creation
  3. Stop all vsock-proxy services on the Backup instance
    1. This will practically isolate the backup AO from the network

Recovery from Replica

This flow allows to restore an AO instance by relying on the state of an operational AO replica. An usual case for this flow is the need to restart a Primary instance (e.g. because of an upgrade) using the Secondary as recovery instance, and viceversa.

Requirements

  • The AO instance to restore (fresh instance) is in its initial state.
  • The AO instance to recover from is operational (recovery instance), i.e. its light client is in sync and it can validate and sign BSA operations
  • The fresh instance has an AO enclave code version greater than or equal to the recovery instance.

Steps

  1. Light client initialization via one of the following
    1. Via operational instance
      1. Access the operational instance:
        1. Execute the ao-cli status command
        2. An attested status receipt file is stored locally (status_attestation.json)
      2. Access the fresh instance:
        1. Copy the attested status receipt file
        2. Execute the ao-cli init-ethereum command providing the status receipt in the --status-attestation flag
        3. 💡 Note: AWS attestations are short-lived (~2-3 hours on average). An attestation or response receipt can be verified via ao-cli verify-response --response <response_file>
    2. Via Trusted Checkpoint from Lombard
      1. Access the fresh instance:
        1. Execute the ao-cli init-ethereum command providing the checkpoint with --checkpoint-lombard
  2. Peer key recovery
    1. Access the fresh instance:
      1. Execute the ao-cli restore-peer-request command
      2. An attested request file is saved locally
    2. Access the recovery instance:
      1. Copy the attested request file
      2. Execute ao-cli restore-peer-respond command
      3. An attested response file is saved locally
    3. Access the fresh instance:
      1. Copy the attested response file
      2. Execute ao-cli restore-peer-complete command
      3. The AO public key is displayed and it signals the replication process completion.

Recovery from KMS Key Backup

This flow allows to restore an AO instance by relying on a key backup created in initialization phase encrypted by a Lombard owned AWS KMS key. The flow is suitable to recover any instance but it requires coordination with the Lombard team.

Requirements

  • The AO instance to restore (fresh instance) is in its initial state
  • The AO Key Backup from step 7 of initialization flow
  • A vsock-proxy configured towards the AWS KMS region Lombard key resides in

Steps

  1. Communicate to Lombard the PCR0 value of the instance to recover
    1. Access the fresh instance
    2. Execute nitro-cli describe-enclaves
    3. The output JSON string has the PCR0 value under the Measurements field (output from previous step could be filtered by chaining the command with | jq '.[0].Measurements.PCR0' to print out the single value if jq is available).
    4. Communicate it to Lombard team
  2. Light client initialization via one of the following
    1. Via operational instance
      1. Access the operational instance:
        1. Execute the ao-cli status command
        2. An attested status receipt file is stored locally (status_attestation.json)
      2. Access the fresh instance:
        1. Copy the attested status receipt file
        2. Execute the ao-cli init-ethereum command providing the status receipt in the --status-attestation flag
        3. 💡 Note: AWS attestations are short-lived (~2-3 hours on average). An attestation or response receipt can be verified via ao-cli verify-response --response <response_file>
    2. Via Trusted Checkpoint from Lombard
      1. Access the fresh instance:
        1. Execute the ao-cli init-ethereum command providing the checkpoint with --checkpoint-lombard
  3. Wait for Lombard confirmation about PCR0 value communication
  4. Backup decryption
    1. Access the fresh instance:
      1. Make sure to locally have the encrypted backup
      2. Execute the ao-cli restore-kms command providing:
        1. the encrypted backup --backup-receipt
        2. the VSock port proxying requests to KMS --kms-proxy-port
        3. the KMS region key resides in --kms-region
        4. the flag --skip-chain-trust-verification since backup attestation verification is not needed and AWS NSM attestations are short lived.
      3. The CLI prints out the public key of the recovered identity

Recovery from Offline Replica

This flow allows to restore an AO instance by relying on the Backup instance created at deployment stage. Such instance is kept offline to preserve it from runtime failure that could bring it down, so the recovery procedure needs to provide all the information the Backup instance cannot fetch on its own.

Requirements

  • The AO instance to restore (fresh instance) is in its initial state.
  • The AO Backup instance
  • The fresh instance has an AO enclave code version greater than or equal to the recovery instance.
  • The AO Backup instance has an AO enclave code version that has not expired

Steps

  1. Collect the PCR0 value of the Backup instance. Can be skipped if already known.
    1. Access the Backup instance
    2. Execute nitro-cli describe-enclaves
    3. The output JSON string has the PCR0 value under the Measurements field (output from previous step could be filtered by chaining the command with | jq '.[0].Measurements.PCR0' to print out the single value if jq is available).
  2. Light client initialization via one of the following
    1. Via operational instance
      1. Access the operational instance:
        1. Execute the ao-cli status command
        2. An attested status receipt file is stored locally (status_attestation.json)
      2. Access the fresh instance:
        1. Copy the attested status receipt file
        2. Execute the ao-cli init-ethereum command providing the status receipt in the --status-attestation flag
        3. 💡 Note: AWS attestations are short-lived (~2-3 hours on average). An attestation or response receipt can be verified via ao-cli verify-response --response <response_file>
    2. Via Trusted Checkpoint from Lombard
      1. Access the fresh instance:
        1. Execute the ao-cli init-ethereum command providing the checkpoint with --checkpoint-lombard
  3. Peer key recovery
    1. Access the fresh instance:
      1. Execute the ao-cli restore-peer-request command by setting the --responder-pcr0 <pcr0> flag with the value collected from step 1.
      2. An attested request file is saved locally
    2. Access the recovery instance:
      1. Copy the attested request file
      2. Execute ao-cli restore-peer-respond command
      3. An attested response file is saved locally
    3. Access the fresh instance:
      1. Copy the attested response file
      2. Execute ao-cli restore-peer-complete command
      3. The controller printing out the AO public key signals the replication process completion.

Upgrade

When the AO software running in the Nitro Enclave needs to be updated it is enough to spin up the new instance with the updated code and execute any of the mentioned recovery procedures.

Note In order to perform an upgrade in an autonomous manner, we recommend doing it while the old instance is still running in order to generate a status attestation whenever needed. This allows to initialize the Ethereum light client on the upgraded instance without requesting a Trusted Checkpoint from Lombard Security Consortium.

Controller deployment

After an AO is initialized by key generation or recovery, the controller component can be started.

Note The AO controller from https://github.com/lombard-finance/bsa-arbitration-controller  monitors both Bitcoin and Smart Account Registry on the destination chain (Ethereum) in order to initiate AO interactions whenever a challenge resolution is needed. The README.md provides all the information needed for configuring and querying such a component.

  1. Access the target AWS EC2 instance
  2. Make sure the controller binary is correctly installed via which ao-controller
    1. Binary is supposed to be placed at /opt/arbiter/ao-controller
    2. If the binary is not there or needs to be updated, build it or download a pre-built binary from https://github.com/lombard-finance/bsa-arbitration-controller 
  3. Adjust the controller configuration at /opt/arbiter/ao-controller-conf.yaml based on the template and description available in the README.md of the repository
  4. Start the controller via systemctl start arbiter-controller.service
    1. the service is already enabled for restart upon machine reboot

← Back to BSA Integration Overview

Last updated on
API StatesIndex