TzIndex

All-in-one zero-conf Tezos blockchain indexer
TzIndex is a high-performance Tezos blockchain indexer. It is a fast, convenient, and resource-friendly way to gain insight into the Tezos blockchain and perform extremely fast analytical queries.
Powered by Blockwatch's proprietary columnar database technology.

Tasks

  1. 1.
    Track role, status, and history of accounts
  2. 2.
    Track account balances
  3. 3.
    Track supply
  4. 4.
    Track important events

Features

  • indexes and cross-checks full on-chain state
  • feature-rich REST API with objects, bulk tables and time-series
  • supports protocols up to Granada (v010)
  • auto-detects and locks Tezos network (never mixes data from different networks)
  • indexes all accounts and smart-contracts (including genesis data)
  • follows chain reorgs as they are resolved
  • can passively monitor for new blocks
  • self-heals broken node connections (retries until node RPC comes back up)
  • API supports CORS and HTTP caching
  • high-performance embedded data-store
  • flexible in-memory caching for fast queries
  • automatic database backups/snapshots
  • ZMQ PUB/SUB support
  • configurable HTTP request rate-limiter
  • health-check endpoint and health-check HTTP headers
  • flexible metadata support
  • pruning of unused historic rights and snapshots

Supported Indexes and Data Tables

  • blocks: all blocks including orphans, indexed by hash and height
  • operations: all on-chain operations including contract call data, indexed by hash
  • accounts: running account details like most recent balances, indexed by address
  • contracts: running smart contract details, code and initial storage
  • flows: complete list of balance, freezer and delegation balance updates
  • chain: running blockchain totals
  • supply: running supply totals
  • rights: full list of all baking and endorsing rights
  • elections, votes, proposals and ballots capturing all on-chain governance activities
  • snapshots: balances of active delegates & delegators at all snapshot blocks
  • baker income: per-cycle statistics on baker income, efficiency, etc
  • bigmap: bigmap smart contract storage index, split across 3 tables bigmap_allocs, bigmap_updates and bigmap_values
  • metadata: standardized and custom account/token metadata

Operation Modes

  • Full regular operation mode that builds all indexes (CLI: --full)
  • Light (default) light-weight mode without consensus and governance indexes (CLI: --light)
  • Validate state validation mode for checking accounts and balances each block/cycle (CLI: --validate)
Light mode is best suited for dapps where access to baking-related data is not necessary. Light mode has become default in TzIndex v010+. In light mode we don't generate the following list of indexes: rights, elections, votes, proposals, ballots, snapshots and income. Its not possible to switch between full and light mode without a full database rebuild.
Light mode saves roughly ~50% storage costs and indexing time. Keep in mind that some endpoints and data fields will not be available:
  • /tables and /explorer API endpoints for missing tables will return 409 errors
  • /explorer/block/.. will not contain the block endorser list, rights and snapshot flag
  • /explorer/account/.. will not contain upcoming bake/endorse info
  • /explorer/cycle/.. will not contain snapshot block info
Validate mode works in combination with full and light mode. At each block it checks balances and states of all touched accounts against a Tezos archive node before any change is written to the database. At the end of each cycle, all known accounts in the indexer database are checked as well. This ensures 100% consistency although at the cost of a reduction in indexing speed.

System Requirements

  • Storage: 9GB (full Mainnet index, Aug 2021), 4G (light mode)
  • RAM: 4-24GB (configurable, use more memory for better query latency)
  • CPU: 2+ cores (configurable, use more for better query parallelism)
  • Tezos node in archive mode
Runs against any Tezos Archive Node (also full nodes when cycle 0 history is not yet pruned). This can be a local node or one of the public Tezos RPC nodes on the Internet. Note that syncing from public nodes over the Internet works but may be slow.
IMPORTANT: WHEN USING OCTEZ V12+ YOU MUST RUN YOUR ARCHIVE NODE WITH --metadata-size-limit unlimited.
Requires access to the following Tezos RPC calls:
/chains/main/blocks/{blockid}
/chains/main/blocks/{blockid}/helpers/baking_rights (full mode only)
/chains/main/blocks/{blockid}/helpers/endorsing_rights (full mode only)
/chains/main/blocks/{blockid}/context/raw/json/cycle/{cycle} (full mode only)
/chains/main/blocks/{blockid}/context/constants
/chains/main/blocks/head/header
/monitor/heads/main (optional)
/chains/main/blocks/{blockid}/context/contract/{address} (validate mode only)

Configuration

Config file

On start-up tzindex tries loading its config from the config.jsonfile in the current directory. You may override this name either on the command line using -c myconf.json or by setting the environment variable TZ_CONFIG_FILE=/some/path/myconf.json
Config file sections:
rpc - configures RPC connection to the Tezos node
crawler - configures the blockchain crawl logic
database - configures the embedded database
server - configures the built-in HTTP API server
logging - configures logging for all subsystems
See the default config.json in the docker subfolder for a detailed list of all settings.

Environment variables

Env variables allow you to override settings from the config file or even specify all configuration settings in the process environment. This makes it easy to manage configuration in Docker and friends. Env variables are all uppercase, start with TZ and use an underscore _ as separator between sub-topics.
# in config.json
{ "rpc": { "host" : "127.0.0.1" }}
# same as env variable
TZ_RPC_HOST=127.0.0.1

Command line arguments

A few global settings can only be controlled via cli arguments:
Usage:
tzindex [command]
Available Commands:
help Help about any command
run Run as service
version Print the version number of tzindex
Flags:
-c, --config string config file
--cpus int max number of logical CPU cores to use (default: all) (default -1)
-p, --dbpath path database path
--gogc int trigger GC when used mem grows by N percent (default 20)
-h, --help help for tzindex
--profile-block string write blocking events to file
--profile-cpu string write cpu profile to file
--profile-mutex string write mutex contention samples to file
--profile-rate int block/mutex profiling rate in fractions of 100 (e.g. 100 == 1%) (default 100)
-t, --test test configuration and exit
--v be verbose
--vv debug mode
--vvv trace mode
The main tzindex run command has a few additional options
Usage:
tzindex run [flags]
Flags:
--enable-cors enable API CORS support
-h, --help help for run
--insecure disable RPC TLS certificate checks (not recommended)
--light light mode (use to skip baker and gov data)
--noapi disable API server
--noindex disable indexing
--nomonitor disable block monitor
--nopublish disable report publishing
--norpc disable RPC client
--notls disable RPC TLS support (use http)
--rpcpass string RPC password
--rpcurl string RPC url (default "http://127.0.0.1:8732")
--rpcuser string RPC username
--stop height stop indexing after height
--unsafe disable fsync for fast ingest (DANGEROUS! data will be lost on crashes)
--validate validate account balances

Tuning Guide

Please check back soon.