____   _         ____              
       / __ \ (_)____   / __ \ __  __ ____ 
      / / / // // __ \ / / / // / / // __ \
     / /_/ // // /_/ // /_/ // /_/ // /_/ /
    /_____//_// .___//_____/ \__,_// .___/ 
             /_/                  /_/      

DipDup is a Python framework for building indexers of Tezos smart contracts. It helps developers focus on the business logic instead of writing data storing and serving boilerplate. DipDup-based indexers are selective, which means only required data is requested. This approach allows to achieve faster indexing times and decreased load on APIs DipDup uses.

• Ready to build your first indexer? Head to Quickstart.

• Looking for examples? Check out Demo Projects.

• Want to contribute? See Contribution Guide.

• Have a question? Contact us on Discord, Telegram, or Slack!

This project is maintained by the Baking Bad team.
Development is supported by Tezos Foundation.

Quickstart

This page will guide you through the steps to get your first selective indexer up and running in a few minutes without getting too deep into the details.

Let's create an indexer for the tzBTC FA1.2 token contract. Our goal is to save all token transfers to the database and then calculate some statistics of its holders' activity.

A modern Linux/MacOS distribution with Python 3.10 installed is required to run DipDup.

Create a new project

Interactively (recommended)

You can initialize a hello-world project interactively by choosing configuration options in the terminal. The following command will install cookiecutter and create a new project in the current directory.

python -c "$(curl -sSL https://dipdup.net/install.py)"

From scratch

We advise using the Poetry package manager for new projects. However, it's not a requirement. If you prefer pdb, piptools, pipenv or other tools — use them instead.

# Create a new project
mkdir my-indexer; cd my-indexer
poetry init --python ">=3.10,<3.11"
# Add dipdup as a dependency
poetry add dipdup
# Enter the virtualenv
poetry shell

💡 SEE ALSO

• 2.1. Installation
• Poetry documentation

Write a configuration file

DipDup configuration is stored in YAML files of a specific format. Create a new file named dipdup.yml in your current working directory with the following content:

spec_version: 1.2
package: demo_tzbtc

database:
  kind: sqlite
  path: demo_tzbtc.sqlite3

contracts:
  tzbtc_mainnet:
    address: KT1PWx2mnDueood7fEmfbBDKx1D9BAnnXitn
    typename: tzbtc

datasources:
  tzkt:
    kind: tzkt
    url: https://api.tzkt.io

advanced:
  early_realtime: True
  crash_reporting: True

indexes:
  tzbtc_holders_mainnet:
    kind: operation
    datasource: tzkt
    contracts:
      - tzbtc_mainnet
    handlers:
      - callback: on_transfer
        pattern:
          - destination: tzbtc_mainnet
            entrypoint: transfer
      - callback: on_mint
        pattern:
          - destination: tzbtc_mainnet
            entrypoint: mint

💡 SEE ALSO

• 2.7. Templates and variables
• 14. Config file reference

Initialize project tree

Now it's time to generate typeclasses and callback stubs. Run the following command:

dipdup init

DipDup will create a Python package demo_tzbtc having the following structure:

demo_tzbtc
├── graphql
├── handlers
│   ├── __init__.py
│   ├── on_mint.py
│   └── on_transfer.py
├── hooks
│   ├── __init__.py
│   ├── on_reindex.py
│   ├── on_restart.py
│   ├── on_index_rollback.py
│   └── on_synchronized.py
├── __init__.py
├── models.py
├── sql
│   ├── on_reindex
│   ├── on_restart
│   ├── on_index_rollback
│   └── on_synchronized
└── types
    ├── __init__.py
    └── tzbtc
        ├── __init__.py
        ├── parameter
        │   ├── __init__.py
        │   ├── mint.py
        │   └── transfer.py
        └── storage.py

That's a lot of files and directories! But don't worry, we will need only models.py and handlers modules in this guide.

💡 SEE ALSO

• 2.4. Project structure
• 13. Command-line reference → init

Define data models

Our schema will consist of a single model Holder having several fields:

• address — account address
• balance — in tzBTC
• volume — total transfer/mint amount bypassed
• tx_count — number of transfers/mints
• last_seen — time of the last transfer/mint

Put the following content in the models.py file:

from tortoise import fields

from dipdup.models import Model


class Holder(Model):
    address = fields.CharField(max_length=36, pk=True)
    balance = fields.DecimalField(decimal_places=8, max_digits=20, default=0)
    turnover = fields.DecimalField(decimal_places=8, max_digits=20, default=0)
    tx_count = fields.BigIntField(default=0)
    last_seen = fields.DatetimeField(null=True)

💡 SEE ALSO

• 2.5. Defining models
• Tortoise ORM documentation
• Tortoise ORM examples

Implement handlers

Everything's ready to implement an actual indexer logic.

Our task is to index all the balance updates, so we'll start with a helper method to handle them. Create a file named on_balance_update.py in the handlers package with the following content:

from datetime import datetime
from decimal import Decimal

import demo_tzbtc.models as models


async def on_balance_update(
    address: str,
    balance_update: Decimal,
    timestamp: datetime,
) -> None:
    holder, _ = await models.Holder.get_or_create(address=address)
    holder.balance += balance_update
    holder.turnover += abs(balance_update)
    holder.tx_count += 1
    holder.last_seen = timestamp
    await holder.save()

Three methods of tzBTC contract can alter token balances — transfer, mint, and burn. The last one is omitted in this tutorial for simplicity. Edit corresponding handlers to call the on_balance_update method with data from matched operations:

on_transfer.py

from decimal import Decimal

from demo_tzbtc.handlers.on_balance_update import on_balance_update
from demo_tzbtc.types.tzbtc.parameter.transfer import TransferParameter
from demo_tzbtc.types.tzbtc.storage import TzbtcStorage
from dipdup.context import HandlerContext
from dipdup.models import Transaction


async def on_transfer(
    ctx: HandlerContext,
    transfer: Transaction[TransferParameter, TzbtcStorage],
) -> None:
    if transfer.parameter.from_ == transfer.parameter.to:
        # NOTE: Internal tzBTC transfer
        return

    amount = Decimal(transfer.parameter.value) / (10**8)
    await on_balance_update(
        address=transfer.parameter.from_,
        balance_update=-amount,
        timestamp=transfer.data.timestamp,
    )
    await on_balance_update(
        address=transfer.parameter.to,
        balance_update=amount,
        timestamp=transfer.data.timestamp,
    )

on_mint.py

from decimal import Decimal

from demo_tzbtc.handlers.on_balance_update import on_balance_update
from demo_tzbtc.types.tzbtc.parameter.mint import MintParameter
from demo_tzbtc.types.tzbtc.storage import TzbtcStorage
from dipdup.context import HandlerContext
from dipdup.models import Transaction


async def on_mint(
    ctx: HandlerContext,
    mint: Transaction[MintParameter, TzbtcStorage],
) -> None:
    amount = Decimal(mint.parameter.value) / (10**8)
    await on_balance_update(
        address=mint.parameter.to,
        balance_update=amount,
        timestamp=mint.data.timestamp,
    )

And that's all! We can run the indexer now.

💡 SEE ALSO

• 2.6. Implementing handlers

Run your indexer

dipdup run

DipDup will fetch all the historical data and then switch to realtime updates. Your application data has been successfully indexed!

💡 SEE ALSO

• 13. Command-line reference

Getting started

This part of the docs covers the same features the Quickstart article does but is more focused on details.

Installation

This page covers the installation of DipDup in different environments.

Host requirements

A Linux/MacOS environment with Python 3.10 installed is required to use DipDup. Other UNIX-like systems should work but are not supported officially.

Minimum hardware requirements are 256 MB RAM, 1 CPU core, and some disk space for the database. RAM requirements increase with the number of indexes.

Non-UNIX environments

DipDup currently doesn't work in Windows environments due to incompatibilities in libraries it depends on. Please use WSL or Docker.

We aim to improve cross-platform compatibility in future releases (issue).

💡 SEE ALSO

• 3.8. Improving performance
• What is the Windows Subsystem for Linux?

Local installation

Interactively (recommended)

You can initialize a hello-world project interactively by choosing configuration options in the terminal. The following command will install cookiecutter and create a new project in the current directory.

python -c "$(curl -sSL https://dipdup.net/install.py)"

Poetry

We advise using the Poetry package manager for new projects. However, it's not a requirement. If you prefer pdb, piptools, pipenv or other tools — use them instead.

# Create a new project
mkdir my-indexer; cd my-indexer
poetry init --python ">=3.10,<3.11"
# Add dipdup as a dependency
poetry add dipdup
# Enter the virtualenv
poetry shell

💡 SEE ALSO

• Poetry documentation

pip

Create a new virtual environment and install DipDup in it.

python -m venv .venv
source .venv/bin/activate
pip install dipdup

Docker

See 5.2. Running in Docker page.

Core concepts

Big picture

Initially, DipDup was heavily inspired by The Graph Protocol, but there are several differences. The most important one is that DipDup indexers are completely off-chain.

DipDup utilizes a microservice approach and relies heavily on existing solutions, making the SDK very lightweight and allowing developers to switch API engines on demand.

DipDup works with operation groups (explicit operation and all internal ones, a single contract call) and Big_map updates (lazy hash map structures, read more) — until fully-fledged protocol-level events are not implemented in Tezos.

Consider DipDup a set of best practices for building custom backends for decentralized applications, plus a toolkit that spares you from writing boilerplate code.

DipDup is tightly coupled with TzKT API but can generally use any data provider which implements a particular feature set. TzKT provides REST endpoints and Websocket subscriptions with flexible filters enabling selective indexing and returns "humanified" contract data, which means you don't have to handle raw Michelson expressions.

DipDup offers PostgreSQL + Hasura GraphQL Engine combo out-of-the-box to expose indexed data via REST and GraphQL with minimal configuration. However, you can use any database and API engine (e.g., write API backend in-house).

How it works

From the developer's perspective, there are three main steps for creating an indexer using the DipDup framework:

1. Write a declarative configuration file containing all the inventory and indexing rules.
2. Describe your domain-specific data models.
3. Implement the business logic, which is how to convert blockchain data to your models.

As a result, you get a service responsible for filling the database with the indexed data.

Within this service, there can be multiple indexers running independently.

Atomicity and persistency

DipDup applies all updates atomically block by block. In case of an emergency shutdown, it can safely recover later and continue from the level it ended. DipDup state is stored in the database per index and can be used by API consumers to determine the current indexer head.

Here are a few essential things to know before running your indexer:

• Ensure that the database (or schema in the case of PostgreSQL) you're connecting to is used by DipDup exclusively. Changes in index configuration or models require DipDup to drop the whole database (schema) and start indexing from scratch. You can, however, mark specific tables as immune to preserve them from being dropped.
• Changing index config triggers reindexing. Also, do not change aliases of existing indexes in the config file without cleaning up the database first. DipDup won't handle that automatically and will treat the renamed index as new.
• Multiple indexes pointing to different contracts should not reuse the same models (unless you know what you are doing) because synchronization is done sequentially by index.

Schema migration

DipDup does not support database schema migration: if there's any model change, it will trigger reindexing. The rationale is that it's easier and faster to start over than handle migrations that can be of arbitrary complexity and do not guarantee data consistency.

DipDup stores a hash of the SQL version of the DB schema and checks for changes each time you run indexing.

Handling chain reorgs

Reorg messages signaling chain reorganizations. That means some blocks, including all operations, are rolled back in favor of another with higher fitness. Chain reorgs happen regularly (especially in testnets), so it's not something you can ignore. These messages must be handled correctly -- otherwise, you will likely accumulate duplicate or invalid data.

Singe version 6.0 DipDup processes chain reorgs seamlessly restoring a previous database state. You can implement your rollback logic by editing the on_index_rollback event hook.

Creating config

Developing a DipDup indexer begins with creating a YAML config file. You can find a minimal example to start indexing on the Quickstart page.

General structure

DipDup configuration is stored in YAML files of a specific format. By default, DipDup searches for dipdup.yml file in the current working directory, but you can provide any path with a -c CLI option.

DipDup config file consists of several logical blocks:

Header contains two required fields, package and spec_version. They are used to identify the project and the version of the DipDup specification. All other fields in the config are optional.

Inventory specifies contracts that need to be indexed, datasources to fetch data from and the database to store data in.

Index definitions define the index templates that will be used to index the contract data.

Hook definitions define callback functions that will be called manually or on schedule.

Integrations are used to integrate with third-party services.

Tunables affect the behavior of the whole framework.

Merging config files

DipDup allows you to customize the configuration for a specific environment or a workflow. It works similar to docker-compose, but only for top-level sections. If you want to override a nested property, you need to recreate a whole top-level section. To merge several DipDup config files, provide the -c command-line option multiple times:

dipdup -c dipdup.yml -c dipdup.prod.yml run

Run config export command if unsure about the final config used by DipDup.

Full example

Let's put it all together. Config below is an artificial example but contains almost all available options.

spec_version: 1.2
package: my_indexer

database:
  kind: postgres
  host: db
  port: 5432
  user: dipdup
  password: changeme
  database: dipdup
  schema_name: public
  immune_tables:
    - token_metadata
    - contract_metadata

contracts:
  some_dex:
    address: KT1K4EwTpbvYN9agJdjpyJm4ZZdhpUNKB3F6
    typename: quipu_fa12

datasources:
  tzkt_mainnet:
    kind: tzkt
    url: https://api.tzkt.io
  my_api:
    kind: http
    url: https://my_api.local/v1
  ipfs:
    kind: ipfs
    url: https://ipfs.io/ipfs
  coinbase:
    kind: coinbase
  metadata:
    kind: metadata
    url: https://metadata.dipdup.net
    network: mainnet

indexes:
  operation_index_from_template:
    template: operation_template
    values:
      datasource: tzkt
      contract: some_dex

  big_map_index_from_template:
    template: big_map_template
    values:
      datasource: tzkt
      contract: some_dex
    first_level: 1
    last_level: 46963
    skip_history: never

  factory:
    kind: operation
    datasource: tzkt
    types:
      - origination
    contracts:
      - some_dex
    handlers:
      - callback: on_factory_origination
        pattern:
          - type: origination
            similar_to: some_dex

templates:
  operation_template:
    kind: operation
    datasource: <datasource>
    types:
      - origination
      - transaction
    contracts:
      - <contract>
    handlers:
      - callback: on_origination
        pattern:
          - type: origination
            originated_contract: <contract>
      - callback: on_some_call
        pattern:
          - type: transaction
            destination: <contract>
            entrypoint: some_call

  big_map_template:
    kind: big_map
    datasource: <datasource>
    handlers:
      - callback: on_update_records
        contract: <name_registry>
        path: store.records
      - callback: on_update_expiry_map
        contract: <name_registry>
        path: store.expiry_map


hooks:
  calculate_stats:
    callback: calculate_stats
    atomic: False
    args:
     major: bool

jobs:
  midnight_stats:
    hook: calculate_stats
    crontab: "0 0 * * *"
    args:
      major: True

sentry:
  dsn: https://localhost
  environment: dev
  debug: False

prometheus:
  host: 0.0.0.0

hasura:
  url: http://hasura:8080
  admin_secret: changeme
  allow_aggregations: False
  camel_case: true
  select_limit: 100

advanced:
  early_realtime: True
  merge_subscriptions: False
  postpone_jobs: False
  metadata_interface: False
  skip_version_check: False
  scheduler:
    apscheduler.job_defaults.coalesce: True
    apscheduler.job_defaults.max_instances: 3
  reindex:
    manual: wipe
    migration: exception
    rollback: ignore
    config_modified: exception
    schema_modified: exception
  rollback_depth: 2
  crash_reporting: False

logging: verbose

Project structure

The structure of the DipDup project package is the following:

demo_tzbtc
├── graphql
├── handlers
│   ├── __init__.py
│   ├── on_mint.py
│   └── on_transfer.py
├── hooks
│   ├── __init__.py
│   ├── on_reindex.py
│   ├── on_restart.py
│   ├── on_index_rollback.py
│   └── on_synchronized.py
├── __init__.py
├── models.py
├── sql
│   ├── on_reindex
│   ├── on_restart
│   ├── on_index_rollback
│   └── on_synchronized
└── types
    ├── __init__.py
    └── tzbtc
        ├── __init__.py
        ├── parameter
        │   ├── __init__.py
        │   ├── mint.py
        │   └── transfer.py
        └── storage.py

DipDup will generate all the necessary directories and files inside the project's root on init command. These include contract type definitions and callback stubs to be implemented by the developer.

Type classes

DipDup receives all smart contract data (transaction parameters, resulting storage, big_map updates) in normalized form (read more about how TzKT handles Michelson expressions) but still as raw JSON. DipDup uses contract type information to generate data classes, which allow developers to work with strictly typed data.

DipDup generates Pydantic models out of JSONSchema. You might want to install additional plugins (PyCharm, mypy) for convenient work with this library.

The following models are created at init:

• operation indexes: storage type for all contracts met in handler patterns plus parameter type for all destination+entrypoint pairs.
• big_map indexes: key and storage types for all big map paths in handler configs.

Nested packages

Callback modules don't have to be in top-level hooks/handlers directories. Add one or multiple dots to the callback name to define nested packages:

package: indexer
hooks:
  foo.bar:
    callback: foo.bar

After running the init command, you'll get the following directory tree (shortened for readability):

indexer
├── hooks
│   ├── foo
│   │   ├── bar.py
│   │   └── __init__.py
│   └── __init__.py
└── sql
    └── foo
        └── bar
            └── .keep

The same rules apply to handler callbacks. Note that the callback field must be a valid Python package name - lowercase letters, underscores, and dots.

💡 SEE ALSO

• 2.5. Defining models
• 2.6. Implementing handlers
• 3.2. Hooks
• 3.7. SQL scripts
• 4.1. Hasura integration

Defining models

DipDup uses the Tortoise ORM library to cover database operations. During initialization, DipDup generates a models.py file on the top level of the package that will contain all database models. The name and location of this file cannot be changed.

A typical models.py file looks like the following (example from demo_tezos_domains package):

from typing import Optional

from tortoise import ForeignKeyFieldInstance
from tortoise import fields

from dipdup.models import Model


class TLD(Model):
    id = fields.CharField(max_length=255, pk=True)
    owner = fields.CharField(max_length=36)


class Domain(Model):
    id = fields.CharField(max_length=255, pk=True)
    tld: ForeignKeyFieldInstance[TLD] = fields.ForeignKeyField('models.TLD', 'domains')
    expiry = fields.DatetimeField(null=True)
    owner = fields.CharField(max_length=36)
    token_id = fields.BigIntField(null=True)

    tld_id: Optional[str]


class Record(Model):
    id = fields.CharField(max_length=255, pk=True)
    domain: ForeignKeyFieldInstance[Domain] = fields.ForeignKeyField('models.Domain', 'records')
    address = fields.CharField(max_length=36, null=True)

See the links below to learn how to use this library.

Limitations

There are some limitations introduced to make Hasura GraphQL integration easier.

• Table names must be in snake_case
• Model fields must be in snake_case
• Model fields must differ from table name

💡 SEE ALSO

• Tortoise ORM documentation
• Tortoise ORM examples
• 5.1. Database engines
• 5.6. Backup and restore

Implementing handlers

DipDup generates a separate file with a callback stub for each handler in every index specified in the configuration file.

In the case of the transaction handler, the callback method signature is the following:

from <package>.types.<typename>.parameter.<entrypoint_1> import EntryPoint1Parameter
from <package>.types.<typename>.parameter.<entrypoint_n> import EntryPointNParameter
from <package>.types.<typename>.storage import TypeNameStorage


async def on_transaction(
    ctx: HandlerContext,
    entrypoint_1: Transaction[EntryPoint1Parameter, TypeNameStorage],
    entrypoint_n: Transaction[EntryPointNParameter, TypeNameStorage]
) -> None:
    ...

where:

• entrypoint_1 ... entrypoint_n are items from the according to handler pattern.
• ctx: HandlerContext provides useful helpers and contains an internal state (see ).
• A Transaction model contains transaction typed parameter and storage, plus other fields.

For the origination case, the handler signature will look similar:

from <package>.types.<typename>.storage import TypeNameStorage


async def on_origination(
    ctx: HandlerContext,
    origination: Origination[TypeNameStorage],
)

An Origination model contains the origination script, initial storage (typed), amount, delegate, etc.

A Big_map update handler will look like the following:

from <package>.types.<typename>.big_map.<path>_key import PathKey
from <package>.types.<typename>.big_map.<path>_value import PathValue


async def on_update(
    ctx: HandlerContext,
    update: BigMapDiff[PathKey, PathValue],
)

BigMapDiff contains action (allocate, update, or remove), nullable key and value (typed).

You can safely change argument names (e.g., in case of collisions).

Naming conventions

Python language requires all module and function names in snake case and all class names in pascal case.

A typical imports section of big_map handler callback looks like this:

from <package>.types.<typename>.storage import TypeNameStorage
from <package>.types.<typename>.parameter.<entrypoint> import EntryPointParameter
from <package>.types.<typename>.big_map.<path>_key import PathKey
from <package>.types.<typename>.big_map.<path>_value import PathValue

Here typename is defined in the contract inventory, entrypoint is specified in the handler pattern, and path is in the handler config.

DipDup does not automatically handle name collisions. Use import ... as if multiple contracts have entrypoints that share the same name:

from <package>.types.<typename>.parameter.<entrypoint> import EntryPointParameter as Alias

Templates and variables

Index templates

Templates allow you to reuse index configuration, e.g., for different networks (mainnet/testnet) or multiple contracts sharing the same codebase.

templates:
  my_template:
    kind: operation
    datasource: <datasource>
    contracts:
      - <contract>
    handlers:
      - callback: callback
        pattern:
          - destination: <contract>
            entrypoint: call

Templates have the same syntax as indexes of all kinds; the only difference is that they additionally support placeholders enabling parameterization:

field: <placeholder>

Template above can be resolved in a following way:

contracts:
  some_dex: ...

datasources:
  tzkt: ...

indexes:
  my_template_instance:
    template: my_template
    values:
      datasource: tzkt_mainnet
      contract: some_dex

Any string value wrapped in angle brackets is treated as a placeholder, so make sure there are no collisions with the actual values. You can use a single placeholder multiple times.

Any index implementing a template must have a value for each existing placeholder; the exception raised otherwise. These values are available in the handler context at ctx.template_values.

Environment variables

DipDup supports compose-style variable expansion with optional default value:

database:
  ...
  password: ${POSTGRES_PASSWORD:-changeme}

You can use environment variables throughout the configuration file, except for property names (YAML object keys).

💡 SEE ALSO

• 3.11. Index factories
• 14.16. templates

Advanced usage

In this section, you will find information about advanced DipDup features.

Datasources

Datasources are DipDup connectors to various APIs. The table below shows how different datasources can be used.

Index datasource is the one used by DipDup internally to process specific index (set with datasource: ... in config). Currently, it can be only tzkt. Datasources available in context can be accessed in handlers and hooks via ctx.get_<kind>_datasource() methods and used to perform arbitrary requests. Finally, standalone services implement a subset of DipDup datasources and config directives. You can't use services-specific datasources like tezos-node in the main framework, they are here for informational purposes only.

✴ required ✅ supported ❌ not supported

TzKT

TzKT provides REST endpoints to query historical data and SignalR (Websocket) subscriptions to get realtime updates. Flexible filters allow you to request only data needed for your application and drastically speed up the indexing process.

datasources:
  tzkt_mainnet:
    kind: tzkt
    url: https://api.tzkt.io

The number of items in each request can be configured with batch_size directive. Affects request number and memory usage.

datasources:
  tzkt_mainnet:
    http:
      ...
      batch_size: 10000

The rest HTTP tunables are the same as for other datasources.

Also, you can wait for several block confirmations before processing the operations:

datasources:
  tzkt_mainnet:
    ...
    buffer_size: 1  # indexing with a single block lag

Since 6.0 chain reorgs are processed automatically, but you may find this feature useful for other cases.

Tezos node

Tezos RPC is a standard interface provided by the Tezos node. This datasource is used solely by mempool and metadata standalone services; you can't use it in regular DipDup indexes.

datasources:
  tezos_node_mainnet:
    kind: tezos-node
    url: https://mainnet-tezos.giganode.io

Coinbase

A connector for Coinbase Pro API. Provides get_candles and get_oracle_data methods. It may be useful in enriching indexes of DeFi contracts with off-chain data.

datasources:
  coinbase:
    kind: coinbase

Please note that Coinbase can't replace TzKT being an index datasource. But you can access it via ctx.datasources mapping both within handler and job callbacks.

DipDup Metadata

dipdup-metadata is a standalone companion indexer for DipDup written in Go. Configure datasource in the following way:

datasources:
  metadata:
    kind: metadata
    url: https://metadata.dipdup.net
    network: mainnet | ithacanet

Then, in your hook or handler code:

datasource = ctx.get_metadata_datasource('metadata')
token_metadata = await datasource.get_token_metadata('KT1...', '0')

IPFS

While working with contract/token metadata, a typical scenario is to fetch it from IPFS. DipDup has a separate datasource to perform such requests via public nodes.

datasources:
  ipfs:
    kind: ipfs
    url: https://ipfs.io/ipfs

You can use this datasource within any callback. Output is either JSON or binary data.

ipfs = ctx.get_ipfs_datasource('ipfs')

file = await ipfs.get('QmdCz7XGkBtd5DFmpDPDN3KFRmpkQHJsDgGiG16cgVbUYu')
assert file[:4].decode()[1:] == 'PDF'

file = await ipfs.get('QmSgSC7geYH3Ae4SpUHy4KutxqNH9ESKBGXoCN4JQdbtEz/package.json')
assert file['name'] == 'json-buffer'

HTTP (generic)

If you need to perform arbitrary requests to APIs not supported by DipDup, use generic HTTP datasource instead of plain aiohttp requests. That way you can use the same features DipDup uses for internal requests: retry with backoff, rate limiting, Prometheus integration etc.

datasources:
  my_api:
    kind: http
    url: https://my_api.local/v1

api = ctx.get_http_datasource('my_api')
response = await api.request(
    method='get',
    url='hello',  # relative to URL in config
    weigth=1,  # ratelimiter leaky-bucket drops
    params={
      'foo': 'bar',
    },
)

All DipDup datasources are inherited from http, so you can send arbitrary requests with any datasource. Let's say you want to fetch the protocol of the chain you're currently indexing (tzkt datasource doesn't have a separate method for it):

tzkt = ctx.get_tzkt_datasource('tzkt_mainnet')
protocol_json = await tzkt.request(
    method='get',
    url='v1/protocols/current',
)
assert protocol_json['hash'] == 'PtHangz2aRngywmSRGGvrcTyMbbdpWdpFKuS4uMWxg2RaH9i1qx'

Datasource HTTP connection parameters (ratelimit, retry with backoff, etc.) are applied on every request.

💡 SEE ALSO

• 14.6. datasources

Hooks

Hooks are user-defined callbacks called either from the ctx.fire_hook method or by scheduler (jobs config section, we'll return to this topic later).

Let's assume we want to calculate some statistics on-demand to avoid blocking an indexer with heavy computations. Add the following lines to DipDup config:

hooks:
  calculate_stats:
    callback: calculate_stats
    atomic: False
    args:
     major: bool
     depth: int

A couple of things here to pay attention to:

• An atomic option defines whether hook callback will be wrapped in a single SQL transaction or not. If this option is set to true main indexing loop will be blocked until hook execution is complete. Some statements like REFRESH MATERIALIZED VIEW do not require to be wrapped in transactions, so choosing a value of the atomic option could decrease the time needed to perform initial indexing.
• Values of args mapping are used as type hints in a signature of a generated callback. We will return to this topic later in this article.

Now it's time to call dipdup init. The following files will be created in the project's root:

├── hooks
│   └── calculate_stats.py
└── sql
    └── calculate_stats
        └── .keep

Content of the generated callback stub:

from dipdup.context import HookContext

async def calculate_stats(
    ctx: HookContext,
    major: bool,
    depth: int,
) -> None:
    await ctx.execute_sql('calculate_stats')

By default, hooks execute SQL scripts from the corresponding subdirectory of sql/. Remove or comment out the execute_sql call to prevent this. This way, both Python and SQL code may be executed in a single hook if needed.

💡 SEE ALSO

• 14.8. hooks

Event hooks

Every DipDup project has multiple event hooks (previously "default hooks"); they fire on system-wide events and, like regular hooks, are not linked to any index. Names of those hooks are reserved; you can't use them in config. It's also impossible to fire them manually or with a job scheduler.

on_restart

This hook executes right before starting indexing. It allows configuring DipDup in runtime based on data from external sources. Datasources are already initialized at execution and available at ctx.datasources. You can, for example, configure logging here or add contracts and indexes in runtime instead of from static config.

on_reindex

This hook fires after the database are re-initialized after reindexing (wipe). Helpful in modifying schema with arbitrary SQL scripts before indexing.

on_synchronized

This hook fires when every active index reaches a realtime state. Here you can clear caches internal caches or do other cleanups.

on_index_rollback

Fires when TzKT datasource has received a chain reorg message which can't be processed by dropping buffered messages (buffer_size option).

Since version 6.0 this hook performs a database-level rollback by default. If it doesn't work for you for some reason remove ctx.rollback call and implement your own rollback logic.

💡 SEE ALSO

• 3.5. Reindexing
• 3.7. SQL scripts
• 3.9. Callback context (ctx)

Job scheduler

Jobs are schedules for hooks. In some cases, it may come in handy to have the ability to run some code on schedule. For example, you want to calculate statistics once per hour instead of every time handler gets matched.

Arguments typechecking

DipDup will ensure that arguments passed to the hooks have correct types when possible. CallbackTypeError exception will be raised otherwise. Values of an args mapping in a hook config should be either built-in types or __qualname__ of external type like decimal.Decimal. Generic types are not supported: hints like Optional[int] = None will be correctly parsed during codegen but ignored on type checking.

See 12.8. jobs for details.

Scheduler configuration

DipDup utilizes apscheduler library to run hooks according to schedules in jobs config section. In the following example, apscheduler will spawn up to three instances of the same job every time the trigger is fired, even if previous runs are in progress:

advanced:
  scheduler:
    apscheduler.job_defaults.coalesce: True
    apscheduler.job_defaults.max_instances: 3

See apscheduler docs for details.

Note that you can't use executors from apscheduler.executors.pool module - ConfigurationError exception will be raised.

Reindexing

In some cases, DipDup can't proceed with indexing without a full wipe. Several reasons trigger reindexing; some are avoidable, some are not:

It is possible to configure desirable action on reindexing triggered by the specific reason.

To configure actions for each reason, add the following section to the DipDup config:

advanced:
  ...
  reindex:
    manual: wipe
    migration: exception
    rollback: ignore
    config_modified: exception
    schema_modified: exception

Feature flags

Feature flags allow users to modify some system-wide tunables that affect the behavior of the whole framework. These options are either experimental or unsuitable for generic configurations.

A good practice is to use set feature flags in environment-specific config files.

advanced:
  early_realtime: False
  merge_subscriptions: False
  postpone_jobs: False
  metadata_interface: False
  skip_version_check: False
  crash_reporting: False

Early realtime

By default, DipDup enters a sync state twice: before and after establishing a realtime connection. This flag allows starting collecting realtime messages while sync is in progress, right after indexes load.

Let's consider two scenarios:

1. Indexing 10 contracts with 10 000 operations each. Initial indexing could take several hours. There is no need to accumulate incoming operations since resync time after establishing a realtime connection depends on the contract number, thus taking a negligible amount of time.

2. Indexing 10 000 contracts with 10 operations each. Both initial sync and resync will take a while. But the number of operations received during this time won't affect RAM consumption much.

If you do not have strict RAM constraints, it's recommended to enable this flag. You'll get faster indexing times and decreased load on TzKT API.

Merge subscriptions

Subscribe to all operations/big map diffs during realtime indexing instead of separate channels. This flag helps to avoid the 10.000 subscription limit of TzKT and speed up processing. The downside is an increased RAM consumption during sync, especially if early_realtimm flag is enabled too.

Postpone jobs

Do not start the job scheduler until all indexes are synchronized. If your jobs perform some calculations that make sense only after indexing is fully finished, this toggle can save you some IOPS.

Metadata interface

Without this flag calling ctx.update_contract_metadata and ctx.update_token_metadata will make no effect. Corresponding internal tables are created on reindexing in any way.

Skip version check

Disables warning about running unstable or out-of-date DipDup version.

Crash reporting

Enables sending crash reports to the Baking Bad team. This is disabled by default. You can inspect crash dumps saved as /tmp/dipdup/crashdumps/XXXXXXX.json before enabling this option.

SQL scripts

Put your *.sql scripts to <package>/sql. You can run these scripts from any callback with ctx.execute_sql('name'). If name is a directory, each script it contains will be executed.

Scripts are executed without being wrapped with SQL transactions. It's generally a good idea to avoid touching table data in scripts.

SQL scripts are ignored if SQLite is used as a database backend.

By default, an empty sql/<hook_name> directory is generated for every hook in config during init. Remove ctx.execute_sql call from hook callback to avoid executing them.

Event hooks

Scripts from sql/on_restart directory are executed each time you run DipDup. Those scripts may contain CREATE OR REPLACE VIEW or similar non-destructive operations.

Scripts from sql/on_reindex directory are executed after the database schema is created based on the models.py module but before indexing starts. It may be useful to change the database schema in ways that are not supported by the Tortoise ORM, e.g., to create a composite primary key.

💡 SEE ALSO

• 3.3. Event hooks

Improving performance

This page contains tips that may help to increase indexing speed.

Optimize database schema

Postgres indexes are tables that Postgres can use to speed up data lookup. A database index acts like a pointer to data in a table, just like an index in a printed book. If you look in the index first, you will find the data much quicker than searching the whole book (or — in this case — database).

You should add indexes on columns often appearing in `WHERE`` clauses in your GraphQL queries and subscriptions.

Tortoise ORM uses BTree indexes by default. To set index on a field, add index=True to the field definition:

from tortoise import Model, fields


class Trade(Model):
    id = fields.BigIntField(pk=True)
    amount = fields.BigIntField()
    level = fields.BigIntField(index=True)
    timestamp = fields.DatetimeField(index=True)

Tune datasources

All datasources now share the same code under the hood to communicate with underlying APIs via HTTP. Configs of all datasources and also Hasura's one can have an optional section http with any number of the following parameters set:

datasources:
  tzkt:
    kind: tzkt
    ...
    http:
      retry_count: 10
      retry_sleep: 1
      retry_multiplier: 1.2
      ratelimit_rate: 100
      ratelimit_period: 60
      connection_limit: 25
      batch_size: 10000
hasura:
  url: http://hasura:8080
  http:
    ...

Each datasource has its defaults. Usually, there's no reason to alter these settings unless you use self-hosted instances of TzKT or other datasource.

By default, DipDup retries failed requests infinitely, exponentially increasing the delay between attempts. Set retry_count parameter to limit the number of attempts.

batch_size parameter is TzKT-specific. By default, DipDup limit requests to 10000 items, the maximum value allowed on public instances provided by Baking Bad. Decreasing this value will reduce the time required for TzKT to process a single request and thus reduce the load. By reducing the connection_limit parameter, you can achieve the same effect (limited to synchronizing multiple indexes concurrently).

💡 SEE ALSO

• Tortoise ORM documentation
• Tortoise ORM examples
• 5.1. Database engines
• 5.6. Backup and restore

See 12.4. datasources for details.

Use TimescaleDB for time-series

🚧 UNDER CONSTRUCTION

This page or paragraph is yet to be written. Come back later.

DipDup is fully compatible with TimescaleDB. Try its "continuous aggregates" feature, especially if dealing with market data like DEX quotes.

Cache commonly used models

If your indexer contains models having few fields and used primarily on relations, you can cache such models during synchronization.

Example code:

class Trader(Model):
    address = fields.CharField(36, pk=True)


class TraderCache:
    def __init__(self, size: int = 1000) -> None:
        self._size = size
        self._traders: OrderedDict[str, Trader] = OrderedDict()

    async def get(self, address: str) -> Trader:
        if address not in self._traders:
            # NOTE: Already created on origination
            self._traders[address], _ = await Trader.get_or_create(address=address)
              if len(self._traders) > self._size:
                self._traders.popitem(last=False)

        return self._traders[address]

trader_cache = TraderCache()

Use trader_cache.get in handlers. After sync is complete, you can clear this cache to free some RAM:

async def on_synchronized(
    ctx: HookContext,
) -> None:
    ...
    models.trader_cache.clear()

Perform heavy computations in separate processes

It's impossible to use apscheduler pool executors with hooks because HookContext is not pickle-serializable. So, they are forbidden now in advanced.scheduler config. However, thread/process pools can come in handy in many situations, and it would be nice to have them in DipDup context. For now, I can suggest implementing custom commands as a workaround to perform any resource-hungry tasks within them. Put the following code in <project>/cli.py:

from contextlib import AsyncExitStack

import asyncclick as click
from dipdup.cli import cli, cli_wrapper
from dipdup.config import DipDupConfig
from dipdup.context import DipDupContext
from dipdup.utils.database import tortoise_wrapper


@cli.command(help='Run heavy calculations')
@click.pass_context
@cli_wrapper
async def do_something_heavy(ctx):
    config: DipDupConfig = ctx.obj.config
    url = config.database.connection_string
    models = f'{config.package}.models'

    async with AsyncExitStack() as stack:
        await stack.enter_async_context(tortoise_wrapper(url, models))
        ...

if __name__ == '__main__':
    cli(prog_name='dipdup', standalone_mode=False)  # type: ignore

Then use python -m <project>.cli instead of dipdup as an entrypoint. Now you can call do-something-heavy like any other dipdup command. dipdup.cli:cli group handles arguments and config parsing, graceful shutdown, and other boilerplate. The rest is on you; use dipdup.dipdup:DipDup.run as a reference. And keep in mind that Tortoise ORM is not thread-safe. I aim to implement ctx.pool_apply and ctx.pool_map methods to execute code in pools with magic within existing DipDup hooks, but no ETA yet.

Callback context (ctx)

An instance of the HandlerContext class is passed to every handler providing a set of helper methods and read-only properties.

Reference

class dipdup.context.DipDupContext(datasources: Dict[str, Datasource], config: DipDupConfig, callbacks: CallbackManager, transactions: TransactionManager)¶

Common execution context for handler and hook callbacks.

Parameters:

• datasources – Mapping of available datasources

• config – DipDup configuration

• logger – Context-aware logger instance

async add_contract(name: str, address: str, typename: Optional[str] = None) → None¶

Adds contract to the inventory.

Parameters:

• name – Contract name

• address – Contract address

• typename – Alias for the contract script

async add_index(name: str, template: str, values: Dict[str, Any], state: Optional[Index] = None) → None¶

Adds a new contract to the inventory.

Parameters:

• name – Index name

• template – Index template to use

• values – Mapping of values to fill template with

async execute_sql(name: str) → None¶

Executes SQL script(s) with given name.

If the name path is a directory, all .sql scripts within it will be executed in alphabetical order.

Parameters:

name – File or directory within project’s sql directory

async fire_hook(name: str, fmt: Optional[str] = None, wait: bool = True, *args, **kwargs: Any) → None¶

Fire hook with given name and arguments.

Parameters:

• name – Hook name

• fmt – Format string for ctx.logger messages

• wait – Wait for hook to finish or fire and forget

get_coinbase_datasource(name: str) → CoinbaseDatasource¶

Get coinbase datasource by name

get_http_datasource(name: str) → HttpDatasource¶

Get http datasource by name

get_ipfs_datasource(name: str) → IpfsDatasource¶

Get ipfs datasource by name

get_metadata_datasource(name: str) → MetadataDatasource¶

Get metadata datasource by name

get_tzkt_datasource(name: str) → TzktDatasource¶

Get tzkt datasource by name

async reindex(reason: Optional[Union[str, ReindexingReason]] = None, **context) → None¶

Drops the entire database and starts the indexing process from scratch.

Parameters:

• reason – Reason for reindexing in free-form string

• context – Additional information to include in exception message

async restart() → None¶

Restart process and continue indexing.

async update_contract_metadata(network: str, address: str, metadata: Dict[str, Any]) → None¶

Inserts or updates corresponding rows in the internal dipdup_contract_metadata table to provide a generic metadata interface (see docs).

Parameters:

• network – Network name (e.g. mainnet)

• address – Contract address

• metadata – Contract metadata to insert/update

async update_token_metadata(network: str, address: str, token_id: str, metadata: Dict[str, Any]) → None¶

Inserts or updates corresponding rows in the internal dipdup_token_metadata table to provide a generic metadata interface (see docs).

Parameters:

• network – Network name (e.g. mainnet)

• address – Contract address

• token_id – Token ID

• metadata – Token metadata to insert/update

class dipdup.context.HandlerContext(datasources: Dict[str, Datasource], config: DipDupConfig, callbacks: CallbackManager, transactions: TransactionManager, logger: FormattedLogger, handler_config: HandlerConfig, datasource: TzktDatasource)¶

Execution context of handler callbacks.

Parameters:

• handler_config – Configuration of current handler

• datasource – Index datasource instance

class dipdup.context.HookContext(datasources: Dict[str, Datasource], config: DipDupConfig, callbacks: CallbackManager, transactions: TransactionManager, logger: FormattedLogger, hook_config: HookConfig)¶

Execution context of hook callbacks.

Parameters:

hook_config – Configuration of current hook

async rollback(index: str, from_level: int, to_level: int) → None¶

Rollback index to a given level reverting all changes made since that level.

Parameters:

• index – Index name

• from_level – Level to rollback from

• to_level – Level to rollback to

Internal models

This page describes the internal models used by DipDup. You shouldn't modify data in these models directly.

With the help of these tables, you can set up monitoring of DipDup deployment to know when something goes wrong:

-- This query will return time since the latest block was received by a datasource.
SELECT NOW() - timestamp FROM dipdup_head;

Index factories

DipDup allows creating new indexes in runtime. To begin with, you need to define index templates in the top-level templates section of the config. Then call ctx.add_contract and ctx.add_index methods from any user callback.

DipDup is currently not able to automatically generate types and handlers for template indexes unless there is at least one static instance (14.9.4. template). Add it temporarily setting template values manually to call dipdup init command.

The most common way to spawn indexes is to create an index that tracks the originations of contracts with similar code or originated by a specific contract. A minimal example looks like this:

contracts:
  registry:
    address: KT19CF3KKrvdW77ttFomCuin2k4uAVkryYqh

indexes:
  factory:
    kind: operation
    datasource: tzkt
    types:
      - origination
    handlers:
      - callback: on_factory_origination
        pattern:
          - type: origination
            similar_to: registry

Another solution is to implement custom logic in on_restart hook (see 3.3. Event hooks → on_restart)

💡 SEE ALSO

• 3.9. Callback context (ctx)
• 14.16. templates

Metadata Interface

When issuing a token on Tezos blockchain, there is an important yet not enough covered aspect related to how various ecosystem applications (wallets, explorers, marketplaces, and others) will display and interact with it. It's about token metadata, stored wholly or partially on-chain but intended for off-chain use only.

Token metadata standards

There are several standards regulating the metadata file format and the way it can be stored and exposed to consumers:

• TZIP-21 | Rich Metadata — describes a metadata schema and standards for contracts and tokens
• TZIP-12 | FA2.0 — a standard for a unified token contract interface, includes an article about how to store and encode token metadata
• TZIP-7 | FA1.2 — single asset token standard; reuses the token metadata approach from FA2.0

Keeping aside the metadata schema, let's focus on which approaches for storing are currently standardized, their pros and cons, and what to do if any of the options available do not fit your case.

Basic: on-chain links / off-chain storage

The most straightforward approach is to store everything in the contract storage, especially if it's just the basic fields (name, symbol, decimals):

storage
└── token_metadata [big_map]
    └── 0
        ├── token_id: 0
        └── token_info
            ├── name: ""
            ├── symbol: ""
            └── decimals: ""

But typically, you want to store more like a token thumbnail icon, and it is no longer feasible to keep such large data on-chain (because you pay gas for every byte stored).
Then you can put large files somewhere off-chain (e.g., IPFS) and store just links:

storage
└── token_metadata [big_map]
    └── 0
        ├── token_id: 0
        └── token_info
            ├── ...
            └── thumbnailUri: "ipfs://"

This approach is still costly, but sometimes (in rare cases), you need to have access to the metadata from the contract (example: Dogami).
We can go further and put the entire token info structure to IPFS:

storage
└── token_metadata [big_map]
    └── 0
        ├── token_id: 0
        └── token_info
            └── "": "ipfs://"

It is the most common case right now (example: HEN).

The main advantage of the basic approach is that all the changes applied to token metadata will result in big map diffs that are easily traceable by indexers. Even if you decide to replace the off-chain file, it will cause the IPFS link to change. In the case of HTTP links, indexers cannot detect the content change; thus, token metadata won't be updated.

Custom: off-chain view

The second approach presented in the TZIP-12 spec was intended to cover the cases when there's a need in reusing the same token info or when it's not possible to expose the %token_metadata big map in the standard form. Instead, it's offered to execute a special Michelson script against the contract storage and treat the result as the token info for a particular token (requested). The tricky part is that the script code itself is typically stored off-chain, and the whole algorithm would look like this:

1. Try to fetch the empty string key of the %metadata big map to retrieve the TZIP-16 file location
2. Resolve the TZIP-16 file (typically from IPFS) — it should contain the off-chain view body
3. Fetch the current contract storage
4. Build arguments for the off-chain view token_metadata using fetched storage and
5. Execute the script using Tezos node RPC

Although this approach is more or less viable for wallets (when you need to fetch metadata for a relatively small amount of tokens), it becomes very inefficient for indexers dealing with millions of tokens:

• After every contract origination, one has to try to fetch the views (even if there aren't any) — it means synchronous fetching, which can take seconds in the case of IPFS
• Executing a Michelson script is currently only* possible via Tezos node, and it's quite a heavy call (setting up the VM and contract context takes time)
• There's no clear way to detect new token metadata addition or change — that is actually the most critical one; you never know for sure when to call the view

Off-chain view approach is not supported by TzKT indexer, and we strongly recommend not to use it, especially for contracts that can issue multiple tokens.

DipDup-based solution

The alternative we offer for the very non-standard cases is using our selective indexing framework for custom token metadata retrieval and then feeding it back to the TzKT indexer, which essentially acts as a metadata aggregator. Note that while this can seem like a circular dependency, it's resolved on the interface level: all custom DipDup metadata indexers should expose specific GraphQL tables with certain fields:

query MyQuery {
  token_metadata() {
    metadata    // TZIP-21 JSON
    network     // mainnet or <protocol>net
    contract    // token contract address
    token_id    // token ID in the scope of the contract
    update_id   // integer cursor used for pagination
  }
}

DipDup handles table management for you and exposes a context-level helper.

Tezos Domains example:

await ctx.update_token_metadata(
    network=ctx.datasource.network,
    address=store_records.data.contract_address,
    token_id=token_id,
    metadata={
        'name': record_name,
        'symbol': 'TD',
        'decimals': '0',
        'isBooleanAmount': True,
        'domainData': decode_domain_data(store_records.value.data)
    },
)

TzKT can be configured to subscribe to one or multiple DipDup metadata sources, currently we use in production:

• Generic TZIP-16/TZIP-12 metadata indexer Github | Playground
• Tezos Domains metadata indexer Github | Playground
• Ubisoft Quartz metadata indexer Github | Playground

GraphQL API

In this section, we assume you use Hasura GraphQL Engine integration to power your API.

Before starting to do client integration, it's good to know the specifics of Hasura GraphQL protocol implementation and the general state of the GQL ecosystem.

Queries

By default, Hasura generates three types of queries for each table in your schema:

• Generic query enabling filters by all columns
• Single item query (by primary key)
• Aggregation query (can be disabled)

All the GQL features such as fragments, variables, aliases, directives are supported, as well as batching.
Read more in Hasura docs.

It's important to understand that GraphQL query is just a POST request with JSON payload, and in some instances, you don't need a complicated library to talk to your backend.

Pagination

By default, Hasura does not restrict the number of rows returned per request, which could lead to abuses and heavy load to your server. You can set up limits in the configuration file. See 12.5. hasura for details. But then you will face the need to paginate over the items if the response does not fit into the limits.

Subscriptions

From Hasura documentation:

Hasura GraphQL engine subscriptions are live queries, i.e., a subscription will return the latest result of the query and not necessarily all the individual events leading up to it.

This feature is essential to avoid complex state management (merging query results and subscription feed). In most scenarios, live queries are what you need to sync the latest changes from the backend.

⚠ WARNING

If the live query has a significant response size that does not fit into the limits, you need one of the following:

1. Paginate with offset (which is not convenient)
2. Use cursor-based pagination (e.g., by an increasing unique id).
3. Narrow down request scope with filtering (e.g., by timestamp or level).

Ultimately you can get "subscriptions" on top of live quires by requesting all the items having ID greater than the maximum existing or all the items with a timestamp greater than now.

Websocket transport

Hasura is compatible with subscriptions-transport-ws library, which is currently deprecated by still used by the majority of the clients.

Mutations

The purpose of DipDup is to create indexers, which means you can consistently reproduce the state as long as data sources are accessible. It makes your backend "stateless" in a sense because it's tolerant of data loss.

However, you might need to introduce a non-recoverable state and mix indexed and user-generated content in some cases. DipDup allows marking these UGC tables "immune", protecting them from being wiped. In addition to that, you will need to set up Hasura Auth and adjust write permissions for the tables (by default, they are read-only).

Lastly, you will need to execute GQL mutations to modify the state from the client side. Read more about how to do that with Hasura.

Hasura integration

DipDup uses this optional section to configure the Hasura engine to track your tables automatically.

hasura:
  url: http://hasura:8080
  admin_secret: ${HASURA_ADMIN_SECRET:-changeme}

Under the hood, DipDup generates Hasura metadata from your DB schema and applies it using Metadata API.

Hasura metadata is all about data representation in GraphQL API. The structure of the database itself is managed solely by Tortoise ORM.

Metadata configuration is idempotent: each time you call run or hasura configure command, DipDup queries the existing schema and does the merge if required. DipDup configures Hasura after reindexing, saves the hash of resulting metadata in the dipdup_schema table, and doesn't touch Hasura until needed.

Database limitations

The current version of Hasura GraphQL Engine treats public and other schemas differently. Table schema.customer becomes schema_customer root field (or schemaCustomer if camel_case option is enabled in DipDup config). Table public.customer becomes customer field, without schema prefix. There's no way to remove this prefix for now. You can track related issue (opens new window)at Hasura's GitHub to know when the situation will change. Since 3.0.0-rc1, DipDup enforces public schema name to avoid ambiguity and issues with the GenQL library. You can still use any schema name if Hasura integration is not enabled.

Authentication

DipDup sets read-only permissions for all tables and enables non-authorized access to the /graphql endpoint.

Limit the number of rows

DipDup creates user role that allows performing queries without authorization. Now you can limit the maximum number of rows such queries return and also disable aggregation queries automatically generated by Hasura:

hasura:
  select_limit: 100

Note that with limits enabled, you have to use either offset or cursor-based pagination on the client-side.

Disable aggregation queries

hasura:
  allow_aggregations: False

Convert field names to camel case

For those of you from the JavaScript world, it may be more familiar to use camelCase for variable names instead of snake_case Hasura uses by default. DipDup now allows to convert all fields in metadata to this casing:

hasura:
  camel_case: true

Now this example query to hic et nunc demo indexer...

query MyQuery {
  hic_et_nunc_token(limit: 1) {
    id
    creator_id
  }
}

...will become this one:

query MyQuery {
  hicEtNuncToken(limit: 1) {
    id
    creatorId
  }
}

All fields auto generated by Hasura will be renamed accordingly: hic_et_nunc_token_by_pk to hicEtNuncTokenByPk, delete_hic_et_nunc_token to deleteHicEtNuncToken and so on. To return to defaults, set camel_case to False and run hasura configure --force.

Keep in mind that "camelcasing" is a separate stage performed after all tables are registered. So during configuration, you can observe fields in snake_case for several seconds even if hasura.camel_case flag is set.

💡 SEE ALSO

• 14.7. hasura
• 13. Command-line reference → hasura-configure

REST endpoints

Hasura 2.0 introduced the ability to expose arbitrary GraphQL queries as REST endpoints. By default, DipDup will generate GET and POST endpoints to fetch rows by primary key for all tables:

curl http://127.0.0.1:8080/api/rest/hicEtNuncHolder?address=tz1UBZUkXpKGhYsP5KtzDNqLLchwF4uHrGjw
{
  "hicEtNuncHolderByPk": {
    "address": "tz1UBZUkXpKGhYsP5KtzDNqLLchwF4uHrGjw"
  }
}

However, there's a limitation dictated by how Hasura parses HTTP requests: only models with primary keys of basic types (int, string, and so on) can be fetched with GET requests. An attempt to fetch model with BIGINT primary key will lead to the error: Expected bigint for variable id got Number. A workaround to fetching any model is to send a POST request containing a JSON payload with a single key:

curl -d '{"id": 152}' http://127.0.0.1:8080/api/rest/hicEtNuncToken
{
  "hicEtNuncTokenByPk": {
    "creatorId": "tz1UBZUkXpKGhYsP5KtzDNqLLchwF4uHrGjw",
    "id": 152,
    "level": 1365242,
    "supply": 1,
    "timestamp": "2021-03-01T03:39:21+00:00"
  }
}

We hope to get rid of this limitation someday and will let you know as soon as it happens.

Custom endpoints

You can put any number of .graphql files into graphql directory in your project's root, and DipDup will create REST endpoints for each of those queries. Let's say we want to fetch not only a specific token, but also the number of all tokens minted by its creator:

query token_and_mint_count($id: bigint) {
  hicEtNuncToken(where: {id: {_eq: $id}}) {
    creator {
      address
      tokens_aggregate {
        aggregate {
          count
        }
      }
    }
    id
    level
    supply
    timestamp
  }
}

Save this query as graphql/token_and_mint_count.graphql and run dipdup configure-hasura. Now, this query is available via REST endpoint at http://127.0.0.1:8080/api/rest/token_and_mint_count.

You can disable exposing of REST endpoints in the config:

hasura:
  rest: False

GenQL

GenQL is a great library and CLI tool that automatically generates a fully typed SDK with a built-in GQL client. It works flawlessly with Hasura and is recommended for DipDup on the client-side.

Project structure

GenQL CLI generates a ready-to-use package, compiled and prepared to publish to NPM. A typical setup is a mono repository containing several packages, including the auto-generated SDK and your front-end application.

project_root/
├── package.json
└── packages/
    ├── app/
    │   ├── package.json
    │   └── src/
    └── sdk/
        └── package.json

SDK package config

Your minimal package.json file will look like the following:

{
  "name": "%PACKAGE_NAME%",
  "version": "0.0.1",
  "main": "dist/index.js",
  "types": "dist/index.d.ts",
  "devDependencies": {
    "@genql/cli": "^2.6.0"
  },
  "dependencies": {
    "@genql/runtime": "2.6.0",
    "graphql": "^15.5.0"
  },
  "scripts": {
    "build": "genql --endpoint %GRAPHQL_ENDPOINT% --output ./dist"
  }
}

That's it! Now you only need to install dependencies and execute the build target:

yarn
yarn build

Read more about CLI options available.

Demo

Create a package.json file with

• %PACKAGE_NAME% => metadata-sdk
• %GRAPHQL_ENDPOINT% => https://metadata.dipdup.net/v1/graphql

And generate the client:

yarn
yarn build

Then create new file index.ts and paste this query:

import { createClient, everything } from './dist'

const client = createClient()

client.chain.query
    .token_metadata({ where: { network: { _eq: 'mainnet' } }})
    .get({ ...everything })
    .then(res => console.log(res))

We need some additional dependencies to run our sample:

yarn add typescript ts-node

Finally:

npx ts-node index.ts

You should see a list of tokens with metadata attached in your console.

Deployment and operations

This section contains recipes to deploy and maintain DipDup instances.

Database engines

DipDup officially supports the following databases: SQLite, PostgreSQL, TimescaleDB. This table will help you choose a database engine that mostly suits your needs.

While sometimes it's convenient to use one database engine for development and another one for production, be careful with specific column types that behave differently in various engines.

💡 SEE ALSO

• 3.7. SQL scripts
• 14.5. database → immune-tables

Running in Docker

Base images

DipDup provides multiple prebuilt images for different environments hosted on Docker Hub. Choose the one according to your needs from the table below.

Default DipDup image is suitable for development and testing. It includes some development tools to make package management easier. If unsure, use this image.

-slim image

This image is based on Alpine Linux and has a much smaller size than the default one. As a tradeoff, it doesn't include codegen functionality (unlikely to be useful in production).

-pytezos image

The only difference with the default image is the pre-installed PyTezos library, the same as pip install dipdup -E pytezos. DipDup doesn't provide any further PyPoetry integration. Having some patience you can build a trading robot or something like that using this image. I don't know if anyone is using it. If you're the one on them, please let us know!

Nightly builds (ghcr.io)

In addition to Docker Hub we also publish images on GitHub Packages. Builds are triggered on push to any branch for developers' convenience. Do not use this registry in production!

# Slim image for `aux/arm64` branch
FROM ghcr.io/dipdup-net/dipdup:aux-arm64-slim

Writing Dockerfile

Start with creating .dockerignore for your project if it's missing.

# Ignore all
*

# Add build files
!Makefile
!pyproject.toml
!poetry.lock
!requirements**
!README.md

# Add code
!src

# Add configs
!*.yml

# Ignore caches
**/.mypy_cache
**/.pytest_cache
**/__pycache__

A typical Dockerfile looks like this:

FROM dipdup/dipdup:6
# FROM dipdup/dipdup:6-pytezos
# FROM dipdup/dipdup:6-slim

# Optional: install additional dependencies using poetry
# COPY pyproject.toml poetry.lock .
# RUN install_dependencies

# Optional: install additional dependencies using pip
# COPY requirements.txt .
# RUN install_dependencies requirements.txt

COPY . .

Note that Poetry integration is not available in the slim image.

Deploying with docker-compose

Make sure you have docker run and docker-compose installed.

Example docker-compose.yml file:

version: "3.8"

services:
  dipdup:
    build: .
    depends_on:
      - db
    command: ["-c", "dipdup.yml", "-c", "dipdup.prod.yml", "run"]
    restart: always
    environment:
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-changeme}
      - ADMIN_SECRET=${ADMIN_SECRET:-changeme}
    ports:
      - 127.0.0.1:9000:9000

  db:
    image: postgres:14
    ports:
      - 127.0.0.1:5432:5432
    volumes:
      - db:/var/lib/postgresql/data
    restart: always
    environment:
      - POSTGRES_USER=dipdup
      - POSTGRES_DB=dipdup
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-changeme}
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U dipdup"]
      interval: 10s
      timeout: 5s
      retries: 5

  hasura:
    image: hasura/graphql-engine:v2.11.2
    ports:
      - 127.0.0.1:8080:8080
    depends_on:
      - db
    restart: always
    environment:
      - HASURA_GRAPHQL_DATABASE_URL=postgres://dipdup:${POSTGRES_PASSWORD:-changeme}@db:5432/dipdup
      - HASURA_GRAPHQL_ENABLE_CONSOLE=true
      - HASURA_GRAPHQL_DEV_MODE=true
      - HASURA_GRAPHQL_ENABLED_LOG_TYPES=startup, http-log, webhook-log, websocket-log, query-log
      - HASURA_GRAPHQL_ADMIN_SECRET=${HASURA_SECRET:-changeme}
      - HASURA_GRAPHQL_UNAUTHORIZED_ROLE=user
      - HASURA_GRAPHQL_STRINGIFY_NUMERIC_TYPES=true

volumes:
  db:

Environment variables are expanded in the DipDup config file; Postgres password and Hasura secret are forwarded in this example.

Create a separate dipdup.<environment>.yml file for this stack:

database:
  kind: postgres
  host: db
  port: 5432
  user: ${POSTGRES_USER:-dipdup}
  password: ${POSTGRES_PASSWORD:-changeme}
  database: ${POSTGRES_DB:-dipdup}

hasura:
  url: http://hasura:8080
  admin_secret: ${HASURA_SECRET:-changeme}
  allow_aggregations: false
  camel_case: true

sentry:
  dsn: ${SENTRY_DSN:-""}
  environment: ${SENTRY_ENVIRONMENT:-prod}

prometheus:
  host: 0.0.0.0

Note the hostnames (resolved in the docker network) and environment variables (expanded by DipDup).

Build and run the containers:

docker-compose up -d --build

We recommend lazydocker for monitoring your application.

Deploying with Docker Swarm

🚧 UNDER CONSTRUCTION

This page or paragraph is yet to be written. Come back later.

Example stack:

version: "3.8"

services:
  dipdup:
    image: ${DOCKER_REGISTRY:-ghcr.io}/dipdup-net/dipdup:${TAG:-master}
    depends_on:
      - db
      - hasura
    command: ["-c", "dipdup.yml", "-c", "dipdup.prod.yml", "run"]
    environment:
      - "POSTGRES_USER=dipdup"
      - "POSTGRES_PASSWORD=changeme"
      - "POSTGRES_DB=dipdup"
      - "HASURA_SECRET=changeme"
    networks:
      - dipdup-private
      - prometheus-private
    deploy:
      mode: replicated
      replicas: ${INDEXER_ENABLED:-1}
      labels:
        - prometheus-job=${SERVICE}
        - prometheus-port=8000
      placement: &placement
        constraints:
          - node.labels.${SERVICE} == true
    logging: &logging
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "10"
        tag: "\{\{.Name\}\}.\{\{.ImageID\}\}"

  db:
    image: postgres:14
    volumes:
      - db:/var/lib/postgresql/data
    environment: 
      - POSTGRES_USER=dipdup
      - POSTGRES_DB=dipdup
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-changeme}
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - dipdup-private
    deploy:
      mode: replicated
      replicas: 1
      placement: *placement
    logging: *logging

  hasura:
    image: hasura/graphql-engine:v2.11.2
    depends_on:
      - db
    environment:
      - HASURA_GRAPHQL_DATABASE_URL=postgres://dipdup:${POSTGRES_PASSWORD:-changeme}@db:5432/dipdup
      - HASURA_GRAPHQL_ENABLE_CONSOLE=true
      - HASURA_GRAPHQL_DEV_MODE=false
      - HASURA_GRAPHQL_ENABLED_LOG_TYPES=startup, http-log, websocket-log, query-log
      - HASURA_GRAPHQL_LOG_LEVEL=warn
      - HASURA_GRAPHQL_ENABLE_TELEMETRY=false
      - HASURA_GRAPHQL_ADMIN_SECRET=${HASURA_SECRET}
      - HASURA_GRAPHQL_UNAUTHORIZED_ROLE=user
      - HASURA_GRAPHQL_STRINGIFY_NUMERIC_TYPES=true
    networks:
      - dipdup-private
      - traefik-public
    deploy:
      mode: replicated
      replicas: 1
      labels:
        - traefik.enable=true
        - traefik.http.services.${SERVICE}.loadbalancer.server.port=8080
        - "traefik.http.routers.${SERVICE}.rule=Host(`${HOST}`) && (PathPrefix(`/v1/graphql`) || PathPrefix(`/api/rest`))"
        - traefik.http.routers.${SERVICE}.entrypoints=http,${INGRESS:-ingress}
        - "traefik.http.routers.${SERVICE}-console.rule=Host(`${SERVICE}.${SWARM_ROOT_DOMAIN}`)"
        - traefik.http.routers.${SERVICE}-console.entrypoints=https
        - traefik.http.middlewares.${SERVICE}-console.headers.customrequestheaders.X-Hasura-Admin-Secret=${HASURA_SECRET}
        - traefik.http.routers.${SERVICE}-console.middlewares=authelia@docker,${SERVICE}-console
      placement: *placement
    logging: *logging


volumes:
  db:

networks:
  dipdup-private:
  traefik-public:
    external: true
  prometheus-private:
    external: true

Sentry integration

Sentry is an error tracking software that can be used either as a service or on-premise. It dramatically improves the troubleshooting experience and requires nearly zero configuration. To start catching exceptions with Sentry in your project, add the following section in dipdup.yml config:

sentry:
  dsn: https://...
  environment: dev
  debug: False

You can obtain Sentry DSN from the web interface at Settings -> Projects -> <project_name> -> Client Keys (DSN). The cool thing is that if you catch an exception and suspect there's a bug in DipDup, you can share this event with us using a public link (created at Share menu).

💡 SEE ALSO

• 3.6. Feature flags → crash-reporting

Prometheus integration

DipDup provides basic integration with the Prometheus monitoring system by exposing some metrics.

When running DipDup in Docker make sure that the Prometheus instance is in the same network.

Available metrics

The following metrics are exposed under dipdup namespace:

You can also query 3.10. Internal models for monitoring purposes.

Logging

To control the number of logs DipDup produces, set the logging field in config:

logging: default|verbose|quiet

If you need more fined tuning, perform it in the on_restart hook:

import logging

async def on_restart(
    ctx: HookContext,
) -> None:
    logging.getLogger('some_logger').setLevel('DEBUG')

Backup and restore

DipDup has no built-in functionality to backup and restore database at the moment. Good news is that DipDup indexes are fully atomic. That means you can perform backup with regular psql/pgdump regardless of the DipDup state.

This page contains several recipes for backup/restore.

Scheduled backup to S3

This example is for Swarm deployments. We use this solution to backup our services in production. Adapt it to your needs if needed.

version: "3.8"
services:
  indexer:
    ...
  db:
    ...
  hasura:
    ...

  backuper:
    image: ghcr.io/dipdup-net/postgres-s3-backup:master
    environment:
      - S3_ENDPOINT=${S3_ENDPOINT:-https://fra1.digitaloceanspaces.com}
      - S3_ACCESS_KEY_ID=${S3_ACCESS_KEY_ID}
      - S3_SECRET_ACCESS_KEY=${S3_SECRET_ACCESS_KEY}
      - S3_BUCKET=dipdup
      - S3_PATH=dipdup
      - S3_FILENAME=${SERVICE}-postgres
      - PG_BACKUP_FILE=${PG_BACKUP_FILE}
      - PG_BACKUP_ACTION=${PG_BACKUP_ACTION:-dump}
      - PG_RESTORE_JOBS=${PG_RESTORE_JOBS:-8}
      - POSTGRES_USER=${POSTGRES_USER:-dipdup}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-changeme}
      - POSTGRES_DB=${POSTGRES_DB:-dipdup}
      - POSTGRES_HOST=${POSTGRES_HOST:-db}
      - HEARTBEAT_URI=${HEARTBEAT_URI}
      - SCHEDULE=${SCHEDULE}
    deploy:
      mode: replicated
      replicas: ${BACKUP_ENABLED:-0}
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 5
        window: 120s
      placement: *placement
    networks:
      - internal
    logging: *logging

Automatic restore on rollback

This awesome code was contributed by @852Kerfunkle, author of tz1and project.

<project>/backups.py

...

def backup(level: int, database_config: PostgresDatabaseConfig):
    ...

    with open('backup.sql', 'wb') as f:
        try:
            err_buf = StringIO()
            pg_dump('-d', f'postgresql://{database_config.user}:{database_config.password}@{database_config.host}:{database_config.port}/{database_config.database}', '--clean',
                '-n', database_config.schema_name, _out=f, _err=err_buf) #, '-E', 'UTF8'
        except ErrorReturnCode:
            err = err_buf.getvalue()
            _logger.error(f'Database backup failed: {err}')


def restore(level: int, database_config: PostgresDatabaseConfig):
    ...

    with open('backup.sql', 'r') as f:
        try:
            err_buf = StringIO()
            psql('-d', f'postgresql://{database_config.user}:{database_config.password}@{database_config.host}:{database_config.port}/{database_config.database}',
                '-n', database_config.schema_name, _in=f, _err=err_buf)
        except ErrorReturnCode:
            err = err_buf.getvalue()
            _logger.error(f'Database restore failed: {err}')
            raise Exception("Failed to restore")

def get_available_backups():
    ...


def delete_old_backups():
    ...

<project>/hooks/on_index_rollback.py

...

async def on_index_rollback(
    ctx: HookContext,
    index: Index,
    from_level: int,
    to_level: int,
) -> None:
    await ctx.execute_sql('on_index_rollback')

    database_config: Union[SqliteDatabaseConfig, PostgresDatabaseConfig] = ctx.config.database

    # if not a postgres db, reindex.
    if database_config.kind != "postgres":
        await ctx.reindex(ReindexingReason.ROLLBACK)

    available_levels = backups.get_available_backups()

    # if no backups available, reindex
    if not available_levels:
        await ctx.reindex(ReindexingReason.ROLLBACK)

    # find the right level. ie the on that's closest to to_level
    chosen_level = 0
    for level in available_levels:
        if level <= to_level and level > chosen_level:
            chosen_level = level

    # try to restore or reindex
    try:
        backups.restore(chosen_level, database_config)
        await ctx.restart()
    except Exception:
        await ctx.reindex(ReindexingReason.ROLLBACK)

<project>/hooks/run_backups.py

...

async def run_backups(
    ctx: HookContext,
) -> None:
    database_config: Union[SqliteDatabaseConfig, PostgresDatabaseConfig] = ctx.config.database

    if database_config.kind != "postgres":
        return

    level = ctx.get_tzkt_datasource("tzkt_mainnet")._level.get(MessageType.head)

    if level is None:
        return

    backups.backup(level, database_config)
    backups.delete_old_backups()

<project>/hooks/simulate_reorg.py

...

async def simulate_reorg(
    ctx: HookContext
) -> None:
    level = ctx.get_tzkt_datasource("tzkt_mainnet")._level.get(MessageType.head)

    if level:
        await ctx.fire_hook(
            "on_index_rollback",
            wait=True
            index=None,  # type: ignore
            from_level=level,
            to_level=level - 2,
        )

F.A.Q

How to index the different contracts that share the same interface?

Multiple contracts can provide the same interface (like FA1.2 and FA2 standard tokens) but have a different storage structure. If you try to use the same typename for them, indexing will fail. However, you can modify typeclasses manually. Modify types/<typename>/storage.py file and comment out unique fields that are not important for your index:

# dipdup: ignore

...

class ContractStorage(BaseModel):
    class Config:
        extra = Extra.ignore

    common_ledger: Dict[str, str]
    # unique_field_foo: str
    # unique_field_bar: str

Note the # dipdup: ignore comment on the first line. It tells DipDup not to overwrite this file on init --overwrite-types command.

Don't forget Extra.ignore Pydantic hint, otherwise, storage deserialization will fail.

What is the correct way to process off-chain data?

DipDup provides convenient helpers to process off-chain data like market quotes or IPFS metadata. Follow the tips below to use them most efficiently.

• Do not perform off-chain requests in handers until necessary. Use hooks instead, enriching indexed data on-demand.
• Use generic http datasources for external APIs instead of plain aiohttp requests. This way you can use the same features DipDup uses for internal requests: retry with backoff, rate limiting, Prometheus integration etc.
• Database tables that store off-chain data can be marked as immune, preventing them from being removed on reindexing.

💡 SEE ALSO

• 3.1. Datasources → http-generic
• 14.5. database → immune-tables

One of my indexes depends on another one's indexed data. How to process them in a specific order?

Indexes of all kinds are fully independent. They are processed in parallel, have their message queues, and don't share any state. It is one of the essential DipDup concepts, so there's no "official" way to manage the order of indexing.

Avoid waiting for sync primitives like asyncio.Event or asyncio.Lock in handlers. Indexing will be stuck forever, waiting for the database transaction to complete.

Instead, save raw data in handlers and process it later with hooks when all conditions are met. For example, process data batch only when all indexes in the dipdup_index table have reached a specific level.

How to perform database migrations?

DipDup does not provide any tooling for database migrations. The reason is that schema changes almost always imply reindexing when speaking about indexers. However, you can perform migrations yourself using any tool you like. First, disable schema hash check in config:

advanced:
  reindex:
    schema_modified: ignore

You can also use the schema approve command for a single schema change.

To determine what manual modifications you need to apply after changing models.py, you can compare raw SQL schema before and after the change. Consider the following example:

-    timestamp = fields.DatetimeField()
+    timestamp = fields.DatetimeField(auto_now=True)

dipdup schema export > old
# ...modify `models.py` here...
dipdup schema export > new
diff old new

76c76
<     "timestamp" TIMESTAMP NOT NULL,
---
>     "timestamp" TIMESTAMP NOT NULL  DEFAULT CURRENT_TIMESTAMP,

Now you can prepare and execute an ALTER TABLE query manually or using SQL hooks.

💡 SEE ALSO

• 3.5. Reindexing
• 3.7. SQL scripts

Troubleshooting

This page contains tips for troubleshooting DipDup issues.

Update DipDup to the latest version

DipDup framework evolves rapidly just like Tezos itself does. We recommend keeping your project up-to-date with the latest version of DipDup.

If you're using Poetry, set caret version constraint in pyproject.toml to use the latest release of the current major version:

[tool.poetry.dependencies]
python = ">=3.10,<3.11"
dipdup = "^6.0.0"

Run poetry update dipdup periodically to update to the latest version.

While building Docker images you can use X and X.Y tags to lock to specific major/minor releases:

FROM dipdup/dipdup:6.0

Ensure that config is correct

DipDup config can be correct syntactically but not necessarily semantically. It's especially easy to make a mistake when actively using templates and environment variables. Use config export command to dump config the way DipDup "sees" it, after resolving all links and templates. config env command can help you to find missing environment variables.

dipdup -c dipdup.yml -c dipdup.prod.yml config export
dipdup -c dipdup.yml -c dipdup.prod.yml config env

💡 SEE ALSO

• 13. Command-line reference → dipdup-config-export
• 13. Command-line reference → dipdup-config-env
• 14. Config file reference

Enable debug logging and crash reporting

More logs can give you a clue about the reason for an issue. Enable them in config:

logging: verbose

When an exception occurs DipDup saves crash dumps to /tmp/dipdup/crashdumps/XXXXXXX.json. You can send those dumps to Baking Bad automatically:

advanced:
  crash_reporting: True

Use linters to find errors in your Python code

Exceptions that occurred during callback execution are reraised as CallbackErrors. If you experience this issue, most likely your code is incorrect. Luckily, the Python ecosystem has lots of tools called linters to help you find mistakes. In DipDup we mostly use a combo of flake8 and mypy. You can start using both with zero configuration:

poetry add --with dev flake8 mypy
poetry run flake8
poetry run mypy

You can find a more complex example in DipDup's pyproject.toml or use the cookiecutter template to create a new project with all recommended bells and whistles (see 1. Quickstart → from-template).

Explore contract calls in Better Call Dev

Better Call Dev is a blockchain explorer for Tezos smart contracts. It provides a more human-friendly interface than TzKT to explore exact contract calls and parameter/storage structures.

Try it out when writing index definitions.

Packaging issues

Poetry package manager we recommend using with DipDup is not the most stable software in the world. If you experience issues with it like SolverProblemError, try the following:

• Remove .venv and poetry.lock from the project root
• Remove ~/.cache/pypoetry and ~/.cache/pip/ directories
• Run poetry install and hope for the best.

Got stuck? Ask for help

We are always ready to answer your questions!

If you think you've found a bug, please report it directly to the GitHub Issues. For all other discussions, join our socials:

• Discord (preferred)
• Telegram
• Slack

DipDup contribution guide

🚧 UNDER CONSTRUCTION

This page or paragraph is yet to be written. Come back later.

Preparation

• To set up the development environment, you need to install Poetry package manager and GNU Make.
• Run make help to get available commands.

Git

• Branch names MUST follow prefix/short-descriptions format. Prefixes we currently use: feat for features, fix for bugfixes, docs for documentation, aux for miscellaneous, exp for experiments.

Codestyle

• We use the following combo of linters and formatters: isort, black, flake8, mypy. All linter checks MUST pass before merging code to master.

Releases

• Release versions SHOULD conform to Semantic Versioning. Releases that introduce breaking changes MUST be major ones.
• Only the latest major version is supported in general. Critical fixes CAN be backported to the previous major release. To do so, git checkout from the latest stable tag, bump DipDup version manually and add a new tag.

Documentation

• All changes that affect user experience MUST be documented in CHANGELOG.md file.
• Changelog formatting SHOULD stick to GitLab changelog guidelines.

Demo projects

DipDup repository contains several minimal examples of how to use various features for real-case scenarios. Please, do not use these examples in production unmodified. We have not put a production-grade amount of effort into developing them, so they may contain flaws in indexing logic.

Some projects that started as a demo now evolved into full-fledged applications running in production. Check out 10. Built with DipDup page.

TzBTC token

source: demo_tzbtc

The most basic indexer used in Quickstart. A single operation index to track balances of TzBTC token holders, nothing else.

hic et nunc

source: demo_hic_et_nunc

Indexes trades and swaps of "hic et nunc", one of the most popular NFT marketplaces on Tezos.

Quipuswap

source: demo_quipuswap

Covers all available operations of Quipuswap DEX contracts: trades, transfers, moving liquidity. A more complex example with index templates.

Homebase RegistryDAO

source: demo_registrydao

Homebase enables users to create DAO contracts. In this example indexes are spawned in runtime (3.11. Index factories) for all contracts having the same script.

Tezos Domains (operation)

source: demo_tezos_domains

Tezos Domains is a distributed naming system. You probably have seen those fancy user.tez names while browsing explorers. This is a pretty basic example of how to index them.

Tezos Domains (big_map)

source: demo_tezos_domains_big_map

The same as above, but uses big_map index instead of operation one. The storage structure of this contract is pretty straightforward; we only need to track a single big map. To speed up indexing more, this example contains skip_history: once directive to index only the current state of the contract before switching to realtime processing.

TzColors

source: demo_tzcolors

A very basic indexer of TzColors NFT token and marketplace. Unlike hic et nunc this marketplace provides auction functionality. Other than that, it is pretty much the same.

Built with DipDup

This page is a brief overview of projects which use DipDup as an indexing solution.

Want to see your project at this page? Create an issue on GitHub!

HicDEX

Homepage | API | GitHub

HicDEX is a Tezos indexer for hicetnunc.art marketplace. Indexed data is available with a public GraphQL endpoint.

• Hicdex: the Hic Et Nunc indexer (Medium)

Homebase

Homepage | GitHub

Homebase is a web application that enables users to create and manage/use DAOs on the Tezos blockchain. This application aims to help empower community members and developers to launch and participate in Tezos-based DAOs.

Tezos Profiles

Homepage | API | GitHub

Tezos Profiles enables you to associate your online identity with your Tezos account.

Juster

Homepage | API | GitHub

Juster is an on-chain smart contract platform allowing users to take part in an automated betting market by creating events, providing liquidity to them, and making bets.

tz1and

Homepage | API | GitHub

A Virtual World and NFT Marketplace.

mempool

This is an optional section used by the mempool indexer plugin. It uses contracts and datasources aliases as well as the database connection.

Mempool configuration has two sections: settings and indexers (required).

{% page-ref page="../advanced/mempool-plugin.md" %}

Settings

This section is optional so are all the setting keys.

mempool:
  settings:
    keep_operations_seconds: 172800
    expired_after_blocks: 60
    keep_in_chain_blocks: 10
    mempool_request_interval_seconds: 10
    rpc_timeout_seconds: 10
  indexers:
    ...

keep_operations_seconds

How long to store operations that did not get into the chain. After that period, such operations will be wiped from the database. Default value is 172800 seconds (2 days).

expired_after_blocks

When level(head) - level(operation.branch) >= expired_after_blocks and operation is still on in chain it's marked as expired. Default value is 60 blocks (~1 hour).

keep_in_chain_blocks

Since the main purpose of this plugin is to index mempool operations (actually it's a rolling index), all the operations that were included in the chain are removed from the database after specified period of time. Default value is 10 blocks (~10 minutes).

mempool_request_interval_seconds

How often Tezos nodes should be polled for pending mempool operations. Default value is 10 seconds.

rpc_timeout_seconds

Tezos node request timeout. Default value is 10 seconds.

Indexers

You can index several networks at once, or index different nodes independently. Indexer names are not standardized, but for clarity it's better to stick with some meaningful keys:

 mempool:
   settings:
     ...
   indexers:
     mainnet:
       filters:
         kinds:
           - transaction
         accounts:
           - contract_alias
       datasources:
         tzkt: tzkt_mainnet
         rpc: 
           - node_mainnet
     edonet:
     florencenet: 

Each indexer object has two keys: filters and datasources (required).

Filters

An optional section specifying which mempool operations should be indexed. By default all transactions will be indexed.

kinds

Array of operations kinds, default value is transaction (single item).
The complete list of values allowed:

• activate_account
• ballot
• delegation*
• double_baking_evidence
• double_endorsement_evidence
• endorsement
• origination*
• proposal
• reveal*
• seed_nonce_revelation
• transaction*

* — manager operations.

accounts

Array of contract aliases used to filter operations by source or destination.
NOTE: applied to manager operations only.

Datasources

Mempool plugin is tightly coupled with TzKT and Tezos node providers.

tzkt

An alias pointing to a datasource of kind tzkt is expected.

rpc

An array of aliases pointing to datasources of kind tezos-node
Polling multiple nodes allows to detect more refused operations and makes indexing more robust in general.

metadata

This is an optional section used by the metadata indexer plugin. It uses contracts and datasources aliases as well as the database connection.

Metadata configuration has two required sections: settings and indexers

Settings

metadata:
  settings:
    ipfs_gateways:
      - https://cloudflare-ipfs.com
    ipfs_timeout: 10
    http_timeout: 10
    max_retry_count_on_error: 3
    contract_service_workers: 15
    token_service_workers: 75
  indexers:
    ...

ipfs_gateways

An array of IPFS gateways. The indexer polls them sequentially until it gets a result or runs out of attempts. It is recommended to specify more than one gateway to overcome propagation issues, rate limits, and other problems.

ipfs_timeout

How long DipDup will wait for a single IPFS gateway response. Default value is 10 seconds.

http_timeout

How long DipDup will wait for a HTTP server response. Default value is 10 seconds.

max_retry_count_on_error

If DipDup fails to get a response from IPFS gateway or HTTP server, it will try again after some time, until it runs out of attempts. Default value is 3 attempts.

contract_service_workers

Count of contract service workers which resolves contract metadata. Default value is 5.

token_service_workers

Count of token service workers which resolves token metadata. Default value is 5.

Indexers

You can index several networks at once, or go with a single one. Indexer names are not standardized, but for clarity it's better to stick with some meaningful keys:

metadata:
  settings:
    ...
  indexers:
    mainnet:
      filters:
        accounts:
          - contract_alias
      datasources:
        tzkt: tzkt_mainnet

Each indexer object has two keys: filters and datasources (required).

Filters

accounts

Array of contract aliases used to filter Big_map updates by the owner contract address.

Datasources

Metadata plugin is tightly coupled with TzKT provider.

tzkt

An alias pointing to a datasource of kind tzkt is expected.

dipdup [OPTIONS] COMMAND [ARGS]...

dipdup config [OPTIONS] COMMAND [ARGS]...

dipdup config env [OPTIONS]

dipdup config export [OPTIONS]

dipdup hasura [OPTIONS] COMMAND [ARGS]...

dipdup hasura configure [OPTIONS]

dipdup init [OPTIONS]

dipdup migrate [OPTIONS]

dipdup run [OPTIONS]

dipdup schema [OPTIONS] COMMAND [ARGS]...

dipdup schema approve [OPTIONS]

dipdup schema export [OPTIONS]

dipdup schema init [OPTIONS]

dipdup schema wipe [OPTIONS]

dipdup status [OPTIONS]

Config file reference

💡 SEE ALSO

• 2.3. Creating config

class dipdup.config.AdvancedConfig(reindex: ~typing.Dict[~dipdup.enums.ReindexingReason, ~dipdup.enums.ReindexingAction] = <factory>, scheduler: ~typing.Optional[~typing.Dict[str, ~typing.Any]] = None, postpone_jobs: bool = False, early_realtime: bool = False, merge_subscriptions: bool = False, metadata_interface: bool = False, skip_version_check: bool = False, rollback_depth: int = 2, crash_reporting: bool = False)¶

Feature flags and other advanced config.

Parameters:

• reindex – Mapping of reindexing reasons and actions DipDup performs

• scheduler – apscheduler scheduler config

• postpone_jobs – Do not start job scheduler until all indexes are in realtime state

• early_realtime – Establish realtime connection immediately after startup

• merge_subscriptions – Subscribe to all operations instead of exact channels

• metadata_interface – Expose metadata interface for TzKT

• skip_version_check – Do not check for new DipDup versions on startup

• rollback_depth – A number of levels to keep for rollback

• crash_reporting – Enable crash reporting

class dipdup.config.BigMapHandlerConfig(callback: str, contract: Union[str, ContractConfig], path: str)¶

Big map handler config

Parameters:

• contract – Contract to fetch big map from

• path – Path to big map (alphanumeric string with dots)

initialize_big_map_type(package: str) → None¶

Resolve imports and initialize key and value type classes

class dipdup.config.BigMapIndexConfig(kind: Literal['big_map'], datasource: Union[str, TzktDatasourceConfig], handlers: Tuple[BigMapHandlerConfig, ...], skip_history: SkipHistory = SkipHistory.never, first_level: int = 0, last_level: int = 0)¶

Big map index config

Parameters:

• kind – always big_map

• datasource – Index datasource to fetch big maps with

• handlers – Description of big map diff handlers

• skip_history – Fetch only current big map keys ignoring historical changes

• first_level – Level to start indexing from

• last_level – Level to stop indexing at (Dipdup will terminate at this level)

classmethod strip(config_dict: Dict[str, Any]) → None¶

Strip config from tunables that are not needed for hash calculation.

class dipdup.config.CallbackMixin(callback: str)¶

Mixin for callback configs

Parameters:

callback – Callback name

class dipdup.config.CodegenMixin¶

Base for pattern config classes containing methods required for codegen

locate_arguments() → Dict[str, Optional[Type]]¶

Try to resolve scope annotations for arguments

class dipdup.config.CoinbaseDatasourceConfig(kind: Literal['coinbase'], api_key: Optional[str] = None, secret_key: Optional[str] = None, passphrase: Optional[str] = None, http: Optional[HTTPConfig] = None)¶

Coinbase datasource config

Parameters:

• kind – always ‘coinbase’

• api_key – API key

• secret_key – API secret key

• passphrase – API passphrase

• http – HTTP client configuration

class dipdup.config.ContractConfig(address: str, typename: Optional[str] = None)¶

Contract config

Parameters:

• address – Contract address

• typename – User-defined alias for the contract script

class dipdup.config.DipDupConfig(spec_version: str, package: str, datasources: ~typing.Dict[str, ~typing.Union[~dipdup.config.TzktDatasourceConfig, ~dipdup.config.CoinbaseDatasourceConfig, ~dipdup.config.MetadataDatasourceConfig, ~dipdup.config.IpfsDatasourceConfig, ~dipdup.config.HttpDatasourceConfig]] = <factory>, database: ~typing.Union[~dipdup.config.SqliteDatabaseConfig, ~dipdup.config.PostgresDatabaseConfig] = SqliteDatabaseConfig(kind='sqlite', path=':memory:'), contracts: ~typing.Dict[str, ~dipdup.config.ContractConfig] = <factory>, indexes: ~typing.Dict[str, ~typing.Union[~dipdup.config.OperationIndexConfig, ~dipdup.config.BigMapIndexConfig, ~dipdup.config.HeadIndexConfig, ~dipdup.config.TokenTransferIndexConfig, ~dipdup.config.IndexTemplateConfig]] = <factory>, templates: ~typing.Dict[str, ~typing.Union[~dipdup.config.OperationIndexConfig, ~dipdup.config.BigMapIndexConfig, ~dipdup.config.HeadIndexConfig, ~dipdup.config.TokenTransferIndexConfig]] = <factory>, jobs: ~typing.Dict[str, ~dipdup.config.JobConfig] = <factory>, hooks: ~typing.Dict[str, ~dipdup.config.HookConfig] = <factory>, hasura: ~typing.Optional[~dipdup.config.HasuraConfig] = None, sentry: ~typing.Optional[~dipdup.config.SentryConfig] = None, prometheus: ~typing.Optional[~dipdup.config.PrometheusConfig] = None, advanced: ~dipdup.config.AdvancedConfig = AdvancedConfig(reindex={}, scheduler=None, postpone_jobs=False, early_realtime=False, merge_subscriptions=False, metadata_interface=False, skip_version_check=False, rollback_depth=2, crash_reporting=False), custom: ~typing.Dict[str, ~typing.Any] = <factory>, logging: ~dipdup.enums.LoggingValues = LoggingValues.default)¶

Main indexer config

Parameters:

• spec_version – Version of specification

• package – Name of indexer’s Python package, existing or not

• datasources – Mapping of datasource aliases and datasource configs

• database – Database config

• contracts – Mapping of contract aliases and contract configs

• indexes – Mapping of index aliases and index configs

• templates – Mapping of template aliases and index templates

• jobs – Mapping of job aliases and job configs

• hooks – Mapping of hook aliases and hook configs

• hasura – Hasura integration config

• sentry – Sentry integration config

• prometheus – Prometheus integration config

• advanced – Advanced config

• custom – User-defined Custom config

property oneshot: bool¶

Whether all indexes have last_level field set

property package_path: str¶

Absolute path to the indexer package, existing or default

class dipdup.config.EventHookConfig(callback: str, args: Dict[str, str] = <factory>, atomic: bool = False)¶

class dipdup.config.HTTPConfig(retry_count: Optional[int] = None, retry_sleep: Optional[float] = None, retry_multiplier: Optional[float] = None, ratelimit_rate: Optional[int] = None, ratelimit_period: Optional[int] = None, connection_limit: Optional[int] = None, connection_timeout: Optional[int] = None, batch_size: Optional[int] = None, replay_path: Optional[str] = None)¶

Advanced configuration of HTTP client

Parameters:

• retry_count – Number of retries after request failed before giving up

• retry_sleep – Sleep time between retries

• retry_multiplier – Multiplier for sleep time between retries

• ratelimit_rate – Number of requests per period (“drops” in leaky bucket)

• ratelimit_period – Time period for rate limiting in seconds

• connection_limit – Number of simultaneous connections

• connection_timeout – Connection timeout in seconds

• batch_size – Number of items fetched in a single paginated request (for some APIs)

• replay_path – Development-only option to replay HTTP requests from a file

merge(other: Optional[HTTPConfig]) → HTTPConfig¶

Set missing values from other config

class dipdup.config.HandlerConfig(callback: str)¶

class dipdup.config.HasuraConfig(url: str, admin_secret: Optional[str] = None, create_source: bool = False, source: str = 'default', select_limit: int = 100, allow_aggregations: bool = True, camel_case: bool = False, rest: bool = True, http: Optional[HTTPConfig] = None)¶

Config for the Hasura integration.

Parameters:

• url – URL of the Hasura instance.

• admin_secret – Admin secret of the Hasura instance.

• create_source – Whether source should be added to Hasura if missing.

• source – Hasura source for DipDup to configure, others will be left untouched.

• select_limit – Row limit for unauthenticated queries.

• allow_aggregations – Whether to allow aggregations in unauthenticated queries.

• camel_case – Whether to use camelCase instead of default pascal_case for the field names (incompatible with metadata_interface flag)

• rest – Enable REST API both for autogenerated and custom queries.

• http – HTTP connection tunables

property headers: Dict[str, str]¶

Headers to include with every request

class dipdup.config.HeadHandlerConfig(callback: str)¶

Head block handler config

class dipdup.config.HeadIndexConfig(kind: Literal['head'], datasource: Union[str, TzktDatasourceConfig], handlers: Tuple[HeadHandlerConfig, ...])¶

Head block index config

class dipdup.config.HookConfig(callback: str, args: ~typing.Dict[str, str] = <factory>, atomic: bool = False)¶

Hook config

Parameters:

• args – Mapping of argument names and annotations (checked lazily when possible)

• atomic – Wrap hook in a single database transaction

class dipdup.config.HttpDatasourceConfig(kind: Literal['http'], url: str, http: Optional[HTTPConfig] = None)¶

Generic HTTP datasource config

kind: always ‘http’ url: URL to fetch data from http: HTTP client configuration

class dipdup.config.IndexConfig(kind: str, datasource: Union[str, TzktDatasourceConfig])¶

Index config

Parameters:

datasource – Alias of index datasource in datasources section

hash() → str¶

Calculate hash to ensure config has not changed since last run.

classmethod strip(config_dict: Dict[str, Any]) → None¶

Strip config from tunables that are not needed for hash calculation.

class dipdup.config.IndexTemplateConfig(template: str, values: Dict[str, str], first_level: int = 0, last_level: int = 0)¶

Index template config

Parameters:

• kind – always template

• name – Name of index template

• template_values – Values to be substituted in template (<key> -> value)

• first_level – Level to start indexing from

• last_level – Level to stop indexing at (DipDup will terminate at this level)

class dipdup.config.IpfsDatasourceConfig(kind: Literal['ipfs'], url: str = 'https://ipfs.io/ipfs', http: Optional[HTTPConfig] = None)¶

IPFS datasource config

Parameters:

• kind – always ‘ipfs’

• url – IPFS node URL, e.g. https://ipfs.io/ipfs/

• http – HTTP client configuration

class dipdup.config.JobConfig(hook: ~typing.Union[str, ~dipdup.config.HookConfig], crontab: ~typing.Optional[str] = None, interval: ~typing.Optional[int] = None, daemon: bool = False, args: ~typing.Dict[str, ~typing.Any] = <factory>)¶

Job schedule config

Parameters:

• hook – Name of hook to run

• crontab – Schedule with crontab syntax (* * * * *)

• interval – Schedule with interval in seconds

• daemon – Run hook as a daemon (never stops)

• args – Arguments to pass to the hook

class dipdup.config.MetadataDatasourceConfig(kind: Literal['metadata'], network: MetadataNetwork, url: str = 'https://metadata.dipdup.net', http: Optional[HTTPConfig] = None)¶

DipDup Metadata datasource config

Parameters:

• kind – always ‘metadata’

• network – Network name, e.g. mainnet, ghostnet, etc.

• url – GraphQL API URL, e.g. https://metadata.dipdup.net

• http – HTTP client configuration

class dipdup.config.NameMixin¶

class dipdup.config.OperationHandlerConfig(callback: str, pattern: Tuple[Union[OperationHandlerOriginationPatternConfig, OperationHandlerTransactionPatternConfig], ...])¶

Operation handler config

Parameters:

• callback – Name of method in handlers package

• pattern – Filters to match operation groups

class dipdup.config.OperationHandlerOriginationPatternConfig(type: Literal['origination'] = 'origination', source: Optional[Union[str, ContractConfig]] = None, similar_to: Optional[Union[str, ContractConfig]] = None, originated_contract: Optional[Union[str, ContractConfig]] = None, optional: bool = False, strict: bool = False, alias: Optional[str] = None)¶

Origination handler pattern config

Parameters:

• type – always ‘origination’

• source – Match operations by source contract alias

• similar_to – Match operations which have the same code/signature (depending on strict field)

• originated_contract – Match origination of exact contract

• optional – Whether can operation be missing in operation group

• strict – Match operations by storage only or by the whole code

• alias – Alias for transaction (helps to avoid duplicates)

class dipdup.config.OperationHandlerTransactionPatternConfig(type: Literal['transaction'] = 'transaction', source: Optional[Union[str, ContractConfig]] = None, destination: Optional[Union[str, ContractConfig]] = None, entrypoint: Optional[str] = None, optional: bool = False, alias: Optional[str] = None)¶

Operation handler pattern config

Parameters:

• type – always ‘transaction’

• source – Match operations by source contract alias

• destination – Match operations by destination contract alias

• entrypoint – Match operations by contract entrypoint

• optional – Whether can operation be missing in operation group

• alias – Alias for transaction (helps to avoid duplicates)

class dipdup.config.OperationIndexConfig(kind: ~typing.Literal['operation'], datasource: ~typing.Union[str, ~dipdup.config.TzktDatasourceConfig], handlers: ~typing.Tuple[~dipdup.config.OperationHandlerConfig, ...], types: ~typing.Tuple[~dipdup.enums.OperationType, ...] = (<OperationType.transaction: 'transaction'>,), contracts: ~typing.List[~typing.Union[str, ~dipdup.config.ContractConfig]] = <factory>, first_level: int = 0, last_level: int = 0)¶

Operation index config

Parameters:

• kind – always operation

• handlers – List of indexer handlers

• types – Types of transaction to fetch

• contracts – Aliases of contracts being indexed in contracts section

• first_level – Level to start indexing from

• last_level – Level to stop indexing at (DipDup will terminate at this level)

property address_filter: Set[str]¶

Set of addresses (any field) to filter operations with before an actual matching

property entrypoint_filter: Set[Optional[str]]¶

Set of entrypoints to filter operations with before an actual matching

classmethod strip(config_dict: Dict[str, Any]) → None¶

Strip config from tunables that are not needed for hash calculation.

class dipdup.config.ParameterTypeMixin¶

parameter_type_cls field

class dipdup.config.ParentMixin¶

parent field for index and template configs

class dipdup.config.PatternConfig¶

Base class for pattern config items.

Contains methods for import and method signature generation during handler callbacks codegen.

class dipdup.config.PostgresDatabaseConfig(kind: ~typing.Literal['postgres'], host: str, user: str = 'postgres', database: str = 'postgres', port: int = 5432, schema_name: str = 'public', password: str = '', immune_tables: ~typing.Set[str] = <factory>, connection_timeout: int = 60)¶

Postgres database connection config

Parameters:

• kind – always ‘postgres’

• host – Host

• port – Port

• user – User

• password – Password

• database – Database name

• schema_name – Schema name

• immune_tables – List of tables to preserve during reindexing

• connection_timeout – Connection timeout

class dipdup.config.PrometheusConfig(host: str, port: int = 8000, update_interval: float = 1.0)¶

Config for Prometheus integration.

Parameters:

• host – Host to bind to

• port – Port to bind to

• update_interval – Interval to update some metrics in seconds

class dipdup.config.SentryConfig(dsn: str, environment: Optional[str] = None, server_name: Optional[str] = None, release: Optional[str] = None, debug: bool = False)¶

Config for Sentry integration.

Parameters:

• dsn – DSN of the Sentry instance

• environment – Environment (defaults to production)

• server_name – Server name (defaults to hostname)

• release – Release version (defaults to DipDup version)

• debug – Catch warning messages and more context

class dipdup.config.SqliteDatabaseConfig(kind: Literal['sqlite'], path: str = ':memory:')¶

SQLite connection config

Parameters:

• kind – always ‘sqlite’

• path – Path to .sqlite3 file, leave default for in-memory database (:memory:)

class dipdup.config.StorageTypeMixin¶

storage_type_cls field

class dipdup.config.SubscriptionsMixin¶

subscriptions field

class dipdup.config.TemplateValuesMixin¶

template_values field

class dipdup.config.TokenTransferHandlerConfig(callback: str)¶

class dipdup.config.TokenTransferIndexConfig(kind: ~typing.Literal['token_transfer'], datasource: ~typing.Union[str, ~dipdup.config.TzktDatasourceConfig], handlers: ~typing.Tuple[~dipdup.config.TokenTransferHandlerConfig, ...] = <factory>, first_level: int = 0, last_level: int = 0)¶

Token index config

class dipdup.config.TransactionIdxMixin¶

transaction_idx field to track index of operation in group

Parameters:

transaction_idx –

class dipdup.config.TzktDatasourceConfig(kind: Literal['tzkt'], url: str = 'https://api.tzkt.io', http: Optional[HTTPConfig] = None, buffer_size: int = 0)¶

TzKT datasource config

Parameters:

• kind – always ‘tzkt’

• url – Base API URL, e.g. https://api.tzkt.io/

• http – HTTP client configuration

• buffer_size – Number of levels to keep in FIFO buffer before processing

advanced

advanced:
  early_realtime: False
  merge_subscriptions: False
  postpone_jobs: False
  reindex:
    manual: wipe
    migration: exception
    rollback: ignore
    config_modified: exception
    schema_modified: exception

This config section allows users to tune some system-wide options, either experimental or unsuitable for generic configurations.

CLI flags have priority over self-titled AdvancedConfig fields.

💡 SEE ALSO

• 3.5. Reindexing
• 3.6. Feature flags

contracts

A list of the contract definitions you might use in the indexer patterns or templates. Each contract entry has two fields:

• address — either originated or implicit account address encoded in base58.
• typename — an alias for the particular contract script, meaning that two contracts sharing the same code can have the same type name.

contracts:
  kusd_dex_mainnet:
    address: KT1CiSKXR68qYSxnbzjwvfeMCRburaSDonT2
    typename: quipu_fa12
  tzbtc_dex_mainnet:
    address: KT1N1wwNPqT5jGhM91GQ2ae5uY8UzFaXHMJS
    typename: quipu_fa12
  kusd_token_mainnet:
    address: KT1K9gCRgaLRFKTErYt1wVxA3Frb9FjasjTV
    typename: kusd_token
  tzbtc_token_mainnet:
    address: KT1PWx2mnDueood7fEmfbBDKx1D9BAnnXitn
    typename: tzbtc_token

A typename field is only required when using index templates, but it helps to improve the readability of auto-generated code and avoid repetition.

Contract entry does not contain information about the network, so it's a good idea to include the network name in the alias. This design choice makes possible a generic index parameterization via templates. See 4.5. Templates and variables for details.

If multiple contracts you index have the same interface but different code, see 8.2. Reusing typename for different contracts.

custom

An arbitrary YAML object you can use to store internal indexer configuration.

package: my_indexer
...
custom:
  foo: bar

Access or modify it from any callback:

ctx.config.custom['foo'] = 'buzz'

database

DipDup supports several database engines for development and production. The obligatory field kind specifies which engine has to be used:

• sqlite
• postgres (and compatible engines)

Database engines article may help you choose a database that better suits your needs.

SQLite

path field must be either path to the .sqlite3 file or :memory: to keep a database in memory only (default):

database:
  kind: sqlite
  path: db.sqlite3

PostgreSQL

Requires host, port, user, password, and database fields. You can set schema_name to values other than public, but Hasura integration won't be available.

database:
  kind: postgres
  host: db
  port: 5432
  user: dipdup
  password: ${POSTGRES_PASSWORD:-changeme}
  database: dipdup
  schema_name: public

You can also use compose-style environment variable substitutions with default values for secrets and other fields. See Templates and variables for details.

Immune tables

In some cases, DipDup can't continue indexing with an existing database. See 5.3. Reindexing for details. One of the solutions to resolve reindexing state is to drop the database and start indexing from scratch. To achieve this, either invoke the schema wipe command or set an action to wipe in the advanced.reindex config section.

You might want to keep several tables during schema wipe if data in them is not dependent on index states yet heavy. A typical example is indexing IPFS data — rollbacks do not affect off-chain storage, so you can safely continue after receiving a reorg message.

database:
  immune_tables:
    - token_metadata
    - contract_metadata

immune_tables is an optional array of table names that will be ignored during schema wipe. Once an immune table is created, DipDup will never touch it again; to change the schema of an immune table, you need to perform a migration manually. Check schema export output before doing this to ensure the resulting schema is the same TortoiseORM would generate.

datasources

A list of API endpoints DipDup uses to retrieve indexing data to process.

A datasource config entry is an alias for the endpoint URI; there's no network mention. Thus it's good to add a network name to the datasource alias, e.g. tzkt_mainnet.

tzkt

datasources:
  tzkt:
    kind: tzkt
    url: ${TZKT_URL:-https://api.tzkt.io}
    http:
      retry_count:  # retry infinetely
      retry_sleep:
      retry_multiplier:
      ratelimit_rate:
      ratelimit_period:
      connection_limit: 100
      connection_timeout: 60
      batch_size: 10000
    buffer_size: 0

coinbase

datasources:
  coinbase:
    kind: coinbase

dipdup-metadata

datasources:
  metadata:
    kind: metadata
    url: https://metadata.dipdup.net
    network: mainnet|ghostnet|jakartanet|kathmandunet

ipfs

datasources:
  ipfs:
    kind: ipfs
    url: https://ipfs.io/ipfs

💡 SEE ALSO

• 3.1. Datasources

hasura

This optional section used by DipDup executor to automatically configure Hasura engine to track your tables.

hasura:
  url: http://hasura:8080
  admin_secret: ${HASURA_ADMIN_SECRET:-changeme}
  allow_aggregations: false
  camel_case: true
  rest: true
  select_limit: 100
  source: default

💡 SEE ALSO

• 4.1. Hasura integration
• 13. Command-line reference → dipdup-hasura-configure

hooks

Hooks are user-defined callbacks you can execute with a job scheduler or within another callback (with ctx.fire_hook).

hooks:
  calculate_stats:
    callback: calculate_stats
    atomic: False
    args:
     major: bool
     depth: int

💡 SEE ALSO

• 3.2. Hooks

indexes

Index — is a basic DipDup entity connecting the inventory and specifying data handling rules.

Each index has a unique string identifier acting as a key under the indexes config section:

indexes:
  my_index:
    kind: operation
    datasource: tzkt_mainnet

There can be various index kinds; currently, the following options are supported for the kind field:

• big_map
• head
• operation
• token_transfer

There's also a special template kind, which is used to generate new indexes from a template during startup.

All the indexes have to specify the datasource field, an alias of an existing entry under the datasources section.

Indexing scope

One can optionally specify block levels DipDup has to start and stop indexing at, e.g., there's a new version of the contract, and there's no need to track the old one anymore.

indexes:
  my_index:
    first_level: 1000000
    last_level: 2000000

big_map

big_map index allows querying only updates of a specific big map (or several). In some cases, it can drastically reduce the amount of data transferred and speed up the indexing process.

indexes:
  my_index:
    kind: big_map
    datasource: tzkt
    skip_history: never
    handlers:
      - callback: on_leger_update
        contract: contract1
        path: data.ledger
      - callback: on_token_metadata_update
        contract: contract1
        path: token_metadata

Handlers

Each big_map handler contains three required fields:

• callback — name of the async function with a particular signature; DipDup will try to load it from the module with the same name <package_name>.handlers.<callback>
• contract — Big map parent contract (from the inventory)
• path — path to the Big map in the contract storage (use dot as a delimiter)

Index only the current state of big maps

When skip_history field is set to once, DipDup will skip historical changes only on initial sync and switch to regular indexing afterward. When the value is always, DipDup will fetch all big map keys on every restart. Preferrable mode depends on your workload.

All big map diffs DipDup pass to handlers during fast sync have action field set to BigMapAction.ADD_KEY. Keep in mind that DipDup fetches all keys in this mode, including ones removed from the big map. You can filter out latter by BigMapDiff.data.active field if needed.

head

🚧 UNDER CONSTRUCTION

This page or paragraph is yet to be written. Come back later.

operation

Operation index allows you to query only operations related to your dapp and match them with handlers by content. A single contract call consists of implicit operation and, optionally, internal operations. For each of them, you can specify a handler that will be called when operation group. As a result, you get something like an event log for your dapp.

Filters

DipDup supports filtering operations by source, destination (if applicable), and originated_contract (if applicable).

DipDup fetches only applied operations.

contracts

indexes:
  my_index:
    kind: operation
    datasource: tzkt
    contracts:
      - contract1
      - contract2

In this example, DipDup will fetch all the operations where any of source and destination is equal to either contract1 or contract2 address. contracts field is obligatory, there has to be at least one contract alias (from the inventory).

types

By default, DipDup works only with transactions, but you can explicitly list operation types you want to subscribe to (currently transaction, origination and migration types are supported):

indexes:
  my_index:
    kind: operation
    datasource: tzkt
    contracts:
      - some_contract
    types:
      - transaction
      - origination

Note that in the case of originations, DipDup will query operations where either source or originated contract address is equal to some_contract.

Handlers

Each operation handler contains two required fields:

• callback — name of the async function with a particular signature; DipDup will create it on init and in the module with the same name <package_name>.handlers.<callback>.
• pattern — a non-empty list of items that need to be matched.

indexes:
  my_index:
    kind: operation
    datasource: tzkt
    contracts:
      - some_contract
    handlers:
      - callback: on_call
        pattern:
          - destination: some_contract
            entrypoint: transfer

You can think of operation pattern as a regular expression on a sequence of operations (both external and internal) with global flag enabled (can be multiple matches) and where various operation parameters (type, source, destination, entrypoint, originated contract) are used for matching.

Pattern

Here are the supported filters for matching operations (all optional):

• type — transaction or origination; usually inferred from the existence of other fields
• destination — invoked contract alias (from the inventory)
• entrypoint — invoked entrypoint name
• source — operation sender alias (from the inventory)
• originated_contract — originated contract alias (from the inventory)
• similar_to — originated contract has the same parameter and storage types as the reference one (from the inventory)
• strict — stronger the similar_to filter by comparing the entire code rather than just parameter+storage
• optional — continue matching even if this item is not found (with limitations, see below)

It's unnecessary to match the entire operation content; you can skip external/internal calls that are not relevant. However, there is a limitation: optional items cannot be followed by operations ignored by the pattern.

pattern:
  # Implicit transaction
  - destination: some_contract
    entrypoint: mint

  # Internal transactions below
  - destination: another_contract
    entrypoint: transfer

  - source: some_contract
    type: transaction

You will get slightly different callback argument types depending on whether you specify destination+entrypoint for transactions and originated_contract for originations. Namely, in the first case, DipDup will generate the dataclass for a particular entrypoint/storage, and in the second case not (meaning you will have to handle untyped parameters/storage updates).

template

This index type is used for creating a static template instance.

indexes:
  my_index:
    template: my_template
    values:
      placeholder1: value1
      placeholder2: value2

For a static template instance (specified in the DipDup config) there are two fields:

• template — template name (from templates section)
• values — concrete values for each placeholder used in a chosen template

token_transfer

🚧 UNDER CONSTRUCTION

This page or paragraph is yet to be written. Come back later.

jobs

Add the following section to DipDup config:

jobs:
  midnight_stats:
    hook: calculate_stats
    crontab: "0 0 * * *"
    args:
      major: True
  leet_stats:
    hook: calculate_stats
    interval: 1337  # in seconds
    args:
      major: False

If you're not familiar with the crontab syntax, there's an online service crontab.guru that will help you build the desired expression.

logging

You can configure an amount of logging output by modifying the logging field.

logging: default | quiet | verbose

At the moment these values are equal to setting dipdup log level to INFO, WARNING or DEBUG, but this may change in the future.

package

DipDup uses this field to discover the Python package of your project.

package: my_indexer_name

DipDup will search for a module named my_module_name in PYTHONPATH

This field helps to decouple DipDup configuration file from the indexer implementation and gives more flexibility in managing the source code.

See 2.4. Project structure for details.

prometheus

prometheus:
  host: 0.0.0.0

Prometheus integration options

sentry

sentry:
  dsn: https://...
  environment: dev
  debug: False

💡 SEE ALSO

• 5.3. Sentry integration

spec_version

DipDup specification version is used to determine the compatibility of the toolkit and configuration file format and/or features.

spec_version: 1.2

This table shows which specific SDK releases support which DipDup file versions.

If you're getting MigrationRequiredError after updating the framework, run dipdup migrate command to perform project migration.

templates

indexes:
  foo:
    kind: template
    name: bar
    first_level: 12341234
    template_values:
      network: mainnet

templates:
  bar:
    kind: index
    datasource: tzkt_<network>  # resolves into `tzkt_mainnet`
    ...

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

6.1.3 - 2022-09-21

Added

• sentry: Enable crash-free session reporting.

Fixed

• metadata: Updated protocol aliases.
• sentry: Unwrap CallbackError traceback to fix event grouping.
• sentry: Hide "attempting to send..." message on shutdown.

Other

• ci: Do not build default and -pytezos nightly images.

6.1.2 - 2022-09-16

Added

• config: Added alias field to operation pattern items.
• tzkt: Added quote field gbp.

Fixed

• config: Require aliases for multiple operations with the same entrypoint.
• http: Raise InvalidRequestError on 204 No Content responses.
• tzkt: Verify API version on datasource initialization.
• tzkt: Remove deprecated block field priority.

6.1.1 - 2022-09-01

Fixed

• ci: Lock Pydantic to 1.9.2 to avoid breaking changes in dataclasses.

6.1.0 - 2022-08-30

Added

• ci: Build arm64 images for M1/M2 silicon.
• ci: Build -slim images based on Alpine Linux.
• ci: Introduced official MacOS support.
• ci: Introduced interactive installer (dipdup.net/install.py).

6.0.1 - 2022-08-19

Fixed

• codegen: Fixed invalid models.py template.
• context: Do not wrap known exceptions with CallbackError.
• database: Raise DatabaseConfigurationError when backward relation name equals table name.
• database: Wrap schema wiping in a transaction to avoid orphaned tables in the immune schema.
• hasura: Fixed processing M2M relations.
• sentry: Fixed "invalid value environment" error.
• sentry: Ignore events from project callbacks when crash_reporting is enabled.

6.0.0 - 2022-08-08

This release contains no changes except for the version number.

6.0.0rc2 - 2022-08-06

Added

• config: Added advanced.crash_reporting flag to enable reporting crashes to Baking Bad.
• dipdup: Save Sentry crashdump in /tmp/dipdup/crashdumps/XXXXXXX.json on a crash.

Fixed

• config: Do not perform env variable substitution in commented-out lines.

Removed

• cli: --logging-config option is removed.
• cli: All run command flags are removed. Use the advanced section of the config.
• cli: cache show and cache clear commands are removed.
• config: http.cache flag is removed.

6.0.0-rc1 - 2022-07-26

Added

• cli: Added config export --full flag to resolve templates before printing config.
• config: Added advanced.rollback_depth field, a number of levels to keep in a database for rollback.
• context: Added rollback method to perform database rollback.
• database: Added an internal ModelUpdate model to store the latest database changes.

Fixed

• prometheus: Fixed updating dipdup_index_handlers_matched_total metric.

Changed

• codegen: on_index_rollback hook calls ctx.rollback by default.
• database: Project models must be subclassed from dipdup.models.Model
• database: bulk_create and bulk_update model methods are no longer supported.

Removed

• hooks: Removed deprecated on_rollback hook.
• index: Do not try to avoid single-level rollbacks by comparing operation hashes.

5.2.5 - 2022-07-26

Fixed

• index: Fixed crash when adding an index with new subscriptions in runtime.

5.2.4 - 2022-07-17

Fixed

• cli: Fixed logs being printed to stderr instead of stdout.
• config: Fixed job scheduler not starting when config contains no indexes.

5.2.3 - 2022-07-07

Added

• sentry: Allow customizing server_name and release tags with corresponding fields in Sentry config.

Fixed

• cli: Fixed hasura configure command crash when models have empty Meta.table.
• config: Removed secrets from config __repr__.

5.2.2 - 2022-07-03

Fixed

• hasura: Fixed metadata generation.

5.2.1 - 2022-07-02

Fixed

• cli: Fixed setting default logging level.
• hasura: Fixed metadata generation for relations with a custom field name.
• hasura: Fixed configuring existing instances after changing camel_case field in config.

5.2.0 - 2022-06-28

Added

• config: Added logging config field.
• config: Added hasura.create_source flag to create PostgreSQL source if missing.

Fixed

• hasura: Do not apply table customizations to tables from other sources.

Deprecated

• cli: --logging-config option is deprecated.
• cli: All run command flags are deprecated. Use the advanced section of the config.
• cli: cache show and cache clear commands are deprecated.
• config: http.cache flag is deprecated.

5.1.7 - 2022-06-15

Fixed

• index: Fixed token_transfer index not receiving realtime updates.

5.1.6 - 2022-06-08

Fixed

• cli: Commands with --help option no longer require a working DipDup config.
• index: Fixed crash with RuntimeError after continuous realtime connection loss.

Performance

• cli: Lazy import dependencies to speed up startup.

Other

• docs: Migrate docs from GitBook to mdbook.

5.1.5 - 2022-06-05

Fixed

• config: Fixed crash when rollback hook is about to be called.

5.1.4 - 2022-06-02

Fixed

• config: Fixed OperationIndexConfig.types field being partially ignored.
• index: Allow mixing oneshot and regular indexes in a single config.
• index: Call rollback hook instead of triggering reindex when single-level rollback has failed.
• index: Fixed crash with RuntimeError after continuous realtime connection loss.
• tzkt: Fixed origination subscription missing when merge_subscriptions flag is set.

Performance

• ci: Decrease the size of generic and -pytezos Docker images by 11% and 16%, respectively.

5.1.3 - 2022-05-26

Fixed

• database: Fixed special characters in password not being URL encoded.

Performance

• context: Do not reinitialize config when adding a single index.

5.1.2 - 2022-05-24

Added

• tzkt: Added originated_contract_tzips field to OperationData.

Fixed

• jobs: Fixed jobs with daemon schedule never start.
• jobs: Fixed failed jobs not throwing exceptions into the main loop.

Other

• database: Tortoise ORM updated to 0.19.1.

5.1.1 - 2022-05-13

Fixed

• index: Ignore indexes with different message types on rollback.
• metadata: Add ithacanet to available networks.

5.1.0 - 2022-05-12

Added

• ci: Push X and X.Y tags to the Docker Hub on release.
• cli: Added config env command to export env-file with default values.
• cli: Show warning when running an outdated version of DipDup.
• hooks: Added a new hook on_index_rollback to perform per-index rollbacks.

Fixed

• index: Fixed fetching migration operations.
• tzkt: Fixed possible data corruption when using the buffer_size option.
• tzkt: Fixed reconnection due to websockets message size limit.

Deprecated

• hooks: The on_rollback default hook is superseded by on_index_rollback and will be removed later.

5.0.4 - 2022-05-05

Fixed

• exceptions: Fixed incorrect formatting and broken links in help messages.
• index: Fixed crash when the only index in config is head.
• index: Fixed fetching originations during the initial sync.

5.0.3 - 2022-05-04

Fixed

• index: Fixed crash when no block with the same level arrived after a single-level rollback.
• index: Fixed setting initial index level when IndexConfig.first_level is set.
• tzkt: Fixed delayed emitting of buffered realtime messages.
• tzkt: Fixed inconsistent behavior of first_level/last_level arguments in different getter methods.

5.0.2 - 2022-04-21

Fixed

• context: Fixed reporting incorrect reindexing reason.
• exceptions: Fixed crash with FrozenInstanceError when an exception is raised from a callback.
• jobs: Fixed graceful shutdown of daemon jobs.

Improved

• codegen: Refined on_rollback hook template.
• exceptions: Updated help messages for known exceptions.
• tzkt: Do not request reindexing if missing subgroups have matched no handlers.

5.0.1 - 2022-04-12

Fixed

• cli: Fixed schema init command crash with SQLite databases.
• index: Fixed spawning datasources in oneshot mode.
• tzkt: Fixed processing realtime messages.

5.0.0 - 2022-04-08

This release contains no changes except for the version number.

5.0.0-rc4 - 2022-04-04

Added

• tzkt: Added ability to process realtime messages with lag.

4.2.7 - 2022-04-02

Fixed

• config: Fixed jobs config section validation.
• hasura: Fixed metadata generation for v2.3.0 and above.
• tzkt: Fixed get_originated_contracts and get_similar_contracts methods response.

5.0.0-rc3 - 2022-03-28

Added

• config: Added custom section to store arbitrary user data.

Fixed

• config: Fixed default SQLite path (:memory:).
• tzkt: Fixed pagination in several getter methods.
• tzkt: Fixed data loss when skip_history option is enabled.

Removed

• config: Removed dummy advanced.oneshot flag.
• cli: Removed docker init command.
• cli: Removed dummy schema approve --hashes flag.

5.0.0-rc2 - 2022-03-13

Fixed

• tzkt: Fixed crash in methods that do not support cursor pagination.
• prometheus: Fixed invalid metric labels.

5.0.0-rc1 - 2022-03-02

Added

• metadata: Added metadata_interface feature flag to expose metadata in TzKT format.
• prometheus: Added ability to expose Prometheus metrics.
• tzkt: Added missing fields to the HeadBlockData model.
• tzkt: Added iter_... methods to iterate over item batches.

Fixed

• tzkt: Fixed possible OOM while calling methods that support pagination.
• tzkt: Fixed possible data loss in get_originations and get_quotes methods.

Changed

• tzkt: Added offset and limit arguments to all methods that support pagination.

Removed

• bcd: Removed bcd datasource and config section.

Performance

• dipdup: Use fast orjson library instead of built-in json where possible.

4.2.6 - 2022-02-25

Fixed

• database: Fixed generating table names from uppercase model names.
• http: Fixed bug that leads to caching invalid responses on the disk.
• tzkt: Fixed processing realtime messages with data from multiple levels.

4.2.5 - 2022-02-21

Fixed

• database: Do not add the schema argument to the PostgreSQL connection string when not needed.
• hasura: Wait for Hasura to be configured before starting indexing.

4.2.4 - 2022-02-14

Added

• config: Added http datasource to making arbitrary http requests.

Fixed

• context: Fixed crash when calling fire_hook method.
• context: Fixed HookConfig.atomic flag, which was ignored in fire_hook method.
• database: Create missing tables even if Schema model is present.
• database: Fixed excess increasing of decimal context precision.
• index: Fixed loading handler callbacks from nested packages (@veqtor).

Other

• ci: Added GitHub Action to build and publish Docker images for each PR opened.

4.2.3 - 2022-02-08

Fixed

• ci: Removed black 21.12b0 dependency since bug in datamodel-codegen-generator is fixed.
• cli: Fixed config export command crash when advanced.reindex dictionary is present.
• cli: Removed optionals from config export output so the result can be loaded again.
• config: Verify advanced.scheduler config for the correctness and unsupported features.
• context: Fixed ignored wait argument of fire_hook method.
• hasura: Fixed processing relation fields with missing related_name.
• jobs: Fixed default apscheduler config.
• tzkt: Fixed crash occurring when reorg message is the first one received by the datasource.

4.2.2 - 2022-02-01

Fixed

• config: Fixed ipfs datasource config.

4.2.1 - 2022-01-31

Fixed

• ci: Added black 21.12b0 dependency to avoid possible conflict with datamodel-codegen-generator.

4.2.0 - 2022-01-31

Added

• context: Added wait argument to fire_hook method to escape current transaction context.
• context: Added ctx.get_<kind>_datasource helpers to avoid type casting.
• hooks: Added ability to configure apscheduler with AdvancedConfig.scheduler field.
• http: Added request method to send arbitrary requests (affects all datasources).
• ipfs: Added ipfs datasource to download JSON and binary data from IPFS.

Fixed

• http: Removed dangerous method close_session.
• context: Fixed help message of IndexAlreadyExistsError exception.

Deprecated

• bcd: Added deprecation notice.

Other

• dipdup: Removed unused internal methods.

4.1.2 - 2022-01-27

Added

• cli: Added schema wipe --force argument to skip confirmation prompt.

Fixed

• cli: Show warning about deprecated --hashes argument
• cli: Ignore SIGINT signal when shutdown is in progress.
• sentry: Ignore exceptions when shutdown is in progress.

4.1.1 - 2022-01-25

Fixed

• cli: Fixed stacktraces missing on exception.
• cli: Fixed wrapping OSError with ConfigurationError during config loading.
• hasura: Fixed printing help messages on HasuraError.
• hasura: Preserve a list of sources in Hasura Cloud environments.
• hasura: Fixed HasuraConfig.source config option.

Changed

• cli: Unknown exceptions are no longer wrapped with DipDupError.

Performance

• hasura: Removed some useless requests.

4.1.0 - 2022-01-24

Added

• cli: Added schema init command to initialize database schema.
• cli: Added --force flag to hasura configure command.
• codegen: Added support for subpackages inside callback directories.
• hasura: Added dipdup_head_status view and REST endpoint.
• index: Added an ability to skip historical data while synchronizing big_map indexes.
• metadata: Added metadata datasource.
• tzkt: Added get_big_map and get_contract_big_maps datasource methods.

4.0.5 - 2022-01-20

Fixed

• index: Fixed deserializing manually modified typeclasses.

4.0.4 - 2022-01-17

Added

• cli: Added --keep-schemas flag to init command to preserve JSONSchemas along with generated types.

Fixed

• demos: Tezos Domains and Homebase DAO demos were updated from edo2net to mainnet contracts.
• hasura: Fixed missing relations for models with ManyToManyField fields.
• tzkt: Fixed parsing storage with nested structures.

Performance

• dipdup: Minor overall performance improvements.

Other

• ci: Cache virtual environment in GitHub Actions.
• ci: Detect CI environment and skip tests that fail in GitHub Actions.
• ci: Execute tests in parallel with pytest-xdist when possible.
• ci: More strict linting rules of flake8.

4.0.3 - 2022-01-09

Fixed

• tzkt: Fixed parsing parameter with an optional value.

4.0.2 - 2022-01-06

Added

• tzkt: Added optional delegate_address and delegate_alias fields to OperationData.

Fixed

• tzkt: Fixed crash due to unprocessed pysignalr exception.
• tzkt: Fixed parsing OperationData.amount field.
• tzkt: Fixed parsing storage with top-level boolean fields.

4.0.1 - 2021-12-30

Fixed

• codegen: Fixed generating storage typeclasses with Union fields.
• codegen: Fixed preprocessing contract JSONSchema.
• index: Fixed processing reindexing reason saved in the database.
• tzkt: Fixed processing operations with default entrypoint and empty parameter.
• tzkt: Fixed crash while recursively applying bigmap diffs to the storage.

Performance

• tzkt: Increased speed of applying bigmap diffs to operation storage.

4.0.0 - 2021-12-24

This release contains no changes except for the version number.

4.0.0-rc3 - 2021-12-20

Fixed

• cli: Fixed missing schema approve --hashes argument.
• codegen: Fixed contract address used instead of an alias when typename is not set.
• tzkt: Fixed processing operations with entrypoint default.
• tzkt: Fixed regression in processing migration originations.
• tzkt: Fixed filtering of big map diffs by the path.

Removed

• cli: Removed deprecated run --oneshot argument and clear-cache command.

4.0.0-rc2 - 2021-12-11

Migration

• Run dipdup init command to generate on_synchronized hook stubs.

Added

• hooks: Added on_synchronized hook, which fires each time all indexes reach realtime state.

Fixed

• cli: Fixed config not being verified when invoking some commands.
• codegen: Fixed generating callback arguments for untyped operations.
• index: Fixed incorrect log messages, remove duplicate ones.
• index: Fixed crash while processing storage of some contracts.
• index: Fixed matching of untyped operations filtered by source field (@pravin-d).

Performance

• index: Checks performed on each iteration of the main DipDup loop are slightly faster now.

4.0.0-rc1 - 2021-12-02

Migration

• Run dipdup schema approve command on every database you want to use with 4.0.0-rc1. Running dipdup migrate is not necessary since spec_version hasn't changed in this release.

Added

• cli: Added run --early-realtime flag to establish a realtime connection before all indexes are synchronized.
• cli: Added run --merge-subscriptions flag to subscribe to all operations/big map diffs during realtime indexing.
• cli: Added status command to print the current status of indexes from the database.
• cli: Added config export [--unsafe] command to print config after resolving all links and variables.
• cli: Added cache show command to get information about file caches used by DipDup.
• config: Added first_level and last_level optional fields to TemplateIndexConfig. These limits are applied after ones from the template itself.
• config: Added daemon boolean field to JobConfig to run a single callback indefinitely. Conflicts with crontab and interval fields.
• config: Added advanced top-level section.