4.0.0

⚠ Breaking Changes

  • run --oneshot option is removed. The oneshot mode (DipDup stops after the sync is finished) applies automatically when last_level field is set in the index config.
  • clear-cache command is removed. Use cache clear instead.

⚠ Migration from 3.x

  • Run dipdup init command to generate on_synchronized hook stubs.
  • Run dipdup schema approve command on every database you want to use with 4.0.0. Running dipdup migrate is not necessary since spec_version hasn't changed in this release.

What's New

Performance optimizations

Overall indexing performance has been significantly improved. Key highlights:

  • Configuration files are loaded 10x times faster. The more indexes in the project, the more noticeable difference is.
  • Significantly reduced CPU usage in realtime mode.
  • Datasource default HTTP connection options optimized for a reasonable balance between resource consumption and indexing speed.

Also, two new flags were added to improve DipDup performance in several scenarios: merge_subscriptions and early_relatime. See this paragraph for details.

Configurable action on reindex

There are several reasons that trigger reindexing:

reasondescription
manualReindexing triggered manually from callback with ctx.reindex.
migrationApplied migration requires reindexing. Check release notes before switching between major DipDup versions to be prepared.
rollbackReorg message received from TzKT, and can not be processed.
config_modifiedOne of the index configs has been modified.
schema_modifiedDatabase schema has been modified. Try to avoid manual schema modifications in favor of SQL hooks.

Now it is possible to configure desirable action on reindexing triggered by the specific reason.

actiondescription
exception (default)Raise ReindexingRequiredError and quit with error code. The safest option since you can trigger reindexing accidentally, e.g., by a typo in config. Don't forget to set up the correct restart policy when using it with containers.
wipeDrop the whole database and start indexing from scratch. Be careful with this option!
ignoreIgnore the event and continue indexing as usual. It can lead to unexpected side-effects up to data corruption; make sure you know what you are doing.

To configure actions for each reason, add the following section to DipDup config:

advanced:
  ...
  reindex:
    manual: wipe
    migration: exception
    rollback: ignore
    config_modified: exception
    schema_modified: exception

New CLI commands and flags

command or flagdescription
cache showGet information about file caches used by DipDup.
config exportPrint config after resolving all links and variables. Add --unsafe option to substitute environment variables; default values from config will be used otherwise.
run --early-realtimeEstablish a realtime connection before all indexes are synchronized.
run --merge-subscriptionsSubscribe to all operations/big map diffs during realtime indexing. This flag helps to avoid reaching TzKT subscriptions limit (currently 10000 channels). Keep in mind that this option could significantly improve RAM consumption depending on the time required to perform a sync.
statusPrint the current status of indexes from the database.

advanced top-level config section

This config section allows users to tune system-wide options, either experimental or unsuitable for generic configurations.

fielddescription
early_realtime
merge_subscriptions
postpone_jobs
Another way to set run command flags. Useful for maintaining per-deployment configurations.
reindexConfigure action on reindexing triggered. See this paragraph for details.

CLI flags have priority over self-titled AdvancedConfig fields.

aiosignalrcore replaced with pysignalr

It may not be the most noticeable improvement for end-user, but it still deserves a separate paragraph in this article.

Historically, DipDup used our own fork of signalrcore library named aiosignalrcore. This project aimed to replace the synchronous websocket-client library with asyncio-ready websockets. Later we discovered that required changes make it hard to maintain backward compatibility, so we have decided to rewrite this library from scratch. So now you have both a modern and reliable library for SignalR protocol and a much more stable DipDup. Ain't it nice?

Changes since 3.1.3

This is a combined changelog of -rc versions released since the last stable release until this one.

Added

  • cli: Added run --early-realtime flag to establish a realtime connection before all indexes are synchronized.
  • cli: Added'run --merge-subscriptions` flag to subscribe to all operations/big map diffs during realtime indexing.
  • cli: Added status command to print the current status of indexes from the database.
  • cli: Added config export [--unsafe] command to print config after resolving all links and variables.
  • cli: Added cache show command to get information about file caches used by DipDup.
  • config: Added first_level and last_level optional fields to TemplateIndexConfig. These limits are applied after ones from the template itself.
  • config: Added daemon boolean field to JobConfig to run a single callback indefinitely. Conflicts with crontab and interval fields.
  • config: Added advanced top-level section.
  • hooks: Added on_synchronized hook, which fires each time all indexes reach realtime state.

Fixed

  • cli: Fixed config not being verified when invoking some commands.
  • cli: Fixed crashes and output inconsistency when piping DipDup commands.
  • cli: Fixed missing schema approve --hashes argument.
  • cli: Fixed schema wipe --immune flag being ignored.
  • codegen: Fixed contract address used instead of an alias when typename is not set.
  • codegen: Fixed generating callback arguments for untyped operations.
  • codegen: Fixed missing imports in handlers generated during init.
  • coinbase: Fixed possible data inconsistency caused by caching enabled for method get_candles.
  • hasura: Fixed unnecessary reconfiguration in restart.
  • http: Fixed increasing sleep time between failed request attempts.
  • index: Fixed CallbackError raised instead of ReindexingRequiredError in some cases.
  • index: Fixed crash while processing storage of some contracts.
  • index: Fixed incorrect log messages, remove duplicate ones.
  • index: Fixed invocation of head index callback.
  • index: Fixed matching of untyped operations filtered by source field (@pravin-d).
  • tzkt: Fixed filtering of big map diffs by the path.
  • tzkt: Fixed get_originated_contracts and get_similar_contracts methods whose output was limited to HTTPConfig.batch_size field.
  • tzkt: Fixed lots of SignalR bugs by replacing aiosignalrcore library with pysignalr.
  • tzkt: Fixed processing operations with entrypoint default.
  • tzkt: Fixed regression in processing migration originations.
  • tzkt: Fixed resubscribing when realtime connectivity is lost for a long time.
  • tzkt: Fixed sending useless subscription requests when adding indexes in runtime.

Changed

  • cli: schema wipe command now requires confirmation when invoked in the interactive shell.
  • cli: schema approve command now also causes a recalculation of schema and index config hashes.
  • index: DipDup will recalculate respective hashes if reindexing is triggered with config_modified: ignore or schema_modified: ignore in advanced config.

Removed

  • cli: Removed deprecated run --oneshot argument and clear-cache command.

Performance

  • config: Configuration files are loaded 10x times faster.
  • index: Checks performed on each iteration of the main DipDup loop are slightly faster now.
  • index: Number of operations processed by matcher reduced by 40%-95% depending on the number of addresses and entrypoints used.
  • tzkt: Improved performance of response deserialization.
  • tzkt: Rate limit was increased. Try to set connection_timeout to a higher value if requests fail with ConnectionTimeout exception.