> For the complete documentation index, see [llms.txt](https://cantonindexer.infradao.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cantonindexer.infradao.com/troubleshooting.md). # Troubleshooting Each entry gives the message you see, its cause, and the fix. `cinder` prints errors as `Error: ...` followed by a `Caused by:` list. The line that names the cause is usually the last one. ## macOS blocks cinder **You see:** macOS refuses to open `cinder` and says it cannot check it for malware. **Cause:** the macOS binaries are not signed with an Apple Developer ID and not notarized. The Apple silicon binary carries only an ad-hoc signature, and the Intel binary has none. `codesign -dv ./cinder-aarch64-darwin` shows `Signature=adhoc`. macOS checks downloaded apps that are not notarized before it opens them. **Fix:** after the first refused attempt, open System Settings, then Privacy & Security, and click **Open Anyway**. Or remove the download mark from the file: ```sh xattr -d com.apple.quarantine /usr/local/bin/cinder ``` ## Unknown field `node` in the configuration file ``` Error: parsing the configuration file /home/you/project/cinder.toml Caused by: TOML parse error at line 2, column 1 | 2 | node = "http://127.0.0.1:8020" | ^^^^ unknown field `node`, expected `pqs_url` ``` **Cause:** your `cinder` is older than v0.10.0, the release that added `deploy`, and it refuses any key it does not know. **Fix:** upgrade `cinder` (`brew upgrade cinder`, or download the latest release). Deploying needs a `cinder` that has the `deploy` command: an older one answers `error: unrecognized subcommand 'deploy'`. A current `cinder` gives the same error for any other key it does not know, ending in `` expected `pqs_url` or `node` `` . Check the key's spelling. ## No PQS connection URL ``` Error: no PQS connection URL: set `pqs_url` in a configuration file at one of /home/you/project/cinder.toml, /home/you/.config/cinder/config.toml ``` **Cause:** `cinder` found no configuration file. The message lists the paths it tried. If it found a file without `pqs_url`, the message names that file: ``` Error: no PQS connection URL: set `pqs_url` in the configuration file /home/you/project/cinder.toml ``` **Fix:** create `./cinder.toml` with `pqs_url`, or pass `--config PATH`. The URL cannot come from a flag or an environment variable. Libpq environment variables such as `PGPASSWORD` and `PGSSLMODE` fill in only what the URL leaves out. ## `pool timed out while waiting for an open connection` ``` Error: connecting to PQS Caused by: 0: connecting to PQS 1: pool timed out while waiting for an open connection ``` **Cause:** `cinder` could not open a connection to the host and port in `pqs_url`. It retries for 30 seconds before it fails with this message, also when the connection is refused. A host name that does not resolve fails at once, with `failed to lookup address information`. **Fix:** check the host and port, and that your network reaches them (a VPN, a firewall, a port-forward that has stopped). ## `database "..." does not exist` **Cause:** the server in `pqs_url` has no database with that name. **Fix:** correct the database name, the part after the port in `pqs_url` (`pqs` in `postgres://user:password@pqs-host:5432/pqs`). ## `password authentication failed for user "..."` **Cause:** the user or password in `pqs_url` is wrong. **Fix:** correct them. Characters such as `@`, `:` and `/` in a password must be percent-encoded in the URL. ## TLS errors when connecting to PQS | Message | Cause | Fix | | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | | `server does not support TLS` | `sslmode=require` (or stricter) and the server has TLS off. | Turn TLS on at the server, or drop `sslmode`. | | `invalid peer certificate: UnknownIssuer` | `sslmode=verify-ca` or `verify-full`, and the server's certificate comes from a CA outside the Mozilla bundle built into `cinder`. | Add `&sslrootcert=/path/to/ca.crt` to `pqs_url`. | See [TLS to PQS](/install.md#tls-to-pqs) for the `sslmode` values. ## `relation "__contract_tpe" does not exist` or `function oldest_offset() does not exist` **Cause:** `pqs_url` reaches a Postgres database that is not a PQS database. `cinder types` reads PQS's type registry and `preview` and `run` call PQS's SQL functions, so they fail on any other database. **Fix:** point the database name in `pqs_url` at the PQS database. ## `the source has no readable offsets: nothing has been ingested yet` **Cause:** PQS is reachable but has not stored any transaction yet. **Fix:** wait for PQS to ingest data, and check its logs if it does not. ## An `exercises` filter writes nothing **Check:** ```sh cinder preview --source exercises --limit 1 cinder preview --source lifecycle --limit 1 ``` If `lifecycle` prints a record and `exercises` prints nothing, your PQS stores no exercises. **Cause:** PQS runs with its default datasource, `TransactionStream`, which does not store exercises. **Fix:** run PQS with `--pipeline-datasource=TransactionTreeStream` (configuration key `pipeline.datasource`). Digital Asset does not support switching an existing PQS database between modes; plan to reseed it. ## A preview shows records but a run writes nothing **Cause:** a preview reads every offset PQS holds. A run reads only 1000 offsets from its start unless you pass `--to`. The run logs the range it reads. **Fix:** pass `--from` and `--to` that cover the records you saw in the preview (their `.offset` field). ## Deploy errors | Message | Cause | Fix | | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | | ``no node URL: set `node` in ...`` | The configuration file has no `node`. | Add `node = "http://host:8020"` or an `https://` URL. | | `a node URL starts with http:// or https://` | `node` has no scheme or another one. | Write the full URL. | | `connecting to the node at ...` followed by `Connection refused` | Nothing listens at that address. | Check the node is running and `NO_AUTH_GRPC_ADDR`, ports and firewalls. In a container it must be a non-loopback address, such as `0.0.0.0:8020`. | | `invalid peer certificate: UnknownIssuer` | An `https://` node whose certificate is not from a CA in the Mozilla bundle. `cinder` has no option to trust another CA. | Serve the node's TLS with a certificate from a public CA. | | `received corrupt message of type InvalidContentType` | An `https://` URL for a node that serves plain gRPC. | Use `http://`, or put a proxy that terminates TLS in front of the node. | | `a filter named "..." is already registered; replace it to redeploy (AlreadyExists)` | The name is taken. | Deploy with `--replace`. This deletes the filter's documents and indexes again from the start. | | a message ending in `canceling statement due to lock timeout (Unavailable)` | A `--replace` waited more than 10 seconds for the filter's running commit. | Run the deploy again. | ## A deployed filter stopped: status `failed` **Check:** in the node's store: ```sql select f.name, s.status, s.failure from indexer.filters f join indexer.filter_status s on s.filter_id = f.id; ``` **Cause:** the filter hit an error that a retry cannot fix, such as a record its `select` cannot evaluate under `on_error = "halt"`. `failure` holds the reason. Errors that can pass, such as a lost connection to PQS, do not mark the filter `failed`: the node restarts its task with growing waits instead (see `RESPAWN_BACKOFF_SECS` in [Install](/install.md#run-the-node)). **Fix:** correct the filter or the cause, then deploy it with `--replace`. **Cause: an offset outside PQS's range.** `failure` ends in ``` Offset is before the oldest known offset ``` or `Offset is after the latest known offset `. PQS refuses a read with a bound below `oldest_offset()` or above `latest_offset()` (SQLSTATE `P0001`), and a retry cannot fix that, so the filter is marked `failed`. This happens when: * PQS was pruned past the cursor of a filter that lagged or was stopped; * `start_offset` is outside PQS's range, for example copied from another validator: offsets are per participant; * PQS was reseeded or restored to a point behind the cursor. **Fix:** correct `start_offset` if it is the cause, then deploy with `--replace`. A replace deletes the filter's documents and indexes again from `start_offset`, or from the oldest offset PQS holds now: what PQS has pruned cannot be indexed again. Back up the filter's schema first, for example with `pg_dump -n target_v` against the store. ## A sort over the documents is slow Run the query under `explain` against the store and read the plan: ```sql explain select document->>'validator' as validator from target1_v1.documents where archived_at_offset is null order by (document->>'_offset')::numeric desc, (document->>'_node_id')::numeric desc, contract_id desc limit 10; ``` **Cause:** Postgres uses an index for an `order by` only when the index has the same expressions, in the same direction or all reversed. * canton-indexer v0.10.0 creates no index on `_offset` in a lifecycle table. Any sort over `_offset` reads the whole table: the plan shows `Seq Scan` and `Sort`, and the time grows with the table. * Unreleased code (\[future release]) adds the index `documents_position` on `(document->>'_offset')::numeric`, `(document->>'_node_id')::numeric` and `contract_id`. What the plan shows depends on the `order by`: | `order by` | Plan | | --------------------------------------------------- | ---------------------------------------------------------------- | | the three keys as above, all `desc` (or all `asc`) | `Index Scan Backward using documents_position` (or `Index Scan`) | | the same keys, `contract_id` in the other direction | `Incremental Sort` over the index scan | | `::bigint` casts instead of `::numeric` | `Seq Scan` and `Sort`: the expressions differ | * An `exercises` table has its primary key on `(event_offset, event_node_id)` in every version, so `order by event_offset desc, event_node_id desc` shows `Index Scan Backward using documents_pkey`. **Fix:** write the `order by` as above. On v0.10.0 the sort stays a full scan until a release carries the index. With `where archived_at_offset is null`, the scan skips the archived rows it meets (`Rows Removed by Filter` under `explain analyze`), and an archive's `_offset` is its archive offset, so many recent archives slow the first rows. ## Sources * Apple, Safely open apps on your Mac: * Release v0.9.0 binaries (inspect with `codesign -dv`): * Release v0.10.0, the first with `cinder deploy`: * Release notes listing every release: * PQS datasource option and its default, PQS v3.5.8: * Switching PQS modes needs a reseed (Canton Network forum): * sqlx `PgConnectOptions` (`sslmode`, `sslrootcert`): * Filter format at v0.9.0 (preview and run ranges): * Sort plans: Postgres 17 (the store image of canton-indexer's compose file), run on 2026-11-03 with the store DDL of canton-indexer v0.10.0 and of its main branch, over 200,000 rows.