Merge pull request #135 from dusk-network/agent/operator-reliability-runbook

Document installer-backed node maintenance and recovery

Author: HDauven

src/content/docs/developer/integrations/tx-lifecycle.md

@@ -38,7 +38,7 @@ Do not use **included**, **removed**, **accepted**, or **confirmed** as a paymen
 | How are transactions prioritized? | Candidate selection iterates by descending `gasPrice`. Equal-price ordering is not a client contract. |
 | How does replacement work? | A transaction that conflicts on a spending ID replaces the existing transaction only when its gas price is strictly higher. Raising only the gas limit is insufficient. Moonlight uses account and nonce as its spending ID; Phoenix conflicts on spent nullifiers. |
 | What happens when the mempool is full? | A higher-priced transaction can evict the lowest-priced entry. Node operators configure the capacity; the default is 10,000 transactions. |
-| When do transactions expire? | Expiry is local node policy, not a field in the transaction. The default residence is three days, checked hourly, and both values are configurable. Clients must not treat three days as a network guarantee. |
+| When do transactions expire? | Expiry is local node policy, not a field in the transaction. Rusk's built-in defaults are three days with hourly checks, while node-installer `v0.5.22` configures mainnet and testnet nodes for 30 minutes with checks every five minutes. Clients must use the policy of their node instead of treating either value as a network guarantee. |
 | What happens to future Moonlight nonces? | A transaction with a nonce gap is briefly staged outside the real mempool while the node waits for intermediate nonces. It emits `deferred`, is invisible to `mempoolTxs`, and is admitted only if the gap is filled before retries end. |
 | Can every node publish transactions? | A full Rusk node can receive and rebroadcast transactions. Its operator can disable or restrict public HTTP access, apply ACLs, or rate-limit propagation. |
 

src/content/docs/operator/archive-node.md

@@ -1,111 +1,103 @@
 ---
 title: Run an archive node
-description: Learn about Dusk archive nodes that store and give access to Dusk’s historical data.
+description: Install and operate a Rusk node with finalized historical indexes.
 ---
 
-Archive nodes extend the functionality of [Provisioners](/operator/provisioner) by also preserving a complete historical record of the Dusk blockchain. While provisioners focus on the current state and consensus duties, archive mode adds long-term historical data access for applications, users, researchers, and auditors.
+An archive node runs the archive build of Rusk and retains finalized historical indexes in addition to normal chain state. Applications use it for queries such as `moonlightHistory`, `fullMoonlightHistory`, and `finalizedEvents`.
 
-Archive nodes are commonly run as data-serving infrastructure and do not need to stake DUSK. However, archive mode is built on top of the regular node stack, so an archive node can also be configured to participate in consensus if you choose to stake and run it that way.
+Archive mode does not require staking. An archive node can also participate in consensus, but production API infrastructure is usually kept separate from provisioner duties so query load and maintenance do not compete with consensus.
 
-In short, archive nodes:
-- Provide additional historical data (such as events emitted by contracts) that is not stored by a Provisioner node
-- Expose archive-only GraphQL queries such as `moonlightHistory`, `fullMoonlightHistory`, and `finalizedEvents`
-- Can also participate in consensus by staking DUSK, though that is optional
+## Requirements
 
-:::tip[Run an Archive node]
-If you want to quickly launch & run an archive node, you can use the <a href="https://github.com/dusk-network/node-installer" target="_blank">node installer</a> by following [the archive guide](/operator/archive-node).
-:::
-
-## Archive Node Specifications
+Use Ubuntu 24.04 LTS and begin with at least:
 
-Archive nodes store and serve historical data and require large storage capacity, efficient processing for concurrent requests, and fast Internet connectivity. The following recommended specifications serve as a baseline for Archive Nodes. Over time, storage requirements will increase.
+| CPU | RAM | Storage | Network |
+|---|---|---|---|
+| 4 cores at 2 GHz | 8 GB | 500 GB | 100 Mbps |
 
-| CPU            | RAM  | Storage | Network Connection |
-| :------------- | :--- | :------ | :----------------- |
-| 4 cores; 2 GHz | 8 GB | 500 GB  | 100 Mbps           |
+This is an initial planning baseline, not a capacity guarantee. Archive storage grows over time; monitor actual utilization and expansion lead time.
 
-## Run an archive node
-> A step-by-step guide to setting up a Dusk archive node.
+## Install
 
-The following guide will explain you how to install and setup an archive node on Ubuntu 24.04 through the [node installer](https://github.com/dusk-network/node-installer). This installer will set up Rusk as a service on your server, preconfigure parts of the node, and provide a couple of helper scripts.
+Run the released node installer with the archive feature:
 
-Install Rusk with the archive feature enabled by pasting the following command in your server terminal:
 ```sh
-curl --proto '=https' --tlsv1.2 -sSfL https://github.com/dusk-network/node-installer/releases/latest/download/node-installer.sh | sudo bash -s -- --feature archive
+curl --proto '=https' --tlsv1.2 -sSfL \
+  https://github.com/dusk-network/node-installer/releases/latest/download/node-installer.sh \
+  | sudo bash -s -- --feature archive
 ```
 
-This will install an archive node with the network being set to mainnet.
-
-## Configure Rusk
+Add `--network testnet` before `--feature archive` for a testnet archive node.
 
-You now should have successfully installed Rusk.
-
-A quick check with:
-
-```sh
-ruskquery version
-```
-
-Should tell you, that you are running the latest installer version.
-
-## Start your node
-
-If you've configured everything correctly, you can now start rusk:
-```sh
-service rusk start
-```
+Start Rusk and monitor synchronization:
 
-Your node will now start syncing. You can check if it indeed is by running:
 ```sh
+sudo systemctl start rusk
+systemctl is-active rusk
+ruskquery peers
 ruskquery block-height
 ```
 
-It is best to wait until your node is synced up. You can find the latest block height on [the block explorer](https://explorer.dusk.network/). Alternatively, consider [fast-syncing](/operator/guides/fast-sync) for a quicker method.
+Compare the height with the matching [network](/operator/networks/).
+
+An archive expected to answer complete history must synchronize from genesis or restore a trusted backup that includes the archive databases. The installer's `download_state` snapshot restores execution state, but it does not backfill archive indexes for blocks before that snapshot.
 
-## Enable http
+## Enable the HTTP API
 
-If you want to serve archive data to the outside world, your node needs to enable the http capabilities. This can be done by adding
+The installed API is not intended to be public by default. For local access or a reverse proxy on the same host, add this to `/opt/dusk/conf/rusk.toml`:
 
 ```toml
-# rusk.toml
 [http]
 listen = true
-listen_address = '0.0.0.0:8080'
+listen_address = '127.0.0.1:8080'
 ```
 
-To your rusk.toml file in your system's `/opt/dusk/conf` folder. That's it.
+Then restart Rusk:
 
-Now you can query the archive for data with an external client application.
+```sh
+sudo systemctl restart rusk
+```
 
-## Test archive endpoint
+Use `0.0.0.0:8080` only when the API must accept traffic on an external interface. Restrict it with a firewall, access policy, rate limits, or a controlled reverse proxy. Public `8080/tcp` access is not required for synchronization or consensus.
 
-You can check which GraphQL queries are available by retrieving the schema (SDL). On RUES, an empty body returns the schema:
-```bash
-curl -s -X POST "http://<your-node-host>:8080/on/graphql/query"
-```
+:::caution[Updates regenerate rusk.toml]
+Rerun the installer with `--feature archive` during every update. The installer regenerates `/opt/dusk/conf/rusk.toml`, so review the new file and reapply the intended `[http]` settings before returning the API to service.
+:::
 
-This should now return a different schema than a normal node returns. Archive-enabled nodes expose additional historical queries such as `moonlightHistory`, `fullMoonlightHistory`, and `finalizedEvents`.
+## Verify archive queries
 
-A regular node also exposes `checkBlock`, but the `onlyFinalized: true` behavior below is archive-only.
+List the archive fields exposed by canonical GraphQL:
 
-In order to test this endpoint, you can run the following command.
+```sh
+curl -fsS -X POST "http://127.0.0.1:8080/graphql" \
+  -H "Content-Type: application/json" \
+  --data-raw '{"query":"{ __schema { queryType { fields { name } } } }"}' \
+  | jq -r '.data.__schema.queryType.fields[].name' \
+  | grep -E '^(moonlightHistory|fullMoonlightHistory|finalizedEvents|checkBlock)$'
+```
+
+Query a known block:
 
-```bash
-curl -s -X POST "http://<your-node-host>:8080/on/graphql/query" \
-  --data-raw '{ block(height: 1) { header { hash } } }' | jq .
+```sh
+curl -fsS -X POST "http://127.0.0.1:8080/graphql" \
+  -H "Content-Type: application/json" \
+  --data-raw '{"query":"{ block(height: 1) { header { height hash } } }"}' \
+  | jq .
 ```
 
-Then use the returned block hash with `checkBlock`:
+To verify finalized-block lookup, replace `<BLOCK_HASH>` with the returned hash:
 
-```bash
-curl -s -X POST "http://<your-node-host>:8080/on/graphql/query" \
-  --data-raw 'query { checkBlock(height: 1, hash: "<block_hash>", onlyFinalized: true) }' | jq .
+```sh
+curl -fsS -X POST "http://127.0.0.1:8080/graphql" \
+  -H "Content-Type: application/json" \
+  --data-raw '{"query":"{ checkBlock(height: 1, hash: \"<BLOCK_HASH>\", onlyFinalized: true) }"}' \
+  | jq .
 ```
 
-On a non-archive node, `onlyFinalized: true` returns an error.
+Archive history is populated as blocks finalize. Verify the earliest range your application requires before declaring the service ready.
 
-## Stake with archive node
+See [Scan Moonlight deposits](/developer/integrations/historical_events/) for a bounded history consumer and [HTTP API](/developer/integrations/http-api/) for query and access-policy details.
 
-It is possible to stake and participate in consensus while the archive node is running. This is usually not recommended, but is possible since the archive is built on top of a normal provisioner node and therefore has all the capabilities to do so.
+## Stake from an archive node
 
-You can read the [node wallet guide](/operator/guides/node-wallet-setup) for a step-by-step instruction on setting up the wallet, depending on if you want the archive node to participate in consensus too or not.
+Staking is optional. If this host will also be a provisioner, follow [Set up the node wallet](/operator/guides/node-wallet-setup/) and apply the same monitoring, key-isolation, and update procedures as any other provisioner.

src/content/docs/operator/faq.md

@@ -63,7 +63,9 @@ Yes. The node only needs the consensus key file (`consensus.keys`). Export it fr
 Example (adjust user/IP):
 
 ```bash
-scp consensus.keys duskadmin@<node-ip>:/opt/dusk/conf/
+scp consensus.keys duskadmin@<node-ip>:~/consensus.keys
+ssh duskadmin@<node-ip> \
+  'sudo install -o root -g dusk -m 640 ~/consensus.keys /opt/dusk/conf/consensus.keys && rm -f ~/consensus.keys'
 ```
 
 #### What if I lose access to my server or keys?
@@ -80,13 +82,15 @@ rusk-wallet stake --amt 3000
 
 #### How can I recover my node if the state is corrupted?
 
-Reload from a snapshot and restart:
+For a default mainnet or testnet node, reload from a snapshot and restart:
 
 ```bash
-download_state
-service rusk start
+sudo download_state
+sudo systemctl start rusk
 ```
 
+For an archive node that requires complete history, follow the archive-specific [re-sync guidance](/operator/guides/manual-resync/#restore-a-published-state).
+
 #### How can I run a Dusk node on Docker?
 
 We don't support a production-ready Docker image. For ephemeral (non-persistent) usage:
@@ -111,37 +115,31 @@ Without the node installer, configure `rusk.toml`:
 [kadcast]
 public_address = "<MY_WAN_IPV4>:<NEW_PORT>"
 listen_address = "<MY_LAN_IPV4>:<NEW_PORT>" # Optional
-bootstrapping_nodes = ["165.22.193.63:9000", "167.172.175.19:9000"]
 ```
 
+Retain the bootstrapping nodes supplied for the selected network.
+
 #### How do I change the HTTP API port?
 
 In `rusk.toml`:
 
 ```toml
 [http]
 listen = true
-listen_address = "0.0.0.0:8080"
+listen_address = "127.0.0.1:8081"
 ```
 
-Or via CLI:
-
-```bash
---http-listen-addr 0.0.0.0:8081
-```
+The installer regenerates `rusk.toml` during upgrades, so review and reapply this setting afterward. Bind an external interface only when required, and restrict public API traffic.
 
 #### How can I get data from testnet or mainnet nodes?
 
 See [/developer/integrations/http-api](/developer/integrations/http-api) and [/developer/integrations/historical_events](/developer/integrations/historical_events).
 
-#### How can I relay my internal port 8080 when using RUES?
-
-```bash
-socat tcp-listen:8081,reuseaddr,fork tcp:localhost:8080
-```
-
 #### How can I perform a liveness check on my node?
 
 - `ruskquery block-height` (should increase over time)
 - `ruskquery peers`
-- `systemctl status rusk`
+- `systemctl is-active rusk`
+- `ruskquery info`
+
+See [Maintain and monitor a node](/operator/maintenance-monitoring/) for network-tip comparison, resources, logs, and alerts.

src/content/docs/operator/guides/fast-sync.md

@@ -1,34 +1,55 @@
 ---
-title: Fast-sync your node
-description:  Speed up your Dusk node synchronization by leveraging pre-available snapshot.
+title: Fast-sync a node
+description: Replace local chain state with a published mainnet or testnet snapshot.
 ---
 
-The node installer comes with an easy to use fast syncing tool. To significantly reduce the time required to sync your node to the latest published state, you can use the `download_state` command. This command stops your node and replaces its current state with the latest published state from one of Dusk's archive nodes.
+The node installer provides `download_state` for mainnet and testnet. It downloads a published state, stops Rusk, replaces the local state and chain database, and leaves the service stopped for operator verification.
 
-## Available states
+:::caution
+Fast sync replaces current chain state under `/opt/dusk/rusk`. It does not replace consensus keys or node configuration, but you should still verify you selected the intended network before confirming.
+:::
+
+Fast sync does not backfill archive indexes for blocks before the snapshot. Do not use a state snapshot alone to bootstrap an archive node that promises complete historical data.
+
+## List snapshots
+
+The command detects the network from `/opt/dusk/conf/rusk.toml`:
 
-To see the available published states, run:
 ```sh
 download_state --list
 ```
 
-## Download state
+Override detection only when necessary:
+
+```sh
+download_state --network testnet --list
+```
+
+## Replace state
+
+Download the latest snapshot:
 
-To install the latest state, simply run:
 ```sh
-download_state
+sudo download_state
 ```
 
-Once it tells you the operation is complete, run the following command to start your node again:
+Or select one of the listed block heights:
+
 ```sh
-service rusk start
+sudo download_state <BLOCK_HEIGHT>
 ```
 
-This process bootstraps your node with the latest available state snapshot, allowing it to sync the remaining blocks much faster than starting from genesis.
+The tool downloads to a unique temporary file and removes it afterward. A successful command still leaves Rusk stopped.
+
+## Restart and verify
 
-:::note
-If you are experiencing errors in downloading the state, it might be due to some remnants of previous state syncing. Try to clean up with:
 ```sh
-sudo rm /tmp/state.tar.gz
+sudo systemctl start rusk
+systemctl is-active rusk
+ruskquery peers
+ruskquery block-height
 ```
-:::
+
+Check the height again after roughly 30 seconds. The node should continue syncing from the snapshot toward the current [network tip](/operator/networks/).
+
+If the service fails, inspect `systemctl status rusk`, `/var/log/rusk.log`, and `/var/log/rusk_recovery.log` before retrying. See [Re-sync a node](/operator/guides/manual-resync/) for the decision process.