Knowledge Graph health & migration

The KG consolidation pipeline is a queue-driven background worker. When it gets stuck — embedder timeouts, schema drift, slow disk — the agent stops being able to write decisions to the graph. Five MCP admin tools surface the pipeline’s state and let an operator unblock it without touching the database file:

Tool	Line	Use it when…
`okto_pulse_kg_health`	12484	You want a single-call summary: queue depth, oldest pending age, dead-letter count, decay tick freshness, and relevance score health.
`okto_pulse_kg_dead_letter_list`	12527	Consolidations are failing and you need to see what bounced.
`okto_pulse_kg_dead_letter_reprocess`	12570	The underlying issue is fixed and you want to retry the failed entries.
`okto_pulse_kg_migrate_schema`	12643	You upgraded Pulse and need to bring the board’s `graph.lbug` to the current schema version.
`okto_pulse_kg_tick_run_now`	12716	You don’t want to wait for the daily decay tick to recompute relevance scores.

Permissions are dotted-string flags in the granular registry. kg_health and kg_dead_letter_list are read-only and ride on the standard kg.query.* / kg.admin.settings_read flags most presets ship with. The two that mutate state — kg_dead_letter_reprocess and kg_migrate_schema (and the historical-consolidation CLI path) — are gated by kg.admin.historical_consolidation and the broader kg.admin.* namespace. Operator presets layer these in deliberately. Source: okto-pulse-core/src/okto_pulse/core/mcp/server.py:12484–12857 and core/infra/permissions.py:PERMISSION_REGISTRY under the kg.admin key. Citations: 80-pulse-feature-inventory.md:493–501. For consolidation flow itself, see consolidation. For the schema, see overview.

`okto_pulse_kg_health`

The board’s pipeline health summary. The Pulse dashboard polls this every 30 seconds.

Input:
  board_id: "brd_abc123"

The MCP tool returns a 12-field aggregate computed in-process (cheap to poll). Implemented in okto-pulse-core/src/okto_pulse/core/services/kg_health_service.py:get_kg_health. Real shape:

Output:
{
  "queue_depth":             12,
  "oldest_pending_age_s":    41.8,
  "dead_letter_count":       0,
  "total_nodes":             4178,
  "default_score_count":     312,
  "default_score_ratio":     0.0747,
  "avg_relevance":           0.612,
  "top_disconnected_nodes":  [
    {"node_id": "ent_91", "node_type": "Entity",     "degree": 0},
    {"node_id": "lrn_03", "node_type": "Learning",   "degree": 0}
  ],
  "schema_version":          "1.0",
  "contradict_warn_count":   2,
  "last_decay_tick_at":      "2026-05-07T03:00:00Z",
  "nodes_recomputed_in_last_tick": 388
}

Field meanings (per docstring at server.py:12484 and the service implementation):

Field	What it means
`queue_depth`	Pending consolidation rows in the SQLite queue. High values mean enqueues are landing faster than the worker drains them.
`oldest_pending_age_s`	Age, in seconds, of the oldest pending consolidation row. `null` when the queue is empty.
`dead_letter_count`	Rows that exceeded `kg_queue_max_attempts` (default 5). Inspect with `kg_dead_letter_list`.
`total_nodes`	Total node count in the board’s `graph.lbug`.
`default_score_count` / `default_score_ratio`	Nodes still at the default `relevance_score` (never recomputed). Ratio above ~0.7 means the decay tick isn’t keeping up — see `kg_tick_run_now`.
`avg_relevance`	Mean `relevance_score` across all nodes.
`top_disconnected_nodes`	Lowest-degree nodes (by edge count). Useful for spotting orphaned consolidations.
`schema_version`	Health response schema version. Use `kg_migrate_schema` or `kg_schema_info` when checking the graph schema itself.
`contradict_warn_count`	Running count of `contradict_penalty` cap events. A spike means the curator should reconcile.
`last_decay_tick_at`	Most recent decay-tick run.
`nodes_recomputed_in_last_tick`	Number of nodes recomputed by the most recent decay tick.

CLI equivalent — note that the CLI uses a different layered check set rather than this 12-field aggregate:

okto-pulse verify-pipeline brd_abc123 --json

cli.py:683–744 (cmd_verify_pipeline) runs 5 layered checks: queue depth, graph file presence + node count, graph-vs-SQLite ref mirror, outbox staleness, global discovery file. Exit code 0 if healthy, 1 if any layer fails. Use the CLI for monitoring scripts; use kg_health from agents.

`okto_pulse_kg_dead_letter_list`

Consolidation entries that exceeded kg_queue_max_attempts (default 5) land in the dead-letter table. Pulse never auto-reprocesses them — an operator must inspect and replay.

Input:
  board_id: "brd_abc123"
  limit:    50
  offset:   0

Output:
{
  "board_id": "brd_abc123",
  "total":    3,
  "entries": [
    {
      "id":              "dl_001",
      "source_type":     "spec",
      "source_id":       "spec_007",
      "first_failed_at": "2026-05-06T22:11:04Z",
      "attempts":        5,
      "last_error":      "embedder.timeout: sentence-transformers exceeded 30s",
      "session_id":      "kgs_01HV..."
    },
    {
      "id":              "dl_002",
      "source_type":     "card",
      "source_id":       "card_91",
      "first_failed_at": "2026-05-07T01:22:18Z",
      "attempts":        5,
      "last_error":      "graph.lock_timeout: failed to acquire write lock"
    }
  ]
}

`okto_pulse_kg_dead_letter_reprocess`

Move dead-letter entries back to the active queue for another attempt. Use after the underlying cause is resolved (embedder up, disk space available, schema migrated).

Input:
  board_id: "brd_abc123"
  entry_ids: ["dl_001", "dl_002"]    # or omit to reprocess all

Output:
{
  "board_id":  "brd_abc123",
  "requeued":  2,
  "skipped":   0,
  "entry_ids": ["dl_001", "dl_002"]
}

The reprocess increments the attempts counter back to 0 for each requeued entry. If the same root cause persists, entries will land back in dead-letter after another kg_queue_max_attempts failures.

`okto_pulse_kg_migrate_schema`

Run schema migrations on a board’s graph.lbug. Use after upgrading okto-pulse-core to a version with a higher schema version than the file on disk.

Input:
  board_id: "brd_abc123"
  target_version: "0.3.3"           # optional — defaults to the runtime's current schema version
  dry_run: false

Output:
{
  "board_id":      "brd_abc123",
  "from_version":  "0.3.1",
  "to_version":    "0.3.3",
  "migrations_applied": [
    {"id": "0001_add_belongs_to_multi_pair", "took_ms": 412},
    {"id": "0002_hnsw_metric_to_cosine",     "took_ms": 1840}
  ],
  "node_count_before": 4178,
  "node_count_after":  4178,
  "ok": true
}

Migrations are idempotent: a board already at target_version returns migrations_applied: [] and ok: true.

Always take a copy of ~/.okto-pulse/boards/{board_id}/graph.lbug before running a migration with dry_run: false. Migrations rewrite the file in place. The okto-pulse kg backfill --apply flow is a safer rebuild path when a migration corrupts data.

`okto_pulse_kg_tick_run_now`

Trigger the decay tick worker immediately instead of waiting for the schedule (default daily, kg_decay_tick_interval_minutes = 1440). The tick recomputes relevance_score for nodes whose last_recomputed_at is older than kg_decay_tick_staleness_days (default 7).

Input:
  board_id: "brd_abc123"

Output:
{
  "board_id":         "brd_abc123",
  "started_at":       "2026-05-07T15:30:11Z",
  "completed_at":     "2026-05-07T15:30:14Z",
  "nodes_recomputed": 312,
  "nodes_skipped_fresh": 3866
}

The decay formula is documented in kg/workers/kg_decay_tick.py. Note that find_similar_decisions uses a separate search reranking formula at retrieval time — do not conflate the two (80-pulse-feature-inventory.md:957).

Hot-reloadable settings

You don’t need to restart Pulse to change pipeline tuning:

Setting group	Hot-reload mechanism
`kg_queue_*` (worker count, claim timeout, max attempts, alert threshold, recovery scan)	APScheduler re-reads the value with a 5-second debounce. No action required.
`kg_decay_tick_*` (interval, staleness, max age)	`PUT /settings/runtime` — applies on next tick boundary.

Source: 80-pulse-feature-inventory.md:790. The full settings table lives in Knowledge Graph and the inventory.

CLI fallbacks

The MCP tools are the primary surface, but three CLI commands cover deeper recovery scenarios:

`okto-pulse verify-pipeline <board_id>`

cli.py:683–744. Wraps the same 5 checks as kg_health but exits 1 on failure — useful in CI / monitoring scripts.

okto-pulse verify-pipeline brd_abc123 --json

`okto-pulse kg backfill <board_id>`

cli.py:747–902. Runs the Layer 1 deterministic KG worker against every artifact on the board.

# Dry run: report what would be emitted, no writes
okto-pulse kg backfill brd_abc123

# Apply: enqueue all artifacts, drain the consolidation queue
okto-pulse kg backfill brd_abc123 --apply

# Filter to one artifact type
okto-pulse kg backfill brd_abc123 --apply --artifact-type spec

This is the recovery path when the graph is structurally out of sync (e.g., after a partial migration or a manual file restore). It rebuilds the deterministic skeleton; it does not replay cognitive-agent decisions.

`okto-pulse kg dedup-entities <board_id>`

cli.py:908–936. Consolidate duplicate nodes per (node_type, source_artifact_ref).

okto-pulse kg dedup-entities brd_abc123 --dry-run
okto-pulse kg dedup-entities brd_abc123

kg dedup-entities writes by default. Always run with --dry-run first.

Underlying REST endpoints

The MCP tools wrap REST endpoints exposed by the API server. They are documented here for completeness — most callers should prefer the MCP tools.

Method	Path	Equivalent MCP tool
`GET`	`/kg/health`	`kg_health`
`GET`	`/kg/queue/health`	`kg_health` (subset)
`POST`	`/kg/tick/run-now`	`kg_tick_run_now`
`GET`	`/kg/dead-letter/...`	`kg_dead_letter_list`
`POST`	`/kg/dead-letter/reprocess`	`kg_dead_letter_reprocess`

Source: 80-pulse-feature-inventory.md:729–735.

Common operational scenarios

”Consolidations stopped landing”

Call kg_health — check queue_depth, oldest_pending_age_s, and dead_letter_count.
If dead_letter_count > 0, call kg_dead_letter_list to see error reasons.
Fix the root cause (embedder, disk, schema mismatch).
Call kg_dead_letter_reprocess with the entry ids.
Re-call kg_health. Expect dead_letter_count: 0 and queue_depth draining.

”Just upgraded Pulse, dashboard shows schema drift”

Inspect the graph schema with kg_schema_info or run the migration tool when a schema error points to drift.
Take a backup of graph.lbug.
Call kg_migrate_schema with default target_version (= runtime version).
Re-check the graph schema. Expect it to match the runtime schema.

”Relevance scores look stale”

Call kg_tick_run_now. Expect nodes_recomputed > 0.
If always 0 nodes recomputed, increase kg_decay_tick_staleness_days lower bound or check that nodes are being touched at all.

”Need to rebuild the deterministic skeleton from scratch”

CLI only: okto-pulse kg backfill <board_id> (dry-run) — review.
okto-pulse kg backfill <board_id> --apply.
Call kg_health — confirm graph_node_refs is balanced.

Next steps

Consolidation

The 7 transactional write primitives the queue is feeding.

Archive & retention

Cascading entity archive, supersedence as soft-archive, and KG retention policy.

Get started

Concepts

Configuration

Reference

Permissions & Agents

Guides

KG Operations

Operations

Help

Knowledge Graph health & migration

Knowledge Graph health & migration

`okto_pulse_kg_health`

`okto_pulse_kg_dead_letter_list`

`okto_pulse_kg_dead_letter_reprocess`

`okto_pulse_kg_migrate_schema`

`okto_pulse_kg_tick_run_now`

Hot-reloadable settings

CLI fallbacks

`okto-pulse verify-pipeline <board_id>`

`okto-pulse kg backfill <board_id>`

`okto-pulse kg dedup-entities <board_id>`

Underlying REST endpoints

Common operational scenarios

”Consolidations stopped landing”

”Just upgraded Pulse, dashboard shows schema drift”

”Relevance scores look stale”

”Need to rebuild the deterministic skeleton from scratch”

Next steps

Consolidation

Archive & retention

​Knowledge Graph health & migration

​okto_pulse_kg_health

​okto_pulse_kg_dead_letter_list

​okto_pulse_kg_dead_letter_reprocess

​okto_pulse_kg_migrate_schema

​okto_pulse_kg_tick_run_now

​Hot-reloadable settings

​CLI fallbacks

​okto-pulse verify-pipeline <board_id>

​okto-pulse kg backfill <board_id>

​okto-pulse kg dedup-entities <board_id>

​Underlying REST endpoints

​Common operational scenarios

​”Consolidations stopped landing”

​”Just upgraded Pulse, dashboard shows schema drift”

​”Relevance scores look stale”

​”Need to rebuild the deterministic skeleton from scratch”

​Next steps

Consolidation

Archive & retention

Knowledge Graph health & migration

`okto_pulse_kg_health`

`okto_pulse_kg_dead_letter_list`

`okto_pulse_kg_dead_letter_reprocess`

`okto_pulse_kg_migrate_schema`

`okto_pulse_kg_tick_run_now`

Hot-reloadable settings

CLI fallbacks

`okto-pulse verify-pipeline <board_id>`

`okto-pulse kg backfill <board_id>`

`okto-pulse kg dedup-entities <board_id>`

Underlying REST endpoints

Common operational scenarios

”Consolidations stopped landing”

”Just upgraded Pulse, dashboard shows schema drift”

”Relevance scores look stale”

”Need to rebuild the deterministic skeleton from scratch”

Next steps