claude-code — parsidion
Building vault-deduplicator
with Claude Code
From a one-liner question to a fully-automated AI dedup pipeline
user ~/Repos/parsidion $ claude
Claude Code  —  parsidion
✓ Vault context loaded  ·  1,531 notes indexed
  • Add --no-index flag to vault-merge for batch workflows
  • Run vault-merge --scan to find 50 near-duplicate pairs
  • Launch 10 parallel Haiku agents to evaluate & merge 46 pairs
  • Create the vault-deduplicator agent codifying the workflow
  • Sync all docs and commit
vault-merge parallel agents 46 merges haiku model
Does vault-merge re-index?
A simple question reveals a missing feature
user → does vault-merge trigger re-index after a merge?
claude → grep -r "update_index" skills/parsidion/scripts/vault_merge.py
No matches found
No — vault-merge does not trigger a re-index after merging.
You'd need to run it manually:

$ uv run --no-project ~/.claude/skills/parsidion/scripts/update_index.py

Want me to add an automatic re-index call at the end of a successful merge?
user → yes, but add a --no-index option to disable it
Gap identified: batch workflows need --no-index to skip per-merge rebuilds and do one final rebuild at the end — much faster for 50+ merges.
Adding --no-index to vault-merge
Three targeted edits to vault_merge.py
import argparse, re, shutil, struct, sqlite3 +import subprocess import sys from pathlib import Path
def _rebuild_index() -> None: index_script = Path(__file__).parent / "update_index.py" if not index_script.exists(): index_script = Path.home() / ".claude/skills/parsidion/scripts/update_index.py" subprocess.run(["uv", "run", str(index_script)], check=True, ...)
parser.add_argument("--no-index", action="store_true", help="Skip rebuilding the vault index after a successful merge.") # after git_commit_vault(): if not args.no_index: _rebuild_index()
✓ vault_merge.py updated · synced to ~/.claude/skills/parsidion/scripts/
vault-merge --scan
Embedding similarity reveals 50 near-duplicate pairs
user $ vault-merge --scan
Found 50 near-duplicate pair(s) (threshold=0.92): 1. [0.9995] burning/ship-fractal-implementation-0929 burning/ship-fractal-implementation-0843 → vault-merge ship-fractal-implementation-0929 ship-fractal-implementation-0843 2. [0.9980] burning/ship-fractal-implementation-0929 burning/ship-fractal-implementation 3. [0.9977] qdrant/capacity-planning-essentials-0928 qdrant/capacity-planning-essentials ... 50. [0.9833] Patterns/terminal-emulator-core-architecture-patterns Patterns/terminal-emulator-core-architecture-patterns-0801
All pairs scored ≥ 0.9833 — well above the 0.92 threshold. Most are timestamped variants (e.g. foo-0929 vs foo) created when the summarizer wrote the same insight across multiple sessions.
Identifying Chain Dependencies
Some notes appear in multiple pairs — order matters
Before launching agents, I identified 6 chain groups where the same stem appears as both NOTE_A in one pair and NOTE_B in another. These must be processed sequentially within a single agent.
ship-fractal: 3 variants 0929 + 0843 → then base + 0929 par-fractal: 3 variants 0804 + 0803 → then base + 0804 pydantic-fix: 3 variants 0758 + 0759 → then base + 0758 ai-service-client: 3 variants 0802 + base → then 0803 + base tmux-click-jitter: 3 variants 1028 + base → then base + tmux/ parsidion-arch: 3 variants 0806 + base → then Patterns/ + Projects/
◆ 10 parallel Haiku agents
◆ Chains stay within 1 agent
◆ Independent pairs batched 5/agent
◆ --no-index on every merge
◆ One index rebuild at the end
◆ Skip redundant pairs after chain
◆ Cross-folder pairs: careful eval
◆ Prefer base name as NOTE_A
6 chains ~36 independent pairs 4 pairs needing careful review
10 Parallel Haiku Agents
Each agent reads both notes, decides validity, executes merge
claude → Agent × 10 (model: haiku, parallel)
1
ship-fractal chain
⟳ running
2
qdrant/research batch
⟳ running
3
debugging batch
⟳ running
4
tmux + parsidion chains
⟳ running
5
patterns batch
⟳ running
6
pydantic + ai-service chains
⟳ running
7
par-fractal chain + colbert
⟳ running
8
debugging/patterns batch
⟳ running
9
cross-named + special
⟳ running
10
patterns batch 6
⟳ running
Each agent: (1) Read both notes  (2) Evaluate same topic?  (3) If YES → vault-merge A B --no-index --execute  (4) Report decision + reason
Agents 1–4 Complete
Ship-fractal chain, qdrant batch, debugging batch, tmux+parsidion chains
🔗
Agent 1 — ship-fractal chain
2 merged · 0929+0843 then base+0929
✓
Agent 2 — qdrant/research (pairs 3,5,6,7,8)
5 merged · all identical content
✓
Agent 3 — debugging/qdrant (pairs 9,10,12,13,15)
5 merged · same session_id on each pair
🔗
Agent 4 — tmux chain + parsidion arch chain
3 merged · 1 skipped (tmux/ note missing)
Agent 4 detail: Pair 14: tmux-click-jitter-1028 → tmux-click-jitter ✓ merged Pair 11: tmux-click-jitter (Debugging/) ↔ click-jitter (tmux/) ⚠ note missing Pair 16: parsidion-arch-0806 → parsidion-arch ✓ merged Pair 32: parsidion-arch-gap (Patterns/) → parsidion-arch ✓ merged
15 merged so far 1 skipped
Agents 5–7 Complete
Patterns batch, pydantic+ai-service chains, par-fractal chain
✓
Agent 5 — patterns batch (pairs 17-21)
5 merged · synknot, mandelbrot, parsplat, frontmatter
🔗
Agent 6 — pydantic chain + ai-service chain
5 merged · 6 notes consolidated (chains)
🔗
Agent 7 — par-fractal chain + colbert/brush/yaml
6 merged · 0804+0803 then base+0804
Agent 6 chain detail: pydantic-fix: 0758+0759 → base+0758 (3 notes → 1) ✓ pydantic-pitfalls: 0759 → base (2 notes → 1) ✓ ai-service: 0802+base → 0803+base (3 notes → 1) ✓ Agent 7 chain detail: par-fractal: 0804+0803 → base+0804 (3 notes → 1) ✓
31 merged so far chains preserved ordering
Agents 8–10 Complete
Debugging/patterns, cross-named specials, final patterns batch
✓
Agent 8 — agentchrome, arkit, par-term, codable, ai-writing
5 merged · identical session_id on all pairs
⚠
Agent 9 — cross-named pairs (careful eval)
5 merged · same session_id confirmed same content
✓
Agent 10 — secret-env, mainactor, fastapi, claudevault, terminal
5 merged · classic summarizer duplicates
Agent 9 cross-name decisions: path-traversal-vault-boundary-fixes ↔ vault-path-traversal-boundary-fixes ✓ MERGED same session_id SEC-001/005 image-tagging-pipeline-architecture ↔ ai-image-tagging-pipeline-architecture ✓ MERGED same par_image_tag project
46 merged total 4 skipped ~170 wikilinks updated
All Agents Done — Rebuilding Index
Final tally and one rebuild to rule them all
50
pairs scanned
46
notes merged
4
skipped / invalid
claude → uv run --no-project ~/.claude/skills/parsidion/scripts/update_index.py
Updated CLAUDE.md: 1531 notes indexed, 490 tags; 8 MANIFEST.md file(s) generated Embeddings: incremental rebuild launched in background
--no-index saved ~46 sequential index rebuilds. Instead: one parallel batch of merges, one final rebuild. Significant time savings on large vaults.
1,531 notes 490 tags embeddings rebuilding in background
Creating vault-deduplicator Agent
Codifying the workflow so any future session can run it
user → create a new agent for the dedupe process we just ran, following existing naming conventions
--- name: vault-deduplicator description: > Use when you need to find and merge near-duplicate notes in ~/ClaudeVault/. Scans, evaluates, merges, rebuilds. Trigger on: "deduplicate the vault", "find duplicate notes", "merge duplicate vault notes", "vault has duplicates" ... model: haiku color: cyan ---
  • Step 1 — vault-merge --scan
  • Step 2 — Identify chain dependencies
  • Step 3 — Batch parallel Agent launches (Haiku)
  • Step 4 — One final update_index.py
✓ Follows vault-* naming pattern ✓ Haiku model (utility task) ✓ Cyan color (distinct from others) ✓ Synced to ~/.claude/agents/
Syncing All Docs
7 parallel agents check each doc for implementation drift
user → update docs · also update skill to be aware of it
claude → Agent × 8 (7 docs + SKILL.md update)
ARCHITECTURE.md ✓ Added vault-deduplicator to ToC, Mermaid diagram, component details, and both file layout trees README.md ✓ No drift — already accurate MCP.md ✓ Added --migrate-subfolders/--execute flags; removed brittle model size MCPL.md ✓ Fixed --activate/--deactivate to accept index or path EMBEDDINGS.md ✓ Documented auto-detection of dimension mismatch on incr. rebuild EMBEDDINGS_EVAL.md ✓ Fixed "sortable by column" → "pre-sorted by MRR descending" AGENTCHROME.md ✓ Updated fallback: curl OR Claude Code Web Fetch tool
SKILL.md: Added vault-deduplicator agent table + ## Vault Merge section with --no-index batch-merge pattern · synced to ~/.claude/skills/
Commit & Ship
9 files, one atomic commit
user → commit all
On branch main · ahead of origin/main by 19 commits Changes to commit: M docs/AGENTCHROME.md M docs/ARCHITECTURE.md M docs/EMBEDDINGS.md M docs/EMBEDDINGS_EVAL.md M docs/MCP.md M docs/MCPL.md M skills/parsidion/SKILL.md M skills/parsidion/scripts/vault_merge.py A agents/vault-deduplicator.md 9 files changed, 232 insertions(+), 13 deletions(-)
[main a8a3c5d] feat(vault): add vault-deduplicator agent and vault-merge --no-index flag Merged: vault-deduplicator agent · --no-index flag · 7 doc syncs · SKILL.md update
What We Built
A fully-automated vault deduplication pipeline
  • vault-merge --no-index flag for batch workflows
  • _rebuild_index() auto-called after merge
  • vault-deduplicator agent (haiku, cyan)
  • 10 parallel Haiku agents merged 46/50 pairs
  • ~170 wikilinks updated across vault
  • 1,531 notes re-indexed, 490 tags
  • 7 docs synced, SKILL.md updated
vault-merge --scan
vault-merge A B --no-index --execute
vault-merge A B --execute (auto re-index)
vault-deduplicator agent trigger
update_index.py final rebuild
46 duplicate pairs merged vault-deduplicator agent --no-index batch pattern commit a8a3c5d