Docs / Knowledge Base / Agent memory and world-model repos

Knowledge Base

Contents

Agent memory and world-model repos

Part: Engineering · Chapter: Agent memory and world-model repos · Edition: 2026-10-06
Keywords: temporal knowledge graph, bi-temporal edge, episode, entity resolution, MinHash/LSH, swarm simulation, graph of agents, computer-use agent, MCP, identity quality, hybrid retrieval
Glossary: bi-temporal edge, episode (agent memory), entity resolution, identity quality, MinHash / LSH, reciprocal rank fusion, swarm simulation, graph of agents, computer-use agent — see GLOSSARY

At a glance

  • Five open repos, read at a pinned commit on 2026-10-06: Graphiti (temporal knowledge graph memory), MiroFish (multi-agent social simulation to “forecast”), screenpipe (local screen and audio capture as memory), Cua (computer-use agents in VMs and containers), Strix (a graph of agents in a Docker sandbox; read at architecture level only).
  • Only Graphiti has a real concept-graph method: typed entities and facts, episodes as provenance, deterministic-then-LLM entity resolution, and bi-temporal edges that are invalidated, never deleted.
  • None of the five does part-of-speech tagging, dependency parsing, RDF/OWL, or entailment. Their NLP is LLM extraction plus embeddings plus BM25. On this desk, Stanza + Universal Dependencies (the certified clock in Concepts governance) is already more linguistic than any of them.
  • None has an argument-map or propositional structure. MiroFish’s “forecast” is a narrative report written by a tool-using agent over a simulated social feed; nothing is scored or calibrated.
  • What transfers to Situation Monitor (SM) and CandleScope (CS): bi-temporal fields and episode ids on every edge, a tri-state identity map (unique / ambiguous / not found), typed edge signatures, and permission-bounded agent tools.

Online references (live)

Verified 2026-10-06. Each URL below returned HTTP 200; titles are the page titles.

Index card (the Knowledge Base) — how five 2026 agent repos represent a world (entities, facts, time, identity) and run agents over it, read against the IEML thesis in the Engineering index and the gates in Concepts governance.

Method: shallow clone, git rev-parse HEAD, then targeted reads of the files cited below. Paths are relative to each repo root. Nothing was run. Where a claim comes from a README rather than code, the text says so.


0. Pinned sources

Graphiti

Commit (HEAD, 2026-10-06)
aa5bb2706929fce502d99d8c7c4ddbb77bff4995
License
Apache-2.0
One-line role
Temporal knowledge graph (“context graph”) for agent memory

MiroFish

Commit (HEAD, 2026-10-06)
7657031ac01184afe2cb220f5ee3545573b5e843
License
AGPL-3.0
One-line role
Seed text → ontology → Zep graph → OASIS agent simulation → report agent

screenpipe

Commit (HEAD, 2026-10-06)
a8a2c3748713b09a4462985b2e9861465f478d87
License
Screenpipe Commercial License (source-available; personal / non-commercial use free; earlier versions MIT)
One-line role
Event-driven screen + audio capture into local SQLite, exposed to agents

Cua

Commit (HEAD, 2026-10-06)
12365b36d41417cae34eb0f90b49e8eaf023ac51
License
MIT by default; Cua Spaces and some crates FSL-1.1-MIT; optional perception extension AGPL-3.0 (OmniParser) — see LICENSING.md
One-line role
Computer-use driver, sandboxes, agent loops, benchmarks

Strix

Commit (HEAD, 2026-10-06)
f1386cad37964197a483ff43c25161f9d933f050
License
Apache-2.0
One-line role
Graph of LLM agents with a Docker sandbox and an MCP registry

License matters for reuse. SM is a network service: copying MiroFish code (AGPL-3.0) or Cua’s perception extension (AGPL-3.0) into it carries source-offer obligations. screenpipe source is not under an open-source license. Graphiti, Strix, and the MIT parts of Cua can be adapted with attribution. Steal methods, not code, unless the license is Apache or MIT.


1. Graphiti — temporal knowledge graph memory

At a glance

  • Unit of input: an episode (message, json, text, or fact_triple) with a reference_time.
  • Output: EntityNodes and EntityEdges (facts) in a property graph (Neo4j, FalkorDB, Kuzu, or Amazon Neptune), every fact linked back to the episodes that produced it.
  • Time: four timestamps per fact. Contradictions invalidate the old edge; nothing is deleted.
  • Identity: candidates by embedding search, then exact name, then MinHash/LSH fuzzy match behind an entropy gate, then an LLM judge for whatever is left.

Architecture

Graphiti ingestion and retrieval pipelineadd_episode(name, body, source, reference_time, entity_types, edge_types, edge_type_map) → EpisodicNode → branches: extract_nodes → resolve_extracted_nodes; extract_edges; attributes + summaries on entities; optional: communities · sagas → GraphDriver → search(): BM25 + cosine + BFS → rerank → MCP server.add_episode(name, body, source, reference_time, entity_types, edge_types,edge_type_map)EpisodicNoderaw content, valid_at = reference_timeextract_nodes → resolve_extracted_nodesLLM, or GLiNER2 locally for NERcandidates: cosine search on name_embedding1. exact normalized name · 2. MinHash/LSH Jaccard ≥ 0.9 (entropy-gated) · 3. LLM dedupe prompt forunresolved / ambiguousextract_edgesLLM; edge types limited by (source type, target type) signature→ resolve_extracted_edge: duplicate? contradicted? timestamps?→ resolve_edge_contradictions → old.invalid_at, old.expired_atattributes + summaries on entitiesLLM, small modeloptional: communities · sagaslabel propagation · episode chainsGraphDriverNeo4j | FalkorDB | Kuzu | Neptune (Cypher + fulltext + vector)search(): BM25 + cosine + BFS → rerankRRF | MMR | node distance | episode mentions | cross-encoderMCP serveradd_memory, search_nodes, search_memory_facts, get_episodes, add_triplet, …
Graphiti ingestion and retrieval pipelineadd_episode(name, body, source, reference_time, entity_types, edge_types, edge_type_map) → EpisodicNode → branches: extract_nodes → resolve_extracted_nodes; extract_edges; attributes + summaries on entities; optional: communities · sagas → GraphDriver → search(): BM25 + cosine + BFS → rerank → MCP server.add_episode(name, body, source,reference_time, entity_types, edge_types,edge_type_map)EpisodicNoderaw content, valid_at = reference_timeextract_nodes → resolve_extracted_nodesLLM, or GLiNER2 locally for NERcandidates: cosine search on name_embedding1. exact normalized name · 2. MinHash/LSHJaccard ≥ 0.9 (entropy-gated) · 3. LLM dedupeprompt for unresolved / ambiguousextract_edgesLLM; edge types limited by (source type, targettype) signature→ resolve_extracted_edge: duplicate?contradicted? timestamps?→ resolve_edge_contradictions →old.invalid_at, old.expired_atattributes + summaries on entitiesLLM, small modeloptional: communities · sagaslabel propagation · episode chainsGraphDriverNeo4j | FalkorDB | Kuzu | Neptune (Cypher + fulltext +vector)search(): BM25 + cosine + BFS → rerankRRF | MMR | node distance | episode mentions |cross-encoderMCP serveradd_memory, search_nodes, search_memory_facts,get_episodes, add_triplet, …
Figure 1. Graphiti: an episode is stored, entities and edges are extracted and resolved, then indexed in a graph driver, searched and exposed over MCP.

Method by topic

Ontology / schema. Four node classes in graphiti_core/nodes.py: EpisodicNode, EntityNode, CommunityNode, SagaNode. Edge classes in graphiti_core/edges.py: EpisodicEdge (episode MENTIONS entity), EntityEdge (RELATES_TO, the fact), CommunityEdge, HasEpisodeEdge, NextEpisodeEdge. A custom ontology is a dict of Pydantic models passed per call (entity_types, edge_types) plus an edge_type_map: dict[(source_type, target_type), list[edge_type]] (signature in graphiti_core/graphiti.py, add_episode). The model docstring is the type description the extraction prompt sees (_build_entity_types_context, utils/maintenance/node_operations.py). The MCP server ships a default ontology: Requirement, Preference, Procedure, Location, Event, Object, Topic, … (mcp_server/src/models/entity_types.py). There is no class hierarchy and no reasoning over the schema. The edge signature map acts like rdfs:domain / rdfs:range used as a filter at extraction time, not as an inference rule, so it adds no consequences.

Concept graph and identity resolution. resolve_extracted_nodes (utils/maintenance/node_operations.py) runs in this order:

  1. Candidate retrieval per extracted node by cosine similarity on name_embedding (_semantic_candidate_search).
  2. _resolve_with_similarity (utils/maintenance/dedup_helpers.py): lowercase + whitespace collapse; one exact match → merge; several exact matches → ambiguous, escalate.
  3. Entropy gate: names shorter than 6 characters with fewer than 2 tokens, or with Shannon character entropy below 1.5, skip fuzzy matching (short or repetitive names produce unreliable shingles).
  4. Fuzzy: 3-gram shingles, 32-permutation MinHash (blake2b), LSH bands of 4, accept Jaccard ≥ 0.9.
  5. Everything unresolved goes to one LLM dedupe call (prompts/dedupe_nodes.py).

Merges are recorded as a uuid_map plus duplicate_pairs. Edges are resolved the same way against edges with the same endpoints; the dedupe prompt (prompts/dedupe_edges.py) returns both duplicate_facts and contradicted_facts, and states that a fact can be a duplicate and contradicted at once (same relation, updated value).

Temporal / bi-temporal. EntityEdge carries created_at (when the system learned it), expired_at (when the system stopped believing it), valid_at and invalid_at (when it was true in the world), plus reference_time from the episode (edges.py). Missing world times are filled by a small-model call that reads the fact and the episode time (_extract_edge_timestamps, utils/maintenance/edge_operations.py). resolve_edge_contradictions keeps both edges when the intervals do not overlap; otherwise, when the old edge started earlier, it sets old.invalid_at = new.valid_at and old.expired_at = now. Old facts stay queryable (“what was true in March?”).

Reasoning. No logic engine. Contradiction is an LLM judgement over fact strings, and time arithmetic is done in Python. Each fact is an atomic proposition with a validity interval. That is the material for a temporal propositional logic, but no connectives or rules are applied to it.

NLP. No tokenizer, POS tagger, or parser. Extraction is a structured-output LLM prompt (prompts/extract_nodes.py, prompts/extract_edges.py, prompts/extract_nodes_and_edges.py). Optional local NER with GLiNER2 (205M–340M parameters, CPU), with relations still on the LLM (llm_client/gliner2_client.py). Token counts are estimated at 4 characters per token (utils/content_chunking.py). Embedders: OpenAI, Azure OpenAI, Gemini, Voyage (embedder/). Rerankers: OpenAI, Gemini, BGE (cross_encoder/).

Graph DB. Driver layer for Neo4j (default dependency), FalkorDB and FalkorDBLite (embedded), Kuzu (embedded), and Neptune with OpenSearch (graphiti_core/driver/, pyproject.toml). Queries are Cypher; BFS over RELATES_TO|MENTIONS up to a depth (search/search_utils.py). Fulltext and range indices are created per provider (driver/neo4j_driver.py). Search methods are cosine, BM25, and BFS; rerankers are RRF, MMR, node distance, episode-mention count, and cross-encoder (search/search_config.py). Communities come from label propagation (utils/maintenance/community_operations.py).

Agent orchestration. Graphiti is a memory component, not an agent. mcp_server/src/graphiti_mcp_server.py exposes add_memory, search_nodes, search_memory_facts, get_episodes, get_entity_edge, delete_entity_edge, delete_episode, add_triplet, build_communities, summarize_saga, clear_graph, and get_status. group_id partitions the graph per tenant or user.

Maps to SM / CS

Episode = provenance

SM / CS today
Article (newsdb.articles), org file, brief
Fit
Strong. SM edges do not carry source ids today (see Concepts governance §6; org edges lose file ids).

episodes: list[str] on each edge

SM / CS today
GraphDeck edge = id, from, to, relation, label
Fit
Gives the article ids and org file ids the query-corpus blast radius needs.

Bi-temporal edge

SM / CS today
Concept cache keyed on brief hash; map.json overwritten by save_map; CS EventEnvelope has wall_time + logical_time only
Fit
Strong for map and concept edges. CS has transaction time but no valid time on the SM brief it ingests.

Exact → entropy-gated fuzzy → LLM

SM / CS today
summary_grounding.name_in_text: casefold + accent fold, _EQUIV_GROUPS, distinctive-token alias
Fit
Medium. SM already has the exact and alias steps. Add the ambiguity state and the entropy gate.

edge_type_map

SM / CS today
_clause, co-occurrence, implies write any keyword pair
Fit
Strong as a conservative filter on the rule table.

Hybrid search + RRF

SM / CS today
Redis-cached concept graph, no search API over it
Fit
Later. Useful once edges persist.

Adopt / don’t

  • Adopt: the four timestamps, invalidate-not-delete, episode ids on edges, the entropy gate, an explicit “ambiguous → escalate” state, the typed edge signature.
  • Adopt with a gate: contradiction invalidation is a non-monotonic update. It changes the answer to “current facts” queries, so in the Concepts governance vocabulary it has a blast radius and goes through plan → certificate → apply like an insert.
  • Don’t (yet): run the full LLM pipeline per article. Each episode makes several LLM calls (extract, dedupe, timestamps, attributes, summaries). The cost at SM article volume was not measured.

2. MiroFish — swarm simulation as a forecast

At a glance

  • Pipeline, from the README “Workflow”: graph building → environment setup → simulation → report generation → deep interaction.
  • The knowledge graph is Zep Cloud (zep-cloud==3.25.0 in backend/requirements.txt); Zep is the hosted product built on Graphiti. MiroFish ships no graph engine of its own.
  • The simulation engine is OASIS (camel-oasis==0.2.5, camel-ai==0.2.78), a Twitter-like and a Reddit-like platform run in parallel.
  • The “prediction” is a Markdown report written by a ReACT-style agent with four retrieval tools. There is no probability, score, or calibration in backend/app/services/report_agent.py.

Architecture

MiroFish simulation pipelineseed docs (PDF/MD/TXT) + "prediction requirement" (natural language) → OntologyGenerator (LLM) → GraphBuilder → OasisProfileGenerator + SimulationConfigGenerator (LLM) → OASIS env.step(actions) → ReportAgent: plan_outline → per-section ReACT loop with tools → report.md + chat with any simulated agent.seed docs (PDF/MD/TXT) + "prediction requirement" (natural language)OntologyGenerator (LLM)exactly 10 entity types (8 specific + Person + Organization fallbacks)+ 6–10 edge types with source_targetsGraphBuildersplit 500 chars / overlap 50 → Zep graph.set_ontology → batch add episodesOasisProfileGenerator + SimulationConfigGenerator (LLM)per agent: activity_level, active_hours, sentiment_bias, stance ∈ {supportive, opposing, neutral, observer},influence_weightper world: minutes_per_round, echo_chamber_strength, …OASIS env.step(actions)Twitter: CREATE_POST, LIKE_POST, REPOST, FOLLOW, QUOTE_POST, DO_NOTHINGReddit: CREATE_POST, CREATE_COMMENT, LIKE/DISLIKE post+comment, SEARCH_POSTS, SEARCH_USER,TREND, REFRESH, …↺ ZepGraphMemoryUpdater: each agent action → text episode → back into the Zep graphReportAgent: plan_outline → per-section ReACT loop with toolsinsight_forge (LLM sub-queries + graph search) · panorama_search (incl. expired facts) · quick_search ·interview_agentsreport.md + chat with any simulated agent
MiroFish simulation pipelineseed docs (PDF/MD/TXT) + "prediction requirement" (natural language) → OntologyGenerator (LLM) → GraphBuilder → OasisProfileGenerator + SimulationConfigGenerator (LLM) → OASIS env.step(actions) → ReportAgent: plan_outline → per-section ReACT loop with tools → report.md + chat with any simulated agent.seed docs (PDF/MD/TXT) + "predictionrequirement" (natural language)OntologyGenerator (LLM)exactly 10 entity types (8 specific + Person +Organization fallbacks)+ 6–10 edge types with source_targetsGraphBuildersplit 500 chars / overlap 50 → Zep graph.set_ontology→ batch add episodesOasisProfileGenerator +SimulationConfigGenerator (LLM)per agent: activity_level, active_hours,sentiment_bias, stance ∈ {supportive, opposing,neutral, observer}, influence_weightper world: minutes_per_round,echo_chamber_strength, …OASIS env.step(actions)Twitter: CREATE_POST, LIKE_POST, REPOST, FOLLOW,QUOTE_POST, DO_NOTHINGReddit: CREATE_POST, CREATE_COMMENT,LIKE/DISLIKE post+comment, SEARCH_POSTS,SEARCH_USER, TREND, REFRESH, …↺ ZepGraphMemoryUpdater: each agent action →text episode → back into the Zep graphReportAgent: plan_outline → per-section ReACTloop with toolsinsight_forge (LLM sub-queries + graph search) ·panorama_search (incl. expired facts) · quick_search ·interview_agentsreport.md + chat with any simulated agent
Figure 2. MiroFish: seed documents become an ontology and a Zep graph, agents are profiled and simulated in OASIS, and a report agent writes up the run.

Method by topic

Ontology / schema. The ontology is generated per run by an LLM (backend/app/services/ontology_generator.py). The system prompt (Chinese) restricts entities to actors that can speak on social media: people, companies, institutions, media, platforms, groups. It forbids abstract concepts, topics, and stances as entities. The validator enforces a hard cap (MAX_ONTOLOGY_TYPES = 10, backend/app/utils/ontology.py), forces the two fallback types Person and Organization into the last slots, normalizes names to PascalCase (entities) and UPPER_SNAKE_CASE (relations), and reserves attribute names such as name_embedding. The result is also emitted as Python Pydantic class code (generate_python_code) for the Zep SDK.

Concept graph and identity. Delegated to Zep. MiroFish reads nodes and edges back through zep_entity_reader.py and zep_tools.py; EdgeInfo exposes is_expired / is_invalid, so the Graphiti bi-temporal fields surface in the report tools (“panorama” search can include expired facts).

Temporal. Simulated time is rounds (minutes_per_round, default 60) and per-agent active_hours. Agent actions are written back as new episodes (zep_graph_memory_updater.py, AgentActivity.to_episode_text), so the graph evolves with the simulation.

Reasoning / argumentation. Agents do not argue in a structured sense. Each agent acts on a feed under a stance label and an influence weight; “debate” is posts, quotes, reposts, and follows. The forecast is the report agent’s synthesis of retrieved graph facts and of interviews (an INTERVIEW manual action sent to selected agents, backend/scripts/run_twitter_simulation.py). There are no claims, warrants, or support/attack edges, no propositional form, and no aggregation rule (vote, market, or scoring) that turns behaviour into a probability.

NLP. Character chunking (500 / 50) in backend/app/utils/file_parser.py; everything else is LLM calls through an OpenAI-compatible client. No embeddings of its own (Zep does that), no tokenizer, no POS tagging.

Graph DB. Zep Cloud (hosted). No Cypher in the repo.

Agent orchestration. Flask API + background tasks; OASIS runs as subprocess scripts with IPC (simulation_ipc.py, simulation_runner.py); the report agent parses tool calls from text and strips fabricated tool results (_strip_fake_tool_results).

Maps to SM / CS

LLM-generated, capped ontology with fallback types

SM / CS
shared/ontology.json fixed categories; sub lists unused
Fit
Cap + explicit fallback is a good discipline. A per-run generated ontology is the opposite of a certified rule table.

Stance enum per actor

SM / CS
Argument map supports / opposes; MAP roles
Fit
Weak. Stance is a persona parameter, not an edge.

Simulation → report

SM / CS
HomoSapiens TEvEP / RUT scenarios; CS signals
Fit
Exploratory scenario rehearsal for HomoSapiens only. Not for CS trading: an uncalibrated narrative is not a signal.

Report agent tools over a temporal graph

SM / CS
SM agent_surface (get_world_brief, …)
Fit
The tool split (deep / panorama incl. history / quick / interview) is a usable pattern.

Adopt / don’t

  • Adopt: capped ontology with mandatory fallback types; “entities must be actors” as an explicit admission rule; panorama search that can include expired facts.
  • Don’t: treat simulated-crowd output as a forecast, or feed it into CS. Nothing in the repo measures accuracy. AGPL-3.0, so no code copy into SM.

3. screenpipe — capture as memory

At a glance

  • Event-driven capture (app switch, click, typing pause, scroll, clipboard) of a screenshot paired with the accessibility tree; OCR only as a fallback. Audio via local Whisper or Deepgram, with speaker diarization. (README “Core features”, “Technical architecture”.)
  • Storage is local SQLite with FTS5 and media on disk; REST on localhost:3030; an MCP server (packages/screenpipe-mcp); “pipes” are scheduled agents defined as pipe.md.
  • The interesting method is in crates/screenpipe-semantic: deterministic semantic projections over accessibility trees with an explicit identity-quality grade.

Architecture

screenpipe capture and memory pipelinebranches: OS events → capture → OCR fallback; audio → Whisper / Deepgram → diarization → speaker embeddings → SQLite (frames, ocr, audio, memories) + FTS5 (unicode61) → branches: screenpipe-semantic → REST :3030 · MCP server · pipes · team-memory files.OS events → capture → OCR fallbackscreenshot + a11y tree, same timestampaudio → Whisper / Deepgram → diarization → speaker embeddingsmatch with marginSQLite (frames, ocr, audio, memories) + FTS5 (unicode61)→ background workers: FTS backfill, PII redactionscreenpipe-semanticper-app parsers → SemanticItem{kind, item_key, identity_quality, actor, occurred_at, source_nodes}PersonResolver: email / E.164 / native id → person_key (unique | ambiguous | not_found)REST :3030 · MCP server · pipes · team-memory filespipe.md + YAML permissions · YAML front matter + Markdown
screenpipe capture and memory pipelinebranches: OS events → capture → OCR fallback; audio → Whisper / Deepgram → diarization → speaker embeddings → SQLite (frames, ocr, audio, memories) + FTS5 (unicode61) → branches: screenpipe-semantic → REST :3030 · MCP server · pipes · team-memory files.OS events → capture → OCR fallbackscreenshot + a11y tree, same timestampaudio → Whisper / Deepgram → diarization→ speaker embeddingsmatch with marginSQLite (frames, ocr, audio, memories) + FTS5(unicode61)→ background workers: FTS backfill, PII redactionscreenpipe-semanticper-app parsers → SemanticItem{kind,item_key, identity_quality, actor, occurred_at,source_nodes}PersonResolver: email / E.164 / native id →person_key (unique | ambiguous | not_found)REST :3030 · MCP server · pipes · team-memoryfilespipe.md + YAML permissions · YAML front matter +Markdown
Figure 3. screenpipe: screen and audio capture feed a local SQLite store with full-text search, a semantic layer, and REST / MCP surfaces.

Method by topic

Ontology / schema. A small closed kind set: SemanticKind = ConversationMessageDocumentTaskCalendarEventPage (crates/screenpipe-semantic/src/model.rs). A SemanticItem has local_id, parent_local_id, kind, item_key, identity_quality, title, body, actor, occurred_at, timestamp_precision, status, metadata, and source_nodes (the accessibility nodes it came from). Parent links are validated before persistence. The memories table (crates/screenpipe-db/src/migrations/20260310000000_create_memories.sql) holds content, source, source_context, tags, importance.

Identity map. Two methods worth copying:

  1. IdentityQuality = Stable | Derived | Ephemeral (model.rs). Stable: an app-native id that survives screen changes. Derived: computed from visible fields, “consumers must treat cross-run entity grouping as approximate”. Ephemeral: valid inside one parse run only, such as a positional index.
  2. PersonResolver (crates/screenpipe-semantic/src/messaging.rs): exact match on normalized email, E.164 phone, or a native (source, value) id. Result is Unique { person_key } | Ambiguous { candidate_count } | NotFound. Display names are deliberately not part of the API “so they cannot accidentally become cross-source join keys”.

Speaker identity uses the same idea for embeddings: a best-match distance threshold (0.55) plus a runner-up margin (0.08); a match inside the margin returns Ambiguous instead of mutating a profile (crates/screenpipe-db/src/db/speakers.rs).

Temporal. Every capture is timestamped; message times are normalized with an explicit precision and parse status (normalize_message_time, TimeParseStatus). No valid-time / transaction-time split.

Reasoning. None in the core. Pipes run an external coding agent over the API.

NLP. SQLite FTS5 with the unicode61 tokenizer; query-side expansion that splits camelCase and digit boundaries that OCR glued together (crates/screenpipe-db/src/text_normalizer.rs); word-level Jaccard for cross-device transcript dedup (text_similarity.rs); tiktoken-rs for token budgets (crates/screenpipe-semantic/Cargo.toml). No POS tagging.

Graph DB. None. Relational + FTS.

Agent orchestration and safety. Pipe permissions in YAML front matter (allow-apps, deny-apps, deny-windows, content-type limits, time ranges, allow-raw-sql, allow-frames), enforced in three places: skill gating, agent interception, and server middleware with per-pipe tokens (README “Pipe data permissions”; the README calls it “not prompt-based”). PII redaction runs as a background reconciliation worker so capture stays fast (crates/screenpipe-redact/src/lib.rs).

Maps to SM / CS

IdentityQuality

SM / CS
Concept node ids are keyword strings; argument-map ids are usl: slugs
Fit
Strong. Grade each id: article _id / org file id = stable; folded keyword = derived; clause index = ephemeral.

PersonResolver tri-state, no display-name joins

SM / CS
name_in_text returns a bool; aliases by distinctive token
Fit
Strong. Return unique / ambiguous / not found, and never join two stories on a display name alone.

Threshold + runner-up margin

SM / CS
Zero-shot labels: > 0.7 kept, else labels[0] written
Fit
Strong. The margin rule is the opposite of “write the top label anyway”.

Pipe permissions, enforced outside the prompt

SM / CS
agent_surface.py /mcp (localhost JSON-RPC)
Fit
Medium. A manifest per tool consumer.

Background redaction / backfill

SM / CS
SM grammar indexing on its own queue
Fit
Already the SM pattern.

Adopt / don’t

  • Adopt: identity-quality grades, the tri-state resolver, the margin rule, permission manifests enforced in code.
  • Don’t: copy code (source-available commercial license). Screen capture itself has no SM use.

4. Cua — computer-use agents

At a glance

  • Cua Driver: tools to inspect and operate native desktop apps and browsers, reached through a CLI, MCP (cua-driver mcp), or typed SDKs over one native runtime (UniFFI over a C ABI) (libs/cua-driver/README.md).
  • Sandboxes: cua sb create ubuntu (a gVisor container), local VMs (Lume on macOS), and remote hosts through cua-spacesd (README “Cua SDK and CLI”).
  • Agent library libs/python/agent/cua_agent: ComputerAgent with per-model loops (Anthropic, OpenAI, Gemini, UI-TARS, OpenCUA, Qwen-VL, OmniParser, …) and a composed grounded loop.
  • CUA-S1: small “System 1” models that score bounded decisions (forms) instead of generating tokens; Cua Bench: tasks with evaluators that report a reward.

Architecture

Cua computer-use stackagent (Claude Code / Cursor / own loop) → cua-driver (MCP or CLI) → sandbox → cua_agent.ComputerAgent.agent (Claude Code / Cursor / own loop)MCP or CLIcua-driverpermission mode fixed at launch: standard | bounded (reviewed manifest) | unrestricted (explicit flag)observe: accessibility tree · browser state · screenshot (capture_id)optional cua-perception worker: screenshot → text/icon regions (AGPL extension)act: click/type with the same one-use capture_id as the observationsandboxgVisor container | VM (Lume, QEMU) | remote host (cua-spacesd :3211)cua_agent.ComputerAgentloops/composed_grounded.py: grounding model (where is "the red submit button"?) + thinking model(what next?)callbacks/: budget_manager · image_retention · pii_anonymization · trajectory_saver · operator_validator ·otel
Cua computer-use stackagent (Claude Code / Cursor / own loop) → cua-driver (MCP or CLI) → sandbox → cua_agent.ComputerAgent.agent (Claude Code / Cursor / own loop)MCP or CLIcua-driverpermission mode fixed at launch: standard | bounded(reviewed manifest) | unrestricted (explicit flag)observe: accessibility tree · browser state ·screenshot (capture_id)optional cua-perception worker: screenshot →text/icon regions (AGPL extension)act: click/type with the same one-use capture_id asthe observationsandboxgVisor container | VM (Lume, QEMU) | remote host(cua-spacesd :3211)cua_agent.ComputerAgentloops/composed_grounded.py: grounding model(where is "the red submit button"?) + thinking model(what next?)callbacks/: budget_manager · image_retention ·pii_anonymization · trajectory_saver ·operator_validator · otel
Figure 4. Cua: an agent drives cua-driver over MCP or CLI inside a sandbox; the ComputerAgent loop is shown separately.

Method by topic

Ontology / schema. The world model is the UI: accessibility tree, DOM, and screenshot regions. The perception RFC (rfcs/3931-cua-perception-and-jev-use.md) keeps capture, perception, decision, action, verification as separate typed stages with explicit ownership, and a pixel action must carry the one-use capture_id of the observation it came from (libs/cua-driver/README.md, “Optional visual perception”).

Identity. Element identity is per capture: the capture_id binds an action to one observation. This is the same discipline as screenpipe’s Ephemeral grade.

Temporal. Trajectories are saved per step (callbacks/trajectory_saver.py) for replay and for training data (Cua Bench exports trajectories).

Reasoning. Planning lives in the external model. The composed loop separates grounding (locate) from reasoning (decide) (loops/composed_grounded.py). CUA-S1 frames small decisions as scoring over structured candidates; application code orders the actions (README “CUA-S1”).

NLP. tiktoken for budgets (libs/python/agent/pyproject.toml); element descriptions in natural language for the grounded tool schema. No linguistic analysis.

Graph DB. None.

Orchestration and sandboxing. Permission modes are a property of the process, chosen before start (a running daemon must restart to change them). bounded admits only the tools and resources in a reviewed manifest. Attaching to a logged-in browser profile is an explicit grant (--grant existing-profile). Callbacks enforce budget and PII anonymization around the loop.

Maps to SM / CS

Permission modes + reviewed manifest

SM / CS
SM /mcp agent surface (localhost); CS auto-trader
Fit
Strong pattern. A bounded manifest for any agent that can reach CS order intents.

Observation id carried by the action

SM / CS
CS EventEnvelope.causal_parents; approval state
Fit
Strong. An order intent must cite the signal / snapshot event it came from (CS already has causal_parents).

Capture → perceive → decide → act → verify

SM / CS
CS signal → order-intent → approval → execution-report → reconciliation families
Fit
Already the CS shape (packages/engine/events.py).

Bench with evaluator reward

SM / CS
SM summary grounding tests; CS backtests
Fit
A QA agent for CS can be a bench task: a fixture, an evaluator, a reward.

Adopt / don’t

  • Adopt: launch-time permission modes, observation-bound actions, separate grounding from deciding, trajectory saving.
  • Don’t: install the AGPL perception extension in anything SM serves.

5. Strix — graph of agents (architecture only)

This section covers the README and the orchestration code: the agent loop, the agent graph, the tool registry, the sandbox runtime, and LLM configuration. Strix’s prompt templates, skills, and security knowledge files were not opened.

At a glance

  • Built on the OpenAI Agents SDK (openai-agents[litellm]) with LiteLLM for providers (pyproject.toml).
  • An AgentCoordinator owns a tree of agents: statuses, parent links, mailboxes, budgets, and resume snapshots (strix/core/agents.py).
  • Agents create, message, wait for, and stop other agents through tools (strix/tools/agents_graph/tools.py).
  • Tools run in a Docker sandbox backend chosen by STRIX_RUNTIME_BACKEND (strix/runtime/backends.py); MCP connections are held in a per-run McpRegistry (strix/tools/mcp/registry.py).

Architecture

Strix multi-agent scanrunner (per scan) → AgentCoordinator — single owner of graph state → branches: root agent → execution loop → report.runner (per scan)→ McpRegistry · sandbox session (Docker) · budgetAgentCoordinator — single owner of graph statestatuses: running | waiting | completed | stopped | crashed | failed | budget_pausedparent_of · names · metadata · mailboxes (AgentRuntime.mailbox + wake event) · wait_kind: user | agents |stalledroot agenttools: create_agent · send_message_to_agent · wait_for_agents · view_agent_graph · agent_finish ·stop_agent+ notes · todo · thinking · shell · browser · proxy · reporting · load_skill · mcp …→ child agents (spawn_child_agent; respawn_subagents on resume)execution looprun_agent_loop → cycles → recovery on transient model errors → compaction when context overflowsreportfindings deduplicated (deterministic identity first, then an LLM judge) → SARIF / PDF
Strix multi-agent scanrunner (per scan) → AgentCoordinator — single owner of graph state → branches: root agent → execution loop → report.runner (per scan)→ McpRegistry · sandbox session (Docker) · budgetAgentCoordinator — single owner of graph statestatuses: running | waiting | completed | stopped |crashed | failed | budget_pausedparent_of · names · metadata · mailboxes(AgentRuntime.mailbox + wake event) · wait_kind:user | agents | stalledroot agenttools: create_agent · send_message_to_agent ·wait_for_agents · view_agent_graph ·agent_finish · stop_agent+ notes · todo · thinking · shell · browser · proxy ·reporting · load_skill · mcp …→ child agents (spawn_child_agent;respawn_subagents on resume)execution looprun_agent_loop → cycles → recovery on transientmodel errors → compaction when context overflowsreportfindings deduplicated (deterministic identity first,then an LLM judge) → SARIF / PDF
Figure 5. Strix: a runner starts a coordinator that owns the agent graph; a root agent spawns children; the execution loop ends in a deduplicated report.

Method by topic

  • Ontology / schema. None for knowledge. The schema is the agent graph itself (statuses, parent links, wait kinds) and typed tool argument schemas, with argument coercion and bounded tool outputs (strix/agents/factory.py).
  • Identity / dedup. Report deduplication checks a deterministic identity first (for dependency findings: package, version, manifest path, CVE) and asks an LLM judge only when that does not decide (strix/report/dedupe.py, function names only).
  • Temporal. Resume snapshots and a resume epoch for budget pauses (core/agents.py).
  • Reasoning. Delegated to the model; “thinking”, “todo”, and “notes” tools externalize state.
  • Memory. Session history with provider-agnostic compaction: older turns become one checkpoint summary, recent turns stay verbatim, and tool-call / tool-result pairs are preserved (strix/llm/compaction.py).
  • Sandboxing. Docker backend with a manifest, exposed ports, and bind mounts (runtime/backends.py, runtime/docker_client.py).

Maps to SM / CS

Coordinator with explicit statuses and mailboxes

SM / CS
CS auto-trader QA agent; SM RQ workers
Fit
Pattern for a supervised QA agent with sub-checks (venue, risk, journal).

Deterministic identity before LLM judge

SM / CS
Graphiti, screenpipe (same idea)
Fit
Third independent instance of the same rule.

Compaction preserving tool pairs

SM / CS
SM map chat (apply_edit) sessions
Fit
Useful if map chat grows long.

Adopt / don’t

  • Adopt: status machine + mailbox for multi-agent QA; deterministic-first dedup; compaction that keeps tool pairs valid.
  • Don’t: anything domain-specific. Strix’s purpose (offensive security testing) has no SM or CS use.

6. Cross-cutting comparison

Ontology / schema

Graphiti
Pydantic entity + edge types; (src,tgt) → edges map; no hierarchy
MiroFish
LLM-generated, 10 types, Person/Organization fallback
screenpipe
Closed SemanticKind set
Cua
UI tree + typed stages
Strix
Agent-graph state + tool schemas
SM today
shared/ontology.json categories; Grammar v.1 roles; map kinds

Concept graph

Graphiti
Entities + fact edges + communities
MiroFish
Via Zep
screenpipe
Semantic items with parents
Cua
—
Strix
—
SM today
build_summary_concepts, org_concepts, map.json

Identity resolution

Graphiti
Embed → exact → entropy-gated MinHash/LSH → LLM
MiroFish
Via Zep
screenpipe
Stable / Derived / Ephemeral; tri-state resolver
Cua
capture_id
Strix
Deterministic id → LLM judge
SM today
fold, _EQUIV_GROUPS, name_in_text, dedupe_articles

Time

Graphiti
Bi-temporal edges, invalidate
MiroFish
Rounds; actions → episodes
screenpipe
Timestamps + precision
Cua
Trajectories
Strix
Resume epochs
SM today
Cache TTL; changelog; grammar_hash

Reasoning / argument

Graphiti
LLM contradiction + interval rule
MiroFish
None (narrative report)
screenpipe
None
Cua
External model
Strix
External model
SM today
role_of, _clause, implies, Pearson; argument map

NLP

Graphiti
LLM extraction; GLiNER2 NER; embeddings; BM25
MiroFish
LLM; char chunks
screenpipe
FTS5 unicode61; OCR/ASR
Cua
Natural-language element descriptions
Strix
—
SM today
Stanza tokenize/POS/lemma/depparse (UD); zero-shot mDeBERTa; BART summaries

Graph DB

Graphiti
Neo4j / FalkorDB / Kuzu / Neptune, Cypher
MiroFish
Zep Cloud
screenpipe
SQLite
Cua
—
Strix
—
SM today
Mongo + Redis; Neo4j lab only (agents/graph, localhost)

Agent / MCP

Graphiti
MCP server for memory
MiroFish
ReACT report agent
screenpipe
MCP server; pipes with permissions
Cua
Driver MCP; permission modes
Strix
Agent graph; MCP registry; Docker
SM today
/mcp JSON-RPC, localhost (agent_surface.py)

6.1 Relation to IEML

IEML gives every expression an address (USL) whose meaning comes from its position in paradigms (IEML). None of the five repos has an address for meaning. Their identities are UUIDs (Graphiti), app keys (screenpipe), or session ids (Cua, Strix). Graphiti’s resolution decides same entity? by string and embedding similarity; IEML decides same meaning? by construction. On SM, the argument map’s usl: slugs are a naming convention, not parsed USL (the SM repo note docs/homosapiens-ieml-network/ARCHITECTURE.md states this). The transferable piece is screenpipe’s grading: a usl: slug is a Derived id until a parser certifies it.

IEML’s nine-role phrase (Grammar v.1) is closer to Graphiti’s typed edge signature than to anything else here: both say which kinds may fill which slot. Graphiti enforces the slot at extraction. IEML defines it in the grammar.

6.2 Relation to the semantic web

None of the five uses RDF, OWL, SHACL, or SPARQL. Graphiti is a labelled property graph queried with Cypher. Its typed edge map is a domain/range constraint used as a filter, so it is conservative by construction: it can only remove candidate facts, never derive new ones. That is the safe half of the silent-policy lesson in Concepts governance §2. The unsafe half, an axiom that adds consequences, has no counterpart in these repos because none of them infers.

A contrasting pattern gates RDF/OWL/JSON-LD assets with SHACL, and can keep RDF/SPARQL as a bridge layer over a typed-atom kernel. A practical split for SM: a property graph with bi-temporal edges for the operational concept graph, and RDF/SHACL only where a certificate is required.

6.3 Relation to propositional logic

A Graphiti fact edge is an atomic proposition p = relation(subject, object) with a validity interval [valid_at, invalid_at). Contradiction handling implements one rule: if p_new contradicts p_old and valid_at(p_old) < valid_at(p_new), then invalid_at(p_old) := valid_at(p_new). The contradiction itself is decided by an LLM, not by a truth table. There are no connectives, no ¬, ∧, → over facts, and no proof. SM’s argument map has typed relations (supports, opposes, implies, evidence_for) that read like connectives but are asserted by chat, and the implies produced by summary_concepts is a lexical join, not material implication. Turning either into propositional logic needs a declared rule table and a checker, which is the Concepts governance programme.

6.4 Relation to graph databases

Graphiti is the only repo that abstracts several graph DBs. Its driver set (Neo4j default; FalkorDB, FalkorDBLite, and Kuzu as embedded options) is the measured shortlist if SM moves concept edges out of Redis. SM’s existing Neo4j lab (agents/graph/README.md) is localhost-only and seeded from map.json with MERGE statements. The engineering-index rule — vendor-neutral defaults, measure before adopting specialty stores (Engineering index §1) — applies: no store was benchmarked in this pass.

6.5 Relation to modern agent systems

Three repos expose MCP (Graphiti memory, screenpipe history, Cua driver); Strix consumes MCP. Across them, safety is enforced outside the prompt: screenpipe’s per-pipe tokens, Cua’s launch-time permission modes and one-use capture ids, Strix’s Docker sandbox and budgets. That matches an evidence-then-replay control-plane habit, and SM’s existing rule that the agent surface wraps public GETs only.


7. What to take to SM and CS

Effort: S ≤ 1 day, M a few days, L a week or more. Each item is an idea from the cited code, not something implemented.

1

Episode ids on every derived edge (article id, org file id, occurrence id)

Source
Graphiti EntityEdge.episodes
SM / CS target
build_summary_concepts, build_org_file_concepts, slim_concept_graph; enables the query-corpus blast radius
Effort
M
2
Idea
Bi-temporal fields (created_at, expired_at, valid_at, invalid_at) with invalidate-not-delete
Source
Graphiti resolve_edge_contradictions
SM / CS target
map.json edges via save_map; persisted org concept edges; CS: valid time of the SM brief in sm_situation_signal
Effort
M
3
Idea
Tri-state identity map (unique / ambiguous / not found) with identity-quality grades and an entropy gate
Source
screenpipe PersonResolver, IdentityQuality; Graphiti _has_high_entropy
SM / CS target
summary_grounding.name_in_text / _EQUIV_GROUPS; concept node ids
Effort
M
4

Typed edge signature map as a conservative rule-table filter

Source
Graphiti edge_type_map; MiroFish source_targets
SM / CS target
_clause, co-occurrence, implies in summary_concepts.py
Effort
S
5

Threshold + runner-up margin → “ambiguous” instead of writing the top label

Source
screenpipe classify_speaker_candidates
SM / CS target
nlp.py category stamp (labels[0] fallback)
Effort
S
6

Launch-time permission manifest for agent tools

Source
Cua permission modes; screenpipe pipe permissions
SM / CS target
SM agent_surface.py /mcp; any CS agent near order intents
Effort
S–M
7

Coordinator + mailbox + status machine for a QA agent

Source
Strix AgentCoordinator
SM / CS target
CS auto-trader QA
Effort
M
8

Capped ontology with mandatory fallback types

Source
MiroFish _validate_and_process
SM / CS target
shared/ontology.json (fallback category instead of top-label promotion)
Effort
S
9

Hybrid retrieval (BM25 + cosine + graph BFS, RRF) over persisted edges

Source
Graphiti search/
SM / CS target
Later: agent get_concepts once edges carry certificate ids
Effort
L

Do not take: simulated-crowd forecasting into CS (no calibration in MiroFish); per-article full LLM graph extraction before its cost is measured; any AGPL or source-available code into SM.


References

Repositories read at the commits in §0 on 2026-10-06. URLs verified the same day.

  1. Graphiti. Zep Software. Apache-2.0. https://github.com/getzep/graphiti — graphiti_core/nodes.py, edges.py, graphiti.py, utils/maintenance/{node_operations,edge_operations,dedup_helpers,community_operations}.py, prompts/{extract_nodes,extract_edges,dedupe_nodes,dedupe_edges}.py, search/search_config.py, search/search_utils.py, llm_client/gliner2_client.py, driver/, mcp_server/src/.
  2. MiroFish. AGPL-3.0. https://github.com/666ghj/MiroFish — README.md, backend/requirements.txt, backend/app/services/{ontology_generator,graph_builder,simulation_config_generator,zep_graph_memory_updater,zep_tools,report_agent}.py, backend/app/utils/ontology.py, backend/scripts/run_twitter_simulation.py.
  3. OASIS: Open Agent Social Interaction Simulations with One Million Agents. CAMEL-AI. https://github.com/camel-ai/oasis
  4. screenpipe. Screenpipe Commercial License. https://github.com/screenpipe/screenpipe — README.md, LICENSE.md, crates/screenpipe-semantic/src/{lib,model,messaging}.rs, crates/screenpipe-db/src/{text_normalizer,text_similarity}.rs, crates/screenpipe-db/src/db/speakers.rs, crates/screenpipe-db/src/migrations/20260310000000_create_memories.sql, crates/screenpipe-redact/src/lib.rs, crates/screenpipe-team-memory/src/lib.rs.
  5. Cua. MIT / FSL-1.1-MIT / AGPL extension per LICENSING.md. https://github.com/trycua/cua — README.md, LICENSING.md, libs/cua-driver/README.md, rfcs/3931-cua-perception-and-jev-use.md, libs/python/agent/cua_agent/{agent.py,loops/composed_grounded.py,callbacks/}.
  6. Strix. Apache-2.0. https://github.com/usestrix/strix — README.md (overview and “Graph of Agents”), pyproject.toml, strix/core/{agents,execution}.py, strix/agents/factory.py, strix/tools/agents_graph/tools.py, strix/tools/mcp/registry.py, strix/runtime/backends.py, strix/llm/compaction.py, strix/report/dedupe.py (function names).
  7. OpenAI Agents SDK. https://github.com/openai/openai-agents-python
  8. Model Context Protocol. https://modelcontextprotocol.io/
  9. GLiNER2: Unified Schema-Based Information Extraction. Fastino AI. https://github.com/fastino-ai/GLiNER2
  10. SQLite FTS5 Extension. https://www.sqlite.org/fts5.html
  11. Temporal database (valid time, transaction time, bitemporal). Wikipedia. https://en.wikipedia.org/wiki/Temporal_database

IEML and semantic-web sources stay in IEML, Concepts governance, and the Engineering index.


See also

Glossary terms introduced

bi-temporal edge, computer-use agent, entity resolution, episode (agent memory), graph of agents, identity quality, MinHash / LSH, reciprocal rank fusion, swarm simulation

Back to top ↑

All entries Glossary Foundations