Benchmarks
Dataset: ten LoCoMo conversational histories with 1,540 questions.3 Campaign-reported categories: single-hop 282, multi-hop 841, temporal 321, open-domain 96, retained as reported; the inherited category constants swap single-hop and multi-hop, so every table here says “as labelled”.4 Accuracy is an LLM-judge verdict against the gold answer; recall@k is presence of a gold-answer fragment among the top-k retrieved messages, a substring proxy, not evidence-set recall.5 The gold-answer acceptance gate and gold-category routing found by the audit were removed; every current number comes from the leakage-free harness.3 Independent ownership is instantiated by two evidence holders, one per speaker.6
| Retrieval | Accuracy | Recall@10 | Recall@50 | s/q | Chip |
|---|---|---|---|---|---|
| flat (original single-store retrieval) | 64.9% | 39.4 | 61.7 | 6.4 | A measured retrieval |
| hybrid (speaker boost + embedding back-fill) | 67.0% | 44.7 | 61.7 | 8.2 | A measured retrieval |
| graph (time chain + speaker + dialogue links) | 66.7% | 39.4 | 61.7 | 7.9 | A measured retrieval |
| local_pairs (per-agent routing + pair back-fill) | 73.8% | 46.8 | 62.8 | 7.9 | A measured retrieval |
The +8.9-point flat → local_pairs gain survives all four judges: +11.3 Qwen lenient, +9.6 Qwen strict, +4.6 substring; p < .001 for the three LLM judges, p = .01 for substring (two-sided sign tests; wins/losses 32/7, 45/13, 38/11, 18/5).8 The treatment changes owner selection and context together and does not isolate routing alone.100The gain is measured for per-agent routing and pair-node back-fill together; the repository record reports the end-to-end gain only for the two combined.9 The s/q column is not a product specification. |
|||||
| Measure | Value | Chip |
|---|---|---|
| Messages containing the speaker’s own name | 0.1% | A measured retrieval |
| Messages containing the other speaker’s name | 33.4% | A measured retrieval |
| Pure-embedding recall, name as asked | 26.6 | A measured retrieval |
| Pure-embedding recall, name swapped in the question | 34.8 | A measured retrieval |
In two-speaker conversations a speaker’s own name appears in 0.1% of their messages and the other speaker’s name in 33.4%, so a name in the question is an embedding-level pointer to the wrong speaker; swapping the name raises pure-embedding recall from 26.6 to 34.8.6 Routing the question to the named speaker’s own store removes the problem: single-hop recall@10 rises from 39.4 to 46.8 and matches oracle routing, because the gold message is spoken by the other agent only 2.9% of the time.610 Scope: LoCoMo two-speaker dialogues only.
| Condition | Recall@10 single-hop (as labelled) | Recall@10 multi-hop (as labelled) |
|---|---|---|
| pooled typed index | 39.4 | 46.8 |
| routed to named holder | 46.8 | 50.3 |
| oracle routing | 46.5 | 51.0 |
| per-store score federation | 36.9 | 42.4 |
Owner routing approaches oracle routing; naive score federation underperforms pooling. Pure-embedding single-hop recall rises 25.9 → 46.8 with owner routing.12 A measured retrieval |
||
Under review A retrieval-only comparison of recall@10 across pooled, routed, oracle and federated conditions is reported in the manuscript under review and will be published here after the review decision, with the date recorded in the corrections log.
The shipped stack (owner routing, dialogue-pair back-fill, open-domain flags) scores 72.2% on 744 of 1,540 questions (conversations 1 to 5) under the lenient local Gemma judge, leakage-free: single-hop 73.9% (n = 142), multi-hop 72.8% (400), temporal 73.7% (156), open-domain 56.5% (46), all as labelled.13 Conversations 6 to 10 and the adversarial category are not yet run.14
| Grader | Overall (n = 584) | Chip |
|---|---|---|
| Gemma lenient | 73.8 | A measured retrieval |
| Qwen lenient | 65.6 | A measured retrieval |
| Qwen strict | 42.5 | A measured retrieval |
| substring | 26.7 | A measured retrieval |
| Answer file | n | Gemma lenient | Qwen lenient | Qwen strict | substring |
|---|---|---|---|---|---|
| flat, single-hop (as labelled) | 282 | 64.9 | 51.4 | 21.6 | 13.5 |
| shipped, single-hop (as labelled) | 282 | 73.8 | 62.8 | 31.2 | 18.1 |
| open-domain prompt treatment | 96 | 57.3 | 39.6 | 10.4 | 9.4 |
Judge model alone moves scores about 12 points at identical prompt; prompt leniency about 30 points; κ against the campaign judge is 0.72 (Qwen lenient), 0.30 (strict), 0.17 (substring).19 The 51-point spread is as wide as the published LoCoMo leaderboard spread.20 Any headline number must name its judge.21 A measured retrieval |
|||||
Published because it shaped the design.22
Expanding the answer context from 15/30 to 50/50 memories raises gold presence from 71% to 73% while accuracy falls from 78% to 69% and accuracy-given-gold falls from 88.7% to 78.1% on 100 matched questions.29 A vector/entity/pair fusion raises recall@10 from 46.8 to 53.7 over 1,219 questions, yet matched single-hop accuracy falls 78 → 68 (3 wins, 13 losses) while multi-hop moves 75 → 76.30 Reported as a candidate-dilution finding on small matched subsets, not a general law. Diagnostic: when the gold memory is in the final context the model is right 88.7% of the time; it is absent 29% of the time; 25.5% of single-hop gold answers are not a substring of any message. Retrieval is the only lever left.31 A measured retrieval
External comparison · checked 2026-09-17
LoCoMo is the common dataset, but there is no single 2026 leaderboard protocol: reader models, judges, retrieval depth, reranking and question subsets all differ. So this chart plots ten memory systems with one published LoCoMo result each, and names the protocol that produced every one of them. Only LoCoMo figures are plotted; where a source does not confirm the benchmark, the figure is left out rather than quietly mixed in. Read the protocol before you read the percentage: the same system moves by more than thirty points depending only on who ran the test.
Mycelic · local-first cloud or self-host
Read the protocol with the percentage. On the latency and locality axes the two measures come from different protocols — accuracy from the uniform third-party harness, latency from each vendor’s own report — and the statistics differ in kind: a mean on a 282-question slice, a p50, an operational range. So the frontier maps the trade-off space rather than ranking it, which is exactly what the first view is for. W figures as published by each source, not independently re-measured here
| System | Accuracy | Retrieval latency, as published | Raw memory lives |
|---|---|---|---|
| Backboard | 90.00% | not published | not documented |
| Hindsight · Gemini-3 | 89.61% | not published for this configuration | open source + cloud |
| Hindsight · OSS-120B | 85.67% | 50–100 ms typical, no reranking (a range, not a percentile) | open source + cloud |
| Hindsight · OSS-20B | 83.18% | as above | open source + cloud |
| Memobase | 75.78% | not published | not documented |
| Zep | 75.14% | 87 ms p50 / 155 ms p95, 1,540 questions | enterprise deployment options |
| Mycelic · own harness | 73.8% | 65 ms mean retrieval stage, 282 questions, one local machine | local-first |
| Mem0 Graph | 68.44% | not published separately | on-prem enterprise; OSS project |
| Mem0 | 66.88% | 880 ms p50, 1,540 questions | on-prem enterprise; OSS project |
Harness rows: Hindsight public benchmark repository, checked 2026-09-17. Open source table → Mycelic row: this site’s own leakage-free harness, 282 single-hop questions, Gemma-lenient judge — a different protocol, shown for reference, never as a same-protocol result.16 The engine is open source: github.com/anovruzov/NeuralGraph →
Mycelic · local-first cloud or self-host published range or upper percentile
Do not read this as a speed ranking. The statistics are not uniform: Mycelic is a mean on a 282-question slice, Zep and Mem0 publish p50 (Zep also p95), and Hindsight publishes an operational range. The next step is to run all systems on the same machine, question set, reader and judge.
| System | Entry price | Graph / advanced memory | Local / self-hosted | Best fit |
|---|---|---|---|---|
| Mycelic | $0 software today; no hosted plan | Local graph + lineage research engine | Local-first | Research, privacy-first local agents, lineage experiments |
| Mem0 | Free Hobby; Starter $19/mo | Pro $249/mo includes graph memory | Enterprise on-prem; OSS project available | Fastest path for general app developers wanting a broad memory SDK |
| Zep | Flex $125/mo | Graph/temporal memory included; Flex Plus $375/mo adds observations | Enterprise deployment options | Production cloud agents prioritising published low latency |
| Hindsight | Usage based | Retain $10/M input tok; Recall $0.75/M output tok; Reflect $0.05/call | Open-source + cloud | Teams wanting strong open memory research and transparent usage pricing |
| AWS AgentCore Memory | Usage based | $0.25/1k events; long-term storage $0.75/1k records/mo built-in; retrieval $0.50/1k | AWS-managed | AWS-native production stacks and integrated governance |
Read the buyer’s guide: which memory system should you use? →
The only end-to-end latency measurements in the programme are those of the research harness on one local machine with one local model: 6.4 to 8.2 seconds per question with reranking.32 The manuscript reports 68.6% with reranking and 61.9% without (6.7 points) at 8.2 s versus 0.44 s end to end; it treats these as rounded values, not an exact matched effect.33 The −4 points for −56% rerank latency belongs to the listwise-reranker experiment above and is not the with/without-reranking pair.25 The previous site’s “<50 ms p95 retrieval”, “~1.3 s end-to-end” and “~100 to 200 ms embedding” figures had no source and are withdrawn.34 A product latency specification will be published only when measured on a stated hardware and model configuration. Every bar below is this system against itself: no other memory system has been run on this harness or this hardware, so none appears in the chart (see “Where this sits next to other memory systems” below for why a cross-system number is not shown as one).
No other memory system has been run on this repository’s leakage-free harness, so there is no measured head-to-head number to publish, for accuracy or for latency; that is a gap, not a modesty pose.120 What exists instead is a set of numbers each vendor or paper reports for its own system, on its own harness, judge and prompt: Mem0’s paper reports Mem0 around 67 (J-score), Mem0-graph around 68, Zep 65.99, full-context around 73; Mem0’s own later marketing claims 92.5; independent replication in the Memori paper found Memori 81.95, Zep 79.09, Mem0 62.47 on the same benchmark; Zep’s own rebuttal after a denominator-fix issue reports 75.14 ± 0.17; A-Mem is listed at 48.4 inside Mem0’s own table.121 Mem0’s own May 2026 post about this spread: “None of these numbers were generated using the same model stack, judge model, or retrieval configuration.”122 This site agrees with that sentence, which is why it does not compute a rank from the numbers in the table below; the comparison that carries meaning is judge sensitivity, not a leaderboard row.
| System | Self-reported figure(s) | Reported by |
|---|---|---|
| Mem0 / Mem0-graph | ≈67 / ≈68 (paper); 62.47 (independent); 92.5 (later marketing) | Mem0 paper (arXiv 2504.19413); Memori paper (arXiv 2603.19935); mem0.ai/research, Sept 2026121 |
| Zep / Graphiti | 65.99 (paper); 58.44 (post-correction); 75.14 ± 0.17 (rebuttal); 79.09 (independent); 94.7 (later marketing) | Mem0 paper; getzep/zep-papers issue #5, May 2025; Zep rebuttal; Memori paper; Zep marketing121 |
| Memori | 81.95 | Memori paper (arXiv 2603.19935), self-reported121 |
| A-Mem | 48.4 | reported inside Mem0’s own comparison table121 |
| ZeroMemory / ByteRover / Dakera / EverMemOS | 96.1 / 92.2 / 88.2 / 93.05 | each vendor’s own self-reported figure, no paper found121 |
| This repository, shipped local_pairs | 73.8% (Gemma lenient) · 62.8% (Qwen lenient) · 31.2% (Qwen strict) · 18.1% (substring) | measured on this site’s own leakage-free harness, four judges shown rather than one30 |
No latency figure appears in that table for any other system: none of the sources above report seconds-per-question, so none is estimated here.121
The December 2025 result of 66.7% overall (66.9% on the 744-question cohort; single-hop 52.1%, multi-hop 74.1%, as labelled) used a GPT-4o lenient judge and had two leakage channels: the gold answer served as an acceptance gate in the speaker-profile path and the gold category label was used for routing.35 These numbers exist to be compared against, not cited; the +5.3 between 66.9% and 72.2% is cross-judge and indicative only.36 W withdrawn
Ten memories descending from one upstream source have ten replicas but one independent root; remove the root and all ten fall.37 A failure domain is any unit (device, organisation, region, shared source) whose failure removes every derivation path that depends on it.38 The survivability margin of a claim is the minimum-cost failure set that intersects every authorized derivation path, a hitting-set object, not a new general min-cut result.39 This track asks whether counting independent roots rather than replicas changes what a collective can still answer after failure.
The cross-node coordinator enforces, in code and tests: NeuralGraph owns local knowledge; the coordinator never gets unrestricted access to private graph internals; cross-node exchange happens only through typed, validated artifacts (ClaimEnvelope, RetrievalTrace, EvidenceExport), never through raw graph reads or a global merged store; policy runs before propagation; failure injection is reversible and payload-free.4041 Documented limitations: a REDACTED policy status fails closed to DENIED; there is no per-source-memory authorization.42 Naming note: an older single-node retrieval module in the repository is also called Tesseract; the two share a name for historical reasons only, and this page is about the coordinator.43
| Gate | What it requires | Status |
|---|---|---|
| 0 | Contract and baseline recovery | PASS44 |
| 1 | Minimal distributed reconstruction: knowledge split across at least two independent holders; no single node holds the full answer (mechanically asserted); coordinated reconstruction by a deterministic rule-based synthesiser; traceable lineage roots on every claim; reconstruction and lineage survive closing and reopening real per-node SQLite storage | PASS; 109 deterministic tests4546 |
| 2 | Fair baselines (centralised, replicated, federated, decentralised) | NOT STARTED; no comparison baseline exists in the coordinator47 |
| 3 | Pareto evidence | NOT STARTED, blocked on Gate 248 |
| 4 | Scale and failure persistence | NOT STARTED49 |
| 5 | Generalisation | NOT STARTED50 |
| 6 | Standalone architecture strength | NOT STARTED51 |
In plain terms: the coordinator has passed a first gate; it has no baselines and no scale evidence yet.
| Strategy | node fail | mem del | lineage root | auth revoke | route rm | edge corrupt | stale | partition | worst domain | mean | bytes |
|---|---|---|---|---|---|---|---|---|---|---|---|
| isolated local | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.000 | 1,371 |
| centralised | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.111 | 3,020 |
| fixed distributed replication | 1.00 | 1.00 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.556 | 4,943 |
| source-count repair | 1.00 | 1.00 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.556 | 4,941 |
| full replication | 1.00 | 1.00 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.00 | 0.667 | 9,877 |
| random path diversification | 1.00 | 1.00 | 0.67 | 0.67 | 1.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.704 | 5,327 |
| lineage-aware repair | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.778 | 5,448 |
| oracle min-cut | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.00 | 1.00 | 0.889 | 6,461 |
“bytes” is serialised transfer volume, not resident-memory savings; forgetting shown elsewhere is unrecovered after repair; full replication survives partition while the repair strategies do not; the scope is a small-N deterministic simulator that validates the mechanism, not 1K to 10K-agent populations.53 Invariants: replica counting survives lineage-root failure at 0.00 while lineage-aware repair survives at 1.00; only the oracle survives worst-single-domain failure; verdicts are invariant across K in {2,3,5,8} and H in {2,3}; lineage-aware repair loses to source-count repair in exactly one of six node-failure cells.54 Statistics: 7/9 versus 6/9 gives Fisher p = 1.00, versus 5/9 p = 0.62; the Wilson 95% interval for 7/9 is [45.3%, 93.7%]; an underpowered mechanism demonstration, not a percentage-point performance claim.55The manuscript’s significance statistics for this fixture are under review; the repository record gives the 30-seed means only. |
|||||||||||
Reported in the manuscript; artifacts are not in the current repository. Combined questioning reaches 0.871 accuracy versus 0.834 for random questioning with roughly a third fewer questions (10 seeds × 2,000 claims), and at budget 2 exceeds random at budget 8 (paired +0.0097 ± 0.0032, positive in 7 of 8 seeds).56 But it is worse than random at avoiding confident error in 15 of 23 cells, uncertainty-only triggering leaves confident error at 0.028, and lineage selection alone has 3 wins, 8 ties and 12 losses: acquisition benefits do not establish calibration benefits.57
Reported in the manuscript. N in {100; 1,000; 10,000; 100,000} simulated agents in teams, failure domains, organisations and regions; each evidence item has one origin; about 35% of claims have a single origin; 25% of facts are revised; semantic values are symbols; there are no LLM calls, no natural-language question generation and no deployed network processes.58 Manuscript ladder (always prefixed “manuscript”): B0 isolated; B1 centralised store (a performance baseline, not decentralised); B2 broad replication; B3 / B4 random fixed-budget replication / gossip; B5 diversity-blind replica-count placement; B6 lineage-aware fabric; B7 = B6 plus continual questioning at the same final storage; B3Q = random placement plus the same questioning.59 Failures: random churn, whole-domain, organisation or region loss, targeted attacks, corruption, two- and four-way partitions; the white-box attacker knows each policy’s placement and greedily removes independent support, a lower bound on an optimal adversary.6061 30 seeds at N ≤ 10,000, 10 at 100,000; 97.2 CPU-minutes on one machine without a GPU, which is simulator compute, not population inference cost.62
| Outcome | Difference | 95% CI | Holm p |
|---|---|---|---|
| Knowledge survival | +.000 | [−.001, .001] | .59 |
| Task accuracy | +.010 | [.009, .012] | < 1e-6 |
| Independent-support survival | +.046 | [.046, .047] | < 1e-6 |
| Contradiction-detection F1 | +.155 | [.149, .161] | < 1e-6 |
| Contradiction resolution | +.003 | [−.001, .008] | .33 |
Knowledge survival, accuracy, independent-support survival and F1 are distinct outcomes; large standardised effects reflect low seed-to-seed variance in a controlled simulator and do not imply comparable deployment gains; storage counts published replicas only.64 Placement, not aggregation, is the operative mechanism in this simulator (B6 .452 versus B6 with count aggregation .451), so “lineage-aware aggregation” is not credited separately.65 |
|||
Absolute values for reference: manuscript-B6 .679 / .451 / .550 / .506 versus manuscript-B3 .678 / .441 / .504 / .351 (survival / accuracy / ISS / F1).66 Under 20% origin corruption on top of 30% churn: accuracy +.019, detection F1 +.199. Under coordinated corruption survival is B6 .750 versus B3 .751 and B7 falls to .733; under 70% whole-domain loss survival differs by +.001 (p = .16).67
| System (manuscript ladder) | Knowledge survival | Replicas per claim |
|---|---|---|
| B6 lineage-aware fabric | .679 | 4.0 |
| B3 random replication | .678 | 4.0 |
| B4 gossip | .749 † | 48.4 |
| B1 centralised store (performance baseline) | .876 | 36.3 |
| B2 broad replication | .870 | 193.7 |
† B4 gossip: .749 is the Table 3 value under the 50% white-box attack;66 the prose (§5.6 ll.206-207) gives B4 .669 at the same storage in the sentence that follows the 90%-churn comparison.68 We read .669 as the 90%-churn value; the manuscript does not label it explicitly, so neither is shown as a headline; see discrepancies. Broad replication reaches .870 at about 48 times the storage; at 90% churn it retains .843 versus B6 .300; a budget sweep k = 1..32 (15 seeds) shows B6 and B3 survival rising identically from .205 to about .877 while independent-support survival at k = 32 is .982 (B6), .892 (B3), .566 (B5). The fabric cannot claim dominance over these baselines; their higher costs purchase substantial survival; lineage-blind duplication saturates independent support.69 |
||
| Condition | B6 accuracy | B7 accuracy (B6 + questioning) | B3Q accuracy (random placement + questioning) |
|---|---|---|---|
| 50% random churn | .574 | .763 | — |
| 50% organisation loss | .575 | .764 | B3 + .245 (exceeds B7’s +.189) |
| 50% white-box attack | .451 | .594 | .628 |
| 20% origin corruption | .475 | .640 | — |
| four-way partition reconnect | .382 | .695 | .792 |
Cost: 14.5 versus 8.0 messages per claim (about 1.81×) at unchanged storage of four replicas; not a comparison at equal message budget.71 B7 − B3Q under attack is −.034 [−.036, −.032], Holm p < 1e-6, while B7 retains more independent support (.550 versus .507): a quality–accuracy trade-off, not dependence of discovery gains on lineage-aware placement.72 After partition reconnect, detection F1 falls .761 → .622 while resolution accuracy rises .299 → .548: refreshing evidence can remove visible disagreement, so detection and resolution are reported separately.73 Re-verification is privileged: an uncorrupted origin re-observes current symbolic truth, so its benefit measures the value of a successful refresh, not the ability or cost of formulating good questions.7475
At 50% random churn, per-claim storage is exactly 4 for B3, B6 and B7 at every tested population because the budget imposes it, not an emergent compression law; knowledge survival for B6 is .8218 / .8204 / .8209 / .8212 and for B7 .8826 / .8808 / .8813 / .8816 at N = 100 / 1,000 / 10,000 / 100,000; evaluation time rises from .0039 s to .27 s.82 Wording rule: “behaviour is stable across four simulated scales”, never “scales to 100,000 agents”; four points do not establish asymptotic complexity or network latency; mutual reachability outside explicit partitions is an assumption, not a discovery protocol; private evidence, lineage metadata, authorization and transport overhead are not measured.83
Measured conversational retrieval involves two holders; the repair fixture has nine capabilities; the initial questioning probe is synthetic. The campaign establishes behaviour through 100,000 simulated agents under a declared distribution model; it does not establish that 100,000 deployed LLM agents can discover questions, reason over text, maintain authorization or coordinate within these costs. No experiment validates the entire continual-discovery architecture end to end. The greedy attacker is a lower bound; recovery time is a logical-round proxy; the world generator is a single modelling family; the 50,700 rows are not 50,700 independent worlds. Privacy and authorization are architectural constraints, not experimentally established guarantees.84
A synthetic acquisition probe and a population-scale symbolic simulation campaign (100 to 100,000 simulated agents, no LLM calls) are described in a manuscript under review. In the lab’s own words: at equal storage, lineage-aware placement changed evidential quality rather than raw availability; broad replication bought more availability at far higher storage; continual questioning raised accuracy at a message cost, and the same gain appeared without lineage-aware placement. Tables, effect sizes and the where-it-loses list will be published here after the review decision, with the date recorded in the corrections log.85
Can a five-layer hierarchy of local agents (user → team → department → region → executive) discover, verify, compress, revise and safely propagate organisational knowledge better than centralised alternatives, while exposing less raw private data?88 The benchmark is adversarial by construction: centralised baselines receive strictly more information than Mycelic, every raw record with exact attributes and full text, and the benchmark is built so that Mycelic can lose.8990
A synthetic organisation of N workers evaluating AI models; no production data from any company is used and no customer is named.91 Each interaction carries nine structured attributes (model family and version, task family, context length, input format, domain, language, difficulty, tool) and outcomes (a score and 18 error labels).91 The world hides effects, conjunctions of one to four attribute values under which one error label is elevated by 0.2 to 0.45, of kinds local, cross-team, global, temporal, contradiction and decoy.92
| Layer (bottom → top) | Condition 1 | Condition 2 | If both hold |
|---|---|---|---|
| worker | some worker holds at least 1.25 × n_min matching evidence | no layer below; the condition is vacuous | minimum discovery layer = worker (a single worker can infer the effect) |
| team | some team holds at least 1.25 × n_min | no single worker holds more than half the evidence | minimum discovery layer = team |
| department | some department holds at least 1.25 × n_min | no single worker or team holds more than half the evidence | minimum discovery layer = department |
| region | some region holds at least 1.25 × n_min | no single unit below holds more than half the evidence | minimum discovery layer = region |
| executive | the organisation holds at least 1.25 × n_min | no single unit below holds more than half the evidence | minimum discovery layer = executive |
Every cross-team and global finding therefore carries a verified property: no individual worker can infer it alone. The table states the rule; a worked example with counts will replace it when a campaign has results to draw from.93 |
|||
For every effect the minimum discovery layer is computed, not assumed: α = 1e-8, power 0.8, the lowest layer at which some unit holds at least 1.25 × the minimum sample, and no single unit below that layer may hold more than half the evidence; this makes “no individual worker can infer it alone” a verified property of every cross-team and global finding.93
| Layer | Moves up | Moves down | Boundary and bookkeeping |
|---|---|---|---|
| worker device | structured observations only | answers to QuestionArtifacts as one-cell sketches | DLP boundary around each device: no text leaves a device; cells with fewer than k contributors are suppressed; canary tokens planted in sensitive records measure leakage exactly |
| team node | a Sketch (per-cell counts with distinct worker, team, department and region columns) and Claims, each with a signature (cell, label), statistics, a SupportRecord, a LineageRecord, validity, a status and an HMAC signature | QuestionArtifacts: bounded requests for order-(k+1) cell counts | independent-support discount after correlation: exact copies count 0; the same worker counts 0 beyond the first; same team 0.5; same department 0.8; otherwise 1.0 |
| department and region nodes | pooled and synthesised claims; Conflict objects opened and resolved by independent-support ratio or region split; claims a recent window refutes are superseded | QuestionArtifacts | every system re-tests accepted claims each round and withdraws those no longer significant |
| executive node | — | QuestionArtifacts | claim status in {proposed, accepted, quarantined, superseded, unresolved}; derivation operator in {observe, pool, synthesize, revise, answer_question} |
A device emits structured observations only; no text leaves a device under DLP.94 A team node promotes a Sketch (per-cell counts with distinct worker, team, department and region columns) and Claims, each with a signature (cell, label), statistics, a SupportRecord with independent support after correlation discount, a LineageRecord with parent claim ids, hashed root worker ids, path and a derivation operator in {observe, pool, synthesize, revise, answer_question}, validity, a status in {proposed, accepted, quarantined, superseded, unresolved} and an HMAC signature.95 Higher nodes pool, synthesise, open Conflict objects, resolve them by independent-support ratio or region split, and supersede claims a recent window refutes; every system re-tests accepted claims each round and withdraws those no longer significant.9697 QuestionArtifacts travel downward as bounded requests for order-(k+1) cell counts and are answered by one-cell sketches.98 Independent-support discount: exact copies count 0; the same worker counts 0 beyond the first; same team 0.5; same department 0.8; otherwise 1.0.99 Cells with fewer than k contributors are suppressed; canary tokens planted in sensitive records measure leakage exactly.94100 This describes what the benchmark measures, not product features.
| System | What it is | Receives | Lineage |
|---|---|---|---|
| bench-B0_isolated | each worker searches only its own data | own records | n/a |
| bench-B1_central_keyword | central inverted index; beam search under a query budget | all raw records | none |
| bench-B2_central_rag | hashed bag-of-words retrieval | all raw records | none |
| bench-B3_central_llm_summary (+ _scoped) | frontier-profile extractor builds sketches per chunk; all text to cloud | all raw records | none |
| bench-B4_majority_vote | workers with matching records vote | all raw records | replica count |
| bench-B5_flat_agents (+ _lineage) | one aggregator with a fan-in byte budget; the _lineage variant isolates topology from lineage | sketches (+ claims with lineage) | none / yes |
| bench-B6_hier_no_lineage | Mycelic topology; claims accepted by replica count | sketches + claims | replica count |
| bench-B7_mycelic | lineage-aware five-layer hierarchy with DLP | sketches + claims with lineage | yes |
| bench-B8_mycelic_security | + local classifier + lineage-aware verifier | same | yes |
| bench-B9_mycelic_questioning | + expected-information-gain questioning | same + answers | yes |
| bench-ORACLE_central_stats | exhaustive search on exact data; a reference, not an upper bound | everything | exact |
All systems share the same hypothesis test (BH-corrected), the same claim bookkeeping and the same analysis cadence; they differ only in what information reaches the test.102 |
|||
Comparisons are paired by seed: paired bootstrap 95% CI (10,000 resamples), paired t, Wilcoxon signed-rank, Cohen’s d_z, Holm correction within a family; never a p-value without an effect size.109110 Tier 1: 100 to 1,000 workers, 30 seeds; Tier 2: 10,000 workers / 100,000 interactions, at least 10 seeds; Tier 3: 50,000 to 100,000 workers, 3 to 5 seeds.111 Simulator cost on a 4-vCPU container without a GPU: a tier-1 hierarchy run about 18 to 25 s at 0.35 GB; tier 2 about 310 s plus 150 s evaluation at 3.4 GB; simulator cost, not product performance.112
No aggregator, baseline or agent imports ground truth; every system uses the same hypothesis engine; centralised baselines always see the full raw records; the simulated frontier model dominates the simulated edge model on every fidelity parameter (the configuration validator refuses otherwise); failed runs are recorded, never deleted; every reported number comes from results/processed via the experiments scripts; every manifest states the model-backend limitation.113114 To replace the simulated profiles with a measurement, a validation script measures a real model’s profile against any OpenAI-compatible or Ollama endpoint and the figures label rows as synthetic or measured.115
The figures and tables generated from results/processed by the benchmark’s own scripts, with manifests; the failure narrative; answers to the research questions; and a dated entry in the corrections log.116117 Until then this section carries no number from the running campaign. Corrections log →
Each marker above points to one note below. Paths are relative to the root of the NeuralGraph repository, which is public to read at github.com/anovruzov/NeuralGraph with no licence selected (all rights reserved); they are given as text, with line numbers checked against main on 2026-09-17, and files under mycelic-org-benchmark/ are on branch claude/mycelic-org-benchmark-9cnf35, not yet merged into main. “Manuscript” is the paper under review, cited by section and printed margin line; notes that support only manuscript-derived copy are held out of the deployed page together with that copy. “website/” is the previous site, cited only for the claims withdrawn from it.