Back to Blog
Galen Guan

Semantica Deep Dive: Auditability Nailed, Correctness Never Attempted

Semantica Deep Dive: Auditability Nailed, Correctness Never Attempted

Which signals do you actually rely on when picking a dependency? The star curve, how complete the README is, whether the CI badge is green, whether there's a docs site and a Discord, how carefully the CHANGELOG is written. Those signals are widely used because they usually correlate — a team willing to write a 75 KB README is usually willing to write decent tests.

semantica-agi/semantica is a sample that maxes out every one of those signals and then falsifies them one by one. It hit #1 on GitHub Trending on 2026-08-10 and reached 5,247 stars within two days. The README's front page calls it "The Open Source Palantir for AI Agents," and the stated buyers are finance, healthcare, legal, and government — regulated industries.

I installed v0.6.5 into a clean Python 3.11 virtualenv, ran its own Quick Start, ran its own 4,400 tests, and read all four reasoning engines line by line. Every terminal output below is an actual run.

What it claims to be

Semantica isn't positioned as "yet another GraphRAG." The claim is bigger: a deterministic infrastructure layer sitting underneath your LLM, vector store, and agent framework — graph construction, reasoning, and provenance all without an LLM in the loop.

The front-page capability list: Context Graphs, Decision Intelligence (every decision a first-class queryable object), end-to-end W3C PROV-O provenance, deterministic reasoning (forward chaining + Rete + Datalog + SPARQL), SHACL/OWL/SKOS ontology governance, conflict detection, entity resolution, point-in-time snapshots, and a storage layer covering both RDF and labeled property graphs.

Nothing else in open source covers that much surface at once. 178,456 lines of Python, 349 source files, 29 ingestors, 9 graph backends.

Breadth isn't the problem in itself. GitNexus, which I took apart earlier, has a 12-stage indexing pipeline — no less complex — but every stage maps to a verifiable artifact. The question is whether breadth comes with matching verification density.

Semantica capability stack: claimed vs measured (v0.6.5, clean Python 3.11 env)

Start with what's real: the provenance layer

The fair way to take apart a project is to find what it genuinely got right first. Semantica's provenance module is one of the most complete implementations I've seen in open source, and it is not marketing copy.

ProvenanceManager emits valid PROV-O Turtle including qualified relations like qualifiedAssociation and qualifiedGeneration — plenty of projects claiming PROV-O support stop at wasGeneratedBy. Every record carries checksum + previous_checksum, forming a tamper-evident hash chain:

>>> pm.verify_chain()
{'valid': True, 'total_entries': 2, 'broken_links': []}

ex:acme_corp a prov:Entity ;
    prov:generatedAtTime "2026-08-12T08:51:41"^^xsd:dateTime ;
    prov:qualifiedAssociation [ a prov:Association ;
            prov:agent ex:semantica ;
            prov:hadRole ex:role_generator ] ;
    prov:qualifiedGeneration [ a prov:Generation ;
            prov:activity ex:entity_tracking ] ;
    prov:wasAttributedTo ex:semantica .

Three more things are equally real:

Backend coverage. On the RDF side: Oxigraph (embedded, no external server), Blazegraph, Jena, RDF4J, Anzo. On the LPG side: Neo4j, FalkorDB, Apache AGE, AWS Neptune. Each backend runs 1,000–1,800 lines and was clearly written with care. Supporting RDF and LPG side by side is close to unique in open source — Cognee, which I dissected earlier, only covers the LPG + vector half.

The Datalog reasoner. datalog_reasoner.py implements proper unification: binding-conflict detection, arity checks, incremental binding dicts. This has to be stated separately from the Rete situation below — quality in this project is highly uneven, and a blanket dismissal would be wrong.

Real security investment. Dedicated ingest/ssrf.py, sparql_escaping.py, query_sanitize.py; CI running CodeQL, Checkov, and Microsoft Defender for DevOps, plus a workflow whose only job is verifying that every GitHub Action is pinned to a commit SHA. Multiple GHSA advisories — SPARQL injection, Cypher label injection, SSRF DNS rebinding, ReDoS, WebSocket Origin bypass — have been disclosed and fixed.

The 179 KB CHANGELOG deserves a mention too. It openly writes things like "the resulting AttributeError was silently swallowed" and "metadata-only filtering was silently non-functional out of the box." That kind of self-incriminating record is a plus.

These are real assets. Precisely because they're real, the rest matters more.

Three headline claims that fail on measurement

1. The Rete engine fires any rule on any fact

The README promises "Deterministic Reasoning: forward chaining, Rete network, Datalog, and SPARQL with fully explainable paths, not black boxes."

The entire value of Rete is discrimination: the alpha network filters facts by condition, the beta network joins on variable bindings, and only genuinely matching combinations reach a terminal node. In semantica/reasoning/rete_engine.py, those two discrimination functions look like this:

def _matches(self, fact: Fact) -> bool:
    """Check if fact matches condition."""
    # Simple matching - can be enhanced
    return True

def _can_join(self, left_fact: Fact, right_fact: Fact) -> bool:
    """Check if facts can be joined."""
    # Simple join logic - can be enhanced
    return True

Two unconditional return Trues. This isn't a crude Rete implementation; it's a Rete-shaped shell with the discrimination logic removed. Measured:

# Rule:  is_manager(X) -> is_employee(X)
# Fact:  likes_pizza(bob)     <- entirely unrelated to the rule's condition

facts added:    ['likes_pizza(bob)']
rule condition: ['is_manager(X)']
matches fired:  [('manager_is_employee', ['likes_pizza(bob)'], confidence=1.0)]

=> the engine inferred is_employee(bob) at confidence 1.0

A rule about managers fired on a fact about pizza, at confidence 1.0, and that inference gets written into the graph with a PROV-O provenance stamp on it.

For a product sold into "the underwriting agent's approval has to survive a regulator's why months later," this isn't a defect — it's negative value. It manufactures false inferences carrying a complete audit trail. A genuine black box at least doesn't claim to be deterministic.

2. Precedent search is all-false-positive in one config and all-false-negative in the other

Decision Intelligence is the flagship feature: store every agent decision as a first-class object, then retrieve past precedents by similar scenario.

The score is 0.7 × content similarity + 0.3 × structural similarity. The structural term is where it breaks:

def _calculate_structural_similarity_for_decision(self, decision_id: str, scenario: str) -> float:
    ...
    similar_nodes = self.find_similar_nodes(decision_id, similarity_type="structural", top_k=5)

Note scenario in the signature — it's the query, and the body never uses it. The function measures how structurally similar this decision node is to other nodes in the graph, and since every decision node has identical structure (isolated, no edges), structural similarity is always 1.0. Every decision therefore scores 0.3 × 1.0 = 0.3, exactly equal to the default min_similarity threshold of find_similar_decisions.

I stored three unrelated decisions (HIPAA cloud selection / German payroll vendor / adopt Kubernetes) and queried:

advanced_analytics=True     <- the config the README Quick Start recommends
   'HIPAA cloud'                  -> [0.5, 0.3, 0.3]
   'banana pizza quantum unicorn' -> [0.3, 0.3, 0.3]   <- all three match
   ''  (empty query)              -> [0.3, 0.3, 0.3]   <- all three match

advanced_analytics=False
   'HIPAA cloud'                  -> []   <- even the genuine match is missed
   'banana pizza quantum unicorn' -> []

With advanced analytics on, every query returns everything, including the empty string. With it off, content similarity alone never clears 0.3 and every query returns nothing. Neither configuration is usable.

3. "Deterministic, LLM-free extraction" produces semantic garbage

This is the project's central technical claim. I tested it on the most standard M&A sentence I could write:

"Acme Corp, headquartered in Berlin, acquired WidgetCo for $4.2 billion in March 2024. CEO Jane Doe said the deal closes in Q3."

Scenario A: straight out of the box (the README never mentions downloading a spaCy model)

spaCy model en_core_web_sm not found. ML method will fallback.

Entities: Acme Corp/ORG, Jane Doe/PERSON, 2024/DATE
          missed Berlin, WidgetCo, $4.2 billion, March, Q3
Relations: Acme Corp --related_to--> Jane Doe
           Acme Corp --related_to--> 2024        <- pure co-occurrence noise

With the model absent it silently degrades to regex extraction, printing one warning to stdout. The return shape is identical, so the caller receives no programmatic failure signal at all.

Scenario B: after manually running python -m spacy download en_core_web_sm

Entities: 7/7 correct (Acme Corp, Berlin, WidgetCo, $4.2 billion, March 2024, Jane Doe, Q3)
Relations: $4.2 billion --located_in--> March 2024   (confidence=0.7)

Entity recognition, borrowed from spaCy, is good. But exactly one relation comes out, and it says "$4.2 billion is located in March 2024" — a triple that is not semantically coherent in any reading, entering the knowledge graph at confidence 0.7.

Meanwhile the one fact that actually matters, acquired(Acme Corp, WidgetCo), is never extracted in either scenario.

The implementation explains it: the primary path for "deterministic relation extraction" is co-occurrence statistics plus regex templates, with predicates guessed from a small pattern table. Nearly all the value of a knowledge graph lives in the relations; entities are just nodes.

How one false fact gets certified as audit evidence by a W3C standard

Two deployment notes worth knowing: RelationExtractor downloads model files from the HuggingFace Hub on init (I observed the Fetching 5 files requests), which matters for a library marketed as self-hosted for regulated environments; and every spaCy-extracted entity carries confidence 1.0 while every regex hit carries 0.7 — those are constants, not calibrated values. For a product that writes confidence into an audit record, that's misleading.

Why all this survives: CI never runs the tests

The repo has 258 test files, 87,841 lines of test code, 6,714 assertions. That investment is real.

But across nine GitHub Actions workflows, not one executes pytest. ci.yml does three things: install frontend deps, run 3 JS tests, build the wheel and check that static assets made it in. The benchmarks/ directory that benchmark.yml references does not exist in the repository.

So I ran the suite myself, on the same Python 3.11 the CI pins, with dependencies installed:

160 failed, 4061 passed, 161 skipped, 186 deselected  in 104s

Failures include core functionality (not just missing optional deps):
  tests/kg/test_algorithms.py::test_betweenness_centrality
  tests/kg/test_algorithms.py::test_closeness_centrality
  tests/kg/test_algorithms.py::test_eigenvector_centrality
  tests/kg/test_algorithms.py::test_louvain_communities   <- community detection returns empty
  tests/semantic_extract/test_extractors.py (all batch paths)
  tests/seed/test_seed_manager.py::test_load_from_csv

Top error types (all genuine defects, not environment issues):
  17 x ProcessingError
  14 x TypeError: 'NoneType' object is not subscriptable
  14 x TypeError: '>' not supported between MagicMock and MagicMock
  11 x TypeError: object.__new__(X): X is not a type object

The README carries a CI badge and it is green. It's green because it tests nothing.

That single fact explains how everything above survived. This isn't "the tests are bad" — it's "the tests were never a gate." Eighty thousand lines of test code sit in the repo and only mean anything when a human happens to run them.

Other engineering problems found along the way

Docs and the real API diverge widely. Copying straight from the README and module docstrings:

>>> graph.get_graph_metrics()   # listed as an "Enhanced Method" in the module docstring
AttributeError: 'ContextGraph' object has no attribute 'get_graph_metrics'

>>> graph.check_decision_rules({"category": "vendor_selection"})  # verbatim from the README
{'compliant': False, 'violations': ['Confidence too low: 0',
                                    'Invalid outcome: None',
                                    'Missing required field: decision_maker']}
# The README uses it as a filter; the implementation treats the dict as
# "the decision under validation." No error — just a plausible-looking,
# entirely meaningless compliance verdict.

Issue #920 already concedes that "the ContextGraph module docstring example is not runnable." That's especially bad in an AI-coding-assistant context, where docstrings are the primary input.

Dependency weight is out of control. torch, transformers, spacy, opencv-python, librosa, gensim, faiss-cpu, umap-learn, matplotlib, seaborn, plotly, ipywidgets are all mandatory core dependencies, not extras. pip install semantica pulls 134 packages — 1.8 GB measured on macOS ARM, and 4–6 GB on Linux x86_64 where you get CUDA torch. A "context and provenance infrastructure" library that requires PyTorch and OpenCV to record one decision is effectively unusable in container images and serverless.

Related: it will not install on Python 3.14 at all (gensim has no cp314 wheel and the source build fails — reproduced); requires-python = ">=3.8" is false (numpy≥2.0.2 needs 3.9+, scikit-learn≥1.7 needs 3.10+), with the real usable range being 3.10–3.13; and PyPI metadata says Development Status :: 5 - Production/Stable at version 0.6.5.

Silent failure at scale. except Exception appears 1,256 times, 40 of them followed directly by pass. When structural similarity computation fails it returns 0.0 instead of raising — the error collapses into a legitimate-looking low score, and the caller cannot distinguish "not similar" from "the computation crashed."

Breadth-first generation artifacts. Module docstrings run 80 lines listing every method by name (some of which don't exist); the entire semantica/evals/ module contains __status__ = "coming_soon" — meaning a project whose whole premise is extraction quality ships no extraction-accuracy benchmark at all; and the repo-root mcp/ directory (1,597 lines, more capable) is not packaged into the wheel — pip gives you semantica/mcp_server/ (617 lines) instead, so two MCP implementations coexist.

How it compares

Project Stars PyPI/month Positioning Advantage over Semantica Semantica's edge
semantica 5.2k ~4.4k Context graphs + decision provenance + deterministic reasoning — —
mem0ai/mem0 63.1k 4,079k Agent memory layer Maturity, ecosystem, 3 orders of magnitude more real usage; small API surface PROV-O provenance and RDF/OWL standards, which mem0 lacks
getzep/graphiti 29.8k 1,575k Temporal knowledge-graph memory Industrial bi-temporal model, incremental updates, published benchmarks More backends (full RDF family), not tied to Neo4j
HKUDS/LightRAG 38.8k 313k Graph-enhanced RAG Paper-backed with retrieval benchmarks; lean implementation Ontology/SHACL/conflict detection; LightRAG only does retrieval
microsoft/graphrag 35.4k 71k LLM-driven graph RAG Microsoft backing, community-summary method with published evaluation Claims LLM independence (see above) and full self-hosting
topoteretes/cognee 30.0k 196k AI memory / ECL pipeline Closest positioning, tighter pipeline, lighter dependencies More complete RDF/W3C stack and compliance export
neo4j/neo4j-graphrag-python 1.3k — Official GraphRAG SDK Neo4j-maintained, predictable behavior, minimal dependencies Not tied to one backend, far broader feature surface

(Data as of 2026-08-12)

How to read that table: Semantica genuinely wins on the feature checklist axis — no other open-source project covers RDF+LPG dual paradigms, PROV-O, SHACL, OWL, SKOS, Datalog, a visual workbench, MCP, CLI, and REST in one package. What it loses is depth and verification on each of them. The competitors are narrower, but the narrow part has been benchmarked and hammered by millions of downloads — Graphify's "build the knowledge graph, outsource everything else" approach has a much shorter feature list and a much more complete verified surface.

The table also surfaces something else worth stealing.

Stars aren't usage: monthly downloads per star

mem0 gets 64.6 monthly downloads per star, Graphiti 52.9, and Semantica 0.85 — nearly two orders of magnitude lower. Trending brings attention, not adoption. That ratio belongs in your open-source evaluation checklist: stars are a function of marketing, downloads are a function of use, and the wider the divergence, the wider the gap between README and code usually is.

Scorecard

Dimension Score Basis
Provenance & audit 8/10 Valid PROV-O + hash chain + verify_chain(); rare completeness in open source
Backend & ecosystem coverage 8/10 9 graph stores, 29 ingestors, MCP/CLI/REST/visualization all present
Security engineering 7/10 Dedicated SSRF/injection modules, CodeQL/Checkov, action pinning, prompt GHSA response
Graph & storage correctness 6/10 Backends written carefully, but centrality and community-detection tests fail locally
Documentation accuracy 4/10 README examples don't match the API; docstrings list methods that don't exist
Dependency & deployment 3/10 Mandatory torch/opencv, 1.8–6 GB, no lightweight path, false Python-version claim
Engineering & test discipline 2/10 80k lines of tests + zero CI test steps + 160 local failures
Extraction quality (LLM-free path) 2/10 Relation extraction is co-occurrence + regex, emits semantically false triples
Reasoning-engine correctness 1/10 Rete discrimination logic is empty; any rule matches any fact
Decision-retrieval relevance 1/10 All-false-positive or all-false-negative depending on config

As a complete platform: 3/10. semantica.provenance taken on its own: 8/10.

Conclusion: worth watching, not worth trusting

Semantica built the most complete shell any open-source project has put around the real problem of AI auditability, and left the core — is the reasoning correct, is the retrieval relevant, is the extraction accurate — in an unverified state, with a CI that doesn't run tests concealing the fact.

Concretely:

Where it's worth considering. Use the provenance module alone: when you need to bolt a W3C PROV-O audit chain onto an existing pipeline, semantica.provenance is directly usable (or read its schemas.py and reimplement, avoiding the dependency tree). Use it when you need a real RDF semantic stack: if your compliance requirement names OWL/SHACL/SPARQL rather than "a vector store with some graph," its triple-store layer is one of the few open-source options. And use it for prototypes and concept demos — the Explorer UI, cookbook, and CLI make "explainable AI governance" quick to show.

Where not to use it. Any decision path that reaches production. Compliance systems in regulated industries — which is exactly the marketed use case, and exactly where a wrong-but-provenanced conclusion does the most damage. Deployments sensitive to image size or cold start. And "I just want agent memory," where mem0 and Graphiti are orders of magnitude more mature.

If you use it anyway, at minimum: pin Python 3.11/3.12; run python -m spacy download en_core_web_sm immediately after install (otherwise you silently get regex results); run pytest tests -m "not integration" once and triage the failures touching your modules (CI won't do it for you); treat rete_engine as unusable and reach for its datalog_reasoner or swap in experta/durable_rules; always route relation extraction through the LLM path; and audit the HuggingFace Hub egress before any offline deployment.

The last point is the transferable finding, and it outlives Semantica itself: when a project combines very high feature breadth, a very low downloads-per-star ratio, and a CI that runs no functional tests, the README needs to be repriced against the code. All three signals are publicly checkable and take about ten minutes together — far cheaper than the hours this cost me.

This verdict can flip. Wire pytest into CI, turn evals from "coming soon" into a real benchmark, fix Rete and precedent search, and one or two releases would do it. The foundation is there: the provenance layer and the RDF stack are genuine assets, and the maintainer is responsive on issues. I'll keep watching.

Sources