A knowledge graph you can check

Today is the kickoff of the NVIDIA AWS Bio-Agent Knowledge Graph Construction Challenge, and I am one of the participants. The brief is short: build AI agents that construct, maintain and query biomedical knowledge graphs. Results are announced on 15 December. I am entering solo, in the closed-source tier, with a pipeline that grew out of LibrisVault.

The hard part

A biomedical knowledge graph connects genes, diseases, chemicals and phenotypes through typed relations: this gene is associated with that disease, this drug treats that condition. Once those connections exist across thousands of papers, questions that would take a week of reading become a query. The problem is building them. Curation by hand is slow while automatic extraction tends to produce edges nobody can trace back to a source. A graph whose edges you cannot check is hard to trust. Nobody uses a graph they do not trust.

LibrisVault already turns documents into a linked Markdown knowledge base, but its graph was internally consistent and externally ungrounded. Pages were linked to each other, not to the identifiers the rest of the field uses. Closing exactly that gap is what this challenge is about for me.

What happens to a paper

A paper enters as a PDF, a PubMed Central article or an abstract. Its text is extracted in a sandbox without network access, and its PubMed ID and license are recorded. Then two things run.

An ingest agent writes a page for the paper and one for each gene, disease or chemical it discusses, and grounds each entity page to a standard identifier through a local index of six ontologies: HGNC for genes, MONDO for diseases, HP for phenotypes, ChEBI for chemicals, GO for processes, and MeSH as a second identifier.

An extraction step then lists the paper’s entities again, grounds them, and puts every pair of grounded entities to a model with one question: does this paper relate the two, and if so how? The answer is either none or a predicate from the BioLink model, the shared schema of biomedical knowledge graphs, together with the sentence that supports it, quoted verbatim. Code, not the model, then decides what is written: whether the predicate fits the two categories, whether the sentence exists, which page each end belongs to.

What lands in the vault is a table on the paper’s own page:

| [[SQSTM1]] | biolink:causes | [[Paget disease of bone]] (MESH:D010001) |  | "Mutations of the p62/Sequestosome 1 gene (p62/SQSTM1) account for both sporadic and familial forms of Paget's disease of bone (PDB)." |

One row is one claim from one paper. A researcher who disagrees deletes the line, and the next export no longer carries it. The whole vault is plain Markdown under git, one commit per paper, and each commit records the model and pipeline version that produced it. The export goes to KGX, the exchange format of biomedical graphs, validated against BioLink, and from there into Amazon Neptune.

How to benchmark a pipeline

The pipeline is measured against BioRED, a public corpus of PubMed abstracts in which experts annotated entities and relations. I develop on half of its development set and keep a change only if it also holds on the other half, which I never tuned on. One run scatters by several points on its own, so every comparison is paired over the same documents. The prompts contain no examples from the benchmark, and the test set stays untouched until the final evaluation.

That discipline already decided most of what the pipeline looks like today. Several ideas that looked good on the first half did not survive the second. Two changes held up on both halves. Together they moved the result more than everything else combined.

What comes next

The next steps include running a corpus of 50 open-access papers through the pipeline, the BioRED test evaluation, and a public release of the corpus, the graph and its export.

The thing I care about most will not change: every edge in the graph should be one click away from the sentence it was drawn from.