AI-Native Series · Curriculum Engineering

I Compiled 58 Snowflake Courses Into an Agent. It Failed Me First.

By Paul Jialiang Wu · agentic-portfolio-lovat.vercel.app · 2026-08-03

A course catalogue is a menu, not a map. What it takes to turn one into the other — and why the honest answer is mostly a list of things you don't know. ~9 min.

Snowflake tells you where the exam stops. Almost nobody reads it.

On the page for the SnowPro Advanced: MLOps Engineer certification, past the marketing, there is a sentence describing who should sit the exam. The last clause is the most honest thing on the whole site:

“Knowledge of general ML engineering concepts and cloud infrastructure is assumed but not tested.”
— Snowflake, SnowPro Advanced: MLOps Engineer candidate profile [1]

Read that again, because it is a vendor voluntarily drawing the boundary of its own instrument. The exam measures Snowflake-specific operations. It assumes the engineering underneath and does not check it. That is not a flaw — it is unusually good epistemic hygiene, and it is the exact discipline that goes missing the moment anyone builds a tool on top of a course catalogue.

I went looking for such a tool. There isn't one.

Everyone built agents for the warehouse. Nobody built one for the curriculum.

Snowflake's own engineering org has shipped an impressive agentic toolchain. Snowflake-Labs/mcp is a Model Context Protocol server exposing Cortex AI, object management and SQL orchestration [2]. cocoplus — the most-starred of the set — describes itself, verbatim, as “an Agentic Operating System for Snowflake Coco” that brings “structured, multi-agent workflows to data engineering projects” [3]. coco-skills ships curated agent skills for their Cortex Code CLI [4]. There is a managed MCP endpoint in the docs [5].

Every one of them operates on the warehouse: run the query, build the pipeline, ship the agent. Not one operates on the curriculum: which of the courses makes a person able to do that, in what order, at what cost, and how you would know they got there.

That gap is not an oversight, it's a category. And it's load-bearing, because the curriculum is where staffing decisions actually get made. So I built the missing layer — snowflake-os, inside my FDE-os toolkit — and gave myself one rule: it may only claim what it can show.

The mental model: a menu is not a map

A catalogue is a menu. It tells you what exists and what each item costs. That is genuinely useful and completely insufficient, because a menu can't tell you three things a map can:

• an ordering — what has to come before what;
• a destination — which subset gets you to this job;
• an arrival test — a way to tell whether you're actually there.

Menus can be claimed. Maps can be checked. That distinction is not new: in 1990 George Miller proposed a four-level framework for assessing clinical competence, rising from knows through knows how and shows how to does — his argument being that medical assessment leaned too hard on written knowledge tests and not enough on observed performance [6][7]. It is still the reference model in medical education thirty-six years later, which is a decent survival test for an idea.

A course completion sits near the bottom of that ladder. A badge — a third party attesting you did the graded lab — climbs a rung. Doing the job is the top rung, and no catalogue on earth can measure it. The trick isn't to pretend otherwise. It's to be explicit about which rung you're standing on.

The mechanism, in one sentence

Type every completion by who can verify it, then compute coverage only from the verifiable ones. A completion counts if it carries a badge_url, a completion_id, or a named verifier. A date, a screenshot, or "trust me" gets reported in full and scored zero.

The test that matters is what happens to a liar. I fed the gate a contract claiming all 58 courses, with no evidence attached to any of them. Coverage across all five core competencies: 0.0. Verdict: NO-GO. Exit code 2.

First principle: a system that cannot fail cannot certify. A learning plan that always says yes is a brochure with a progress bar.

What the catalogue actually contains

The ingest is deliberately boring: one request per second, read-only, against public pages, into a committed offline snapshot. Every record stores the URL it came from, the SHA-256 of the exact bytes parsed, and a timestamp — 94 records, all with provenance. The result:

And then the part I care about more.

Four things it refuses to know

A knowledge base that reports only what it found is a marketing asset. The interesting column is what it looked for and didn't get:

1. The role → course mapping does not exist. 0 of 12. All twelve role-journey pages describe the role in prose and link to the catalogue, not to individual courses. The single most useful edge in the entire graph — "if you want to be an ML engineer, take these" — is not published anywhere machine-readable. I supply it myself, and every mapping is tagged with its basis: title (the course title literally names the competency), objective (a published exam objective does), or curated (my judgment). Currently 22 of 71 mappings are curated, and a build gate fails if that ever exceeds 40%. It is very easy to "cover" a role by inventing mappings; the ceiling makes that a red build instead of a nice-looking number.
2. Specialty exam pricing is unknown. Every exam page repeats the same FAQ quoting $175 for Core and $375 for the Advanced series. Specialty prices appear nowhere. The parser therefore records not measured — it refuses to borrow the plausible number sitting two inches away on the same page. That refusal is a test in the suite.
3. One learning track publishes no sequence. Four of five list their courses in order. The SnowPro Core study track links the catalogue and the exam registration page, and nothing else.
4. Nobody's real competence. Effort hours are Snowflake's estimates, not measurements of your time. And "does the job" — Miller's top rung — is measured on an engagement, not in a catalogue. The tool says so in its own output.

Then it failed my work, three times

This is the part I'd have skipped if the tool were a demo instead of a gate.

1. The integrity gate caught my curation on its first run

I wrote ten checks over the knowledge base — dangling references, missing provenance, "measured" flags with no value behind them. The very first execution came back 8 of 9, failing competency-reachable: I had defined a Migration & modernisation competency that no role required. An orphan. I'd written the words "most engagements start as somebody else's legacy stack" in its own description and then wired it to nothing. I fixed the data, not the check. A validator that has never once fired on its author probably isn't checking anything.

2. The readiness gate was wrong, and it was wrong in the flattering direction's opposite

My first coverage formula scored a genuinely well-prepared candidate at 21%. The bug was in my model, not their preparation: Snowflake teaches the same material in multiple delivery modes — "Snowflake Platform Training" ships as both an on-demand and an instructor-led course — and I was counting both sides of every such pair as separate requirements. The denominator was inflated with duplicates. Introducing explicit alternatives groups (any one member satisfies the requirement, costed at the cheapest) moved the same candidate to 55%. Same evidence, same person; my arithmetic had been the problem. An assessment tool whose denominator is wrong is worse than no tool, because it produces a specific-looking number.

3. Eleven knowledge viewers in my own repo had been rendering blank

My new catalogue graph rendered as an empty white page — no console error, nothing. The cause: I'd HTML-escaped the JSON payload embedded in a <script type="application/json"> tag. A browser does not decode HTML entities inside a script element, so JSON.parse received literal " characters and threw during initialisation — which left the const in its temporal dead zone, so even probing the variable afterwards gave a confusing ReferenceError rather than an honest one.

The fix is one line. The finding was that the same line existed in knowledgefy.py, my repo's shared renderer — and had been silently blanking eleven already-shipped knowledge viewers. They had been broken for weeks.

So: I am the person who writes articles about verifying your claims, who built a gate that refuses to accept unverified evidence, and who had spent several weeks confidently linking colleagues to eleven beautifully rendered white rectangles. Each one 300KB. Each one containing the complete knowledge graph, correctly generated, perfectly escaped, and entirely invisible. The files passed every test I had, because every test I had asked "did it generate?" and none asked "can a human see it?"

I only found it because I opened the page in an actual browser instead of trusting that valid-looking HTML renders. The gate that caught it was a person looking at a screen — which is a humbling thing to write in an article about automated verification, and precisely why it's in here.

Anti-pattern: "the file generated without error, so the artifact works." Generation success is not rendering success, and a silent failure with no console output is the most expensive kind.

Patterns and anti-patterns

Patterns worth stealing:

• Key off stable signals, never CSS classes. The site is a Gatsby build whose class names are content hashes that rotate every deploy. The parser anchors only on URL path codes (OD-ESS-DWW) and visible label text ("Course Number"). Rename the styling, it still parses; rename a label, it reports not measured rather than guessing.
• Separate the networked step from the gate. One command touches the internet. Everything else reads a committed snapshot, so CI is offline, deterministic, and can't be broken by someone else's deploy.
• Point, don't fork. The six Snowflake-Labs repos are registered as pinned SHAs with an honest note on what each is bad at. Two are quarantined: coco-skills ships no licence at all, and subagent-cortex-code's is unidentifiable by GitHub. Absent a licence, default copyright applies — you may link and describe, not copy. That's enforced by a check, not by remembering.

Anti-patterns I walked into:

• Trusting a link's literal text. The curated track pages link legacy slugs like uni-essdww101 that 301-redirect to the canonical OD-ESS-DWW. Matching the literal slug produced an empty track sequence — not an error, just a confident, plausible, wrong answer. Silent wrongness beats loud wrongness only for the person shipping it.
• A percentage with an unexamined denominator (see: 21% vs 55%).
• Believing a generated file works because it generated.

What you can actually do with it

The whole thing is offline, standard-library-only, and gated by 43 tests. Three commands do the real work:

$ pathfinder.py plan --role forward-deployed-engineer --free-only
PLAN · role:forward-deployed-engineer
  constraints: free_only
  24 courses · 37.75h · 24 free / 0 paid

1. Platform foundations & architecture  [core]  6h
   · OD-LVLUP-101     0.5h  FREE  Level Up: First Concepts  [title]
   ...
   order: Snowflake's published track order
...
HONEST EDGES
  · 5/38 mappings are curated (13%) — snowflake-os judgment, not published by Snowflake
  · --free-only excluded 12 paid course(s): ILT-DE, ILT-GENAI, ILT-NAF, ...
  · a plan predicts effort; only readiness.py scores completion

(Two different curated shares appear in this article, so to be explicit about the denominators: 22 of 71 is every mapping in the whole knowledge base; 5 of 38 is only the mappings this particular free-only FDE plan touched. The build gate enforces the ceiling against the first number, the global one.)

Ordering is evidence-first: where Snowflake published a sequence (Badge 1 → Badge 6), that order wins. Where it published none, the fallback is easiest-and-shortest-first, and the plan says which rule it used per step. Then readiness.py issues GO or NO-GO on verified evidence, and a workflow composes them into a staffing verdict where a refusal always ships the shortest path to yes. A "no" that doesn't come with a route is just an opinion.

The part that generalises

Almost nothing here is about Snowflake. Every vendor with a certification program publishes a menu and calls it a path. Every organisation staffing against those certifications is doing Miller's bottom rung and reporting it as the top one. The reusable moves:

1. Type your evidence. Not "did they complete it" but "who could check that they did." The gap between those two questions is where every inflated skills matrix lives.
2. Publish your unmeasured column. The four things my tool refuses to know are more useful than the 58 it indexed, because they tell you exactly where your confidence should stop.
3. Cap your own judgment and let a build fail on it. Curation is unavoidable when upstream publishes nothing. Unlabelled curation is just opinion wearing a schema.

Snowflake wrote four honest words on a certification page: assumed but not tested. Most tooling built on top of credentials does the opposite — it tests nothing and assumes everything, then renders it as a percentage. The least you can do is be as candid as the exam you're measuring against.

References

1. Snowflake. SnowPro Advanced: MLOps Engineer — candidate profile. learn.snowflake.com/en/certifications/snowpro-advanced-mlopsengineer/ (accessed 2026-08-03). Quoted verbatim.
2. Snowflake-Labs. mcp — MCP Server for Snowflake. github.com/Snowflake-Labs/mcp (Apache-2.0; accessed 2026-08-03).
3. Snowflake-Labs. cocoplus. github.com/Snowflake-Labs/cocoplus (MIT; accessed 2026-08-03). Repository description quoted verbatim via the GitHub REST API.
4. Snowflake-Labs. coco-skills — Agent Skills for Cortex Code. github.com/Snowflake-Labs/coco-skills (no licence file as of 2026-08-03).
5. Snowflake. Snowflake-managed MCP server. docs.snowflake.com (accessed 2026-08-03). Cited, not independently tested — no Snowflake account was under test.
6. Miller, G. E. (1990). The assessment of clinical skills/competence/performance. Academic Medicine, 65(9 Suppl), S63–S67. doi:10.1097/00001888-199009000-00045. pubmed.ncbi.nlm.nih.gov/2400509/. Paywalled — described, not quoted.
7. Witheridge, A., Ferns, G., & Scott-Smith, W. (2019). Revisiting Miller's pyramid in medical education: the gap between traditional assessment and diagnostic reasoning. International Journal of Medical Education, 10, 191–192. pmc.ncbi.nlm.nih.gov/articles/PMC7246123/ (open access; used for the pyramid's four levels and its continued standing).
8. Snowflake. Course catalogue and Hands-On Essentials track. learn.snowflake.com/en/courses/, /pages/hands-on-essentials-track/ (snapshot taken 2026-08-03; all catalogue counts in this article derive from that snapshot).

Provenance note: every count in this article (58 courses, 430.25h, 68.25 free hours, 12 exams from 19 pages, 94 provenanced records, 22/71 curated mappings, 43 tests) is computed from the committed snapshot described in [8], not estimated. The two direct quotations, [1] and [3], were verified verbatim against their live sources on 2026-08-03.

More in the AI-Native series

• You Can't Self-Issue Reputation — claimed → observed → attested; why a self-made artifact stalls one rung short of trust. The direct ancestor of this article's evidence ladder.
• My AI Caught the Fake Citations — the same discipline pointed at research claims, including what it couldn't do.
• The Machine That Says No — why a gate that can't refuse isn't a gate.

All of it lives in the Writing section on the home page.

Part of the AI-Native series. The tool's own honest edges are printed in its output, not buried in a footnote. You own the Publish button.