Jeongho Nam
#function-calling#qwen#seminar
TL;DR
- AutoBe — AI backend auto-generation agent
- Production-grade backend from natural language conversation
- 4 AST types + 4-tier compiler validation + self-healing loops
- Schema specs are the new prompts
- Typia — The infrastructure that turns 0% into 100%
- A single type automates schema, parser, validator, and feedback generator
- Lenient JSON parsing + schema-based type coercion + precise validation feedback
- Combined with AutoBe to complete harness engineering
- In Praise of Function Calling
- Types eliminate ambiguity; schemas constrain through absence
- Model-neutral, mechanically verifiable, deterministically convergent
- Applicable to all engineering domains with validators — semiconductors, chemical processes, control systems, etc.
- Qwen — Why small models are the best QA engineers
- Smaller models are better at exposing system vulnerabilities
- R&D cost reduction, vendor independence, open ecosystem virtuous cycle
- 6.75% is not failure — it’s the first input to the loop
- scores 6.75% on first-try tool calling
- AutoBe’s self-healing harness turns that into 100% compilation success
- If you can verify, you converge
6.75%.
That’s the first-try function calling success rate when is asked to generate API data types for a shopping mall backend. 93 out of 100 attempts produce invalid structured output.
This isn’t surprising. NESTFUL (EMNLP 2025) measured GPT-4o at 28% accuracy on nested tool call sequences. JSONSchemaBench (ICLR 2025) tested constrained decoding frameworks on 10,000 real-world schemas and found 3–41% coverage on the hardest ones. BoundaryML went further, arguing that structured outputs actively degrade model reasoning — that forcing JSON format makes the model dumber. The consensus is clear: function calling works for flat, simple schemas. For anything with recursive nesting or deep structural complexity, don’t bother.
But if you want to make AI output deterministic — parse it, validate it, and correct it in a loop until it converges — there is no alternative to structured output. Free-form text can’t be mechanically verified. Natural language can’t be compiled. Without structure, there’s no feedback loop, and without a feedback loop, there’s no guarantee. So we didn’t have the luxury of giving up on function calling. We had to make it work on the exact kind of complex, recursive schemas the industry had written off.
AutoBe is the result. It’s an open-source AI agent that takes a single natural language conversation and generates a complete backend — requirements analysis, database schema, API specification, E2E tests, and implementation code. Hook up that 6.75% model and what happens? Final compilation success rate: 99.8%+. All five Qwen models.
The answer wasn’t a better model or a smarter prompt. It was a harness — type schemas that constrain outputs, compilers that verify results, and structured feedback that pinpoints exactly where and why something went wrong so the LLM can correct itself. A deterministic loop wrapping a probabilistic model. The engineering outside the model, not inside, that made the difference.
This talk dissects that engineering.
Chapter 2 examines AutoBe’s architecture: a 5-phase pipeline running through 4 AST types and 4-tier compilers, with self-healing loops that systematically correct LLM mistakes.
Chapter 3 delves into Typia, the heart of that structure. The TypeScript compiler analyzes a single type from source code and generates schema, parser, validator, and feedback generator — all automatically. The concrete mechanism that flipped Qwen 3.5’s 0% to 100% lives here.
Chapter 4 steps back to ask a bigger question. Does this pattern work beyond backends? Semiconductors, chemical processes, architecture, control systems — anywhere deterministic validators exist in engineering.
And Chapter 5 answers why this story belongs at Qwen Meetup. Small models aren’t a weakness. They’re the harness system’s best QA engineers.
AutoBe is an open-source AI agent that generates production-grade backends from natural language. Developed by Wrtn Technologies .
“Build me a shopping mall backend with products, carts, orders, and payments.” From this single sentence, AutoBe generates:
Most AI coding agents tell the LLM “write this code” and save the returned text directly as source files. AutoBe is different.
AutoBe uses function calling. Instead of generating free-form text, the LLM fills in predefined structures — JSON Schema. It’s filling out a form, not writing on a blank page. Once the LLM fills the form, compilers validate and transform it into actual code. The LLM fills structures; compilers write code.
This approach applies across the entire 5-phase waterfall pipeline.
Each AST strictly limits what the LLM can generate — ’s field types allow only 7 options (), making physically impossible. Schema specs are the new prompts — unambiguous, model-independent, mechanically verifiable.
But the structures the LLM fills are far from simple. The that defines DTO types is a recursive union of 10 variants:
10 variants, infinitely recursive nesting. First-try success rate: 6.75%.
The testing phase raises complexity further — captures E2E test logic with 30+ recursive variants:
Programming-language complexity in a single function call.
When compilation fails, AutoBe doesn’t stop. It runs a self-healing loop:
Four compilers — Database, OpenAPI, Test, TypeScript — each validate at a different level and return structured diagnostics: exact location, target, and cause of every error. The Correct agent receives the original output + diagnostics and makes targeted fixes. Successful parts are preserved; only failures are corrected.
On top of this, Typia’s validation feedback (Chapter 3) adds precise correction at the function calling level. The combination of compiler-level and function calling-level validation is the driving force behind the 99.8%+ compilation rate.
AutoBe currently tests against five Qwen models. All achieve successful compilation.
From 397B to 35B. Same schema, same pipeline, same result.
Chapter 2 described what AutoBe builds — but not how it survives 6.75%. Schema generation, broken JSON recovery, type coercion, precise error feedback — every piece of infrastructure that makes function calling work on complex types despite the industry consensus that it can’t. Who handles all of it?
Typia . Making function calling reliable on recursive union types required going deeper than runtime libraries can reach. Runtime reflection can’t see TypeScript types — they’re erased at compilation. Zod-style schema builders choke on recursive unions. The only path was to operate at the compiler level itself — analyze types directly from source code and generate every piece of infrastructure from that single source of truth.
That’s what Typia is. A compiler library that directly leverages the TypeScript compiler’s type analyzer to automatically generate JSON Schema, validators, parsers, and feedback generators at compile time. Define one type, and the compiler handles the rest. It’s the result of choosing to solve the problem at the deepest layer available, because every shallower approach hit a wall.
Let’s examine in detail how it turns ’s 6.75% success rate and ’s 0% success rate into 100%.
Function calling requires JSON Schema to tell the LLM “give me data in this structure.” Normally, developers define types, separately write schemas, and keep the two synchronized forever.
Typia automates this process. Define a TypeScript type, and Typia automatically generates validation code and JSON Schema at compile time — not through runtime reflection, but by directly leveraging the TypeScript compiler’s type analyzer.
Let’s see the principle first. When you call , type information is analyzed at compile time and transformed into optimized validation code:
A single line — — transforms at compile time into optimized code containing UUID regex, email regex, integer checks, and range checks. It overcomes TypeScript’s limitation of erasing type information at runtime through a compiler plugin.
This principle applies directly to function calling. generates JSON Schema through the same type analysis:
JSDoc comments become fields. The LLM reads these descriptions to decide what values to generate. Type constraints become validation rules. becomes a ”> 18” rule, and becomes an email format check. A single type definition simultaneously generates LLM guidance and validation rules.
At the class level, can schematize an entire API:
The type is the schema. The constraints the LLM sees and the constraints the validator applies are always identical — because they come from the same source.
This is the key point. The schema generated by the Typia compiler from source code types powers every runtime function that follows. The schema that references when recovering broken JSON and coercing types, the schema that uses as the comparison target when diagnosing errors — they’re all the same schema, automatically generated from types at compile time. Because it’s compiler output, not manually written, types and schemas can never diverge.
The 10 variants of and 30+ variants of from Chapter 2. Why is the first-try success rate so low?
Recursive union types cause combinatorial explosion. 10 variants nested 3 levels deep create 1,000 possible paths. With 30 variants, that’s 27,000. The probability of the LLM choosing the correct path in one try is structurally low.
Moreover, subtle errors are frequent in union types:
These errors are “structurally correct but semantically wrong,” making it difficult to provide accurate feedback with simple JSON Schema validation.
6.75% is the natural result of this structural complexity. The issue isn’t the first try — it’s what happens after failure.
LLMs are language models, not JSON generators. They wrap output in Markdown code blocks, prepend chatter like “I’d be happy to help!”, leave brackets unclosed, forget to quote keys, and write instead of . The Qwen 3.5 series goes further: on every (union type) field, it 100% consistently double-stringifies the value. Not occasionally — every union field, every attempt, without exception.
rejects all of this. Here’s a real example from production — all seven problems in a single response:
One call to fixes all seven problems:
The last one is the killer. The Qwen 3.5 series double-stringifies every field, 100% of the time — 0% success rate on union types without this. It’s not Qwen-only either; Claude does the same on . eliminates all of them. Zero model changes, zero prompt tuning.
Even after parsing and coercion, values themselves can be wrong. Negative prices, strings that aren’t emails, decimals where integers should be.
Typia’s detects schema violations and tells you exactly where and why something is wrong:
“The price inside product inside order should be ≥ 0, but you gave -100.”
renders these errors as inline comments on top of the LLM’s original JSON:
should be a string but got a number. should be ≥ 0. should be a positive integer. is not a valid email. should be a boolean. 5 errors, each with exact path and expected type.
The LLM sees exactly where it went wrong on its own JSON. Instead of rewriting everything, it only needs to fix the 5 marked fields. Precise, structured, immediately actionable feedback.
Combining everything into a single loop:
recovers broken JSON and performs initial type coercion. catches schema violations. renders errors in a format the LLM can read. The LLM self-corrects and retries.
This is the complete loop that turns 6.75% into 100%.
- Only Typia integrates parse, coerce, and validate by compiler skills.
- Only Typia handles union types correctly.
Typia (function calling level):
AutoBe (system level):
The type is the schema, the validator, and the prompt. The harness is everything around it.
“Structured outputs create false confidence.” The criticism is accurate — when you use structured output without a harness. Every failure the industry observed is what happens when you treat function calling as a feature to toggle on, rather than as infrastructure to build around.
Natural language evolved to be ambiguous. Metaphor, nuance, politeness, humor — all operate on top of ambiguity. “Just make it pretty” works between humans.
Programming languages were designed to eliminate ambiguity. “Just make it pretty” doesn’t compile.
When people communicate in natural language, misunderstandings arise. When they communicate through types, there are none.
Expressing constraints through prompts:
“The age field should be a positive integer greater than 18. Don’t use string types for number fields. All required fields must be present…”
Is “greater than 18” >18 or ≥18? You can’t know whether the LLM followed this rule without manually inspecting the output. As schemas grow, these rules multiply endlessly.
Expressing constraints through types:
is >18. It’s an integer. It’s required. No ambiguity, mechanically verifiable.
In domains requiring precision, type constraints provide certainty that natural language instructions cannot.
If you’ve built a prompt-based AI agent, you’ve written prohibition rules:
“Don’t think of a pink elephant.” The first thing that comes to mind is a pink elephant. When you tell an LLM “don’t do X,” X gets placed at the center of attention. To avoid a forbidden pattern, the model must first recall that pattern, which paradoxically increases its generation probability. This is the essence of token prediction.
Even knowing this, you can’t avoid prohibition rules in prompts. “Don’t do X” is the only way natural language can express constraints.
With schemas, this problem disappears.
No need to say “don’t use the type” — if doesn’t exist in the schema, the LLM physically cannot generate it. No need to say “don’t create utility functions” — if there’s no slot for utility functions, that’s the end of it. When field types are limited to — 7 choices — there’s no path for the LLM to write .
Not prohibition, but absence. Prompts prohibit what you don’t want. Schemas allow only what you do want.
This is function calling’s deepest advantage: instead of fighting the model’s tendencies, it makes unwanted outputs structurally impossible.
Prompt engineering is inherently model-dependent. A prompt optimized for GPT behaves differently on Claude, and differently again on Qwen. Rewriting prompts with each new model is routine.
Function calling-based approaches are model-neutral. JSON Schema means the same thing regardless of which model reads it. The validation feedback loop absorbs performance differences between models. Strong models converge in 1–2 attempts, weaker models take 3–4, but both reach 100%.
AutoBe running Qwen, GLM, DeepSeek, and OpenAI models with the same schema, the same pipeline and achieving 100% compilation across all of them is proof of this neutrality. No model-specific prompt tuning was ever performed.
This changes the nature of model selection. From “Can this model do this task?” — a capability question — to “Which model is most cost-effective?” — a cost optimization problem: .
This isn’t theoretical. Every major vendor has demonstrated prompt fragility across model versions:
OpenAI: GPT-4 → GPT-4o caused widespread prompt regressions — same prompts suddenly produced different outputs. GPT-4 → GPT-5 required prompt rewrites at such scale that OpenAI had to ship a Prompt Optimizer tool . And GPT-4o is being retired on 2026.03.31 — every application using it must migrate.
Anthropic: Claude 3.x → 4.x introduced breaking changes every major version — prefill removed, tool versions changed, response style shifted.
Every vendor, every version: prompts must be rewritten. Model-specific tricks accumulate as vendor lock-in and technical debt.
Type schemas don’t break across versions. JSON Schema is an industry standard — zero rewrite required.
A single thread runs through everything.
Function calling’s fundamental advantage is that it brings LLM output into the domain of software engineering.
Free-form text output makes correctness an AI problem. Parsing is fuzzy. Validation is fuzzy. Correction is fuzzy.
Structured output makes correctness an engineering problem:
The model is still probabilistic. It still makes mistakes. But because the structure wrapping the model is deterministic, the process converges to 100%.
Type schema + deterministic validator + structured feedback = harness
Prompt engineering tries to make the probabilistic part reliable. Function calling makes the deterministic part perfect. In domains requiring precision, the latter wins: 6.75% → 100%.
This pattern applies to every domain where output is mechanically verifiable — not just software.
Run the cheapest validator first, fix errors, move to the next tier. Every domain here shares the same structure as AutoBe: recursive union types, hierarchical decomposition, deterministic validators refined over decades.
Note: These domain examples were AI-recommended. I’m a developer, not a domain expert — please treat the specifics as reference material.
Not a coincidence — hierarchical decomposition is how engineers manage complexity, and it always produces recursive union types. The same structure as AutoBe’s and .
This doesn’t work everywhere. Creative writing, emotional intelligence, strategic decisions — there’s no validator for “a good novel.” Without a validator, there’s no feedback loop. This is a solution for domains where accuracy is non-negotiable and mechanically verifiable.
AutoBe’s entire pipeline is function calling. The only criterion is how accurately a model fills complex JSON Schemas. At the small/medium scale, Qwen was the only open-weight model that could handle this complexity — even MoE models with 3B active parameters process schemas containing 10+ recursive union variants.
For customers, model cost is a non-issue — even the most expensive model is cheaper than hiring a developer. For us developing AutoBe, it’s different. Thousands of generate-compile-feedback cycles per iteration. Commercial models at this scale would be financial ruin. Local Qwen models made the journey from 6.75% to 100% possible.
Large models “correctly guess” ambiguous parts of schemas and pass through — our mistakes stay hidden. Small models expose everything:
The 10% success rate was the most valuable result. Every failure pointed to a system vulnerability, and each fix strengthened the pipeline for all models. Large models make mistakes less frequently, not never. In production, “rarely” means outage.
When even a 3B-active model can’t break your system, no model will.
We started at 6.75%. The industry said complex function calling doesn’t work, and our results agreed.
But there was no alternative — deterministic AI output requires structured output — so we built the harness, one failure mode at a time. Lenient parsing because JSON broke. Type coercion because types were wrong. Validation feedback because values were wrong. Compiler pipelines because the system needed consistency.
AutoBe achieved 99.8%+ compilation across all five Qwen models. Not through better prompts, but through the accumulated engineering of every way things went wrong.
Three things: type schemas that constrain outputs, compilers that verify results, and structured feedback that corrects errors. These three form a deterministic loop wrapping probabilistic models.
This pattern is not limited to code generation. The same structure can be built in every engineering domain where deterministic validators exist — semiconductors, chemical processes, control systems.
Communicate through types and there are no misunderstandings. Constrain through schemas and there are no pink elephants. With a deterministic loop, even 6.75% becomes 100%.
6.75% is not a failure — it’s the first input to the loop. If you can verify, you converge.
About AutoBe: AutoBe is an open-source AI agent developed by Wrtn Technologies . It generates production-grade backend applications from natural language.
About Typia: Typia is a compiler library that automatically generates runtime validators, JSON Schema, and function calling schemas from TypeScript types.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。