Here is the Medium-style article for Part 7 of your series, focusing on Just-In-Time (JIT) Tool Loading. This is a more advanced architectural pattern that is essential for larger production systems.

In our last article, we discussed how to filter tools for security. But what if you have so many tools that you can’t even list them all in the first place?

In a large enterprise, you might have thousands of automation scripts, database queries, and API integrations. If you try to give an AI agent 5,000 tools at once, two things will happen:

1. Context Overflow: You will hit the token limit before the user even finishes their sentence.
2. Confusion: The model will get “lost” in the options, leading to higher error rates.

Today, we are solving this with Just-In-Time (JIT) Tool Loading. Instead of giving the agent everything upfront, we keep it lightweight and “mount” tools onto the agent only when they are needed.

1. The “Ghost Tool” Concept

A Ghost Tool is a function that is defined in your code but is not registered when you create your agent.

# This tool is a "Ghost"—the Agent doesn't know it exists... yet.
@tool
def emergency_restart(target: str, confirmation_code: str):
    """Restart a critical target. Only available via JIT Middleware."""
    return f"⚠️ EMERGENCY RESTART triggered for {target}."

By keeping high-risk or rare tools “off the menu,” we save valuable context window space and keep the AI focused.

2. The JIT Injector: Intent-Based Mounting

The magic happens in the Middleware. We use @wrap_model_call to intercept the user's message. If the message contains keywords like "restart," the middleware dynamically injects the

emergency_restart tool into the request.

@wrap_model_call
def jit_injector(request: ModelRequest, handler):
    last_msg = request.state["messages"][-1].content.lower()
    if "restart" in last_msg:
        print("⚡ JIT INJECTOR: Mounting 'emergency_restart' for this turn.")
                # Add the dynamic tool to the model's "menu" dynamically
        updated_tools = [*request.tools, emergency_restart]
        return handler(request.override(tools=updated_tools))
    return handler(request)

Think of this like a “Just-In-Time” manufacturing line. The tool only appears on the conveyor belt when the order calls for it.

3. The JIT Executor: Wiring the Execution

Since the agent didn’t have this tool when it was born, we need to tell it how to run it when the AI decides to call it. We use @wrap_tool_call for this.

@wrap_tool_call
def jit_executor(request: ToolCallRequest, handler):
    tool_name = request.tool_call["name"]
        if tool_name == "emergency_restart":
        print(f"⚡ JIT EXECUTOR: Running dynamic code for {tool_name}...")
        # Explicitly tell LangChain which function to execute
        return handler(request.override(tool=emergency_restart))
            return handler(request)

4. The Benefits: Pure Efficiency

When we run this “Ghost Agent,” it behaves like a lightweight specialist most of the time.

• Scenario 1 (Status Check): The user asks “Check status.” The agent only sees the
• get_status tool. The prompt is short, fast, and cheap.
• Scenario 2 (Emergency): The user asks “Perform an emergency restart.” The JIT Injector kicks in. Suddenly, the
• emergency_restart tool is available. The AI calls it, the JIT Executor runs it, and the task is complete.

As soon as the turn is over, the tool is “unmounted.” The agent returns to its lightweight state.

Conclusion: Engineering for Scale

Scale isn’t about giving an AI more data; it’s about giving it the right data at the right time.

JIT Tool Loading is the secret to building agents that can scale to thousands of capabilities without becoming slow or hallucination-prone. By separating the discovery of tools from the definition of tools, you can build a massive library of AI-powered operations that stay “in the shadows” until the moment they are needed.

💬 What do you think?
Drop your thoughts, questions, or suggestions in the comments below!

Check out my YouTube channel for more exciting content! [YouTube Channel Link — Harsha Selvi]

Disclaimer: This text has been rephrased using AI tools, and some parts are derived from various sources to provide a comprehensive overview.

#AI #LangChain #Architecture #AIEngineering #Python #CloudComputing #Scalability #ModernSoftwareEngineering