Why This Article

After months of building LLM systems that call external tools—from simple calculators to complex API chains—I learned that most tutorials show you how to implement tool calling but skip the messy details of what actually works in practice.

This article shares the experiments I ran, the mistakes I made, and the patterns that survived production.

The Promise vs. Reality of Tool Calling

The Tutorial Version:

  1. Define your tools with JSON schemas
  2. Pass them to the LLM
  3. LLM magically picks the right tool
  4. Execute and return results
  5. Profit! 🎉

The Reality:

  • LLMs hallucinate non-existent tools
  • Parameters get extracted incorrectly
  • Tools fail in unexpected ways
  • Coordination overhead tanks performance
  • Debugging is a nightmare

Let me walk you through what I learned fixing each of these.

Experiment 1: Which Tool Selection Strategy Actually Works?

I built three different approaches and tested them on 100 real user queries.

Approach 1: Let the LLM Choose (Prompt-Based)

My first naive attempt:

def select_tool_naive(query, available_tools):
    tools_desc = "\n".join([f"- {t.name}: {t.description}" for t in available_tools])

    prompt = f"""
    User query: {query}

    Available tools:
    {tools_desc}

    Which tool should I use? Respond with just the tool name.
    """

    tool_name = llm.generate(prompt).strip()
    return tool_name

What went wrong:

  • 11% hallucination rate (invented tools like “super_calculator”)
  • Slow (2.1s average)
  • Sometimes returned explanations instead of tool names
  • Inconsistent formatting

Approach 2: Semantic Similarity (Embeddings)

Next, I tried matching query embeddings to tool description embeddings:

def select_tool_semantic(query, tools):
    query_embedding = embed_model.encode(query)

    best_match = max(tools, key=lambda t:
        cosine_similarity(query_embedding, t.embedding)
    )

    return best_match

Results:

  • ⚡ Fast (0.3s)
  • ❌ But only 76% accurate
  • Failed on edge cases where semantic similarity misleading

Example failure:

  • Query: “What’s 2+2?”
  • Matched to: web_search (because “what’s” seemed like a search query)
  • Should have been: calculator

Approach 3: Native Function Calling (OpenAI/Claude)

Finally tried the built-in function calling APIs:

def select_tool_native(query, tools):
    functions = [tool_to_openai_function(t) for t in tools]

    response = openai.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": query}],
        functions=functions,
        function_call="auto"
    )

    if response.choices[0].message.function_call:
        return response.choices[0].message.function_call.name
    return None

Results:

  • ✅ 94% accuracy
  • ✅ 1.2s latency (reasonable)
  • ✅ Structured parameter extraction included
  • ✅ No hallucinations (tool name must match defined functions)

Lesson 1: Use native function calling. Don’t roll your own.

The providers spent millions optimizing this. Your custom prompt won’t beat it.

Experiment 2: Parameter Extraction is Sneakier Than You Think

Even with 94% tool selection accuracy, I still had a 9% failure rate overall. Why?

The Date Parsing Nightmare

# User asks: "What's the weather tomorrow?"
# LLM extracts: {"location": "unknown", "date": "tomorrow"}
# Weather API expects: {"location": "lat,lon", "date": "2025-03-21"}

What I tried:

Attempt 1: Detailed parameter descriptions

ToolParameter(
    name="date",
    type="string",
    description="Date in ISO 8601 format (YYYY-MM-DD)"
)

Result: LLM still passed “tomorrow” 40% of the time

Attempt 2: Add examples

ToolParameter(
    name="date",
    description="Date in YYYY-MM-DD format. Examples: '2025-03-21', '2025-12-25'"
)

Result: Improved to 85% correct

Attempt 3: Validation + Retry

def execute_with_retry(tool, params, max_retries=2):
    for attempt in range(max_retries):
        try:
            validated_params = tool.validate_params(params)
            return tool.execute(**validated_params)
        except ValidationError as e:
            # Ask LLM to fix parameters
            correction_prompt = f"""
            Parameter validation failed: {e}
            Original: {params}
            Please provide corrected parameters.
            """
            params = llm.generate(correction_prompt)

    raise ParameterExtractionError()

Result: 91% success rate

Lesson 2: Validation + retry > perfect prompting

You can’t prompt your way out of all parameter errors. Build validation and recovery.

Experiment 3: Multi-Tool Chains Break in Surprising Ways

Simple example: “Find Apple’s stock price and calculate 15% of it”

Expected:

  1. get_stock_price(symbol="AAPL") → $178.50
  2. calculator(expression="178.50 * 0.15") → $26.78

What Actually Happened (First Implementation)

15% of attempts: Tools called in wrong order

# LLM tried calculator FIRST with placeholder
calculator(expression="STOCK_PRICE * 0.15")  # ❌ No value for STOCK_PRICE

8% of attempts: Infinite loops

# LLM called get_stock_price... then called it again... and again
get_stock_price("AAPL")
get_stock_price("AAPL")  # Why?
get_stock_price("AAPL")  # Seriously, why?

5% of attempts: Skipped intermediate step entirely

# User: "Find Apple stock and calculate 15%"
# LLM response: "15% of Apple's stock price is approximately $26"
# (No tools called! Just hallucinated the number)

The Fix: Explicit Chain-of-Thought

def execute_chain_with_cot(query, tools, max_steps=10):
    conversation = [{"role": "user", "content": query}]

    system_prompt = """
    You must solve this step-by-step:
    1. Identify what information you need
    2. Use tools to gather that information
    3. Use results from step 2 in subsequent calculations

    After each tool call, explain what you learned and what to do next.
    """

    for step in range(max_steps):
        response = llm.chat(conversation, functions=tools)

        if response.function_call:
            # Execute tool
            result = execute_tool(response.function_call)

            # Add result to conversation
            conversation.append({
                "role": "function",
                "name": response.function_call.name,
                "content": str(result)
            })

            # Explicitly ask: what next?
            conversation.append({
                "role": "user",
                "content": "Got it. What's your next step?"
            })
        else:
            # LLM has final answer
            return response.content

Result: 85% success rate on 2-tool chains (up from 70%)

Lesson 3: Make the LLM narrate its reasoning between tool calls

Forcing explicit “what I learned, what I’ll do next” drastically reduced errors.

Experiment 4: Error Recovery Is Harder Than I Thought

Tools fail. A lot.

  • APIs go down (503 errors)
  • Rate limits hit (429 errors)
  • Timeouts
  • Malformed responses

My Naive Error Handler

try:
    result = tool.execute(**params)
except Exception as e:
    return f"Error: {str(e)}"

Problem: LLM sees “Error: Service unavailable” and… does nothing useful.

Better Approach: Typed Errors with Recovery Strategies

class ToolExecutor:
    def execute_with_recovery(self, tool, params):
        try:
            return tool.execute(**params)

        except ValidationError as e:
            # Parameter problem → retry with correction
            return self.retry_with_param_correction(tool, params, e)

        except RateLimitError as e:
            # Hit rate limit → try fallback tool
            fallback = self.get_fallback_tool(tool.name)
            if fallback:
                return fallback.execute(**params)
            raise

        except TimeoutError as e:
            # Service slow → retry once, then fail gracefully
            if not hasattr(self, '_retried'):
                self._retried = True
                time.sleep(2)
                return self.execute_with_recovery(tool, params)
            return {"error": "timeout", "suggestion": "try_again_later"}

Results:

  • 78% of failures recovered automatically
  • User experience: graceful degradation instead of cryptic errors

Lesson 4: Different errors need different recovery strategies

Don’t treat all exceptions the same. Validation errors → retry. Service errors → fallback.

What I Wish I’d Known Before Starting

1. Start Simple, Add Tools Gradually

I built 20 tools upfront. Bad idea.

  • LLM confused between similar tools
  • Hard to debug which tool caused failures
  • Parameter validation nightmare

Better: Start with 3-5 core tools. Add more only when needed.

2. Tool Descriptions Matter More Than You Think

Bad description:

Tool(
    name="calculator",
    description="Does math"
)

Good description:

Tool(
    name="calculator",
    description="Evaluates mathematical expressions. Use for: arithmetic, percentages, unit conversion. Returns numeric result. Example: '2 + 2' → 4"
)

Specificity and examples reduced selection errors by 30%.

3. Log Everything

def execute_tool_with_logging(tool, params):
    logger.info({
        "tool": tool.name,
        "params": params,
        "timestamp": time.time()
    })

    result = tool.execute(**params)

    logger.info({
        "tool": tool.name,
        "result": result,
        "duration": time.time() - start
    })

    return result

When things break (they will), logs showing the exact sequence of tool calls save hours.

4. Cost Adds Up Fast

Function calling increases token usage:

  • Tool descriptions in every prompt
  • Multiple back-and-forth calls for chains
  • Error recovery retries

On one project: 40% higher API costs than single-shot generation.

Mitigation:

  • Cache tool results when possible
  • Use cheaper models for tool selection
  • Batch similar queries

Benchmarks: What You Can Expect

Based on my experiments across multiple projects:

Metric Realistic Expectation
Tool selection accuracy 90-95% (with native function calling)
Parameter extraction 85-92% (with validation + retry)
2-tool chains success 80-85%
3+ tool chains success 65-75%
Error recovery rate 70-80%

If you’re hitting these numbers, you’re doing well.

Where Tool Calling Shines (and Where It Doesn’t)

✅ Great For:

  • Customer support: Looking up orders, checking status
  • Data retrieval: Fetching real-time info (weather, stocks, etc.)
  • Computation: Calculations, code execution
  • Simple workflows: 2-3 tool chains

❌ Struggles With:

  • Long chains: 5+ tools in sequence gets unreliable
  • Complex logic: Nested conditionals, loops
  • Stateful operations: Database transactions with rollbacks
  • Real-time requirements: Each LLM call adds latency

Current Open Questions I’m Exploring

  1. Can we teach LLMs to compose tools? Create a new weather_and_outfit tool from weather + outfit_recommender

  2. How do we handle tool versioning? API changes break existing tool definitions

  3. Can tools be discoverable? LLM explores available tools and learns which work best

  4. What about tool security? Preventing malicious prompt injections that abuse tools

Resources That Helped Me

  • OpenAI Function Calling Docs: Best practical guide
  • LangChain Tool Framework: Good abstractions, some overhead
  • AutoGPT source code: Real-world multi-tool chaining examples

This learning came from building:


Have you built tool-calling systems? What worked (or didn’t) for you?

Reach out on LinkedIn or GitHub - I’d love to compare notes!