Why This Article
After months of building LLM systems that call external tools—from simple calculators to complex API chains—I learned that most tutorials show you how to implement tool calling but skip the messy details of what actually works in practice.
This article shares the experiments I ran, the mistakes I made, and the patterns that survived production.
The Promise vs. Reality of Tool Calling
The Tutorial Version:
- Define your tools with JSON schemas
- Pass them to the LLM
- LLM magically picks the right tool
- Execute and return results
- Profit! 🎉
The Reality:
- LLMs hallucinate non-existent tools
- Parameters get extracted incorrectly
- Tools fail in unexpected ways
- Coordination overhead tanks performance
- Debugging is a nightmare
Let me walk you through what I learned fixing each of these.
Experiment 1: Which Tool Selection Strategy Actually Works?
I built three different approaches and tested them on 100 real user queries.
Approach 1: Let the LLM Choose (Prompt-Based)
My first naive attempt:
def select_tool_naive(query, available_tools):
tools_desc = "\n".join([f"- {t.name}: {t.description}" for t in available_tools])
prompt = f"""
User query: {query}
Available tools:
{tools_desc}
Which tool should I use? Respond with just the tool name.
"""
tool_name = llm.generate(prompt).strip()
return tool_name
What went wrong:
- 11% hallucination rate (invented tools like “super_calculator”)
- Slow (2.1s average)
- Sometimes returned explanations instead of tool names
- Inconsistent formatting
Approach 2: Semantic Similarity (Embeddings)
Next, I tried matching query embeddings to tool description embeddings:
def select_tool_semantic(query, tools):
query_embedding = embed_model.encode(query)
best_match = max(tools, key=lambda t:
cosine_similarity(query_embedding, t.embedding)
)
return best_match
Results:
- ⚡ Fast (0.3s)
- ❌ But only 76% accurate
- Failed on edge cases where semantic similarity misleading
Example failure:
- Query: “What’s 2+2?”
- Matched to:
web_search(because “what’s” seemed like a search query) - Should have been:
calculator
Approach 3: Native Function Calling (OpenAI/Claude)
Finally tried the built-in function calling APIs:
def select_tool_native(query, tools):
functions = [tool_to_openai_function(t) for t in tools]
response = openai.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": query}],
functions=functions,
function_call="auto"
)
if response.choices[0].message.function_call:
return response.choices[0].message.function_call.name
return None
Results:
- ✅ 94% accuracy
- ✅ 1.2s latency (reasonable)
- ✅ Structured parameter extraction included
- ✅ No hallucinations (tool name must match defined functions)
Lesson 1: Use native function calling. Don’t roll your own.
The providers spent millions optimizing this. Your custom prompt won’t beat it.
Experiment 2: Parameter Extraction is Sneakier Than You Think
Even with 94% tool selection accuracy, I still had a 9% failure rate overall. Why?
The Date Parsing Nightmare
# User asks: "What's the weather tomorrow?"
# LLM extracts: {"location": "unknown", "date": "tomorrow"}
# Weather API expects: {"location": "lat,lon", "date": "2025-03-21"}
What I tried:
Attempt 1: Detailed parameter descriptions
ToolParameter(
name="date",
type="string",
description="Date in ISO 8601 format (YYYY-MM-DD)"
)
Result: LLM still passed “tomorrow” 40% of the time
Attempt 2: Add examples
ToolParameter(
name="date",
description="Date in YYYY-MM-DD format. Examples: '2025-03-21', '2025-12-25'"
)
Result: Improved to 85% correct
Attempt 3: Validation + Retry
def execute_with_retry(tool, params, max_retries=2):
for attempt in range(max_retries):
try:
validated_params = tool.validate_params(params)
return tool.execute(**validated_params)
except ValidationError as e:
# Ask LLM to fix parameters
correction_prompt = f"""
Parameter validation failed: {e}
Original: {params}
Please provide corrected parameters.
"""
params = llm.generate(correction_prompt)
raise ParameterExtractionError()
Result: 91% success rate
Lesson 2: Validation + retry > perfect prompting
You can’t prompt your way out of all parameter errors. Build validation and recovery.
Experiment 3: Multi-Tool Chains Break in Surprising Ways
Simple example: “Find Apple’s stock price and calculate 15% of it”
Expected:
get_stock_price(symbol="AAPL")→ $178.50calculator(expression="178.50 * 0.15")→ $26.78
What Actually Happened (First Implementation)
15% of attempts: Tools called in wrong order
# LLM tried calculator FIRST with placeholder
calculator(expression="STOCK_PRICE * 0.15") # ❌ No value for STOCK_PRICE
8% of attempts: Infinite loops
# LLM called get_stock_price... then called it again... and again
get_stock_price("AAPL")
get_stock_price("AAPL") # Why?
get_stock_price("AAPL") # Seriously, why?
5% of attempts: Skipped intermediate step entirely
# User: "Find Apple stock and calculate 15%"
# LLM response: "15% of Apple's stock price is approximately $26"
# (No tools called! Just hallucinated the number)
The Fix: Explicit Chain-of-Thought
def execute_chain_with_cot(query, tools, max_steps=10):
conversation = [{"role": "user", "content": query}]
system_prompt = """
You must solve this step-by-step:
1. Identify what information you need
2. Use tools to gather that information
3. Use results from step 2 in subsequent calculations
After each tool call, explain what you learned and what to do next.
"""
for step in range(max_steps):
response = llm.chat(conversation, functions=tools)
if response.function_call:
# Execute tool
result = execute_tool(response.function_call)
# Add result to conversation
conversation.append({
"role": "function",
"name": response.function_call.name,
"content": str(result)
})
# Explicitly ask: what next?
conversation.append({
"role": "user",
"content": "Got it. What's your next step?"
})
else:
# LLM has final answer
return response.content
Result: 85% success rate on 2-tool chains (up from 70%)
Lesson 3: Make the LLM narrate its reasoning between tool calls
Forcing explicit “what I learned, what I’ll do next” drastically reduced errors.
Experiment 4: Error Recovery Is Harder Than I Thought
Tools fail. A lot.
- APIs go down (503 errors)
- Rate limits hit (429 errors)
- Timeouts
- Malformed responses
My Naive Error Handler
try:
result = tool.execute(**params)
except Exception as e:
return f"Error: {str(e)}"
Problem: LLM sees “Error: Service unavailable” and… does nothing useful.
Better Approach: Typed Errors with Recovery Strategies
class ToolExecutor:
def execute_with_recovery(self, tool, params):
try:
return tool.execute(**params)
except ValidationError as e:
# Parameter problem → retry with correction
return self.retry_with_param_correction(tool, params, e)
except RateLimitError as e:
# Hit rate limit → try fallback tool
fallback = self.get_fallback_tool(tool.name)
if fallback:
return fallback.execute(**params)
raise
except TimeoutError as e:
# Service slow → retry once, then fail gracefully
if not hasattr(self, '_retried'):
self._retried = True
time.sleep(2)
return self.execute_with_recovery(tool, params)
return {"error": "timeout", "suggestion": "try_again_later"}
Results:
- 78% of failures recovered automatically
- User experience: graceful degradation instead of cryptic errors
Lesson 4: Different errors need different recovery strategies
Don’t treat all exceptions the same. Validation errors → retry. Service errors → fallback.
What I Wish I’d Known Before Starting
1. Start Simple, Add Tools Gradually
I built 20 tools upfront. Bad idea.
- LLM confused between similar tools
- Hard to debug which tool caused failures
- Parameter validation nightmare
Better: Start with 3-5 core tools. Add more only when needed.
2. Tool Descriptions Matter More Than You Think
Bad description:
Tool(
name="calculator",
description="Does math"
)
Good description:
Tool(
name="calculator",
description="Evaluates mathematical expressions. Use for: arithmetic, percentages, unit conversion. Returns numeric result. Example: '2 + 2' → 4"
)
Specificity and examples reduced selection errors by 30%.
3. Log Everything
def execute_tool_with_logging(tool, params):
logger.info({
"tool": tool.name,
"params": params,
"timestamp": time.time()
})
result = tool.execute(**params)
logger.info({
"tool": tool.name,
"result": result,
"duration": time.time() - start
})
return result
When things break (they will), logs showing the exact sequence of tool calls save hours.
4. Cost Adds Up Fast
Function calling increases token usage:
- Tool descriptions in every prompt
- Multiple back-and-forth calls for chains
- Error recovery retries
On one project: 40% higher API costs than single-shot generation.
Mitigation:
- Cache tool results when possible
- Use cheaper models for tool selection
- Batch similar queries
Benchmarks: What You Can Expect
Based on my experiments across multiple projects:
| Metric | Realistic Expectation |
|---|---|
| Tool selection accuracy | 90-95% (with native function calling) |
| Parameter extraction | 85-92% (with validation + retry) |
| 2-tool chains success | 80-85% |
| 3+ tool chains success | 65-75% |
| Error recovery rate | 70-80% |
If you’re hitting these numbers, you’re doing well.
Where Tool Calling Shines (and Where It Doesn’t)
✅ Great For:
- Customer support: Looking up orders, checking status
- Data retrieval: Fetching real-time info (weather, stocks, etc.)
- Computation: Calculations, code execution
- Simple workflows: 2-3 tool chains
❌ Struggles With:
- Long chains: 5+ tools in sequence gets unreliable
- Complex logic: Nested conditionals, loops
- Stateful operations: Database transactions with rollbacks
- Real-time requirements: Each LLM call adds latency
Current Open Questions I’m Exploring
-
Can we teach LLMs to compose tools? Create a new
weather_and_outfittool fromweather+outfit_recommender -
How do we handle tool versioning? API changes break existing tool definitions
-
Can tools be discoverable? LLM explores available tools and learns which work best
-
What about tool security? Preventing malicious prompt injections that abuse tools
Resources That Helped Me
- OpenAI Function Calling Docs: Best practical guide
- LangChain Tool Framework: Good abstractions, some overhead
- AutoGPT source code: Real-world multi-tool chaining examples
Related Projects
This learning came from building:
- Neo4j GraphRAG - Tool calling for database queries
- B2B Workflow Automation - API integration tools
- Virtual Peer - Information retrieval tools
Have you built tool-calling systems? What worked (or didn’t) for you?
Reach out on LinkedIn or GitHub - I’d love to compare notes!