INTRODUCTION TO THE PROJECT
This tutorial guides you through creating a sophisticated code assistant that runs entirely on your local machine using a Large Language Model. The assistant will understand your coding requests, generate solutions, create files, execute code, and even debug errors automatically. We will build this system step by step, ensuring you understand each component deeply.
The code assistant we are building has several remarkable capabilities. When you ask it to implement an algorithm or design pattern, it generates the complete Python code, creates a new directory for the project, saves the code to a file, and executes it. If errors occur during execution, the assistant analyzes them, generates corrected code, overwrites the file, and tries again. This cycle continues until the code runs successfully or you interrupt it.
Our implementation uses the LangChain framework, which provides excellent abstractions for working with LLMs and tool calling. For the local LLM, we will use Ollama, which supports both CUDA for NVIDIA GPUs and MPS for Apple Silicon. The assistant can also search the internet when needed, making it more capable than relying solely on training data.
UNDERSTANDING THE ARCHITECTURE
Before diving into code, let us understand the system architecture. The assistant consists of several interconnected components working together harmoniously.
The core component is the LLM inference engine, which processes your requests and generates responses. This engine runs locally using Ollama, ensuring your code and conversations remain private. The inference engine supports hardware acceleration through CUDA on NVIDIA GPUs and MPS on Apple Silicon, making it fast and responsive.
The tool calling system allows the LLM to perform actions beyond generating text. Our assistant has tools for creating directories, writing files, executing Python code, and searching the internet. When the LLM decides it needs to use a tool, it generates a structured request, and our framework executes the corresponding function.
The conversation management system maintains chat history for each session. This allows the assistant to remember context from earlier in the conversation, making interactions more natural and coherent. Each session has its own isolated chat window and history.
The error handling and retry mechanism monitors code execution. When errors occur, the system extracts error messages, feeds them back to the LLM, and requests corrected code. This creates a self-healing loop that continues until success or manual interruption.
SETTING UP THE DEVELOPMENT ENVIRONMENT
Let us begin by setting up our development environment with all necessary dependencies. We need Python 3.9 or higher, and several libraries that provide the foundation for our assistant.
First, create a new directory for your project and set up a virtual environment. This isolates our project dependencies from your system Python installation.
mkdir code_assistant
cd code_assistant
python -m venv venv
source venv/bin/activate
On Windows, the activation command differs slightly. Use the following instead of the source command above.
venv\Scripts\activate
Now install the required packages. We need LangChain for LLM orchestration, Ollama for running local models, DuckDuckGo for internet search, and a few supporting libraries.
pip install langchain langchain-community langchain-ollama duckduckgo-search rich
The rich library provides beautiful terminal formatting, making our chat interface more pleasant to use. LangChain-community contains additional tools and integrations, while langchain-ollama provides the Ollama integration.
Next, install Ollama itself. Visit the Ollama website and download the installer for your operating system. On macOS, you can use Homebrew.
brew install ollama
On Linux, use the installation script provided by Ollama.
curl -fsSL https://ollama.com/install.sh | sh
After installing Ollama, download a capable model. We will use CodeLlama, which is specifically trained for coding tasks and works excellently as a code assistant.
ollama pull codellama:13b-instruct
The 13b-instruct variant provides a good balance between capability and resource usage. If you have more powerful hardware, consider the 34b variant for even better results.
CREATING THE PROJECT STRUCTURE
Let us create a well-organized project structure that follows clean architecture principles. Our project will have several modules, each with a clear responsibility.
Create the following directory structure in your project folder.
code_assistant/
main.py
assistant/
__init__.py
llm_engine.py
tools.py
conversation.py
session_manager.py
utils/
__init__.py
console.py
file_operations.py
The main.py file serves as the entry point for our application. The assistant package contains the core logic, with separate modules for the LLM engine, tools, conversation management, and session handling. The utils package provides helper functions for console output and file operations.
IMPLEMENTING THE LLM ENGINE
The LLM engine is the heart of our assistant. It initializes the local model, manages inference, and coordinates tool calling. Let us implement this component first.
Create the file assistant/llm_engine.py with the following content. This module encapsulates all LLM-related functionality.
from langchain_ollama import ChatOllama
from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder
from typing import List, Any
import torch
class LLMEngine:
"""
The LLM Engine manages the local language model and coordinates
inference with tool calling capabilities. It automatically detects
available hardware acceleration (CUDA or MPS) and configures the
model accordingly.
"""
def __init__(self, model_name: str = "codellama:13b-instruct"):
"""
Initialize the LLM engine with the specified model.
Args:
model_name: The Ollama model to use for inference
"""
self.model_name = model_name
self.device = self._detect_device()
self.llm = self._initialize_llm()
self.agent_executor = None
def _detect_device(self) -> str:
"""
Detect the best available device for inference.
Checks for CUDA (NVIDIA), MPS (Apple Silicon), or falls back to CPU.
Returns:
String indicating the device type
"""
if torch.cuda.is_available():
return "cuda"
elif torch.backends.mps.is_available():
return "mps"
else:
return "cpu"
def _initialize_llm(self) -> ChatOllama:
"""
Initialize the ChatOllama instance with appropriate settings.
Returns:
Configured ChatOllama instance
"""
# Ollama handles device selection internally based on availability
# We configure temperature and other parameters for code generation
llm = ChatOllama(
model=self.model_name,
temperature=0.1, # Low temperature for more deterministic code
num_ctx=8192, # Context window size
num_gpu=1 if self.device != "cpu" else 0,
verbose=False
)
return llm
def create_agent(self, tools: List[Any]) -> None:
"""
Create an agent executor with the provided tools.
The agent can decide when and how to use tools based on user requests.
Args:
tools: List of LangChain tools available to the agent
"""
# Define the system prompt that guides the assistant's behavior
system_prompt = """You are an expert code assistant that helps developers
implement algorithms, design patterns, and solve coding challenges.
When asked to implement code:
1. Generate complete, working Python code
2. Create a new directory for the project if needed
3. Save the code to a .py file
4. Execute the code to verify it works
5. If errors occur, analyze them and generate corrected code
6. Repeat until the code runs successfully
Always provide clear explanations of your implementation choices.
Use the tools available to you to create files and execute code.
When searching the internet, look for recent best practices and examples.
Be thorough, write clean code with proper documentation, and follow
Python best practices including PEP 8 style guidelines."""
# Create the prompt template with placeholders for messages and tools
prompt = ChatPromptTemplate.from_messages([
("system", system_prompt),
MessagesPlaceholder(variable_name="chat_history", optional=True),
("human", "{input}"),
MessagesPlaceholder(variable_name="agent_scratchpad")
])
# Create the tool-calling agent
agent = create_tool_calling_agent(
llm=self.llm,
tools=tools,
prompt=prompt
)
# Create the agent executor that runs the agent with tools
self.agent_executor = AgentExecutor(
agent=agent,
tools=tools,
verbose=True,
handle_parsing_errors=True,
max_iterations=10
)
def invoke(self, user_input: str, chat_history: List[Any]) -> str:
"""
Process user input and generate a response using the agent.
Args:
user_input: The user's message or request
chat_history: Previous messages in the conversation
Returns:
The assistant's response
"""
if self.agent_executor is None:
raise RuntimeError("Agent not initialized. Call create_agent first.")
result = self.agent_executor.invoke({
"input": user_input,
"chat_history": chat_history
})
return result["output"]
This LLM engine implementation provides several important features. The device detection automatically identifies whether CUDA or MPS acceleration is available, ensuring optimal performance on different hardware. The temperature setting of 0.1 makes code generation more deterministic and reliable, which is crucial for a code assistant.
The system prompt carefully guides the assistant's behavior, instructing it to follow a specific workflow when implementing code. This prompt is critical because it shapes how the LLM interprets requests and decides when to use tools.
IMPLEMENTING THE TOOL SYSTEM
Tools give our assistant the ability to interact with the file system and execute code. Let us implement a comprehensive set of tools that enable the complete workflow.
Create the file assistant/tools.py with the following implementation.
from langchain.tools import tool
from typing import Optional
import os
import subprocess
import sys
@tool
def create_directory(path: str) -> str:
"""
Create a new directory at the specified path.
Creates parent directories if they don't exist.
Args:
path: The directory path to create
Returns:
Success or error message
"""
try:
os.makedirs(path, exist_ok=True)
return f"Successfully created directory: {path}"
except Exception as e:
return f"Error creating directory: {str(e)}"
@tool
def write_file(filepath: str, content: str) -> str:
"""
Write content to a file, creating it if it doesn't exist.
Overwrites existing files with the same name.
Args:
filepath: Path to the file to write
content: Content to write to the file
Returns:
Success or error message
"""
try:
# Ensure the directory exists
directory = os.path.dirname(filepath)
if directory and not os.path.exists(directory):
os.makedirs(directory, exist_ok=True)
# Write the content to the file
with open(filepath, 'w', encoding='utf-8') as f:
f.write(content)
return f"Successfully wrote {len(content)} characters to {filepath}"
except Exception as e:
return f"Error writing file: {str(e)}"
@tool
def read_file(filepath: str) -> str:
"""
Read and return the contents of a file.
Args:
filepath: Path to the file to read
Returns:
File contents or error message
"""
try:
with open(filepath, 'r', encoding='utf-8') as f:
content = f.read()
return content
except Exception as e:
return f"Error reading file: {str(e)}"
@tool
def execute_python(filepath: str) -> str:
"""
Execute a Python file and return the output or error messages.
Captures both stdout and stderr.
Args:
filepath: Path to the Python file to execute
Returns:
Execution output or error messages
"""
try:
# Execute the Python file in a subprocess
result = subprocess.run(
[sys.executable, filepath],
capture_output=True,
text=True,
timeout=30 # Prevent infinite loops
)
# Combine stdout and stderr
output = ""
if result.stdout:
output += "STDOUT:\n" + result.stdout + "\n"
if result.stderr:
output += "STDERR:\n" + result.stderr + "\n"
# Include return code
output += f"Return code: {result.returncode}\n"
if result.returncode == 0:
return f"Execution successful:\n{output}"
else:
return f"Execution failed:\n{output}"
except subprocess.TimeoutExpired:
return "Error: Execution timed out after 30 seconds"
except Exception as e:
return f"Error executing Python file: {str(e)}"
@tool
def search_internet(query: str) -> str:
"""
Search the internet for information using DuckDuckGo.
Returns relevant snippets from search results.
Args:
query: The search query
Returns:
Search results or error message
"""
try:
from duckduckgo_search import DDGS
# Perform the search
with DDGS() as ddgs:
results = list(ddgs.text(query, max_results=5))
# Format the results
if not results:
return "No results found for the query."
formatted_results = "Search Results:\n\n"
for i, result in enumerate(results, 1):
formatted_results += f"{i}. {result['title']}\n"
formatted_results += f" {result['body']}\n"
formatted_results += f" URL: {result['href']}\n\n"
return formatted_results
except Exception as e:
return f"Error searching internet: {str(e)}"
def get_all_tools() -> list:
"""
Return a list of all available tools for the agent.
Returns:
List of tool instances
"""
return [
create_directory,
write_file,
read_file,
execute_python,
search_internet
]
Each tool is decorated with the @tool decorator from LangChain, which automatically generates the necessary metadata for the LLM to understand how to use it. The docstrings are particularly important because the LLM reads them to understand what each tool does and when to use it.
The create_directory tool uses os.makedirs with exist_ok=True, which prevents errors if the directory already exists. This makes the tool idempotent, meaning it can be called multiple times safely.
The write_file tool ensures the parent directory exists before writing, preventing errors when creating files in new directories. It also overwrites existing files, which is exactly what we need for the error correction loop.
The execute_python tool runs Python files in a subprocess with a timeout to prevent infinite loops from hanging the assistant. It captures both stdout and stderr, providing complete information about execution results.
The search_internet tool uses DuckDuckGo's search API, which does not require authentication or API keys. This makes it easy to integrate internet search without additional setup.
IMPLEMENTING CONVERSATION MANAGEMENT
Conversation management maintains the chat history and provides context for the LLM. This enables the assistant to reference earlier parts of the conversation and maintain coherent multi-turn interactions.
Create the file assistant/conversation.py with the following implementation.
from langchain.schema import HumanMessage, AIMessage
from typing import List, Tuple
from datetime import datetime
class Conversation:
"""
Manages a single conversation session with chat history.
Maintains the context needed for coherent multi-turn interactions.
"""
def __init__(self, session_id: str):
"""
Initialize a new conversation session.
Args:
session_id: Unique identifier for this conversation
"""
self.session_id = session_id
self.history: List[Tuple[str, str]] = []
self.created_at = datetime.now()
self.last_updated = datetime.now()
def add_message(self, role: str, content: str) -> None:
"""
Add a message to the conversation history.
Args:
role: Either 'human' or 'ai'
content: The message content
"""
self.history.append((role, content))
self.last_updated = datetime.now()
def get_langchain_history(self) -> List:
"""
Convert the conversation history to LangChain message format.
Returns:
List of LangChain message objects
"""
messages = []
for role, content in self.history:
if role == "human":
messages.append(HumanMessage(content=content))
elif role == "ai":
messages.append(AIMessage(content=content))
return messages
def get_formatted_history(self) -> str:
"""
Get a human-readable formatted version of the conversation history.
Returns:
Formatted conversation string
"""
formatted = f"Conversation {self.session_id}\n"
formatted += f"Created: {self.created_at.strftime('%Y-%m-%d %H:%M:%S')}\n"
formatted += f"Last updated: {self.last_updated.strftime('%Y-%m-%d %H:%M:%S')}\n"
formatted += "-" * 60 + "\n\n"
for role, content in self.history:
prefix = "You: " if role == "human" else "Assistant: "
formatted += f"{prefix}{content}\n\n"
return formatted
def clear(self) -> None:
"""
Clear the conversation history while maintaining the session.
"""
self.history = []
self.last_updated = datetime.now()
def get_message_count(self) -> int:
"""
Get the total number of messages in the conversation.
Returns:
Number of messages
"""
return len(self.history)
The Conversation class maintains a simple list of tuples representing the chat history. Each tuple contains the role (human or ai) and the message content. This simple structure is easy to work with and can be converted to LangChain's message format when needed.
The get_langchain_history method converts our internal representation to LangChain message objects, which the agent executor expects. This separation of concerns keeps our conversation management independent of LangChain's specific requirements.
The formatted history method provides a human-readable view of the conversation, useful for debugging or displaying conversation summaries to users.
IMPLEMENTING SESSION MANAGEMENT
The session manager coordinates multiple conversation sessions, allowing users to have separate chat windows for different tasks or topics.
Create the file assistant/session_manager.py with the following implementation.
from typing import Dict, Optional
from .conversation import Conversation
import uuid
class SessionManager:
"""
Manages multiple conversation sessions.
Each session has its own isolated chat history and context.
"""
def __init__(self):
"""
Initialize the session manager with an empty session dictionary.
"""
self.sessions: Dict[str, Conversation] = {}
self.current_session_id: Optional[str] = None
def create_session(self) -> str:
"""
Create a new conversation session with a unique ID.
Returns:
The new session ID
"""
session_id = str(uuid.uuid4())[:8] # Use short UUID for readability
self.sessions[session_id] = Conversation(session_id)
self.current_session_id = session_id
return session_id
def get_session(self, session_id: str) -> Optional[Conversation]:
"""
Retrieve a conversation session by ID.
Args:
session_id: The session identifier
Returns:
Conversation object or None if not found
"""
return self.sessions.get(session_id)
def get_current_session(self) -> Optional[Conversation]:
"""
Get the currently active conversation session.
Returns:
Current Conversation object or None
"""
if self.current_session_id:
return self.sessions.get(self.current_session_id)
return None
def switch_session(self, session_id: str) -> bool:
"""
Switch to a different conversation session.
Args:
session_id: The session to switch to
Returns:
True if successful, False if session not found
"""
if session_id in self.sessions:
self.current_session_id = session_id
return True
return False
def list_sessions(self) -> Dict[str, dict]:
"""
Get information about all active sessions.
Returns:
Dictionary mapping session IDs to session info
"""
session_info = {}
for session_id, conversation in self.sessions.items():
session_info[session_id] = {
"message_count": conversation.get_message_count(),
"created_at": conversation.created_at,
"last_updated": conversation.last_updated,
"is_current": session_id == self.current_session_id
}
return session_info
def delete_session(self, session_id: str) -> bool:
"""
Delete a conversation session.
Args:
session_id: The session to delete
Returns:
True if successful, False if session not found
"""
if session_id in self.sessions:
del self.sessions[session_id]
if self.current_session_id == session_id:
self.current_session_id = None
return True
return False
The SessionManager uses UUIDs to generate unique session identifiers. We truncate the UUID to eight characters for better readability while maintaining sufficient uniqueness for typical usage.
The manager maintains a reference to the current session, making it easy to work with the active conversation without explicitly passing session IDs around. This simplifies the user interface code.
The list_sessions method provides an overview of all active sessions, including metadata like message count and timestamps. This information helps users manage multiple concurrent conversations.
IMPLEMENTING CONSOLE UTILITIES
Beautiful console output makes the assistant more pleasant to use. We will use the rich library to create an attractive terminal interface.
Create the file utils/console.py with the following implementation.
from rich.console import Console
from rich.panel import Panel
from rich.markdown import Markdown
from rich.syntax import Syntax
from rich.table import Table
from typing import Dict
console = Console()
def print_welcome():
"""
Display a welcome message when the assistant starts.
"""
welcome_text = """
# Code Assistant with Local LLM
Welcome to your intelligent code assistant! I can help you:
- Implement algorithms and design patterns
- Generate complete Python programs
- Debug and fix code errors automatically
- Search the internet for coding solutions
- Maintain conversation context across sessions
Type your coding requests, and I'll create working solutions!
Type 'exit' or 'quit' to end the session.
Press Ctrl+C to interrupt code execution loops.
"""
console.print(Panel(Markdown(welcome_text), title="Welcome", border_style="blue"))
def print_user_message(message: str):
"""
Display a user message with formatting.
Args:
message: The user's input message
"""
console.print(f"\n[bold cyan]You:[/bold cyan] {message}")
def print_assistant_message(message: str):
"""
Display an assistant message with formatting.
Args:
message: The assistant's response
"""
console.print(f"\n[bold green]Assistant:[/bold green] {message}\n")
def print_code_block(code: str, language: str = "python"):
"""
Display a code block with syntax highlighting.
Args:
code: The code to display
language: Programming language for syntax highlighting
"""
syntax = Syntax(code, language, theme="monokai", line_numbers=True)
console.print(syntax)
def print_error(message: str):
"""
Display an error message with formatting.
Args:
message: The error message
"""
console.print(f"\n[bold red]Error:[/bold red] {message}\n")
def print_success(message: str):
"""
Display a success message with formatting.
Args:
message: The success message
"""
console.print(f"\n[bold green]Success:[/bold green] {message}\n")
def print_info(message: str):
"""
Display an informational message with formatting.
Args:
message: The info message
"""
console.print(f"\n[bold yellow]Info:[/bold yellow] {message}\n")
def print_session_list(sessions: Dict[str, dict]):
"""
Display a formatted table of active sessions.
Args:
sessions: Dictionary of session information
"""
table = Table(title="Active Sessions")
table.add_column("Session ID", style="cyan")
table.add_column("Messages", style="magenta")
table.add_column("Created", style="green")
table.add_column("Current", style="yellow")
for session_id, info in sessions.items():
table.add_row(
session_id,
str(info["message_count"]),
info["created_at"].strftime("%Y-%m-%d %H:%M"),
"Yes" if info["is_current"] else "No"
)
console.print(table)
def print_divider():
"""
Print a visual divider line.
"""
console.print("\n" + "-" * 80 + "\n")
These console utilities provide a consistent and attractive interface throughout the application. The rich library handles color codes, formatting, and even syntax highlighting automatically.
The print_code_block function is particularly useful for displaying generated code with proper syntax highlighting and line numbers, making it easier to review the assistant's output.
IMPLEMENTING FILE OPERATION UTILITIES
File operation utilities provide helper functions for common file system tasks used throughout the application.
Create the file utils/file_operations.py with the following implementation.
import os
import shutil
from typing import Optional
def ensure_directory_exists(path: str) -> bool:
"""
Ensure a directory exists, creating it if necessary.
Args:
path: Directory path to ensure exists
Returns:
True if successful, False otherwise
"""
try:
os.makedirs(path, exist_ok=True)
return True
except Exception as e:
print(f"Error creating directory {path}: {e}")
return False
def safe_delete_file(filepath: str) -> bool:
"""
Safely delete a file if it exists.
Args:
filepath: Path to the file to delete
Returns:
True if successful or file doesn't exist, False on error
"""
try:
if os.path.exists(filepath):
os.remove(filepath)
return True
except Exception as e:
print(f"Error deleting file {filepath}: {e}")
return False
def safe_delete_directory(dirpath: str) -> bool:
"""
Safely delete a directory and its contents if it exists.
Args:
dirpath: Path to the directory to delete
Returns:
True if successful or directory doesn't exist, False on error
"""
try:
if os.path.exists(dirpath):
shutil.rmtree(dirpath)
return True
except Exception as e:
print(f"Error deleting directory {dirpath}: {e}")
return False
def get_file_extension(filepath: str) -> str:
"""
Get the file extension from a filepath.
Args:
filepath: Path to the file
Returns:
File extension including the dot, or empty string
"""
return os.path.splitext(filepath)[1]
def read_file_safe(filepath: str) -> Optional[str]:
"""
Safely read a file's contents.
Args:
filepath: Path to the file to read
Returns:
File contents or None on error
"""
try:
with open(filepath, 'r', encoding='utf-8') as f:
return f.read()
except Exception as e:
print(f"Error reading file {filepath}: {e}")
return None
def write_file_safe(filepath: str, content: str) -> bool:
"""
Safely write content to a file.
Args:
filepath: Path to the file to write
content: Content to write
Returns:
True if successful, False otherwise
"""
try:
directory = os.path.dirname(filepath)
if directory:
ensure_directory_exists(directory)
with open(filepath, 'w', encoding='utf-8') as f:
f.write(content)
return True
except Exception as e:
print(f"Error writing file {filepath}: {e}")
return False
These utility functions provide safe wrappers around common file operations. They handle errors gracefully and return boolean success indicators, making error handling in the main application code cleaner.
CREATING THE MAIN APPLICATION
Now we bring all components together in the main application file. This orchestrates the entire system and provides the user interface.
Create the file main.py with the following implementation.
#!/usr/bin/env python3
"""
Code Assistant with Local LLM
A sophisticated code assistant that generates, executes, and debugs Python code.
"""
import sys
import signal
from assistant.llm_engine import LLMEngine
from assistant.tools import get_all_tools
from assistant.session_manager import SessionManager
from utils.console import (
console, print_welcome, print_user_message, print_assistant_message,
print_error, print_success, print_info, print_session_list
)
class CodeAssistant:
"""
Main application class that coordinates all components.
"""
def __init__(self):
"""
Initialize the code assistant with all necessary components.
"""
self.session_manager = SessionManager()
self.llm_engine = None
self.running = True
# Set up signal handler for graceful interruption
signal.signal(signal.SIGINT, self.signal_handler)
def signal_handler(self, signum, frame):
"""
Handle Ctrl+C interruption gracefully.
Args:
signum: Signal number
frame: Current stack frame
"""
print_info("Interrupted by user. Type 'exit' to quit or continue chatting.")
self.running = True # Allow continuation after interruption
def initialize(self):
"""
Initialize the LLM engine and create the first session.
"""
print_info("Initializing Code Assistant...")
print_info("Loading local LLM model (this may take a moment)...")
try:
# Initialize the LLM engine
self.llm_engine = LLMEngine(model_name="codellama:13b-instruct")
# Get all available tools
tools = get_all_tools()
# Create the agent with tools
self.llm_engine.create_agent(tools)
# Create the first session
session_id = self.session_manager.create_session()
print_success(f"Assistant initialized successfully!")
print_info(f"Created session: {session_id}")
print_info(f"Using device: {self.llm_engine.device.upper()}")
except Exception as e:
print_error(f"Failed to initialize assistant: {e}")
sys.exit(1)
def process_command(self, user_input: str) -> bool:
"""
Process special commands that start with '/'.
Args:
user_input: The user's input
Returns:
True if a command was processed, False otherwise
"""
if not user_input.startswith('/'):
return False
parts = user_input[1:].split()
command = parts[0].lower() if parts else ""
if command == "sessions":
sessions = self.session_manager.list_sessions()
print_session_list(sessions)
elif command == "new":
session_id = self.session_manager.create_session()
print_success(f"Created new session: {session_id}")
elif command == "switch":
if len(parts) < 2:
print_error("Usage: /switch <session_id>")
else:
session_id = parts[1]
if self.session_manager.switch_session(session_id):
print_success(f"Switched to session: {session_id}")
else:
print_error(f"Session not found: {session_id}")
elif command == "clear":
session = self.session_manager.get_current_session()
if session:
session.clear()
print_success("Conversation history cleared")
elif command == "help":
self.print_help()
else:
print_error(f"Unknown command: {command}")
self.print_help()
return True
def print_help(self):
"""
Print available commands and usage information.
"""
help_text = """
Available Commands:
/sessions - List all active sessions
/new - Create a new session
/switch <id> - Switch to a different session
/clear - Clear current conversation history
/help - Show this help message
Regular Usage:
Simply type your coding request, and the assistant will:
1. Generate the code
2. Create necessary files and directories
3. Execute the code
4. Fix any errors automatically
5. Repeat until successful
Examples:
- "Implement the quicksort algorithm"
- "Create a binary search tree with insert and search methods"
- "Implement the singleton design pattern"
- "Write a function to calculate fibonacci numbers"
Type 'exit' or 'quit' to end the session.
Press Ctrl+C to interrupt long-running operations.
"""
console.print(help_text)
def run(self):
"""
Main application loop that handles user interaction.
"""
print_welcome()
self.initialize()
while self.running:
try:
# Get current session
session = self.session_manager.get_current_session()
if not session:
print_error("No active session. Creating new session...")
session_id = self.session_manager.create_session()
session = self.session_manager.get_current_session()
# Get user input
user_input = console.input("\n[bold cyan]You:[/bold cyan] ").strip()
# Check for exit commands
if user_input.lower() in ['exit', 'quit', 'bye']:
print_info("Thank you for using Code Assistant. Goodbye!")
break
# Skip empty input
if not user_input:
continue
# Process special commands
if self.process_command(user_input):
continue
# Add user message to history
session.add_message("human", user_input)
# Get chat history in LangChain format
chat_history = session.get_langchain_history()
# Process the request with the LLM
print_info("Processing your request...")
try:
response = self.llm_engine.invoke(user_input, chat_history)
# Add assistant response to history
session.add_message("ai", response)
# Display the response
print_assistant_message(response)
except Exception as e:
print_error(f"Error processing request: {e}")
print_info("Please try rephrasing your request or check the error details.")
except KeyboardInterrupt:
print_info("\nInterrupted. Type 'exit' to quit or continue chatting.")
continue
except Exception as e:
print_error(f"Unexpected error: {e}")
print_info("The assistant will continue running. Please try again.")
def main():
"""
Entry point for the application.
"""
assistant = CodeAssistant()
assistant.run()
if __name__ == "__main__":
main()
The main application brings together all components we have built. The CodeAssistant class coordinates the session manager, LLM engine, and user interface.
The signal handler allows users to interrupt long-running operations with Ctrl+C without terminating the entire application. This is essential for stopping the error correction loop when needed.
The command processing system provides special commands prefixed with a slash, similar to many chat applications. These commands let users manage sessions, view history, and control the application without interfering with normal conversation.
The main loop follows a simple pattern: get input, check for commands or exit requests, process with the LLM, display results, and repeat. Error handling at multiple levels ensures the application remains stable even when individual requests fail.
UNDERSTANDING THE COMPLETE WORKFLOW
Let us walk through a complete example to understand how all components work together. Imagine a user asks the assistant to implement the quicksort algorithm.
When the user types their request, the main application adds it to the conversation history and passes it to the LLM engine. The engine formats the request along with the chat history and sends it to the local LLM model.
The LLM analyzes the request and decides it needs to generate code, create a file, and execute it. It generates a tool call to create_directory with a path like "quicksort_implementation". The tool executes and returns a success message.
Next, the LLM generates the quicksort implementation code and calls write_file to save it to "quicksort_implementation/quicksort.py". The tool creates the file with the generated code.
The LLM then calls execute_python to run the code. If the code runs successfully, the execution output is returned and the LLM formulates a response explaining the implementation.
If errors occur during execution, the error messages are captured and returned to the LLM. The LLM analyzes the errors, generates corrected code, and calls write_file again to overwrite the previous version. It then calls execute_python again with the new code.
This cycle continues until the code executes successfully or the user interrupts with Ctrl+C. Throughout the process, all interactions are added to the conversation history, maintaining context for follow-up questions.
TESTING THE ASSISTANT
Before we discuss extensions, let us test the complete system to ensure everything works correctly. Start by running the main application.
python main.py
You should see the welcome message and initialization output. The assistant will load the LLM model, which may take a few moments on first run.
Try a simple request to test basic functionality.
You: Implement a simple hello world function and run it
The assistant should generate code, create a directory and file, execute the code, and show you the output. This tests the complete workflow including tool calling.
Next, test the error correction loop with a deliberately problematic request.
You: Create a function that divides two numbers, but make it fail first then fix it
The assistant should generate code with an error, attempt to run it, see the error, generate corrected code, and try again until successful.
Test the internet search capability.
You: Search for the latest Python best practices for error handling
The assistant should use the search_internet tool and provide relevant results from recent sources.
Test session management with the special commands.
You: /new
You: /sessions
You: /switch <previous-session-id>
These commands should create new sessions, list them, and allow switching between them.
EXTENDING THE ASSISTANT
Now that we have a fully functional code assistant, let us discuss potential extensions and improvements.
One valuable extension would be adding support for multiple programming languages. Currently, the assistant focuses on Python, but the architecture could easily accommodate other languages by adding language-specific execution tools and updating the system prompt.
Implementing a web interface would make the assistant more accessible. You could use frameworks like Flask or FastAPI to create a REST API, then build a web frontend with React or Vue. The session management system already provides the foundation for multi-user support.
Adding code analysis tools would enhance the assistant's capabilities. Integration with tools like pylint, black, or mypy would allow the assistant to check code quality, format code according to standards, and perform static type checking before execution.
Implementing a vector database for code examples would enable semantic search over previous implementations. When asked to implement something similar to a past request, the assistant could reference and adapt previous solutions.
Adding support for package installation would allow the assistant to handle projects requiring external dependencies. A tool that calls pip install could be added, along with virtual environment management.
Implementing project templates would speed up common tasks. The assistant could maintain templates for web applications, CLI tools, data analysis projects, and other common patterns.
Adding git integration would enable version control for generated code. Tools for initializing repositories, committing changes, and creating branches would help users manage their code projects.
Implementing code review capabilities would make the assistant more valuable for learning. It could analyze existing code, suggest improvements, and explain potential issues.
Adding support for Jupyter notebooks would enable interactive data science workflows. The assistant could create and execute notebook cells, visualize data, and explain results.
Implementing collaborative features would allow multiple users to work with the same assistant instance. Shared sessions, code review workflows, and team-based project management could be added.
PERFORMANCE OPTIMIZATION
Several optimizations can improve the assistant's performance and responsiveness.
Model quantization reduces memory usage and improves inference speed. Ollama supports quantized models, and using a 4-bit quantized version of CodeLlama would significantly reduce resource requirements while maintaining good quality.
Caching frequently used responses could speed up common requests. A simple cache keyed by request text could avoid redundant LLM calls for identical questions.
Streaming responses would improve perceived responsiveness. Instead of waiting for the complete response, the assistant could display tokens as they are generated, making it feel faster and more interactive.
Parallel tool execution could speed up workflows that involve multiple independent operations. When the LLM calls multiple tools that do not depend on each other, they could execute concurrently.
Prompt optimization reduces token usage and improves response quality. Carefully crafted prompts that are concise but comprehensive lead to better results with fewer tokens.
Context window management becomes important in long conversations. Implementing strategies to summarize or truncate old messages keeps the context window from filling up while maintaining important information.
SECURITY CONSIDERATIONS
When deploying a code assistant, security must be carefully considered.
Code execution in isolated environments prevents malicious code from affecting the host system. Using containers or virtual machines to execute generated code adds a critical security layer.
Input validation prevents injection attacks and other malicious inputs. Sanitizing user requests and validating tool parameters helps protect against exploitation.
Rate limiting prevents abuse and resource exhaustion. Limiting the number of requests per user and the execution time for code prevents denial of service scenarios.
Access control ensures only authorized users can access the assistant. Implementing authentication and authorization mechanisms protects sensitive functionality.
Audit logging tracks all interactions for security monitoring and debugging. Recording requests, responses, and tool executions helps identify and investigate issues.
Resource limits prevent runaway processes from consuming excessive resources. Setting memory limits, CPU quotas, and execution timeouts protects system stability.
DEPLOYMENT STRATEGIES
Different deployment strategies suit different use cases.
Local deployment runs the assistant entirely on a developer's machine. This provides maximum privacy and control but requires sufficient hardware resources.
Server deployment hosts the assistant on a dedicated server accessible to multiple users. This centralizes resources and enables sharing but requires network infrastructure.
Cloud deployment uses cloud platforms like AWS, Azure, or Google Cloud. This provides scalability and reliability but involves ongoing costs and data privacy considerations.
Containerized deployment uses Docker to package the assistant with all dependencies. This ensures consistent behavior across different environments and simplifies deployment.
Kubernetes orchestration enables scalable, resilient deployments for enterprise use. This handles load balancing, automatic scaling, and fault tolerance.
MONITORING AND MAINTENANCE
Ongoing monitoring and maintenance keep the assistant running smoothly.
Performance monitoring tracks response times, resource usage, and error rates. This helps identify bottlenecks and optimization opportunities.
Error tracking captures and analyzes failures to improve reliability. Implementing comprehensive error logging and alerting helps catch issues early.
Usage analytics provide insights into how users interact with the assistant. Understanding common requests and workflows guides feature development.
Model updates keep the assistant current with the latest capabilities. Regularly updating to newer model versions improves performance and adds features.
Dependency management ensures all libraries remain up to date with security patches. Regular updates prevent vulnerabilities and compatibility issues.
ADDENDUM: HELPFUL RESOURCES
The following resources provide additional information for learning and extending the code assistant.
BOOKS
"Designing Data-Intensive Applications" by Martin Kleppmann provides excellent coverage of system architecture principles applicable to building robust assistants.
"Python for Data Analysis" by Wes McKinney offers deep insights into Python programming patterns useful for implementing tools and utilities.
"Hands-On Large Language Models" by Jay Alammar and Maarten Grootendorst covers LLM fundamentals and practical implementation strategies.
"Building LLM Apps" by Valentina Alto provides practical guidance on creating applications with language models.
WEB RESOURCES
The LangChain documentation at https://python.langchain.com/docs/get_started/introduction provides comprehensive guides and examples for using the framework.
The Ollama documentation at https://ollama.ai/docs explains model management, API usage, and optimization techniques.
The Hugging Face course at https://huggingface.co/course offers free training on transformers and language models.
The PyTorch documentation at https://pytorch.org/docs/stable/index.html covers the underlying framework used by many LLM implementations.
The Python documentation at https://docs.python.org/3/ remains essential for understanding language features and standard library capabilities.
COMMUNITY RESOURCES
The LangChain GitHub repository at https://github.com/langchain-ai/langchain contains examples, discussions, and issue tracking.
The Ollama Discord community provides real-time help and discussions about local LLM deployment.
The r/LocalLLaMA subreddit discusses local language model deployment, optimization, and applications.
Stack Overflow contains thousands of questions and answers about Python, LangChain, and related technologies.
RESEARCH PAPERS
"Attention Is All You Need" by Vaswani et al. introduced the transformer architecture underlying modern LLMs.
"Language Models are Few-Shot Learners" by Brown et al. demonstrated the capabilities of large language models.
"Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" by Wei et al. explores techniques for improving LLM reasoning.
TOOLS AND PLATFORMS
GitHub Copilot provides AI-powered code completion and can inspire features for your assistant.
Cursor IDE demonstrates advanced AI integration in development environments.
Replit AI shows how conversational interfaces can enhance coding workflows.
CONCLUSION
We have built a sophisticated code assistant that runs entirely on local hardware, providing privacy, control, and impressive capabilities. The assistant can understand coding requests, generate implementations, create files, execute code, and automatically debug errors.
The modular architecture we created makes the system easy to understand, maintain, and extend. Each component has a clear responsibility and well-defined interfaces, following clean architecture principles.
The tool calling system gives the LLM agency to perform actions beyond generating text, enabling the complete workflow from request to working code. The error correction loop demonstrates how LLMs can iteratively improve their outputs based on feedback.
Session management and conversation history enable natural, context-aware interactions that feel more like working with a colleague than using a tool.
The extensibility of the system means you can continue adding capabilities, improving performance, and adapting it to your specific needs. Whether you want to add support for more languages, integrate with development tools, or deploy it for team use, the foundation we have built supports these enhancements.
This code assistant demonstrates the power of local LLMs and the potential for AI to enhance developer productivity while maintaining privacy and control. As models continue to improve and hardware becomes more capable, tools like this will become increasingly valuable parts of the development workflow.
No comments:
Post a Comment