INTRODUCTION
The challenge of understanding unfamiliar codebases is a persistent problem in software development. Developers frequently encounter code written by others or revisit their own code after extended periods. Traditional static analysis tools rely on language-specific parsers and abstract syntax trees, which require maintenance for each programming language and version. This article presents an alternative approach that leverages Large Language Models to analyze code files without language-specific parsers. The LLM's training on vast amounts of code enables it to understand syntax, semantics, and patterns across multiple programming languages naturally.
Our system will support both local and remote LLM deployments, accommodating various GPU architectures including NVIDIA CUDA, AMD ROCm, Intel GPUs, and Apple Metal Performance Shaders. This flexibility ensures the tool can run in diverse environments from cloud deployments to local developer machines.
ARCHITECTURAL OVERVIEW
The system consists of several key components working together to provide
comprehensive code analysis. The architecture follows clean architecture
principles with clear separation of concerns and dependency inversion.
The core components include an LLM abstraction layer that provides a unified interface for different LLM backends, a file handling system that reads and preprocesses code files, a prompt engineering module that constructs effective analysis requests, a response parser that extracts structured information from LLM outputs, and a reporting system that formats analysis results.
The abstraction layer is crucial because it allows the system to work with
various LLM providers without changing the core logic. Whether using OpenAI's API, a locally hosted model through Ollama, or a custom deployment on specific hardware, the interface remains consistent.
LLM ABSTRACTION LAYER DESIGN
The foundation of our system is a well-designed abstraction that hides the
complexity of different LLM backends. We define an abstract base class that
all LLM implementations must inherit from. This ensures consistent behavior
regardless of the underlying provider.
from abc import ABC, abstractmethod
from typing import Dict, List, Optional
from dataclasses import dataclass
@dataclass
class LLMConfig:
"""Configuration for LLM backend"""
model_name: str
temperature: float = 0.2
max_tokens: int = 4096
timeout: int = 300
api_key: Optional[str] = None
base_url: Optional[str] = None
device: Optional[str] = None
class LLMBackend(ABC):
"""Abstract base class for all LLM backends"""
def __init__(self, config: LLMConfig):
self.config = config
self.validate_config()
@abstractmethod
def validate_config(self) -> None:
"""Validate configuration parameters"""
pass
@abstractmethod
def generate(self, prompt: str, system_prompt: Optional[str] = None) -> str:
"""Generate response from the LLM"""
pass
@abstractmethod
def is_available(self) -> bool:
"""Check if the backend is available and properly configured"""
pass
The LLMConfig dataclass encapsulates all configuration parameters needed for
any LLM backend. The temperature parameter controls randomness in generation,
with lower values producing more deterministic outputs suitable for code
analysis. The max_tokens parameter limits response length, while timeout
prevents indefinite waiting. Optional parameters like api_key and base_url
support remote services, while device specification enables hardware selection
for local models.
The abstract methods enforce a contract that all implementations must fulfill.
The validate_config method ensures that required parameters are present and
valid before attempting to use the backend. The generate method is the core
interface for obtaining LLM responses, accepting both a main prompt and an
optional system prompt that sets behavioral context. The is_available method
allows the system to check backend readiness before attempting operations.
IMPLEMENTING REMOTE LLM SUPPORT
Remote LLM services like OpenAI, Anthropic, or custom API endpoints provide
powerful models without local hardware requirements. The implementation wraps
HTTP requests with proper error handling and retry logic.
import requests
import time
from typing import Dict, Any
class RemoteLLMBackend(LLMBackend):
"""Backend for remote LLM APIs (OpenAI-compatible)"""
def validate_config(self) -> None:
if not self.config.api_key:
raise ValueError("API key required for remote backend")
if not self.config.base_url:
raise ValueError("Base URL required for remote backend")
def is_available(self) -> bool:
try:
response = requests.get(
f"{self.config.base_url}/models",
headers={"Authorization": f"Bearer {self.config.api_key}"},
timeout=10
)
return response.status_code == 200
except Exception:
return False
def generate(self, prompt: str, system_prompt: Optional[str] = None) -> str:
messages = []
if system_prompt:
messages.append({"role": "system", "content": system_prompt})
messages.append({"role": "user", "content": prompt})
payload = {
"model": self.config.model_name,
"messages": messages,
"temperature": self.config.temperature,
"max_tokens": self.config.max_tokens
}
headers = {
"Authorization": f"Bearer {self.config.api_key}",
"Content-Type": "application/json"
}
max_retries = 3
for attempt in range(max_retries):
try:
response = requests.post(
f"{self.config.base_url}/chat/completions",
json=payload,
headers=headers,
timeout=self.config.timeout
)
response.raise_for_status()
return response.json()["choices"][0]["message"]["content"]
except requests.exceptions.Timeout:
if attempt == max_retries - 1:
raise
time.sleep(2 ** attempt)
except requests.exceptions.RequestException as e:
raise RuntimeError(f"Remote LLM request failed: {str(e)}")
The RemoteLLMBackend implementation follows the OpenAI API specification,
which has become a de facto standard adopted by many providers. The generate
method constructs a messages array with optional system and user messages,
then sends a POST request to the completions endpoint. Error handling includes
exponential backoff for timeout errors, allowing transient network issues to
resolve without immediate failure. The response parsing extracts the generated
text from the nested JSON structure returned by the API.
IMPLEMENTING LOCAL LLM SUPPORT WITH HARDWARE ACCELERATION
Local LLM deployment offers privacy, cost control, and independence from
external services. However, it requires careful hardware management to achieve
acceptable performance. Modern LLM inference frameworks support multiple GPU
architectures through different backends.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from typing import Optional
class LocalLLMBackend(LLMBackend):
"""Backend for locally hosted LLMs with multi-GPU support"""
def __init__(self, config: LLMConfig):
super().__init__(config)
self.model = None
self.tokenizer = None
self.device_map = self._determine_device()
self._load_model()
def _determine_device(self) -> str:
"""Determine the best available device for inference"""
if self.config.device:
return self.config.device
if torch.cuda.is_available():
return "cuda"
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
return "mps"
elif hasattr(torch, "xpu") and torch.xpu.is_available():
return "xpu"
else:
return "cpu"
def _load_model(self) -> None:
"""Load model and tokenizer with appropriate device mapping"""
try:
self.tokenizer = AutoTokenizer.from_pretrained(
self.config.model_name,
trust_remote_code=True
)
load_kwargs = {
"pretrained_model_name_or_path": self.config.model_name,
"trust_remote_code": True,
"torch_dtype": torch.float16 if self.device_map != "cpu" else torch.float32
}
if self.device_map == "cuda":
load_kwargs["device_map"] = "auto"
else:
self.model = AutoModelForCausalLM.from_pretrained(**load_kwargs)
self.model = self.model.to(self.device_map)
self.model.eval()
except Exception as e:
raise RuntimeError(f"Failed to load model: {str(e)}")
def validate_config(self) -> None:
if not self.config.model_name:
raise ValueError("Model name required for local backend")
def is_available(self) -> bool:
return self.model is not None and self.tokenizer is not None
def generate(self, prompt: str, system_prompt: Optional[str] = None) -> str:
if not self.is_available():
raise RuntimeError("Model not loaded")
full_prompt = prompt
if system_prompt:
full_prompt = f"{system_prompt}\n\n{prompt}"
inputs = self.tokenizer(
full_prompt,
return_tensors="pt",
truncation=True,
max_length=self.config.max_tokens
).to(self.device_map)
with torch.no_grad():
outputs = self.model.generate(
**inputs,
max_new_tokens=self.config.max_tokens,
temperature=self.config.temperature,
do_sample=self.config.temperature > 0,
pad_token_id=self.tokenizer.eos_token_id
)
generated_text = self.tokenizer.decode(
outputs[0][inputs["input_ids"].shape[1]:],
skip_special_tokens=True
)
return generated_text.strip()
The LocalLLMBackend implementation handles the complexity of loading and
running models on different hardware. The device determination logic checks
for CUDA support first, which indicates NVIDIA GPUs. If unavailable, it checks
for Apple's Metal Performance Shaders through the mps backend. Intel GPU
support is checked through the xpu backend. Finally, it falls back to CPU
execution if no accelerators are available.
Model loading uses the Transformers library's AutoModelForCausalLM class,
which automatically selects the appropriate model architecture based on the
configuration. The torch_dtype parameter uses float16 precision on GPUs to
reduce memory usage and increase speed, while maintaining float32 on CPU for
numerical stability. The device_map parameter set to "auto" enables automatic
model parallelism across multiple GPUs when using CUDA.
The generate method tokenizes the input prompt, moves tensors to the
appropriate device, and performs inference with gradient computation disabled
for efficiency. The output decoding extracts only the newly generated tokens,
excluding the input prompt from the result.
SUPPORTING AMD ROCM FOR LOCAL INFERENCE
AMD GPUs using the ROCm platform require specific configuration but can
provide excellent performance for LLM inference. The implementation extends
the local backend with ROCm-specific optimizations.
class ROCmLLMBackend(LocalLLMBackend):
"""Specialized backend for AMD GPUs with ROCm"""
def _determine_device(self) -> str:
"""Override device determination for ROCm"""
if self.config.device:
return self.config.device
try:
import torch
if torch.cuda.is_available() and "AMD" in torch.cuda.get_device_name(0):
return "cuda"
except Exception:
pass
return "cpu"
def _load_model(self) -> None:
"""Load model with ROCm-specific optimizations"""
import os
os.environ["PYTORCH_HIP_ALLOC_CONF"] = "max_split_size_mb:512"
super()._load_model()
The ROCm backend inherits from LocalLLMBackend and overrides specific methods
for AMD GPU compatibility. PyTorch's ROCm support uses the same CUDA API
surface, so device selection checks for CUDA availability and verifies the
device name contains "AMD". The environment variable PYTORCH_HIP_ALLOC_CONF
configures memory allocation behavior specific to HIP, the ROCm equivalent
of CUDA, preventing memory fragmentation issues that can occur with large
models.
FILE HANDLING AND PREPROCESSING
Before sending code to the LLM, we must read files and prepare them
appropriately. The file handler manages encoding detection, size limits,
and basic preprocessing to ensure clean input.
import os
import chardet
from pathlib import Path
from typing import Optional, Tuple
class CodeFileHandler:
"""Handles reading and preprocessing code files"""
def __init__(self, max_file_size_mb: int = 10):
self.max_file_size_bytes = max_file_size_mb * 1024 * 1024
def read_file(self, file_path: str) -> Tuple[str, str]:
"""Read file and return content with detected language"""
path = Path(file_path)
if not path.exists():
raise FileNotFoundError(f"File not found: {file_path}")
if not path.is_file():
raise ValueError(f"Not a file: {file_path}")
file_size = path.stat().st_size
if file_size > self.max_file_size_bytes:
raise ValueError(
f"File too large: {file_size} bytes "
f"(max: {self.max_file_size_bytes})"
)
encoding = self._detect_encoding(file_path)
try:
with open(file_path, "r", encoding=encoding) as f:
content = f.read()
except UnicodeDecodeError:
with open(file_path, "r", encoding="utf-8", errors="ignore") as f:
content = f.read()
language = self._detect_language(path)
return content, language
def _detect_encoding(self, file_path: str) -> str:
"""Detect file encoding using chardet"""
with open(file_path, "rb") as f:
raw_data = f.read(10000)
result = chardet.detect(raw_data)
encoding = result["encoding"]
if encoding is None or result["confidence"] < 0.7:
return "utf-8"
return encoding
def _detect_language(self, path: Path) -> str:
"""Detect programming language from file extension"""
extension_map = {
".py": "Python",
".js": "JavaScript",
".ts": "TypeScript",
".java": "Java",
".cpp": "C++",
".c": "C",
".cs": "C#",
".go": "Go",
".rs": "Rust",
".rb": "Ruby",
".php": "PHP",
".swift": "Swift",
".kt": "Kotlin",
".scala": "Scala",
".r": "R",
".m": "Objective-C",
".sh": "Shell",
".sql": "SQL",
".html": "HTML",
".css": "CSS",
".jsx": "JSX",
".tsx": "TSX",
".vue": "Vue",
".dart": "Dart",
".lua": "Lua",
".pl": "Perl",
".ex": "Elixir",
".clj": "Clojure",
".hs": "Haskell",
".ml": "OCaml",
".fs": "F#"
}
extension = path.suffix.lower()
return extension_map.get(extension, "Unknown")
The CodeFileHandler class encapsulates all file operations. The read_file
method performs validation checks before reading, ensuring the path exists,
points to a regular file, and does not exceed size limits. Size limits prevent
memory exhaustion and excessive LLM context usage.
Encoding detection uses the chardet library to analyze the first 10000 bytes
of the file. This sample size balances accuracy with performance. If detection
confidence is below 70 percent or fails entirely, the method defaults to UTF-8
as the most common encoding for source code. The fallback reading with error
handling ignores invalid characters rather than failing completely, ensuring
robustness when dealing with files containing mixed encodings or binary data.
Language detection relies on file extensions mapped to human-readable language
names. While not foolproof, this approach works reliably for standard file
naming conventions and avoids the complexity of content-based detection. The
detected language helps the LLM provide more accurate analysis by establishing
context.
PROMPT ENGINEERING FOR CODE ANALYSIS
The quality of LLM output depends heavily on prompt design. Effective prompts
provide clear instructions, establish context, and specify the desired output
format. Our prompt engineering module constructs specialized prompts for
different analysis tasks.
from typing import Dict, Any
from dataclasses import dataclass
@dataclass
class AnalysisRequest:
"""Represents a code analysis request"""
code_content: str
file_path: str
language: str
analysis_type: str = "comprehensive"
class PromptBuilder:
"""Builds prompts for code analysis tasks"""
def __init__(self):
self.system_prompt = self._build_system_prompt()
def _build_system_prompt(self) -> str:
"""Create system prompt that establishes LLM behavior"""
return """You are an expert code analyst with deep knowledge of
software engineering principles, design patterns, and best practices across
multiple programming languages. Your task is to analyze code thoroughly and
provide clear, accurate, and actionable insights. Focus on understanding the
code's purpose, functionality, structure, and quality. Be specific and cite
examples from the code when making observations."""
def build_analysis_prompt(self, request: AnalysisRequest) -> str:
"""Build comprehensive analysis prompt"""
prompt = f"""Analyze the following {request.language} code from the file
{request.file_path}. Provide a comprehensive analysis covering these aspects:
1. Purpose and Overview: What is the primary purpose of this code? What problem
does it solve? Provide a high-level summary of its functionality.
2. Main Components: Identify and describe the key components such as classes,
functions, modules, or data structures. Explain what each component does and
how they relate to each other.
3. Functionality Details: Describe the specific functionality provided by this
code. What operations can it perform? What are the inputs and outputs? What
are the main algorithms or logic flows?
4. Dependencies and Integrations: What external libraries, frameworks, or
modules does this code depend on? How does it integrate with other systems
or components?
5. Code Quality Observations: Comment on code organization, readability,
naming conventions, documentation, and adherence to best practices. Identify
any potential issues or areas for improvement.
6. Technical Characteristics: Note any interesting technical aspects such as
design patterns used, performance considerations, error handling approaches,
or security measures.
7. Usage Context: Based on the code structure and functionality, describe
typical use cases or scenarios where this code would be employed.
Here is the code to analyze:
```{request.language.lower()}
{request.code_content}
```
Provide your analysis in a clear, structured format with detailed explanations."""
return prompt
def build_summary_prompt(self, request: AnalysisRequest) -> str:
"""Build prompt for concise summary"""
prompt = f"""Provide a concise summary of the following {request.language}
code from {request.file_path}. The summary should include:
- A brief description of what the code does
- The main functionality it provides
- Key components or functions
- Primary use case or purpose
Keep the summary clear and informative but concise, suitable for quick
understanding.
Code to summarize:
```{request.language.lower()}
{request.code_content}
```"""
return prompt
def build_functionality_prompt(self, request: AnalysisRequest) -> str:
"""Build prompt focused on functionality extraction"""
prompt = f"""Extract and describe all functionality provided by this
{request.language} code from {request.file_path}. For each function, method,
or capability:
- State what it does
- Describe its parameters and return values
- Explain its role in the overall system
- Note any side effects or state changes
Be thorough and systematic in documenting all functionality.
Code to analyze:
```{request.language.lower()}
{request.code_content}
```"""
return prompt
The PromptBuilder class separates prompt construction from the core analysis
logic, following the single responsibility principle. The system prompt
establishes the LLM's role and behavioral expectations, setting a professional
and thorough tone for analysis.
The build_analysis_prompt method creates a comprehensive prompt with seven
distinct aspects to analyze. This structured approach ensures consistent,
complete analysis across different code files. The numbered sections guide
the LLM to address each aspect systematically, reducing the likelihood of
omissions. Including the file path and language in the prompt provides
additional context that helps the LLM tailor its analysis appropriately.
Alternative prompt builders for summaries and functionality extraction
demonstrate how different analysis needs require different prompt structures.
The summary prompt emphasizes brevity while maintaining informativeness, while
the functionality prompt focuses specifically on extracting operational
capabilities.
RESPONSE PARSING AND STRUCTURING
LLM responses are typically unstructured text. To make analysis results
programmatically useful, we need to parse and structure the output. The
response parser extracts key information and organizes it into a consistent
format.
from typing import Dict, List, Optional
import re
from dataclasses import dataclass, field
@dataclass
class CodeAnalysis:
"""Structured representation of code analysis results"""
file_path: str
language: str
purpose: str = ""
components: List[str] = field(default_factory=list)
functionality: List[str] = field(default_factory=list)
dependencies: List[str] = field(default_factory=list)
quality_observations: List[str] = field(default_factory=list)
technical_characteristics: List[str] = field(default_factory=list)
usage_context: str = ""
raw_analysis: str = ""
summary: str = ""
class ResponseParser:
"""Parses LLM responses into structured analysis objects"""
def parse_comprehensive_analysis(
self,
response: str,
file_path: str,
language: str
) -> CodeAnalysis:
"""Parse comprehensive analysis response into structured format"""
analysis = CodeAnalysis(
file_path=file_path,
language=language,
raw_analysis=response
)
sections = self._split_into_sections(response)
analysis.purpose = self._extract_purpose(sections)
analysis.components = self._extract_list_items(
sections.get("main components", "")
)
analysis.functionality = self._extract_list_items(
sections.get("functionality details", "")
)
analysis.dependencies = self._extract_list_items(
sections.get("dependencies and integrations", "")
)
analysis.quality_observations = self._extract_list_items(
sections.get("code quality observations", "")
)
analysis.technical_characteristics = self._extract_list_items(
sections.get("technical characteristics", "")
)
analysis.usage_context = sections.get("usage context", "").strip()
analysis.summary = self._generate_summary_from_analysis(analysis)
return analysis
def _split_into_sections(self, response: str) -> Dict[str, str]:
"""Split response into sections based on numbered headers"""
sections = {}
pattern = r'(\d+)\.\s*([^:]+):\s*(.*?)(?=\n\d+\.|$)'
matches = re.finditer(pattern, response, re.DOTALL | re.IGNORECASE)
for match in matches:
section_name = match.group(2).strip().lower()
section_content = match.group(3).strip()
sections[section_name] = section_content
return sections
def _extract_purpose(self, sections: Dict[str, str]) -> str:
"""Extract purpose from sections"""
purpose_section = sections.get("purpose and overview", "")
if purpose_section:
lines = purpose_section.split("\n")
return " ".join(line.strip() for line in lines if line.strip())
return ""
def _extract_list_items(self, section_text: str) -> List[str]:
"""Extract list items from section text"""
if not section_text:
return []
items = []
bullet_pattern = r'^[\s]*[-*•]\s*(.+)$'
lines = section_text.split("\n")
current_item = ""
for line in lines:
match = re.match(bullet_pattern, line)
if match:
if current_item:
items.append(current_item.strip())
current_item = match.group(1)
elif line.strip() and current_item:
current_item += " " + line.strip()
elif line.strip() and not current_item:
current_item = line.strip()
if current_item:
items.append(current_item.strip())
if not items and section_text.strip():
items.append(section_text.strip())
return items
def _generate_summary_from_analysis(self, analysis: CodeAnalysis) -> str:
"""Generate concise summary from structured analysis"""
summary_parts = []
if analysis.purpose:
summary_parts.append(analysis.purpose)
if analysis.components:
comp_summary = f"Main components include {', '.join(analysis.components[:3])}"
if len(analysis.components) > 3:
comp_summary += f" and {len(analysis.components) - 3} others"
summary_parts.append(comp_summary + ".")
if analysis.functionality:
func_summary = f"Provides functionality for {', '.join(analysis.functionality[:2])}"
if len(analysis.functionality) > 2:
func_summary += " among other capabilities"
summary_parts.append(func_summary + ".")
return " ".join(summary_parts)
The ResponseParser class transforms unstructured LLM output into the
CodeAnalysis dataclass. The parsing strategy uses regular expressions to
identify section headers matching the numbered format from our prompt. This
approach is robust to minor formatting variations in LLM output while
maintaining structure.
The section splitting method uses a regex pattern that matches numbered
headers followed by colons, capturing everything until the next numbered
header or end of text. The DOTALL flag allows matching across newlines,
essential for multi-paragraph sections. Converting section names to lowercase
enables case-insensitive matching when extracting specific sections.
List item extraction handles multiple formatting styles including bullet
points with dashes, asterisks, or bullet characters. The method accumulates
multi-line items by continuing to append text until encountering the next
bullet point. This handles cases where the LLM wraps long descriptions across
multiple lines. If no bullet points are found but the section contains text,
the entire section is treated as a single item, ensuring no information is
lost.
The summary generation method creates a concise overview from the structured
analysis by combining the purpose with abbreviated lists of components and
functionality. This provides a quick reference without requiring the full
analysis.
INTEGRATING COMPONENTS INTO THE ANALYSIS ENGINE
With all components defined, we create the main analysis engine that
orchestrates the entire process. This engine manages backend selection,
file reading, prompt construction, LLM invocation, and response parsing.
from typing import Optional, Dict, Any
from enum import Enum
class BackendType(Enum):
"""Enumeration of supported backend types"""
REMOTE = "remote"
LOCAL = "local"
ROCM = "rocm"
class CodeAnalyzer:
"""Main engine for code analysis using LLMs"""
def __init__(
self,
backend_type: BackendType,
config: LLMConfig,
max_file_size_mb: int = 10
):
self.backend = self._create_backend(backend_type, config)
self.file_handler = CodeFileHandler(max_file_size_mb)
self.prompt_builder = PromptBuilder()
self.response_parser = ResponseParser()
if not self.backend.is_available():
raise RuntimeError(f"Backend {backend_type.value} is not available")
def _create_backend(
self,
backend_type: BackendType,
config: LLMConfig
) -> LLMBackend:
"""Factory method to create appropriate backend"""
if backend_type == BackendType.REMOTE:
return RemoteLLMBackend(config)
elif backend_type == BackendType.LOCAL:
return LocalLLMBackend(config)
elif backend_type == BackendType.ROCM:
return ROCmLLMBackend(config)
else:
raise ValueError(f"Unknown backend type: {backend_type}")
def analyze_file(
self,
file_path: str,
analysis_type: str = "comprehensive"
) -> CodeAnalysis:
"""Analyze a code file and return structured results"""
code_content, language = self.file_handler.read_file(file_path)
request = AnalysisRequest(
code_content=code_content,
file_path=file_path,
language=language,
analysis_type=analysis_type
)
if analysis_type == "comprehensive":
prompt = self.prompt_builder.build_analysis_prompt(request)
elif analysis_type == "summary":
prompt = self.prompt_builder.build_summary_prompt(request)
elif analysis_type == "functionality":
prompt = self.prompt_builder.build_functionality_prompt(request)
else:
raise ValueError(f"Unknown analysis type: {analysis_type}")
response = self.backend.generate(
prompt=prompt,
system_prompt=self.prompt_builder.system_prompt
)
if analysis_type == "comprehensive":
analysis = self.response_parser.parse_comprehensive_analysis(
response,
file_path,
language
)
else:
analysis = CodeAnalysis(
file_path=file_path,
language=language,
raw_analysis=response,
summary=response
)
return analysis
def analyze_multiple_files(
self,
file_paths: List[str],
analysis_type: str = "comprehensive"
) -> Dict[str, CodeAnalysis]:
"""Analyze multiple files and return results dictionary"""
results = {}
for file_path in file_paths:
try:
analysis = self.analyze_file(file_path, analysis_type)
results[file_path] = analysis
except Exception as e:
print(f"Error analyzing {file_path}: {str(e)}")
results[file_path] = None
return results
The CodeAnalyzer class serves as the facade for the entire system, providing
a simple interface that hides internal complexity. The constructor accepts
a backend type enumeration, configuration, and file size limit, then
initializes all necessary components. The availability check ensures the
system fails fast if the backend cannot be used, rather than failing later
during analysis.
The factory method for backend creation uses the backend type enumeration to
instantiate the appropriate class. This pattern makes adding new backends
straightforward by adding a new enum value and corresponding instantiation logic without modifying existing code.
The analyze_file method orchestrates the complete analysis workflow. It reads the file, constructs an analysis request, builds the appropriate prompt, invokes the LLM, and parses the response. The analysis type parameter allows callers to request different levels of detail, from quick summaries to comprehensive analysis. Error handling at each step ensures failures arecaught and reported meaningfully.
The analyze_multiple_files method extends functionality to batch processing, useful for analyzing entire codebases. It continues processing even if individual files fail, collecting results in a dictionary keyed by file path. Failed analyses are recorded as None rather than halting the entire batch.
REPORTING AND OUTPUT FORMATTING
Analysis results need clear presentation for human consumption. The reporting system formats structured analysis data into readable reports with multiple output formats.
from typing import TextIO
import json
from datetime import datetime
class AnalysisReporter:
"""Formats and outputs analysis results"""
def generate_text_report(
self,
analysis: CodeAnalysis,
output_file: Optional[str] = None
) -> str:
"""Generate human-readable text report"""
report_lines = []
report_lines.append("=" * 80)
report_lines.append("CODE ANALYSIS REPORT")
report_lines.append("=" * 80)
report_lines.append("")
report_lines.append(f"File: {analysis.file_path}")
report_lines.append(f"Language: {analysis.language}")
report_lines.append(f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
report_lines.append("")
if analysis.summary:
report_lines.append("-" * 80)
report_lines.append("SUMMARY")
report_lines.append("-" * 80)
report_lines.append(analysis.summary)
report_lines.append("")
if analysis.purpose:
report_lines.append("-" * 80)
report_lines.append("PURPOSE AND OVERVIEW")
report_lines.append("-" * 80)
report_lines.append(analysis.purpose)
report_lines.append("")
if analysis.components:
report_lines.append("-" * 80)
report_lines.append("MAIN COMPONENTS")
report_lines.append("-" * 80)
for idx, component in enumerate(analysis.components, 1):
report_lines.append(f"{idx}. {component}")
report_lines.append("")
if analysis.functionality:
report_lines.append("-" * 80)
report_lines.append("FUNCTIONALITY")
report_lines.append("-" * 80)
for idx, func in enumerate(analysis.functionality, 1):
report_lines.append(f"{idx}. {func}")
report_lines.append("")
if analysis.dependencies:
report_lines.append("-" * 80)
report_lines.append("DEPENDENCIES AND INTEGRATIONS")
report_lines.append("-" * 80)
for idx, dep in enumerate(analysis.dependencies, 1):
report_lines.append(f"{idx}. {dep}")
report_lines.append("")
if analysis.quality_observations:
report_lines.append("-" * 80)
report_lines.append("CODE QUALITY OBSERVATIONS")
report_lines.append("-" * 80)
for idx, obs in enumerate(analysis.quality_observations, 1):
report_lines.append(f"{idx}. {obs}")
report_lines.append("")
if analysis.technical_characteristics:
report_lines.append("-" * 80)
report_lines.append("TECHNICAL CHARACTERISTICS")
report_lines.append("-" * 80)
for idx, char in enumerate(analysis.technical_characteristics, 1):
report_lines.append(f"{idx}. {char}")
report_lines.append("")
if analysis.usage_context:
report_lines.append("-" * 80)
report_lines.append("USAGE CONTEXT")
report_lines.append("-" * 80)
report_lines.append(analysis.usage_context)
report_lines.append("")
report_lines.append("=" * 80)
report_text = "\n".join(report_lines)
if output_file:
with open(output_file, "w", encoding="utf-8") as f:
f.write(report_text)
return report_text
def generate_json_report(
self,
analysis: CodeAnalysis,
output_file: Optional[str] = None
) -> str:
"""Generate JSON format report"""
report_data = {
"file_path": analysis.file_path,
"language": analysis.language,
"generated_at": datetime.now().isoformat(),
"summary": analysis.summary,
"purpose": analysis.purpose,
"components": analysis.components,
"functionality": analysis.functionality,
"dependencies": analysis.dependencies,
"quality_observations": analysis.quality_observations,
"technical_characteristics": analysis.technical_characteristics,
"usage_context": analysis.usage_context,
"raw_analysis": analysis.raw_analysis
}
json_text = json.dumps(report_data, indent=2, ensure_ascii=False)
if output_file:
with open(output_file, "w", encoding="utf-8") as f:
f.write(json_text)
return json_text
def generate_markdown_report(
self,
analysis: CodeAnalysis,
output_file: Optional[str] = None
) -> str:
"""Generate Markdown format report"""
report_lines = []
report_lines.append(f"# Code Analysis Report")
report_lines.append("")
report_lines.append(f"**File:** `{analysis.file_path}`")
report_lines.append(f"**Language:** {analysis.language}")
report_lines.append(f"**Generated:** {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
report_lines.append("")
if analysis.summary:
report_lines.append("## Summary")
report_lines.append("")
report_lines.append(analysis.summary)
report_lines.append("")
if analysis.purpose:
report_lines.append("## Purpose and Overview")
report_lines.append("")
report_lines.append(analysis.purpose)
report_lines.append("")
if analysis.components:
report_lines.append("## Main Components")
report_lines.append("")
for component in analysis.components:
report_lines.append(f"- {component}")
report_lines.append("")
if analysis.functionality:
report_lines.append("## Functionality")
report_lines.append("")
for func in analysis.functionality:
report_lines.append(f"- {func}")
report_lines.append("")
if analysis.dependencies:
report_lines.append("## Dependencies and Integrations")
report_lines.append("")
for dep in analysis.dependencies:
report_lines.append(f"- {dep}")
report_lines.append("")
if analysis.quality_observations:
report_lines.append("## Code Quality Observations")
report_lines.append("")
for obs in analysis.quality_observations:
report_lines.append(f"- {obs}")
report_lines.append("")
if analysis.technical_characteristics:
report_lines.append("## Technical Characteristics")
report_lines.append("")
for char in analysis.technical_characteristics:
report_lines.append(f"- {char}")
report_lines.append("")
if analysis.usage_context:
report_lines.append("## Usage Context")
report_lines.append("")
report_lines.append(analysis.usage_context)
report_lines.append("")
report_text = "\n".join(report_lines)
if output_file:
with open(output_file, "w", encoding="utf-8") as f:
f.write(report_text)
return report_text
The AnalysisReporter class provides three output formats to suit different
needs. Text reports use ASCII formatting with section dividers for terminal
display or plain text files. JSON reports enable programmatic consumption by
other tools or storage in databases. Markdown reports provide formatted
documentation suitable for wikis, documentation sites, or version control
systems.
Each format method follows the same pattern of building output incrementally,
checking for the presence of each analysis component before including it. This
ensures reports remain clean when certain analysis aspects are unavailable.
The optional output_file parameter allows direct file writing while also
returning the formatted string for immediate use.
COMPREHENSIVE UNIT TESTING STRATEGY
Thorough testing ensures system reliability and facilitates maintenance. Our
testing strategy covers all components with unit tests achieving complete code
coverage. We use pytest as the testing framework for its powerful features and
clean syntax.
Testing the LLM abstraction layer requires mocking actual LLM calls to avoid
dependencies on external services and ensure deterministic test execution. We
create mock implementations that simulate various response scenarios.
import pytest
from unittest.mock import Mock, patch, MagicMock
import tempfile
import os
class TestLLMConfig:
"""Test suite for LLMConfig dataclass"""
def test_config_creation_with_defaults(self):
"""Test creating config with default values"""
config = LLMConfig(model_name="test-model")
assert config.model_name == "test-model"
assert config.temperature == 0.2
assert config.max_tokens == 4096
assert config.timeout == 300
assert config.api_key is None
assert config.base_url is None
assert config.device is None
def test_config_creation_with_custom_values(self):
"""Test creating config with custom values"""
config = LLMConfig(
model_name="custom-model",
temperature=0.5,
max_tokens=2048,
timeout=600,
api_key="test-key",
base_url="https://api.example.com",
device="cuda"
)
assert config.model_name == "custom-model"
assert config.temperature == 0.5
assert config.max_tokens == 2048
assert config.timeout == 600
assert config.api_key == "test-key"
assert config.base_url == "https://api.example.com"
assert config.device == "cuda"
class TestRemoteLLMBackend:
"""Test suite for RemoteLLMBackend"""
def test_validate_config_missing_api_key(self):
"""Test validation fails without API key"""
config = LLMConfig(
model_name="test-model",
base_url="https://api.example.com"
)
with pytest.raises(ValueError, match="API key required"):
RemoteLLMBackend(config)
def test_validate_config_missing_base_url(self):
"""Test validation fails without base URL"""
config = LLMConfig(
model_name="test-model",
api_key="test-key"
)
with pytest.raises(ValueError, match="Base URL required"):
RemoteLLMBackend(config)
def test_validate_config_success(self):
"""Test validation succeeds with required parameters"""
config = LLMConfig(
model_name="test-model",
api_key="test-key",
base_url="https://api.example.com"
)
backend = RemoteLLMBackend(config)
assert backend.config == config
@patch('requests.get')
def test_is_available_success(self, mock_get):
"""Test availability check succeeds"""
mock_response = Mock()
mock_response.status_code = 200
mock_get.return_value = mock_response
config = LLMConfig(
model_name="test-model",
api_key="test-key",
base_url="https://api.example.com"
)
backend = RemoteLLMBackend(config)
assert backend.is_available() is True
mock_get.assert_called_once()
@patch('requests.get')
def test_is_available_failure(self, mock_get):
"""Test availability check fails on error"""
mock_get.side_effect = Exception("Connection error")
config = LLMConfig(
model_name="test-model",
api_key="test-key",
base_url="https://api.example.com"
)
backend = RemoteLLMBackend(config)
assert backend.is_available() is False
@patch('requests.post')
def test_generate_success(self, mock_post):
"""Test successful generation"""
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {
"choices": [{"message": {"content": "Generated response"}}]
}
mock_post.return_value = mock_response
config = LLMConfig(
model_name="test-model",
api_key="test-key",
base_url="https://api.example.com"
)
backend = RemoteLLMBackend(config)
result = backend.generate("Test prompt", "System prompt")
assert result == "Generated response"
mock_post.assert_called_once()
call_args = mock_post.call_args
assert call_args[1]["json"]["messages"][0]["role"] == "system"
assert call_args[1]["json"]["messages"][1]["role"] == "user"
@patch('requests.post')
def test_generate_with_retry(self, mock_post):
"""Test generation retries on timeout"""
mock_post.side_effect = [
requests.exceptions.Timeout(),
Mock(
status_code=200,
json=lambda: {"choices": [{"message": {"content": "Success"}}]}
)
]
config = LLMConfig(
model_name="test-model",
api_key="test-key",
base_url="https://api.example.com"
)
backend = RemoteLLMBackend(config)
result = backend.generate("Test prompt")
assert result == "Success"
assert mock_post.call_count == 2
@patch('requests.post')
def test_generate_failure_after_retries(self, mock_post):
"""Test generation fails after max retries"""
mock_post.side_effect = requests.exceptions.Timeout()
config = LLMConfig(
model_name="test-model",
api_key="test-key",
base_url="https://api.example.com"
)
backend = RemoteLLMBackend(config)
with pytest.raises(requests.exceptions.Timeout):
backend.generate("Test prompt")
assert mock_post.call_count == 3
class TestCodeFileHandler:
"""Test suite for CodeFileHandler"""
def test_read_file_not_found(self):
"""Test reading non-existent file raises error"""
handler = CodeFileHandler()
with pytest.raises(FileNotFoundError):
handler.read_file("/nonexistent/file.py")
def test_read_file_is_directory(self):
"""Test reading directory raises error"""
handler = CodeFileHandler()
with tempfile.TemporaryDirectory() as tmpdir:
with pytest.raises(ValueError, match="Not a file"):
handler.read_file(tmpdir)
def test_read_file_too_large(self):
"""Test reading oversized file raises error"""
handler = CodeFileHandler(max_file_size_mb=1)
with tempfile.NamedTemporaryFile(mode='w', delete=False) as f:
f.write("x" * (2 * 1024 * 1024))
temp_path = f.name
try:
with pytest.raises(ValueError, match="File too large"):
handler.read_file(temp_path)
finally:
os.unlink(temp_path)
def test_read_file_success(self):
"""Test successful file reading"""
handler = CodeFileHandler()
test_content = "def hello():\n print('Hello, World!')\n"
with tempfile.NamedTemporaryFile(
mode='w',
suffix='.py',
delete=False,
encoding='utf-8'
) as f:
f.write(test_content)
temp_path = f.name
try:
content, language = handler.read_file(temp_path)
assert content == test_content
assert language == "Python"
finally:
os.unlink(temp_path)
def test_detect_language_python(self):
"""Test Python language detection"""
handler = CodeFileHandler()
path = Path("test.py")
language = handler._detect_language(path)
assert language == "Python"
def test_detect_language_javascript(self):
"""Test JavaScript language detection"""
handler = CodeFileHandler()
path = Path("test.js")
language = handler._detect_language(path)
assert language == "JavaScript"
def test_detect_language_unknown(self):
"""Test unknown language detection"""
handler = CodeFileHandler()
path = Path("test.xyz")
language = handler._detect_language(path)
assert language == "Unknown"
class TestPromptBuilder:
"""Test suite for PromptBuilder"""
def test_system_prompt_creation(self):
"""Test system prompt is created"""
builder = PromptBuilder()
assert builder.system_prompt is not None
assert len(builder.system_prompt) > 0
assert "code analyst" in builder.system_prompt.lower()
def test_build_analysis_prompt(self):
"""Test comprehensive analysis prompt building"""
builder = PromptBuilder()
request = AnalysisRequest(
code_content="def test(): pass",
file_path="test.py",
language="Python"
)
prompt = builder.build_analysis_prompt(request)
assert "test.py" in prompt
assert "Python" in prompt
assert "def test(): pass" in prompt
assert "Purpose and Overview" in prompt
assert "Main Components" in prompt
def test_build_summary_prompt(self):
"""Test summary prompt building"""
builder = PromptBuilder()
request = AnalysisRequest(
code_content="def test(): pass",
file_path="test.py",
language="Python"
)
prompt = builder.build_summary_prompt(request)
assert "test.py" in prompt
assert "Python" in prompt
assert "summary" in prompt.lower()
assert "def test(): pass" in prompt
def test_build_functionality_prompt(self):
"""Test functionality prompt building"""
builder = PromptBuilder()
request = AnalysisRequest(
code_content="def test(): pass",
file_path="test.py",
language="Python"
)
prompt = builder.build_functionality_prompt(request)
assert "test.py" in prompt
assert "Python" in prompt
assert "functionality" in prompt.lower()
assert "def test(): pass" in prompt
class TestResponseParser:
"""Test suite for ResponseParser"""
def test_parse_comprehensive_analysis(self):
"""Test parsing comprehensive analysis response"""
parser = ResponseParser()
response = """
1. Purpose and Overview: This code implements a calculator with basic operations.
2. Main Components:
- Calculator class: Main class handling operations
- Operation enum: Defines supported operations
- Result class: Stores calculation results
3. Functionality Details:
- Addition of two numbers
- Subtraction of two numbers
- Multiplication of two numbers
- Division with zero checking
4. Dependencies and Integrations:
- Uses standard library math module
- Integrates with logging framework
5. Code Quality Observations:
- Well-structured and readable
- Good error handling
- Comprehensive documentation
6. Technical Characteristics:
- Uses design pattern: Strategy
- Thread-safe implementation
- Optimized for performance
7. Usage Context: Suitable for applications requiring basic mathematical operations.
"""
analysis = parser.parse_comprehensive_analysis(
response,
"calculator.py",
"Python"
)
assert analysis.file_path == "calculator.py"
assert analysis.language == "Python"
assert "calculator" in analysis.purpose.lower()
assert len(analysis.components) == 3
assert len(analysis.functionality) == 4
assert len(analysis.dependencies) == 2
assert len(analysis.quality_observations) == 3
assert len(analysis.technical_characteristics) == 3
assert "mathematical operations" in analysis.usage_context.lower()
def test_extract_list_items_with_bullets(self):
"""Test extracting list items with bullet points"""
parser = ResponseParser()
text = """
- First item here
- Second item here
- Third item here
"""
items = parser._extract_list_items(text)
assert len(items) == 3
assert items[0] == "First item here"
assert items[1] == "Second item here"
assert items[2] == "Third item here"
def test_extract_list_items_multiline(self):
"""Test extracting multiline list items"""
parser = ResponseParser()
text = """
- First item that spans
multiple lines here
- Second item also
spans multiple lines
"""
items = parser._extract_list_items(text)
assert len(items) == 2
assert "multiple lines" in items[0]
assert "multiple lines" in items[1]
def test_generate_summary_from_analysis(self):
"""Test summary generation from analysis"""
parser = ResponseParser()
analysis = CodeAnalysis(
file_path="test.py",
language="Python",
purpose="This is a test module",
components=["Component1", "Component2"],
functionality=["Function1", "Function2"]
)
summary = parser._generate_summary_from_analysis(analysis)
assert "test module" in summary.lower()
assert "Component1" in summary
assert "Function1" in summary
class TestCodeAnalyzer:
"""Test suite for CodeAnalyzer"""
def test_create_backend_remote(self):
"""Test creating remote backend"""
config = LLMConfig(
model_name="test-model",
api_key="test-key",
base_url="https://api.example.com"
)
with patch.object(RemoteLLMBackend, 'is_available', return_value=True):
analyzer = CodeAnalyzer(BackendType.REMOTE, config)
assert isinstance(analyzer.backend, RemoteLLMBackend)
def test_create_backend_local(self):
"""Test creating local backend"""
config = LLMConfig(model_name="test-model")
with patch.object(LocalLLMBackend, '__init__', return_value=None):
with patch.object(LocalLLMBackend, 'is_available', return_value=True):
analyzer = CodeAnalyzer(BackendType.LOCAL, config)
assert isinstance(analyzer.backend, LocalLLMBackend)
def test_backend_not_available_raises_error(self):
"""Test error when backend not available"""
config = LLMConfig(
model_name="test-model",
api_key="test-key",
base_url="https://api.example.com"
)
with patch.object(RemoteLLMBackend, 'is_available', return_value=False):
with pytest.raises(RuntimeError, match="not available"):
CodeAnalyzer(BackendType.REMOTE, config)
def test_analyze_file_comprehensive(self):
"""Test comprehensive file analysis"""
config = LLMConfig(
model_name="test-model",
api_key="test-key",
base_url="https://api.example.com"
)
test_code = "def hello(): pass"
llm_response = """
1. Purpose and Overview: Simple hello function
2. Main Components:
- hello function
3. Functionality Details:
- Does nothing currently
4. Dependencies and Integrations:
- None
5. Code Quality Observations:
- Very simple
6. Technical Characteristics:
- Basic function
7. Usage Context: Example code
"""
with tempfile.NamedTemporaryFile(
mode='w',
suffix='.py',
delete=False
) as f:
f.write(test_code)
temp_path = f.name
try:
with patch.object(RemoteLLMBackend, 'is_available', return_value=True):
with patch.object(RemoteLLMBackend, 'generate', return_value=llm_response):
analyzer = CodeAnalyzer(BackendType.REMOTE, config)
analysis = analyzer.analyze_file(temp_path)
assert analysis.file_path == temp_path
assert analysis.language == "Python"
assert len(analysis.purpose) > 0
finally:
os.unlink(temp_path)
def test_analyze_multiple_files(self):
"""Test analyzing multiple files"""
config = LLMConfig(
model_name="test-model",
api_key="test-key",
base_url="https://api.example.com"
)
files = []
for i in range(3):
with tempfile.NamedTemporaryFile(
mode='w',
suffix='.py',
delete=False
) as f:
f.write(f"def func{i}(): pass")
files.append(f.name)
try:
with patch.object(RemoteLLMBackend, 'is_available', return_value=True):
with patch.object(RemoteLLMBackend, 'generate', return_value="Analysis"):
analyzer = CodeAnalyzer(BackendType.REMOTE, config)
results = analyzer.analyze_multiple_files(files)
assert len(results) == 3
for file_path in files:
assert file_path in results
finally:
for file_path in files:
os.unlink(file_path)
class TestAnalysisReporter:
"""Test suite for AnalysisReporter"""
def test_generate_text_report(self):
"""Test text report generation"""
reporter = AnalysisReporter()
analysis = CodeAnalysis(
file_path="test.py",
language="Python",
purpose="Test purpose",
components=["Component1"],
functionality=["Function1"],
summary="Test summary"
)
report = reporter.generate_text_report(analysis)
assert "CODE ANALYSIS REPORT" in report
assert "test.py" in report
assert "Python" in report
assert "Test purpose" in report
assert "Component1" in report
def test_generate_json_report(self):
"""Test JSON report generation"""
reporter = AnalysisReporter()
analysis = CodeAnalysis(
file_path="test.py",
language="Python",
purpose="Test purpose",
summary="Test summary"
)
report = reporter.generate_json_report(analysis)
import json
data = json.loads(report)
assert data["file_path"] == "test.py"
assert data["language"] == "Python"
assert data["purpose"] == "Test purpose"
assert data["summary"] == "Test summary"
def test_generate_markdown_report(self):
"""Test Markdown report generation"""
reporter = AnalysisReporter()
analysis = CodeAnalysis(
file_path="test.py",
language="Python",
purpose="Test purpose",
components=["Component1"],
summary="Test summary"
)
report = reporter.generate_markdown_report(analysis)
assert "# Code Analysis Report" in report
assert "test.py" in report
assert "Python" in report
assert "Test purpose" in report
assert "- Component1" in report
def test_generate_report_to_file(self):
"""Test writing report to file"""
reporter = AnalysisReporter()
analysis = CodeAnalysis(
file_path="test.py",
language="Python",
summary="Test summary"
)
with tempfile.NamedTemporaryFile(mode='w', delete=False) as f:
temp_path = f.name
try:
reporter.generate_text_report(analysis, temp_path)
with open(temp_path, 'r') as f:
content = f.read()
assert "CODE ANALYSIS REPORT" in content
assert "test.py" in content
finally:
os.unlink(temp_path)
The test suite provides comprehensive coverage of all components. Each test class focuses on a single component, following the principle of test isolation. Tests use descriptive names that clearly indicate what behavior is being verified. The arrange-act-assert pattern structures each test method for clarity.
Mocking external dependencies like HTTP requests and file system operations ensures tests run quickly and reliably without external dependencies. The patch decorator from unittest.mock replaces real implementations with controllable mock objects during test execution. Temporary files created during tests are properly cleaned up in finally blocks to prevent resource leaks.
Parametrized tests could further reduce duplication when testing similar
scenarios with different inputs. Fixtures could extract common setup code
shared across multiple tests. Integration tests would complement these unit
tests by verifying component interactions in realistic scenarios.
No comments:
Post a Comment