Introduction to Tree-Sitter and Its Revolutionary Approach
Tree-sitter represents a paradigm shift in how we approach parsing programming languages. Unlike traditional parsers that operate in a single pass and produce abstract syntax trees that become stale the moment code changes, tree-sitter was designed from the ground up to handle the dynamic nature of modern code editors. It generates concrete syntax trees that can be incrementally updated as the user types, making it extraordinarily efficient for real-time applications like syntax highlighting, code folding, and intelligent code navigation.
The library was created by Max Brunsfeld at GitHub, initially to power the Atom editor's syntax highlighting capabilities. What makes tree-sitter special is its ability to produce error-tolerant parsers that continue functioning even when the code being parsed contains syntax errors. This is crucial for editor tooling because developers spend most of their time working with code that is temporarily invalid as they type and refactor.
Tree-sitter parsers are written in a declarative grammar format and compiled into efficient C code. The parsing algorithm is based on GLR parsing, which stands for Generalized Left-to-right Rightmost derivation parsing. This approach allows tree-sitter to handle ambiguous grammars and produce all possible parse trees, though in practice the grammars are designed to be unambiguous. The incremental parsing capability means that when you change a single character in a large file, tree-sitter only re-parses the affected portion of the syntax tree, making it blazingly fast.
Understanding the Core Architecture
Before we dive into practical usage, we need to understand the fundamental components that make up the tree-sitter ecosystem. The architecture consists of several layers that work together seamlessly to provide parsing capabilities.
At the foundation, we have the tree-sitter runtime library, which is written in C and provides the core parsing engine. This library knows how to execute the parsing algorithm and manage the syntax tree data structure, but it does not contain any language-specific knowledge. The runtime is language-agnostic and provides a consistent API regardless of which programming language you are parsing.
On top of the runtime, we have language-specific parser libraries. Each supported programming language has its own parser library, which is generated from a grammar definition. For example, there is tree-sitter-python for Python, tree-sitter-javascript for JavaScript, tree-sitter-rust for Rust, and so on. These parser libraries are also written in C and compiled as shared libraries that can be loaded dynamically.
The grammar definitions themselves are written in JavaScript using a special DSL provided by tree-sitter. When you run the tree-sitter CLI tool on a grammar file, it generates C code that implements the parser for that language. This generated code is highly optimized and includes both the lexer and the parser logic.
Finally, we have language bindings that allow you to use tree-sitter from various programming languages. The most commonly used bindings are for Node.js, Python, and Rust. These bindings wrap the C API in a more idiomatic interface for each language, handling memory management and providing convenient abstractions.
Setting Up Your Development Environment
To begin working with tree-sitter, you need to install both the core library and the language-specific parsers you want to use. The installation process varies depending on which language bindings you choose to work with. For this tutorial, we will focus on the Python bindings, as Python provides an excellent balance between ease of use and power.
First, you need to install the Python tree-sitter package. This can be done using pip, Python's package manager. The package provides both the runtime library and the Python bindings that wrap the C API. You should install it in a virtual environment to keep your dependencies isolated from other projects.
After installing the base package, you need to install or build the language parsers you want to use. Tree-sitter parsers can be installed in two ways. You can either install pre-compiled parser libraries if they are available for your platform, or you can build them from source. Building from source requires having a C compiler installed on your system, but it gives you access to the latest versions of all parsers and allows you to customize them if needed.
Let me show you the initial setup code that demonstrates how to import the necessary modules and prepare for parsing:
from tree_sitter import Language, Parser
import tree_sitter_python as tspython
import tree_sitter_javascript as tsjavascript
The imports above bring in the core tree-sitter components and the language-specific modules. The Language class represents a compiled parser for a specific programming language, while the Parser class is the main interface for parsing source code. The language-specific modules like tree_sitter_python contain the compiled parser code for each language.
Creating Your First Parser Instance
Now that we have the necessary imports, we can create a parser instance and configure it to parse a specific language. The parser object is reusable and can be configured to parse different languages by changing its language setting. However, for clarity and to avoid confusion, it is generally better to create separate parser instances for each language you want to parse.
Creating a parser involves two steps. First, you instantiate a Parser object. This creates an empty parser that is not yet configured for any specific language. Second, you set the language for the parser by calling the set_language method and passing it a Language object that represents the compiled grammar for your target language.
Here is how you would create a parser configured for Python:
python_parser = Parser()
python_parser.set_language(tspython.language())
The language() function from the tree_sitter_python module returns a Language object that encapsulates the Python grammar. When you call set_language, the parser is configured with all the rules and patterns needed to recognize Python syntax. At this point, the parser is ready to process Python source code.
You can create multiple parsers for different languages if your application needs to work with multiple programming languages. Each parser maintains its own state and can be used independently:
javascript_parser = Parser()
javascript_parser.set_language(tsjavascript.language())
Now we have two parsers ready to go, one for Python and one for JavaScript. The parsers are completely independent and can be used concurrently if needed.
Parsing Source Code and Obtaining the Syntax Tree
With a configured parser in hand, we can now parse actual source code. The parsing process takes a string or bytes object containing source code and produces a tree object that represents the hierarchical structure of the code. This tree is the concrete syntax tree, which includes all the details from the source code including whitespace, comments, and punctuation.
The parse method is the primary interface for parsing. It accepts the source code as bytes and returns a Tree object. It is important to note that tree-sitter expects the source code to be provided as bytes, not as a string. This is because tree-sitter works at the byte level to support all possible text encodings. If you have a string, you need to encode it to bytes before passing it to the parser.
Let us parse a simple Python function to see how this works:
source_code = b"""
def calculate_fibonacci(n):
if n <= 1:
return n
else:
return calculate_fibonacci(n - 1) + calculate_fibonacci(n - 2)
"""
tree = python_parser.parse(source_code)
The parse method returns a Tree object that contains the root node of the syntax tree. This tree represents the complete hierarchical structure of the source code. The tree is immutable, meaning that once created, it cannot be modified. If you need to parse updated code, you call parse again, and tree-sitter will efficiently reuse parts of the old tree that have not changed.
The Tree object has a root_node property that gives you access to the top-level node of the syntax tree. From this root node, you can traverse the entire tree structure to examine the code's syntax:
root_node = tree.root_node
The root node represents the entire source file. For Python, this is typically a module node. Every node in the tree has a type, which is a string that describes what kind of syntax element it represents. You can access this through the type property:
print(f"Root node type: {root_node.type}")
This would print something like "module" for Python code, indicating that the root node represents a Python module.
Navigating the Syntax Tree Structure
The syntax tree is a hierarchical data structure where each node can have zero or more children. Understanding how to navigate this tree is essential for extracting information from parsed code. Tree-sitter provides several properties and methods on each node to facilitate tree traversal.
Every node has a children property that returns a list of its direct child nodes. These children represent the immediate syntactic constituents of the parent node. For example, a function definition node would have children for the function name, parameters, and body.
Let us explore the structure of our parsed Python function:
def print_tree(node, indent=0):
print(' ' * indent + f'{node.type} [{node.start_point[0]}:{node.start_point[1]} - {node.end_point[0]}:{node.end_point[1]}]')
for child in node.children:
print_tree(child, indent + 2)
print_tree(root_node)
This recursive function walks through the entire tree and prints each node's type along with its position in the source code. The start_point and end_point properties give you the row and column numbers where the node begins and ends. These are tuples of the form (row, column), where both row and column are zero-indexed.
When you run this function on our fibonacci example, you will see the complete hierarchical structure. The root module node contains a function_definition node, which in turn contains nodes for the def keyword, the function name, the parameters, and the function body. The body contains an if_statement, which has its own children representing the condition, the if block, and the else block.
In addition to children, nodes also have properties for navigating to specific relatives in the tree. The parent property gives you the node's parent, allowing you to traverse upward in the tree. The next_sibling and prev_sibling properties let you move horizontally among nodes at the same level. These navigation properties return None when there is no such relative, so you should always check before using them.
Working with Named and Anonymous Nodes
Tree-sitter distinguishes between two categories of nodes in the syntax tree: named nodes and anonymous nodes. This distinction is important for understanding how to work with the tree effectively.
Named nodes represent significant syntactic constructs in the language. These are the nodes you typically care about when analyzing code structure. Examples include function definitions, class declarations, variable assignments, and expressions. Named nodes have meaningful type names that describe what they represent, such as "function_definition", "class_definition", or "binary_expression".
Anonymous nodes, on the other hand, represent literal tokens like keywords, operators, and punctuation. These nodes are called anonymous because they do not have unique type names. Instead, their type is the literal text they represent. For example, the def keyword in Python is an anonymous node with type "def", and the plus operator is an anonymous node with type "+".
You can check whether a node is named using the is_named property:
for child in root_node.children:
if child.is_named:
print(f"Named node: {child.type}")
else:
print(f"Anonymous node: {child.type}")
When traversing the tree, you often want to focus only on named nodes because they represent the semantic structure of the code. Tree-sitter provides convenience properties for this. The named_children property returns only the named children of a node, filtering out all anonymous nodes. Similarly, next_named_sibling and prev_named_sibling let you navigate among named siblings.
This distinction becomes particularly useful when you are building tools that need to understand code structure. For example, if you are building a code formatter, you care about the anonymous nodes because they represent the exact punctuation and keywords. But if you are building a code analyzer that looks for function calls, you primarily care about named nodes.
Extracting Text from Nodes
Once you have navigated to a node of interest in the syntax tree, you often need to extract the actual source code text that the node represents. Tree-sitter provides several ways to do this, each suited for different use cases.
The most straightforward way to get the text for a node is to use the text property. However, this property returns bytes, not a string, because tree-sitter works at the byte level. You need to decode these bytes to get a string:
function_def = root_node.children[0]
function_text = function_def.text.decode('utf-8')
print(f"Function text:\n{function_text}")
This will print the complete text of the function definition, exactly as it appears in the source code, including all whitespace and formatting.
For more precise text extraction, you can use the start_byte and end_byte properties along with the original source code. These properties give you the byte offsets where the node starts and ends in the source:
start = function_def.start_byte
end = function_def.end_byte
extracted_text = source_code[start:end].decode('utf-8')
This approach is particularly useful when you are working with very large files and want to avoid storing the text in every node. Instead, you keep the original source code and extract text on demand using byte offsets.
Another useful property is sexp, which returns a symbolic expression representation of the node and its subtree. This is primarily useful for debugging and understanding the tree structure:
print(function_def.sexp())
The symbolic expression shows the complete tree structure in a Lisp-like format, making it easy to see the relationships between nodes at a glance.
Querying the Syntax Tree with Patterns
While manual tree traversal is powerful, tree-sitter provides an even more elegant way to extract information from syntax trees: the query system. Queries allow you to declaratively specify patterns you want to find in the tree, similar to how regular expressions let you find patterns in text. However, tree-sitter queries operate on the tree structure, not on raw text, making them much more powerful for code analysis.
Queries are written in a special S-expression syntax that describes tree patterns. The query language supports matching node types, capturing specific nodes, and adding predicates to filter matches. Once you have defined a query, you can execute it against a syntax tree to find all matching subtrees.
Let us create a query that finds all function definitions in Python code:
query_text = """
(function_definition
name: (identifier) @function.name
parameters: (parameters) @function.params
body: (block) @function.body)
"""
query = tspython.language().query(query_text)
This query pattern matches function_definition nodes and captures three parts of each function: the name, the parameters, and the body. The @ symbol followed by a name creates a capture, which lets you refer to that specific node in the query results. The capture names are arbitrary and can be anything you choose.
To execute the query against our syntax tree, we use the captures method:
captures = query.captures(root_node)
for node, capture_name in captures:
print(f"Captured {capture_name}: {node.text.decode('utf-8')}")
The captures method returns a list of tuples, where each tuple contains a node that matched a capture and the name of the capture. This lets you process each captured node according to its role in the pattern.
Queries can be much more sophisticated than this simple example. You can use predicates to filter matches based on text content, you can match optional elements with question marks, you can match repeated elements with plus signs and asterisks, and you can use alternation to match multiple possible patterns. The query language is designed to be both expressive and efficient.
Handling Incremental Parsing for Real-Time Updates
One of tree-sitter's most powerful features is its ability to perform incremental parsing. When you make a small change to a large file, tree-sitter can update the syntax tree by only re-parsing the affected region, leaving the rest of the tree intact. This makes it extremely fast for real-time applications like syntax highlighting in editors.
To use incremental parsing, you need to keep track of the previous tree and pass it to the parse method when parsing updated code. Tree-sitter will use the old tree as a reference to determine which parts of the new tree can be reused:
old_tree = tree
modified_code = b"""
def calculate_fibonacci(n):
if n <= 1:
return n
else:
return calculate_fibonacci(n - 1) + calculate_fibonacci(n - 2)
def factorial(n):
if n <= 1:
return 1
return n * factorial(n - 1)
"""
new_tree = python_parser.parse(modified_code, old_tree)
By passing the old tree as the second argument to parse, we enable incremental parsing. Tree-sitter will compare the new source code with the old tree and identify the regions that have changed. Only those regions will be re-parsed, while unchanged regions will be copied from the old tree.
The performance improvement from incremental parsing can be dramatic. For a file with thousands of lines, changing a single character might only require re-parsing a few dozen nodes instead of the entire file. This is what makes tree-sitter suitable for real-time editor features.
You can also manually specify which region of the code has changed using the edit method on the old tree before parsing. This gives tree-sitter even more information to optimize the incremental parse:
old_tree.edit(
start_byte=100,
old_end_byte=100,
new_end_byte=150,
start_point=(5, 0),
old_end_point=(5, 0),
new_end_point=(7, 0)
)
new_tree = python_parser.parse(modified_code, old_tree)
The edit method tells tree-sitter exactly what changed: the byte positions and line/column positions of the edit. This allows tree-sitter to make even better decisions about what needs to be re-parsed.
Error Recovery and Handling Invalid Syntax
Unlike many traditional parsers that fail when they encounter a syntax error, tree-sitter is designed to be error-tolerant. It will always produce a syntax tree, even if the code contains errors. This is crucial for editor tooling because developers spend most of their time working with code that is temporarily invalid.
When tree-sitter encounters a syntax error, it inserts special ERROR nodes into the tree to mark the problematic regions. The parser tries to recover from the error and continue parsing the rest of the file. This means you can still get useful information about the valid parts of the code even when some parts are broken.
Let us parse some invalid Python code to see how this works:
invalid_code = b"""
def broken_function(
print("This is missing a closing parenthesis")
return 42
"""
error_tree = python_parser.parse(invalid_code)
error_root = error_tree.root_node
If we traverse this tree, we will find ERROR nodes marking the locations where the parser could not make sense of the code:
def find_errors(node, errors=None):
if errors is None:
errors = []
if node.type == 'ERROR' or node.is_missing:
errors.append({
'node': node,
'start': node.start_point,
'end': node.end_point,
'text': node.text.decode('utf-8') if node.text else '<missing>'
})
for child in node.children:
find_errors(child, errors)
return errors
errors = find_errors(error_root)
for error in errors:
print(f"Error at {error['start']}: {error['text']}")
The is_missing property indicates whether a node represents a syntax element that the parser expected but did not find in the source code. For example, if you have an if statement without a colon, tree-sitter might insert a missing node for the colon.
This error tolerance makes tree-sitter ideal for building development tools. You can provide syntax highlighting, code folding, and other features even when the user is in the middle of typing and the code is temporarily broken. The valid parts of the code will still be parsed correctly, and the ERROR nodes clearly mark the problematic regions.
Building a Practical Code Analysis Tool
Now that we understand the fundamentals of tree-sitter, let us build a practical tool that demonstrates these concepts in action. We will create a code complexity analyzer that examines Python code and calculates various complexity metrics. This tool will find all functions, count their lines of code, calculate cyclomatic complexity, and identify deeply nested code blocks.
The tool will demonstrate tree traversal, pattern matching with queries, text extraction, and error handling. It will be a complete, production-ready implementation that can analyze real Python codebases.
Our complexity analyzer will work by parsing Python files, finding all function definitions, and analyzing each function's structure. For cyclomatic complexity, we will count decision points like if statements, for loops, while loops, and boolean operators. For nesting depth, we will track how deeply nested the code blocks are within each function.
Let me show you the complete implementation of this tool:
from tree_sitter import Language, Parser, Node
import tree_sitter_python as tspython
from typing import List, Dict, Any
from pathlib import Path
import sys
class CodeComplexityAnalyzer:
"""
Analyzes Python code to calculate various complexity metrics including
cyclomatic complexity, lines of code, and maximum nesting depth.
"""
def __init__(self):
"""Initialize the analyzer with a Python parser."""
self.parser = Parser()
self.parser.set_language(tspython.language())
# Query to find all function definitions
self.function_query = tspython.language().query("""
(function_definition
name: (identifier) @function.name
parameters: (parameters) @function.params
body: (block) @function.body) @function.def
""")
# Query to find complexity-increasing constructs
self.complexity_query = tspython.language().query("""
[
(if_statement) @decision
(elif_clause) @decision
(for_statement) @decision
(while_statement) @decision
(except_clause) @decision
(with_statement) @decision
(boolean_operator) @decision
]
""")
def analyze_file(self, file_path: str) -> Dict[str, Any]:
"""
Analyze a Python file and return complexity metrics for all functions.
Args:
file_path: Path to the Python file to analyze
Returns:
Dictionary containing file-level and function-level metrics
"""
try:
with open(file_path, 'rb') as f:
source_code = f.read()
except IOError as e:
return {
'error': f'Failed to read file: {str(e)}',
'file_path': file_path
}
tree = self.parser.parse(source_code)
root_node = tree.root_node
# Check for parse errors
has_errors = self._check_for_errors(root_node)
# Find all functions and analyze them
functions = self._find_functions(root_node, source_code)
function_metrics = []
for func_info in functions:
metrics = self._analyze_function(func_info, source_code)
function_metrics.append(metrics)
# Calculate file-level statistics
total_functions = len(function_metrics)
total_loc = sum(m['lines_of_code'] for m in function_metrics)
avg_complexity = (sum(m['cyclomatic_complexity'] for m in function_metrics) / total_functions) if total_functions > 0 else 0
max_complexity = max((m['cyclomatic_complexity'] for m in function_metrics), default=0)
return {
'file_path': file_path,
'has_parse_errors': has_errors,
'total_functions': total_functions,
'total_lines_of_code': total_loc,
'average_complexity': round(avg_complexity, 2),
'max_complexity': max_complexity,
'functions': function_metrics
}
def _check_for_errors(self, node: Node) -> bool:
"""
Recursively check if the tree contains any ERROR nodes.
Args:
node: The node to check
Returns:
True if errors are found, False otherwise
"""
if node.type == 'ERROR' or node.is_missing:
return True
for child in node.children:
if self._check_for_errors(child):
return True
return False
def _find_functions(self, root_node: Node, source_code: bytes) -> List[Dict[str, Any]]:
"""
Find all function definitions in the syntax tree.
Args:
root_node: The root node of the syntax tree
source_code: The original source code as bytes
Returns:
List of dictionaries containing function information
"""
functions = []
captures = self.function_query.captures(root_node)
# Group captures by function
current_function = {}
for node, capture_name in captures:
if capture_name == 'function.def':
if current_function:
functions.append(current_function)
current_function = {'node': node}
elif capture_name == 'function.name':
current_function['name'] = node.text.decode('utf-8')
current_function['start_line'] = node.start_point[0] + 1
elif capture_name == 'function.params':
current_function['params'] = node.text.decode('utf-8')
elif capture_name == 'function.body':
current_function['body'] = node
if current_function:
functions.append(current_function)
return functions
def _analyze_function(self, func_info: Dict[str, Any], source_code: bytes) -> Dict[str, Any]:
"""
Analyze a single function and calculate its complexity metrics.
Args:
func_info: Dictionary containing function information
source_code: The original source code as bytes
Returns:
Dictionary containing complexity metrics for the function
"""
func_node = func_info['node']
body_node = func_info['body']
# Calculate lines of code
start_line = func_node.start_point[0]
end_line = func_node.end_point[0]
lines_of_code = end_line - start_line + 1
# Count non-empty, non-comment lines
func_text = func_node.text.decode('utf-8')
actual_lines = [line.strip() for line in func_text.split('\n')]
non_empty_lines = len([line for line in actual_lines if line and not line.startswith('#')])
# Calculate cyclomatic complexity
complexity = self._calculate_cyclomatic_complexity(body_node)
# Calculate maximum nesting depth
max_depth = self._calculate_max_nesting_depth(body_node)
# Count parameters
param_text = func_info.get('params', '()')
param_count = self._count_parameters(param_text)
return {
'name': func_info['name'],
'start_line': func_info['start_line'],
'lines_of_code': lines_of_code,
'non_empty_lines': non_empty_lines,
'cyclomatic_complexity': complexity,
'max_nesting_depth': max_depth,
'parameter_count': param_count,
'complexity_rating': self._rate_complexity(complexity)
}
def _calculate_cyclomatic_complexity(self, node: Node) -> int:
"""
Calculate the cyclomatic complexity of a code block.
Cyclomatic complexity starts at 1 and increases by 1 for each
decision point (if, elif, for, while, except, with, boolean operator).
Args:
node: The node representing the code block
Returns:
The cyclomatic complexity value
"""
complexity = 1 # Base complexity
captures = self.complexity_query.captures(node)
for captured_node, capture_name in captures:
if capture_name == 'decision':
complexity += 1
return complexity
def _calculate_max_nesting_depth(self, node: Node, current_depth: int = 0) -> int:
"""
Calculate the maximum nesting depth of control structures.
Args:
node: The current node being examined
current_depth: The current nesting depth
Returns:
The maximum nesting depth found
"""
nesting_types = {
'if_statement', 'elif_clause', 'else_clause',
'for_statement', 'while_statement', 'with_statement',
'try_statement', 'except_clause', 'finally_clause'
}
max_depth = current_depth
if node.type in nesting_types:
current_depth += 1
max_depth = current_depth
for child in node.children:
child_depth = self._calculate_max_nesting_depth(child, current_depth)
max_depth = max(max_depth, child_depth)
return max_depth
def _count_parameters(self, param_text: str) -> int:
"""
Count the number of parameters in a parameter list.
Args:
param_text: The text of the parameter list
Returns:
The number of parameters
"""
# Remove parentheses and whitespace
param_text = param_text.strip('() \t\n')
if not param_text:
return 0
# Simple count by splitting on commas
# This is a simplified approach; a more robust implementation
# would parse the parameter list properly
params = [p.strip() for p in param_text.split(',')]
return len([p for p in params if p])
def _rate_complexity(self, complexity: int) -> str:
"""
Rate the complexity level based on the cyclomatic complexity value.
Args:
complexity: The cyclomatic complexity value
Returns:
A string rating the complexity level
"""
if complexity <= 5:
return 'Low'
elif complexity <= 10:
return 'Moderate'
elif complexity <= 20:
return 'High'
else:
return 'Very High'
def format_report(self, analysis_result: Dict[str, Any]) -> str:
"""
Format the analysis results into a human-readable report.
Args:
analysis_result: The dictionary returned by analyze_file
Returns:
A formatted string report
"""
if 'error' in analysis_result:
return f"Error analyzing {analysis_result['file_path']}: {analysis_result['error']}"
report_lines = []
report_lines.append('=' * 80)
report_lines.append(f"Code Complexity Analysis Report")
report_lines.append(f"File: {analysis_result['file_path']}")
report_lines.append('=' * 80)
report_lines.append('')
if analysis_result['has_parse_errors']:
report_lines.append('WARNING: This file contains syntax errors!')
report_lines.append('')
report_lines.append('File-Level Metrics:')
report_lines.append(f" Total Functions: {analysis_result['total_functions']}")
report_lines.append(f" Total Lines of Code: {analysis_result['total_lines_of_code']}")
report_lines.append(f" Average Complexity: {analysis_result['average_complexity']}")
report_lines.append(f" Maximum Complexity: {analysis_result['max_complexity']}")
report_lines.append('')
if analysis_result['functions']:
report_lines.append('Function-Level Metrics:')
report_lines.append('')
for func in analysis_result['functions']:
report_lines.append(f"Function: {func['name']} (line {func['start_line']})")
report_lines.append(f" Lines of Code: {func['lines_of_code']}")
report_lines.append(f" Non-Empty Lines: {func['non_empty_lines']}")
report_lines.append(f" Cyclomatic Complexity: {func['cyclomatic_complexity']} ({func['complexity_rating']})")
report_lines.append(f" Max Nesting Depth: {func['max_nesting_depth']}")
report_lines.append(f" Parameter Count: {func['parameter_count']}")
report_lines.append('')
report_lines.append('=' * 80)
return '\n'.join(report_lines)
def main():
"""
Main entry point for the code complexity analyzer.
Analyzes Python files provided as command-line arguments.
"""
if len(sys.argv) < 2:
print("Usage: python analyzer.py <python_file> [<python_file> ...]")
print("Example: python analyzer.py my_script.py")
sys.exit(1)
analyzer = CodeComplexityAnalyzer()
for file_path in sys.argv[1:]:
if not Path(file_path).exists():
print(f"Error: File not found: {file_path}")
continue
if not file_path.endswith('.py'):
print(f"Warning: {file_path} does not appear to be a Python file")
result = analyzer.analyze_file(file_path)
report = analyzer.format_report(result)
print(report)
print()
if __name__ == '__main__':
main()
This complete implementation demonstrates all the key concepts we have covered throughout this tutorial. The CodeComplexityAnalyzer class encapsulates all the functionality needed to parse Python files and extract meaningful metrics from them.
The constructor initializes the parser and defines two queries. The first query finds all function definitions and captures their components. The second query finds all the language constructs that contribute to cyclomatic complexity. By using queries instead of manual tree traversal, we make the code more declarative and easier to understand.
The analyze_file method is the main entry point. It reads the source file, parses it, checks for errors, finds all functions, and analyzes each one. It then aggregates the results into file-level statistics. This method demonstrates error handling, file I/O, and coordination of multiple analysis steps.
The _find_functions method shows how to work with query results. It processes the captures from the function query and groups them by function, building up a dictionary of information about each function. This demonstrates the practical use of tree-sitter's query system.
The _analyze_function method calculates various metrics for a single function. It shows how to extract text from nodes, count lines, and use queries to find specific patterns. The cyclomatic complexity calculation uses a query to find all decision points, while the nesting depth calculation uses recursive tree traversal.
The _calculate_max_nesting_depth method demonstrates recursive tree traversal with state tracking. It maintains a current depth counter and updates it as it encounters nesting constructs. This pattern is common when you need to track context while traversing the tree.
The format_report method shows how to present the analysis results in a human-readable format. It generates a structured text report that clearly presents both file-level and function-level metrics.
The main function provides a command-line interface to the analyzer. It processes multiple files and prints reports for each one. This makes the tool immediately useful as a standalone utility.
To use this analyzer, you would save it to a file and run it from the command line, passing one or more Python files as arguments. The tool will analyze each file and print a detailed complexity report. This demonstrates how tree-sitter can be used to build practical, production-ready code analysis tools.
The implementation follows clean code principles with clear separation of concerns, descriptive names, comprehensive documentation, and proper error handling. Each method has a single, well-defined responsibility, making the code easy to understand and maintain.
This example only scratches the surface of what you can do with tree-sitter. You could extend this analyzer to detect code smells, find security vulnerabilities, enforce coding standards, or generate documentation. The key insight is that tree-sitter gives you a reliable, efficient way to understand code structure, which is the foundation for all kinds of developer tools.
