PREFACE: WHY ERROR REPORTING IS THE SOUL OF A COMPILER
Ask any seasoned programmer what separates a good compiler from a great one, and I can almost guarantee the answer will not be "how fast the generated code runs." It will be something like "it actually tells me what I did wrong." I have been building compilers and language tools for years, and the single most consistent piece of feedback I hear from users is about error messages. Not about optimization quality. Not about compilation speed. About whether the compiler speaks to them like a helpful colleague or like an indifferent bureaucrat stamping rejection forms.
The history of compiler error messages is littered with cautionary tales. Early Fortran compilers would emit a single cryptic code like "ERR 47" and leave the programmer to consult a paper manual. Early C compilers were notorious for cascading errors where a single missing semicolon would produce forty lines of noise. Even today, mature compilers sometimes emit messages that are technically accurate but practically useless. We can do better, and with the tools available to us now, there is really no excuse not to.
The good news is that ANTLR4 and LLVM together provide extraordinarily rich infrastructure for building error reporting that is both precise and deeply informative. ANTLR4 gives us token-level location information and a clean extension point for intercepting parse errors. LLVM gives us a mature debug information system and a diagnostic handler mechanism. The challenge is not finding the raw material; it is assembling it into a coherent system that serves two very different audiences simultaneously: the user of your compiler, who wants to know what is wrong with their source code, and the developer of your compiler, who wants to know what went wrong inside the compiler itself.
This article walks you through the entire journey. We will use a small but realistic example language called MotorLang throughout, a domain-specific language for motor control systems. MotorLang supports integer and floating-point variables, arithmetic expressions, function definitions, and a simple type system. It is complex enough to illustrate all the interesting error scenarios without drowning us in language semantics.
By the end, you will have a complete, working error reporting infrastructure that you can adapt for your own compiler projects. Every piece of code shown here has been carefully checked. Let us get into it.
CHAPTER ONE: SETTING UP THE PROJECT -- DEPENDENCIES AND BUILD SYSTEM
Before we write a single line of compiler code, we need to get our build environment right. There is nothing more demoralizing than spending an afternoon debugging a CMake configuration when you could be building something interesting. So let us get this out of the way cleanly and completely.
Our compiler depends on three major components. The first is ANTLR4, specifically the ANTLR4 C++ runtime library and the ANTLR4 tool itself (a Java jar) for generating the lexer and parser from our grammar. The second is LLVM, version 16 or later, which we use for IR generation, optimization, and code emission. The third is a standard C++17 compiler, either GCC 9+ or Clang 10+.
On Ubuntu 22.04 or Debian Bookworm, you can install the prerequisites like this:
# Install LLVM 16 and its development headers
sudo apt-get install -y llvm-16 llvm-16-dev clang-16
# Install Java for running the ANTLR4 tool
sudo apt-get install -y default-jdk
# Download the ANTLR4 tool jar
wget https://www.antlr.org/download/antlr-4.13.1-complete.jar \
-O /usr/local/lib/antlr-4.13.1-complete.jar
# Install the ANTLR4 C++ runtime from source
git clone https://github.com/antlr/antlr4.git
cd antlr4/runtime/Cpp
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)
sudo make install
cd ../../../..
On macOS with Homebrew, the process is simpler:
brew install llvm antlr4-cpp-runtime
brew install --cask temurin # Java for the ANTLR4 tool jar
wget https://www.antlr.org/download/antlr-4.13.1-complete.jar \
-O /usr/local/lib/antlr-4.13.1-complete.jar
Now let us look at the project structure. I like to keep generated files clearly separated from hand-written source, and I like to keep the diagnostic infrastructure in its own directory since it is genuinely reusable across compiler projects.
motorlang/
CMakeLists.txt
grammar/
MotorLang.g4
src/
diag/
SourceLocation.h
SourceManager.h
SourceManager.cpp
Diagnostic.h
DiagnosticEngine.h
DiagnosticEngine.cpp
TerminalDiagnosticConsumer.h
TerminalDiagnosticConsumer.cpp
JsonDiagnosticConsumer.h
JsonDiagnosticConsumer.cpp
DiagnosticCodes.h
frontend/
AntlrDiagnosticListener.h
ASTBuilder.h
ASTBuilder.cpp
CustomErrorStrategy.h
ast/
AST.h
sema/
SymbolTable.h
SymbolTable.cpp
TypeChecker.h
TypeChecker.cpp
codegen/
IRGenerator.h
IRGenerator.cpp
IRVerification.h
IRVerification.cpp
LLVMDiagnosticBridge.h
LLVMDiagnosticBridge.cpp
CompilerDriver.h
CompilerDriver.cpp
main.cpp
generated/ (populated by ANTLR4 tool at build time)
tests/
ErrorReportingTests.cpp
test_type_errors.ml
test_undeclared.ml
The CMakeLists.txt ties everything together. I have written this to be self-contained: it invokes the ANTLR4 tool automatically during the build, so you never have to remember to regenerate the parser manually after editing the grammar.
cmake_minimum_required(VERSION 3.20)
project(MotorLangCompiler CXX)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
# ---------------------------------------------------------------
# Find LLVM. We require version 16 or later.
# ---------------------------------------------------------------
find_package(LLVM 16 REQUIRED CONFIG)
message(STATUS "Found LLVM ${LLVM_PACKAGE_VERSION}")
message(STATUS "Using LLVMConfig.cmake in: ${LLVM_DIR}")
include_directories(${LLVM_INCLUDE_DIRS})
add_definitions(${LLVM_DEFINITIONS})
# ---------------------------------------------------------------
# Find the ANTLR4 C++ runtime library.
# ---------------------------------------------------------------
find_package(antlr4-runtime REQUIRED)
# ---------------------------------------------------------------
# Locate the ANTLR4 tool jar for grammar compilation.
# ---------------------------------------------------------------
find_file(ANTLR4_JAR
NAMES antlr-4.13.1-complete.jar
PATHS /usr/local/lib /usr/local/share/java
DOC "Path to the ANTLR4 complete jar"
)
if(NOT ANTLR4_JAR)
message(FATAL_ERROR
"antlr-4.13.1-complete.jar not found. "
"Download it from https://www.antlr.org/download.html "
"and place it in /usr/local/lib/")
endif()
# ---------------------------------------------------------------
# Generate the lexer and parser from the grammar file.
# The ANTLR4 tool is invoked at configure time so that the
# generated files are available when the compiler builds.
# ---------------------------------------------------------------
set(GRAMMAR_FILE ${CMAKE_SOURCE_DIR}/grammar/MotorLang.g4)
set(GENERATED_DIR ${CMAKE_SOURCE_DIR}/generated)
file(MAKE_DIRECTORY ${GENERATED_DIR})
add_custom_command(
OUTPUT
${GENERATED_DIR}/MotorLangLexer.cpp
${GENERATED_DIR}/MotorLangLexer.h
${GENERATED_DIR}/MotorLangParser.cpp
${GENERATED_DIR}/MotorLangParser.h
${GENERATED_DIR}/MotorLangBaseVisitor.h
${GENERATED_DIR}/MotorLangVisitor.h
COMMAND java -jar ${ANTLR4_JAR}
-Dlanguage=Cpp
-visitor
-no-listener
-o ${GENERATED_DIR}
${GRAMMAR_FILE}
DEPENDS ${GRAMMAR_FILE}
COMMENT "Generating ANTLR4 lexer and parser from MotorLang.g4"
VERBATIM
)
add_custom_target(GenerateParser
DEPENDS
${GENERATED_DIR}/MotorLangLexer.cpp
${GENERATED_DIR}/MotorLangParser.cpp
)
# ---------------------------------------------------------------
# Collect all source files.
# ---------------------------------------------------------------
set(COMPILER_SOURCES
src/diag/SourceManager.cpp
src/diag/DiagnosticEngine.cpp
src/diag/TerminalDiagnosticConsumer.cpp
src/diag/JsonDiagnosticConsumer.cpp
src/frontend/ASTBuilder.cpp
src/sema/SymbolTable.cpp
src/sema/TypeChecker.cpp
src/codegen/IRGenerator.cpp
src/codegen/IRVerification.cpp
src/codegen/LLVMDiagnosticBridge.cpp
src/CompilerDriver.cpp
src/main.cpp
${GENERATED_DIR}/MotorLangLexer.cpp
${GENERATED_DIR}/MotorLangParser.cpp
)
# ---------------------------------------------------------------
# Build the compiler executable.
# ---------------------------------------------------------------
add_executable(motorlangc ${COMPILER_SOURCES})
add_dependencies(motorlangc GenerateParser)
target_include_directories(motorlangc PRIVATE
src
${GENERATED_DIR}
${antlr4-runtime_INCLUDE_DIRS}
)
# Link LLVM components. We need core, support, IR builder,
# the native target, and the verifier.
llvm_map_components_to_libnames(LLVM_LIBS
core support irreader nativecodegen
passes analysis target
)
target_link_libraries(motorlangc PRIVATE
${LLVM_LIBS}
antlr4-runtime
)
# Enable useful warnings.
target_compile_options(motorlangc PRIVATE
-Wall -Wextra -Wpedantic -Wno-unused-parameter
)
# ---------------------------------------------------------------
# Build the test executable.
# ---------------------------------------------------------------
add_executable(motorlangc_tests
tests/ErrorReportingTests.cpp
src/diag/SourceManager.cpp
src/diag/DiagnosticEngine.cpp
src/diag/TerminalDiagnosticConsumer.cpp
src/diag/JsonDiagnosticConsumer.cpp
)
target_include_directories(motorlangc_tests PRIVATE src)
target_link_libraries(motorlangc_tests PRIVATE ${LLVM_LIBS})
# ---------------------------------------------------------------
# Build instructions (printed at configure time as a reminder).
# ---------------------------------------------------------------
message(STATUS "")
message(STATUS "Build with: cmake --build . --parallel")
message(STATUS "Run tests: ./motorlangc_tests")
message(STATUS "Compile: ./motorlangc <source.ml>")
message(STATUS "")
To build and run the compiler from scratch:
# Clone or create the project directory, then:
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Debug
cmake --build . --parallel
# Run the test suite
./motorlangc_tests
# Compile a MotorLang source file
./motorlangc ../tests/test_type_errors.ml
# Compile with JSON diagnostic output (useful for IDE integration)
./motorlangc --json-output diags.json ../tests/test_type_errors.ml
# Compile in developer mode (shows compiler-internal information)
./motorlangc --developer ../tests/test_type_errors.ml
# Treat all warnings as errors
./motorlangc -Werror ../tests/test_type_errors.ml
CHAPTER TWO: THE GRAMMAR FILE
Every compiler built on ANTLR4 starts with a grammar. Our MotorLang grammar is small enough to be readable in one sitting but rich enough to generate all the interesting error scenarios we want to demonstrate. I have annotated it heavily because the grammar is the contract between the language designer and the parser generator, and that contract deserves documentation.
// grammar/MotorLang.g4
//
// Grammar for MotorLang: a small domain-specific language for
// motor control systems. Supports integer and floating-point
// variables, arithmetic expressions, function definitions,
// conditional statements, and loops.
//
// ANTLR4 naming conventions:
// - Parser rules start with a lowercase letter.
// - Lexer rules start with an uppercase letter.
// - Labeled alternatives (# Label) generate separate visitor methods.
grammar MotorLang;
// ---------------------------------------------------------------
// Parser rules
// ---------------------------------------------------------------
// The top-level rule: a program is a sequence of declarations.
program
: declaration* EOF
;
declaration
: functionDecl
| varDeclStmt
;
functionDecl
: type IDENTIFIER '(' paramList? ')' block
;
paramList
: param (',' param)*
;
param
: type IDENTIFIER
;
block
: '{' statement* '}'
;
statement
: varDeclStmt
| assignStmt
| ifStmt
| whileStmt
| returnStmt
| exprStmt
;
varDeclStmt
: type IDENTIFIER ('=' expr)? ';'
;
assignStmt
: IDENTIFIER '=' expr ';'
;
ifStmt
: 'if' '(' expr ')' block ('else' block)?
;
whileStmt
: 'while' '(' expr ')' block
;
returnStmt
: 'return' expr? ';'
;
exprStmt
: expr ';'
;
// Expressions use labeled alternatives so that ANTLR4 generates
// a separate visitor method for each operator precedence level.
// Higher alternatives have lower precedence (ANTLR4 resolves
// ambiguity by choosing the first matching alternative).
expr
: expr op=('*' | '/') expr # MulDivExpr
| expr op=('+' | '-') expr # AddSubExpr
| expr op=('<' | '>' | '<='
| '>=' | '==' | '!=') expr # CompareExpr
| '(' expr ')' # ParenExpr
| IDENTIFIER '(' argList? ')' # CallExpr
| IDENTIFIER # VarExpr
| INT_LITERAL # IntLiteralExpr
| FLOAT_LITERAL # FloatLiteralExpr
;
argList
: expr (',' expr)*
;
type
: 'int'
| 'float'
| 'double'
| 'bool'
;
// ---------------------------------------------------------------
// Lexer rules
// ---------------------------------------------------------------
// Keywords must appear before IDENTIFIER so that ANTLR4 gives
// them priority during lexing.
KW_IF : 'if' ;
KW_ELSE : 'else' ;
KW_WHILE : 'while' ;
KW_RETURN : 'return' ;
KW_INT : 'int' ;
KW_FLOAT : 'float' ;
KW_DOUBLE : 'double' ;
KW_BOOL : 'bool' ;
IDENTIFIER
: [a-zA-Z_][a-zA-Z_0-9]*
;
// Integer literals: decimal, hexadecimal, and binary.
INT_LITERAL
: [0-9]+
| '0x' [0-9a-fA-F]+
| '0b' [01]+
;
// Floating-point literals: require at least one digit on one side
// of the decimal point.
FLOAT_LITERAL
: [0-9]+ '.' [0-9]*
| '.' [0-9]+
| [0-9]+ ('.' [0-9]*)? [eE] [+-]? [0-9]+
;
// Comments and whitespace are skipped (not passed to the parser).
LINE_COMMENT : '//' ~[\r\n]* -> skip ;
BLOCK_COMMENT : '/*' .*? '*/' -> skip ;
WS : [ \t\r\n]+ -> skip ;
CHAPTER THREE: THE ARCHITECTURE OF AN ERROR REPORTING SYSTEM
Before writing a single line of compiler logic, the thoughtful compiler engineer designs the error reporting infrastructure. I cannot stress this enough: if you bolt error reporting on at the end, you will spend the rest of the project's life paying down that technical debt. The error reporting system is the connective tissue of the entire compiler. Get it right at the start, and every subsequent component slots into it naturally.
The central concept is the source location. Every piece of information that a compiler processes -- every token, every AST node, every IR instruction -- has a home in the original source text. That home is described by a source location, which at minimum records the file name, the line number, and the column number. In a sophisticated system it also records the end position of the relevant text span, so that error messages can underline an entire expression rather than just pointing at a single character.
Let us define a SourceLocation structure that will serve as the foundation for everything that follows. I keep this structure deliberately small because it will be embedded in millions of AST nodes and IR instructions during a typical compilation, and memory pressure matters.
// src/diag/SourceLocation.h
//
// Represents a precise location within a source file.
// Kept small and trivially copyable so it can be embedded
// cheaply in every AST node and IR instruction.
#pragma once
#include <string>
#include <memory>
#include <cstdint>
namespace diag {
struct SourceLocation {
// The path to the source file, stored as a shared pointer
// to avoid copying the string for every token. All tokens
// from the same file share a single string allocation.
std::shared_ptr<const std::string> filePath;
// Line numbers are 1-based, matching what editors display.
uint32_t line = 0;
// Column numbers are 1-based, measured in UTF-8 code units.
uint32_t column = 0;
// The byte offset from the start of the file, useful for
// computing text spans and for LSP-based IDE integration.
uint32_t offset = 0;
// Constructs an invalid (sentinel) location.
SourceLocation() = default;
SourceLocation(std::shared_ptr<const std::string> file,
uint32_t ln, uint32_t col, uint32_t off)
: filePath(std::move(file))
, line(ln)
, column(col)
, offset(off)
{}
bool isValid() const {
return filePath != nullptr && line > 0;
}
std::string toString() const {
if (!isValid()) return "<unknown location>";
return *filePath + ":"
+ std::to_string(line) + ":"
+ std::to_string(column);
}
};
// A SourceRange captures the beginning and end of a syntactic
// construct, enabling the compiler to underline entire expressions
// in error messages rather than just marking a single point.
struct SourceRange {
SourceLocation begin;
SourceLocation end;
bool isValid() const {
return begin.isValid() && end.isValid();
}
};
} // namespace diag
The SourceLocation and SourceRange types are the atoms of our error system. Notice that the file path is stored as a shared pointer. This is a deliberate optimization: in a real compiler you might have hundreds of thousands of tokens, and each one needs to know which file it came from. Copying a full file path string for each token would be wasteful. The shared pointer ensures that all tokens from the same file share a single string allocation, which is both memory-efficient and cache-friendly.
Now we need to think about what an error actually is. An error is not just a location and a message. It has a severity level, a unique diagnostic code, optional notes that provide additional context, and optionally a suggested fix. The fix-it hint is particularly important for IDE integration: modern editors like VS Code can apply fix-it hints automatically with a single click, which dramatically improves the user experience.
// src/diag/Diagnostic.h
//
// Models a single compiler diagnostic (error, warning, note, etc.)
// with full location information and structured metadata.
// This is the central data type of the entire error reporting system.
#pragma once
#include "SourceLocation.h"
#include <string>
#include <vector>
#include <optional>
namespace diag {
// The severity of a diagnostic determines how it is displayed
// and whether it prevents code generation from proceeding.
enum class Severity {
Note, // Supplementary information attached to another diagnostic.
Remark, // Informational message about optimization decisions.
Warning, // Potentially problematic but not fatal.
Error, // A definite problem; compilation cannot succeed.
Fatal, // An unrecoverable error; compilation must stop immediately.
InternalError // A bug in the compiler itself; always accompanied by
// a request to file a bug report.
};
// A FixItHint describes a mechanical transformation that would fix
// the reported problem. IDEs can apply these automatically.
struct FixItHint {
SourceRange rangeToReplace;
std::string replacementText;
// Factory methods make the intent clear at the call site.
static FixItHint makeInsertion(SourceLocation loc,
std::string text) {
FixItHint h;
h.rangeToReplace = { loc, loc };
h.replacementText = std::move(text);
return h;
}
static FixItHint makeReplacement(SourceRange range,
std::string text) {
FixItHint h;
h.rangeToReplace = std::move(range);
h.replacementText = std::move(text);
return h;
}
static FixItHint makeDeletion(SourceRange range) {
FixItHint h;
h.rangeToReplace = std::move(range);
h.replacementText = "";
return h;
}
};
// A single diagnostic message with all its associated metadata.
struct Diagnostic {
// A unique numeric code for this diagnostic, e.g., 3042.
// Codes are grouped by category:
// 1000-1999 = lexer errors (LEX)
// 2000-2999 = parser errors (PAR)
// 3000-3999 = semantic errors (SEM)
// 4000-4999 = type errors (TYP)
// 5000-5999 = backend errors (GEN)
// 9000-9999 = internal errors (ICE)
uint32_t code = 0;
Severity severity = Severity::Error;
// The primary location where the problem was detected.
SourceLocation location;
// The range of source text most relevant to this diagnostic.
SourceRange range;
// The human-readable message. This should be a complete sentence
// that describes the problem without jargon where possible.
std::string message;
// Additional notes that provide context. For example, if a
// variable is used before declaration, a note can point to
// the location where the declaration was expected.
std::vector<Diagnostic> notes;
// Optional mechanical fix suggestions for IDE integration.
std::vector<FixItHint> fixItHints;
// The name of the compiler phase that generated this diagnostic.
// This is invaluable for compiler developers debugging the
// compiler itself, and is shown in --developer mode.
std::string phase;
// For InternalError severity, this captures the C++ source
// location within the compiler where the error was detected.
std::optional<std::string> compilerSourceLocation;
};
} // namespace diag
The Diagnostic structure is deliberately rich. The separation between the primary location and the source range allows us to point at both a specific token (the location) and the broader expression it belongs to (the range). The notes vector enables the cascading diagnostic style used by Clang, where the primary error is followed by supplementary notes that explain the context. The fixItHints vector is what enables IDE integration.
The phase field is something I added after spending a frustrating afternoon trying to figure out whether a spurious error was coming from the type checker or the IR generator. Now every diagnostic carries a breadcrumb that tells me exactly which phase of the compiler produced it. In developer mode, this is printed alongside the message. It sounds like a small thing, but it has saved me hours of debugging.
CHAPTER FOUR: THE DIAGNOSTIC ENGINE AND SOURCE MANAGER
With our data types defined, we need two infrastructure components: the SourceManager, which loads and caches source files and provides line-text lookup, and the DiagnosticEngine, which collects diagnostics, enforces error limits, and dispatches them to consumers.
Let us start with the SourceManager. Its most important job is providing the source text for a given line number, because the terminal consumer needs this to print the offending line with a caret underneath it.
// src/diag/SourceManager.h
//
// Manages source file content and provides efficient line/column
// lookup. The SourceManager is the bridge between raw byte offsets
// and human-readable line:column positions.
#pragma once
#include "SourceLocation.h"
#include <string>
#include <vector>
#include <unordered_map>
#include <memory>
namespace diag {
class SourceManager {
public:
// Load a source file from disk and register it.
// Returns a shared pointer to the canonical file path string,
// which is used as the key in SourceLocation objects.
// Returns nullptr if the file cannot be opened.
std::shared_ptr<const std::string>
loadFile(const std::string& path);
// Register source text that was not read from a file,
// for example, a string passed via -e on the command line
// or source text used in unit tests.
std::shared_ptr<const std::string>
registerInMemorySource(const std::string& virtualPath,
std::string content);
// Retrieve a specific line of source text (1-based line number).
// Returns an empty string if the line does not exist or the
// file has not been loaded.
std::string getLine(const std::string& filePath,
uint32_t lineNumber) const;
// Returns the total number of lines in a loaded file.
uint32_t lineCount(const std::string& filePath) const;
private:
struct FileData {
std::shared_ptr<const std::string> canonicalPath;
std::string content;
// lineOffsets[i] is the byte offset of the first character
// of line (i+1). Index 0 corresponds to line 1.
std::vector<uint32_t> lineOffsets;
};
void buildLineIndex(FileData& data);
std::unordered_map<std::string, FileData> files_;
};
} // namespace diag
// src/diag/SourceManager.cpp
#include "SourceManager.h"
#include <fstream>
#include <sstream>
namespace diag {
std::shared_ptr<const std::string>
SourceManager::loadFile(const std::string& path) {
auto it = files_.find(path);
if (it != files_.end()) {
return it->second.canonicalPath;
}
FileData data;
data.canonicalPath = std::make_shared<const std::string>(path);
std::ifstream stream(path);
if (!stream.is_open()) {
// Register the path even if the file cannot be opened.
// The caller handles the error; we just need the path key.
files_[path] = std::move(data);
return files_[path].canonicalPath;
}
std::ostringstream buf;
buf << stream.rdbuf();
data.content = buf.str();
buildLineIndex(data);
auto result = data.canonicalPath;
files_[path] = std::move(data);
return result;
}
std::shared_ptr<const std::string>
SourceManager::registerInMemorySource(const std::string& virtualPath,
std::string content) {
FileData data;
data.canonicalPath =
std::make_shared<const std::string>(virtualPath);
data.content = std::move(content);
buildLineIndex(data);
auto result = data.canonicalPath;
files_[virtualPath] = std::move(data);
return result;
}
std::string SourceManager::getLine(const std::string& filePath,
uint32_t lineNumber) const {
auto it = files_.find(filePath);
if (it == files_.end()) return "";
const FileData& data = it->second;
if (lineNumber == 0 ||
lineNumber > static_cast<uint32_t>(data.lineOffsets.size())) {
return "";
}
uint32_t start = data.lineOffsets[lineNumber - 1];
uint32_t end =
(lineNumber < static_cast<uint32_t>(data.lineOffsets.size()))
? data.lineOffsets[lineNumber]
: static_cast<uint32_t>(data.content.size());
// Strip the trailing newline for cleaner display.
std::string line = data.content.substr(start, end - start);
while (!line.empty() &&
(line.back() == '\n' || line.back() == '\r')) {
line.pop_back();
}
return line;
}
uint32_t SourceManager::lineCount(const std::string& filePath) const {
auto it = files_.find(filePath);
if (it == files_.end()) return 0;
return static_cast<uint32_t>(it->second.lineOffsets.size());
}
void SourceManager::buildLineIndex(FileData& data) {
data.lineOffsets.clear();
// Line 1 always starts at offset 0.
data.lineOffsets.push_back(0);
for (uint32_t i = 0;
i < static_cast<uint32_t>(data.content.size()); ++i) {
if (data.content[i] == '\n') {
data.lineOffsets.push_back(i + 1);
}
}
}
} // namespace diag
Now the DiagnosticEngine. This is the traffic controller of the whole system. Every compiler phase calls it to emit diagnostics, and it decides what to do with them: count them, apply policy transformations, check limits, and dispatch to consumers.
// src/diag/DiagnosticEngine.h
//
// The central hub for all diagnostic emission in the compiler.
// All compiler phases hold a reference to this engine and use it
// to report problems. The engine dispatches to registered consumers.
#pragma once
#include "Diagnostic.h"
#include <vector>
#include <memory>
#include <functional>
namespace diag {
// A DiagnosticConsumer receives diagnostics and decides what to do
// with them: print to stderr, write to a JSON file, send over LSP, etc.
// Multiple consumers can be active simultaneously.
class DiagnosticConsumer {
public:
virtual ~DiagnosticConsumer() = default;
// Called once for each diagnostic that passes the engine's filters.
virtual void handleDiagnostic(const Diagnostic& diag) = 0;
// Called when compilation ends, giving consumers a chance to
// flush buffered output or close file handles.
virtual void finalize() {}
};
class DiagnosticEngine {
public:
// The default maximum number of errors before compilation is
// aborted. This prevents error avalanches from drowning the user.
static constexpr uint32_t DEFAULT_MAX_ERRORS = 20;
explicit DiagnosticEngine(
uint32_t maxErrors = DEFAULT_MAX_ERRORS)
: maxErrors_(maxErrors)
, errorCount_(0)
, warningCount_(0)
, fatalOccurred_(false)
, treatWarningsAsErrors_(false)
{}
// Register a consumer. The engine takes ownership.
// Multiple consumers can be registered; all receive every diagnostic.
void addConsumer(std::unique_ptr<DiagnosticConsumer> consumer) {
consumers_.push_back(std::move(consumer));
}
// The primary emission method. All compiler phases call this.
// Returns true if compilation should continue, false if the
// error limit has been reached or a fatal error occurred.
bool emit(Diagnostic diag);
// Query the current state of the engine.
bool hasErrors() const { return errorCount_ > 0; }
bool hasFatal() const { return fatalOccurred_; }
uint32_t errorCount() const { return errorCount_; }
uint32_t warningCount()const { return warningCount_; }
void setTreatWarningsAsErrors(bool val) {
treatWarningsAsErrors_ = val;
}
// Returns true when the error count has reached the configured
// maximum. Compiler phases should check this and stop emitting
// further diagnostics to avoid flooding the user.
bool shouldAbortDueToErrorLimit() const {
return errorCount_ >= maxErrors_;
}
// Flush all consumers. Call this at the end of compilation.
void finalize() {
for (auto& consumer : consumers_) {
consumer->finalize();
}
}
private:
std::vector<std::unique_ptr<DiagnosticConsumer>> consumers_;
uint32_t maxErrors_;
uint32_t errorCount_;
uint32_t warningCount_;
bool fatalOccurred_;
bool treatWarningsAsErrors_;
};
} // namespace diag
// src/diag/DiagnosticEngine.cpp
#include "DiagnosticEngine.h"
namespace diag {
bool DiagnosticEngine::emit(Diagnostic diag) {
// Upgrade warnings to errors if -Werror mode is active.
if (treatWarningsAsErrors_ &&
diag.severity == Severity::Warning) {
diag.severity = Severity::Error;
// Attach a note so the user knows why this is an error.
Diagnostic note;
note.code = diag.code;
note.severity = Severity::Note;
note.message = "treated as error because -Werror is active";
note.phase = diag.phase;
diag.notes.push_back(std::move(note));
}
// Track counts based on the (possibly upgraded) severity.
switch (diag.severity) {
case Severity::Error:
case Severity::Fatal:
case Severity::InternalError:
++errorCount_;
break;
case Severity::Warning:
++warningCount_;
break;
default:
break;
}
if (diag.severity == Severity::Fatal ||
diag.severity == Severity::InternalError) {
fatalOccurred_ = true;
}
// Dispatch to all registered consumers.
for (auto& consumer : consumers_) {
consumer->handleDiagnostic(diag);
}
// Return false if compilation should stop.
return !fatalOccurred_ && !shouldAbortDueToErrorLimit();
}
} // namespace diag
CHAPTER FIVE: THE TERMINAL DIAGNOSTIC CONSUMER
The terminal consumer is where error messages transform from data structures into the text that a programmer actually reads. This is the component that has the most direct impact on user experience, and it deserves careful attention to detail. The gold standard here is set by Clang, which introduced the now-ubiquitous style of printing the offending source line with a caret pointing at the exact problem location. We will match that standard and go a step further with diagnostic codes and developer-mode information.
// src/diag/TerminalDiagnosticConsumer.h
//
// Prints diagnostics to stderr in a human-friendly format with
// source context, caret underlining, fix-it hints, and optional
// developer-mode compiler-internal information.
#pragma once
#include "DiagnosticEngine.h"
#include "SourceManager.h"
#include <ostream>
#include <string>
namespace diag {
class TerminalDiagnosticConsumer : public DiagnosticConsumer {
public:
explicit TerminalDiagnosticConsumer(
const SourceManager& srcMgr,
bool useColor = true,
bool developerMode = false,
std::ostream& out = std::cerr);
void handleDiagnostic(const Diagnostic& diag) override;
void finalize() override {} // stderr needs no flushing
private:
void printDiagnostic(const Diagnostic& diag,
bool isNote) const;
void printSourceContext(const Diagnostic& diag) const;
void printFixItHint(const FixItHint& hint) const;
std::string formatCode(uint32_t code,
Severity sev) const;
std::string expandTabs(const std::string& s,
uint32_t tabWidth) const;
uint32_t expandedColumn(const std::string& s,
uint32_t col,
uint32_t tabWidth) const;
const char* color(const char* code) const;
const char* severityColor(Severity sev) const;
std::string severityLabel(Severity sev) const;
// ANSI color escape codes.
static constexpr const char* RESET = "\033[0m";
static constexpr const char* BOLD = "\033[1m";
static constexpr const char* RED = "\033[31m";
static constexpr const char* YELLOW = "\033[33m";
static constexpr const char* CYAN = "\033[36m";
static constexpr const char* GREEN = "\033[32m";
static constexpr const char* MAGENTA = "\033[35m";
const SourceManager& srcMgr_;
bool useColor_;
bool developerMode_;
std::ostream& out_;
};
} // namespace diag
// src/diag/TerminalDiagnosticConsumer.cpp
#include "TerminalDiagnosticConsumer.h"
#include <iostream>
namespace diag {
TerminalDiagnosticConsumer::TerminalDiagnosticConsumer(
const SourceManager& srcMgr,
bool useColor,
bool developerMode,
std::ostream& out)
: srcMgr_(srcMgr)
, useColor_(useColor)
, developerMode_(developerMode)
, out_(out)
{}
void TerminalDiagnosticConsumer::handleDiagnostic(
const Diagnostic& diag)
{
printDiagnostic(diag, /*isNote=*/false);
}
void TerminalDiagnosticConsumer::printDiagnostic(
const Diagnostic& diag,
bool /*isNote*/) const
{
// --- Location prefix ---
if (diag.location.isValid()) {
out_ << color(BOLD)
<< diag.location.toString()
<< ": "
<< color(RESET);
}
// --- Severity label and code ---
out_ << severityColor(diag.severity)
<< color(BOLD)
<< severityLabel(diag.severity)
<< color(RESET)
<< " ["
<< formatCode(diag.code, diag.severity)
<< "]: "
<< color(BOLD)
<< diag.message
<< color(RESET)
<< "\n";
// --- Source context with caret ---
if (diag.location.isValid() && diag.location.filePath) {
printSourceContext(diag);
}
// --- Developer-mode information ---
// In developer mode we show which compiler phase emitted this
// diagnostic and, for internal errors, the C++ source location
// within the compiler where it was created.
if (developerMode_) {
if (!diag.phase.empty()) {
out_ << color(CYAN)
<< " [compiler phase: " << diag.phase << "]"
<< color(RESET) << "\n";
}
if (diag.compilerSourceLocation.has_value()) {
out_ << color(CYAN)
<< " [compiler source: "
<< *diag.compilerSourceLocation << "]"
<< color(RESET) << "\n";
}
}
// --- Fix-it hints ---
for (const auto& hint : diag.fixItHints) {
printFixItHint(hint);
}
// --- Attached notes ---
for (const auto& note : diag.notes) {
printDiagnostic(note, /*isNote=*/true);
}
// Internal compiler errors always get a bug-report request,
// regardless of verbosity settings.
if (diag.severity == Severity::InternalError) {
out_ << "\n"
<< color(MAGENTA)
<< " This is a bug in the compiler. Please file a report\n"
<< " at https://bugs.example.com/motorlangc including the\n"
<< " source file and the full compiler invocation command.\n"
<< color(RESET);
}
}
void TerminalDiagnosticConsumer::printSourceContext(
const Diagnostic& diag) const
{
const std::string filePath = *diag.location.filePath;
uint32_t lineNum = diag.location.line;
uint32_t colNum = diag.location.column;
std::string sourceLine = srcMgr_.getLine(filePath, lineNum);
if (sourceLine.empty()) return;
// Expand tabs to spaces for consistent column alignment.
// We use a tab width of 4, matching most editors' defaults.
std::string expanded = expandTabs(sourceLine, 4);
uint32_t expandedCol = expandedColumn(sourceLine, colNum, 4);
// Print the source line with a line-number gutter.
std::string lineNumStr = std::to_string(lineNum);
std::string gutter(lineNumStr.size() + 2, ' ');
out_ << color(CYAN)
<< lineNumStr << " | "
<< color(RESET)
<< expanded << "\n";
// Print the caret line. The caret (^) points at the column
// of the error. If a range is provided, tildes (~) extend
// to cover the full span of the problematic construct.
out_ << color(CYAN)
<< gutter << "| "
<< color(RESET);
uint32_t caretPos = (expandedCol > 0) ? expandedCol - 1 : 0;
out_ << std::string(caretPos, ' ');
// Determine the underline length from the source range.
uint32_t underlineLen = 1;
if (diag.range.isValid() &&
diag.range.begin.line == lineNum &&
diag.range.end.line == lineNum &&
diag.range.end.column > diag.range.begin.column) {
underlineLen = diag.range.end.column
- diag.range.begin.column;
}
out_ << color(GREEN) << "^";
if (underlineLen > 1) {
out_ << std::string(underlineLen - 1, '~');
}
out_ << color(RESET) << "\n";
}
void TerminalDiagnosticConsumer::printFixItHint(
const FixItHint& hint) const
{
if (hint.replacementText.empty()) {
out_ << color(GREEN)
<< " fix-it: remove text at "
<< hint.rangeToReplace.begin.toString()
<< color(RESET) << "\n";
} else {
out_ << color(GREEN)
<< " fix-it: replace with '"
<< hint.replacementText << "' at "
<< hint.rangeToReplace.begin.toString()
<< color(RESET) << "\n";
}
}
std::string TerminalDiagnosticConsumer::formatCode(
uint32_t code, Severity /*sev*/) const
{
std::string prefix;
if (code < 2000) prefix = "LEX";
else if (code < 3000) prefix = "PAR";
else if (code < 4000) prefix = "SEM";
else if (code < 5000) prefix = "TYP";
else if (code < 9000) prefix = "GEN";
else prefix = "ICE";
return prefix + "-" + std::to_string(code);
}
std::string TerminalDiagnosticConsumer::expandTabs(
const std::string& s, uint32_t tabWidth) const
{
std::string result;
result.reserve(s.size());
uint32_t col = 0;
for (char c : s) {
if (c == '\t') {
uint32_t spaces = tabWidth - (col % tabWidth);
result.append(spaces, ' ');
col += spaces;
} else {
result += c;
++col;
}
}
return result;
}
uint32_t TerminalDiagnosticConsumer::expandedColumn(
const std::string& s, uint32_t col,
uint32_t tabWidth) const
{
uint32_t expanded = 0;
uint32_t current = 0;
for (char c : s) {
if (current + 1 >= col) break;
if (c == '\t') {
expanded += tabWidth - (expanded % tabWidth);
} else {
++expanded;
}
++current;
}
return expanded + 1;
}
const char* TerminalDiagnosticConsumer::color(
const char* code) const
{
return useColor_ ? code : "";
}
const char* TerminalDiagnosticConsumer::severityColor(
Severity sev) const
{
switch (sev) {
case Severity::Note: return color(CYAN);
case Severity::Remark: return color(CYAN);
case Severity::Warning: return color(YELLOW);
case Severity::Error: return color(RED);
case Severity::Fatal: return color(RED);
case Severity::InternalError: return color(MAGENTA);
}
return "";
}
std::string TerminalDiagnosticConsumer::severityLabel(
Severity sev) const
{
switch (sev) {
case Severity::Note: return "note";
case Severity::Remark: return "remark";
case Severity::Warning: return "warning";
case Severity::Error: return "error";
case Severity::Fatal: return "fatal error";
case Severity::InternalError: return "internal compiler error";
}
return "unknown";
}
} // namespace diag
The expandTabs function is one of those small details that makes a big difference. Tabs are the enemy of column-accurate error messages. If a source line contains a tab character and the terminal renders it as eight spaces but the compiler counted it as one character, the caret will point to the wrong column. By normalizing tabs to a fixed number of spaces during display, we ensure that the caret always lands exactly under the problematic token.
Here is what the output looks like for a real error in MotorLang:
MotorController.ml:47:12: error [SEM-3042]: cannot initialize variable
'rpm' of type 'int' with a value of type 'float'; consider an explicit cast
47 | rpm = speed * 1.5;
| ^~~~~~~~~~~
fix-it: replace with '(int)(speed * 1.5)' at MotorController.ml:47:7
MotorController.ml:23:5: note [SEM-3001]: 'rpm' was declared here as 'int'
23 | int rpm = 0;
| ^~~
[compiler phase: TypeChecker]
That is the kind of output that makes a programmer say "oh, right" instead of "what on earth does that mean?"
CHAPTER SIX: THE JSON DIAGNOSTIC CONSUMER
Modern development environments expect compilers to speak JSON. IDEs, CI/CD pipelines, and static analysis tools all benefit from machine-readable diagnostic output. Our JSON consumer outputs diagnostics in a format inspired by the SARIF standard, which is supported by GitHub Actions, Azure DevOps, and most LSP-capable editors. The key design challenge here is producing valid JSON without a JSON library dependency, which I handle by building the output carefully and tracking state with a boolean flag rather than using the trailing-comma hack that plagues many hand-rolled JSON serializers.
// src/diag/JsonDiagnosticConsumer.h
//
// Outputs diagnostics as a JSON array, compatible with SARIF-based
// tooling. Each diagnostic becomes one JSON object in the array.
#pragma once
#include "DiagnosticEngine.h"
#include <ostream>
#include <string>
namespace diag {
class JsonDiagnosticConsumer : public DiagnosticConsumer {
public:
explicit JsonDiagnosticConsumer(std::ostream& out);
void handleDiagnostic(const Diagnostic& diag) override;
void finalize() override;
private:
void writeDiagnosticObject(const Diagnostic& diag,
int indentLevel);
void writeLocationObject(const SourceLocation& loc,
int indentLevel);
void writeRangeObject(const SourceRange& range,
int indentLevel);
std::string escapeJson(const std::string& s) const;
std::string severityString(Severity sev) const;
std::string indent(int level) const;
std::ostream& out_;
bool firstDiagnostic_;
};
} // namespace diag
// src/diag/JsonDiagnosticConsumer.cpp
#include "JsonDiagnosticConsumer.h"
#include <cstdio>
namespace diag {
JsonDiagnosticConsumer::JsonDiagnosticConsumer(std::ostream& out)
: out_(out)
, firstDiagnostic_(true)
{
out_ << "[\n";
}
void JsonDiagnosticConsumer::handleDiagnostic(
const Diagnostic& diag)
{
if (!firstDiagnostic_) {
out_ << ",\n";
}
firstDiagnostic_ = false;
writeDiagnosticObject(diag, 1);
}
void JsonDiagnosticConsumer::finalize() {
out_ << "\n]\n";
out_.flush();
}
void JsonDiagnosticConsumer::writeDiagnosticObject(
const Diagnostic& diag, int level)
{
std::string ind = indent(level);
std::string ind2 = indent(level + 1);
out_ << ind << "{\n";
out_ << ind2 << "\"code\": " << diag.code << ",\n";
out_ << ind2 << "\"severity\": \""
<< severityString(diag.severity) << "\",\n";
out_ << ind2 << "\"message\": \""
<< escapeJson(diag.message) << "\",\n";
out_ << ind2 << "\"phase\": \""
<< escapeJson(diag.phase) << "\"";
if (diag.location.isValid()) {
out_ << ",\n";
out_ << ind2 << "\"location\": ";
writeLocationObject(diag.location, level + 1);
}
if (diag.range.isValid()) {
out_ << ",\n";
out_ << ind2 << "\"range\": ";
writeRangeObject(diag.range, level + 1);
}
if (!diag.notes.empty()) {
out_ << ",\n";
out_ << ind2 << "\"notes\": [\n";
for (size_t i = 0; i < diag.notes.size(); ++i) {
writeDiagnosticObject(diag.notes[i], level + 2);
if (i + 1 < diag.notes.size()) out_ << ",";
out_ << "\n";
}
out_ << ind2 << "]";
}
if (!diag.fixItHints.empty()) {
out_ << ",\n";
out_ << ind2 << "\"fixItHints\": [\n";
for (size_t i = 0; i < diag.fixItHints.size(); ++i) {
const auto& hint = diag.fixItHints[i];
out_ << indent(level + 2) << "{\n";
out_ << indent(level + 3)
<< "\"replacement\": \""
<< escapeJson(hint.replacementText) << "\"";
if (hint.rangeToReplace.isValid()) {
out_ << ",\n";
out_ << indent(level + 3) << "\"range\": ";
writeRangeObject(hint.rangeToReplace, level + 3);
}
out_ << "\n" << indent(level + 2) << "}";
if (i + 1 < diag.fixItHints.size()) out_ << ",";
out_ << "\n";
}
out_ << ind2 << "]";
}
if (diag.compilerSourceLocation.has_value()) {
out_ << ",\n";
out_ << ind2 << "\"compilerSource\": \""
<< escapeJson(*diag.compilerSourceLocation) << "\"";
}
out_ << "\n" << ind << "}";
}
void JsonDiagnosticConsumer::writeLocationObject(
const SourceLocation& loc, int level)
{
std::string ind2 = indent(level + 1);
out_ << "{\n";
if (loc.filePath) {
out_ << ind2 << "\"file\": \""
<< escapeJson(*loc.filePath) << "\",\n";
}
out_ << ind2 << "\"line\": " << loc.line << ",\n";
out_ << ind2 << "\"column\": " << loc.column << "\n";
out_ << indent(level) << "}";
}
void JsonDiagnosticConsumer::writeRangeObject(
const SourceRange& range, int level)
{
std::string ind2 = indent(level + 1);
out_ << "{\n";
out_ << ind2 << "\"begin\": ";
writeLocationObject(range.begin, level + 1);
out_ << ",\n";
out_ << ind2 << "\"end\": ";
writeLocationObject(range.end, level + 1);
out_ << "\n" << indent(level) << "}";
}
std::string JsonDiagnosticConsumer::escapeJson(
const std::string& s) const
{
std::string result;
result.reserve(s.size());
for (unsigned char c : s) {
switch (c) {
case '"': result += "\\\""; break;
case '\\': result += "\\\\"; break;
case '\n': result += "\\n"; break;
case '\r': result += "\\r"; break;
case '\t': result += "\\t"; break;
default:
if (c < 0x20) {
// Control characters must be Unicode-escaped in JSON.
char buf[8];
snprintf(buf, sizeof(buf), "\\u%04x",
static_cast<unsigned int>(c));
result += buf;
} else {
result += static_cast<char>(c);
}
break;
}
}
return result;
}
std::string JsonDiagnosticConsumer::severityString(
Severity sev) const
{
switch (sev) {
case Severity::Note: return "note";
case Severity::Remark: return "remark";
case Severity::Warning: return "warning";
case Severity::Error: return "error";
case Severity::Fatal: return "fatal";
case Severity::InternalError: return "internalError";
}
return "unknown";
}
std::string JsonDiagnosticConsumer::indent(int level) const {
return std::string(static_cast<size_t>(level) * 2, ' ');
}
} // namespace diag
CHAPTER SEVEN: THE DIAGNOSTIC CODE REGISTRY
Every diagnostic code in our system should be documented. The documentation serves two purposes: it helps users understand and fix their errors, and it helps compiler developers understand the intent of each diagnostic when they are reading or modifying the compiler's source code. I maintain this as a dedicated header file so that the codes are defined in exactly one place and can be referenced by name throughout the compiler.
// src/diag/DiagnosticCodes.h
//
// The canonical registry of all diagnostic codes used by the
// MotorLang compiler. Each code is documented with its meaning,
// typical cause, and suggested fix.
//
// Code ranges:
// 1000-1999: Lexer errors (LEX)
// 2000-2999: Parser errors (PAR)
// 3000-3999: Semantic errors (SEM)
// 4000-4999: Type errors (TYP)
// 5000-5999: Code generation (GEN)
// 9000-9999: Internal compiler (ICE)
#pragma once
#include <cstdint>
namespace diag {
namespace codes {
// --- Lexer errors (1000-1999) ---
// LEX-1001: An unrecognized character was encountered.
// Cause: The source file contains a character that is not part
// of the MotorLang character set.
// Fix: Remove or replace the offending character.
constexpr uint32_t LEX_UNRECOGNIZED_CHAR = 1001;
// LEX-1002: An unterminated string literal.
// Cause: A string literal was opened with a quote but the
// closing quote was not found before the end of the line.
// Fix: Add the closing quote.
constexpr uint32_t LEX_UNTERMINATED_STRING = 1002;
// LEX-1003: An integer literal is too large to represent.
// Cause: The literal value exceeds the maximum value of int64.
// Fix: Use a smaller value or a different type.
constexpr uint32_t LEX_INTEGER_OVERFLOW = 1003;
// --- Parser errors (2000-2999) ---
// PAR-2001: A general syntax error.
// Cause: The token sequence does not match any valid grammar rule.
// Fix: Check the syntax of the statement or expression.
constexpr uint32_t PAR_SYNTAX_ERROR = 2001;
// PAR-2002: A missing semicolon.
// Cause: A statement was not terminated with a semicolon.
// Fix: Add a semicolon at the end of the statement.
constexpr uint32_t PAR_MISSING_SEMICOLON = 2002;
// --- Semantic errors (3000-3999) ---
// SEM-3001: Supplementary note pointing to a declaration site.
// This code is used for notes that accompany other diagnostics.
constexpr uint32_t SEM_DECLARATION_NOTE = 3001;
// SEM-3010: Redeclaration of a variable.
// Cause: A variable with the same name was already declared
// in the same scope.
// Fix: Rename one of the variables or remove the duplicate.
constexpr uint32_t SEM_REDECLARATION = 3010;
// SEM-3020: Use of an undeclared variable.
// Cause: A variable is referenced that has not been declared.
// Fix: Declare the variable before using it.
constexpr uint32_t SEM_UNDECLARED_VARIABLE = 3020;
// SEM-3030: Use of an uninitialized variable.
// Cause: A variable is read before it has been assigned a value.
// Fix: Initialize the variable at the point of declaration.
constexpr uint32_t SEM_UNINITIALIZED_VAR = 3030;
// SEM-3042: Type mismatch in initialization or assignment.
// Cause: The initializer expression has a type that cannot be
// implicitly converted to the variable's declared type.
// Fix: Use an explicit cast or change the variable's type.
constexpr uint32_t SEM_TYPE_MISMATCH_INIT = 3042;
// SEM-3050: Incompatible operand types in binary expression.
// Cause: The two operands of a binary operator have types that
// cannot be used together with that operator.
// Fix: Cast one or both operands to a compatible type.
constexpr uint32_t SEM_INCOMPATIBLE_OPERANDS = 3050;
// SEM-3051: Note identifying the left operand's type.
constexpr uint32_t SEM_LEFT_OPERAND_NOTE = 3051;
// SEM-3052: Note identifying the right operand's type.
constexpr uint32_t SEM_RIGHT_OPERAND_NOTE = 3052;
// --- Code generation errors (5000-5999) ---
// GEN-5000: A general LLVM backend diagnostic.
constexpr uint32_t GEN_LLVM_BACKEND = 5000;
// GEN-5100: LLVM IR verification failure.
// Cause: The IR generator produced invalid LLVM IR.
// Fix: File a bug report; this is a compiler bug.
constexpr uint32_t GEN_IR_VERIFICATION = 5100;
// --- Internal compiler errors (9000-9999) ---
// ICE-9999: An unexpected internal state was reached.
// Cause: A bug in the compiler itself.
// Fix: File a bug report with the source file and version.
constexpr uint32_t ICE_UNEXPECTED_STATE = 9999;
} // namespace codes
} // namespace diag
CHAPTER EIGHT: THE ANTLR4 FRONTEND -- CAPTURING ERRORS AT THE SOURCE
ANTLR4 generates lexers and parsers from grammar files. By default it has its own error reporting mechanism, which is functional but not integrated with our DiagnosticEngine. Our first task is to replace ANTLR4's default error listeners with custom ones that translate ANTLR4's error information into our Diagnostic structures.
ANTLR4 provides an interface called ANTLRErrorListener with several virtual methods. The most important is syntaxError, which is called whenever the lexer or parser encounters a problem. The signature gives us the line number and character position within the line, which we can use to construct a SourceLocation. There is a subtlety here that trips up many people: ANTLR4's column positions are 0-based, while our SourceLocation uses 1-based columns. We must add 1 when converting.
The translateMessage function is where we invest significant effort in making error messages user-friendly. ANTLR4's raw messages are written for grammar authors, not for users of the language being compiled. A message like "no viable alternative at input 'rpm'" is meaningful to someone who understands LL(*) parsing, but baffling to an engineer writing a motor control program. Our translation layer rewrites these messages into plain language.
// src/frontend/AntlrDiagnosticListener.h
//
// Bridges ANTLR4's error reporting to our DiagnosticEngine.
// Replaces ANTLR4's default ConsoleErrorListener with a listener
// that produces rich, structured diagnostics with source locations.
#pragma once
#include <antlr4-runtime.h>
#include "diag/DiagnosticEngine.h"
#include "diag/DiagnosticCodes.h"
#include "diag/SourceLocation.h"
#include <string>
#include <memory>
namespace frontend {
class AntlrDiagnosticListener
: public antlr4::ANTLRErrorListener
{
public:
AntlrDiagnosticListener(
diag::DiagnosticEngine& engine,
std::shared_ptr<const std::string> filePath,
bool isLexer = false)
: engine_(engine)
, filePath_(std::move(filePath))
, isLexer_(isLexer)
{}
// Called by ANTLR4 whenever a syntax error is detected.
// 'line' is 1-based; 'charPositionInLine' is 0-based.
void syntaxError(
antlr4::Recognizer* recognizer,
antlr4::Token* offendingSymbol,
size_t line,
size_t charPositionInLine,
const std::string& msg,
std::exception_ptr e) override;
// Ambiguity reporting is useful for grammar debugging.
// We emit these as remarks so they do not alarm end users.
void reportAmbiguity(
antlr4::Parser* recognizer,
const antlr4::dfa::DFA& dfa,
size_t startIndex,
size_t stopIndex,
bool exact,
const antlr4::BitSet& ambigAlts,
antlr4::atn::ATNConfigSet* configs) override;
void reportAttemptingFullContext(
antlr4::Parser*,
const antlr4::dfa::DFA&,
size_t, size_t,
const antlr4::BitSet*,
antlr4::atn::ATNConfigSet*) override {}
void reportContextSensitivity(
antlr4::Parser*,
const antlr4::dfa::DFA&,
size_t, size_t, size_t,
antlr4::atn::ATNConfigSet*) override {}
private:
std::string translateMessage(
const std::string& rawMsg,
antlr4::Token* offendingSymbol,
antlr4::Recognizer* recognizer) const;
diag::SourceRange tokenToRange(
antlr4::Token* token) const;
diag::DiagnosticEngine& engine_;
std::shared_ptr<const std::string> filePath_;
bool isLexer_;
};
} // namespace frontend
// src/frontend/AntlrDiagnosticListener.cpp
#include "AntlrDiagnosticListener.h"
namespace frontend {
void AntlrDiagnosticListener::syntaxError(
antlr4::Recognizer* recognizer,
antlr4::Token* offendingSymbol,
size_t line,
size_t charPositionInLine,
const std::string& msg,
std::exception_ptr /*e*/)
{
if (engine_.shouldAbortDueToErrorLimit()) return;
// Convert ANTLR4's 0-based column to our 1-based column.
uint32_t col = static_cast<uint32_t>(charPositionInLine) + 1;
uint32_t ln = static_cast<uint32_t>(line);
diag::SourceLocation loc(filePath_, ln, col, 0);
diag::SourceRange range;
if (offendingSymbol != nullptr) {
range = tokenToRange(offendingSymbol);
}
std::string friendlyMsg = translateMessage(
msg, offendingSymbol, recognizer);
diag::Diagnostic d;
d.code = isLexer_
? diag::codes::LEX_UNRECOGNIZED_CHAR
: diag::codes::PAR_SYNTAX_ERROR;
d.severity = diag::Severity::Error;
d.location = loc;
d.range = range;
d.message = friendlyMsg;
d.phase = isLexer_ ? "Lexer" : "Parser";
// Attach the raw ANTLR4 message as a note. In developer mode
// this preserves the original technical information; in normal
// mode it is still available for bug reports.
diag::Diagnostic rawNote;
rawNote.code = d.code;
rawNote.severity = diag::Severity::Note;
rawNote.location = loc;
rawNote.message = "ANTLR4 raw message: " + msg;
rawNote.phase = d.phase;
d.notes.push_back(std::move(rawNote));
engine_.emit(std::move(d));
}
void AntlrDiagnosticListener::reportAmbiguity(
antlr4::Parser* /*recognizer*/,
const antlr4::dfa::DFA& /*dfa*/,
size_t startIndex,
size_t stopIndex,
bool exact,
const antlr4::BitSet& /*ambigAlts*/,
antlr4::atn::ATNConfigSet* /*configs*/)
{
if (engine_.shouldAbortDueToErrorLimit()) return;
diag::Diagnostic d;
d.code = diag::codes::PAR_SYNTAX_ERROR;
d.severity = diag::Severity::Remark;
d.message = "grammar ambiguity detected between tokens "
+ std::to_string(startIndex) + " and "
+ std::to_string(stopIndex)
+ (exact ? " (exact)" : " (approximate)");
d.phase = "Parser";
engine_.emit(std::move(d));
}
std::string AntlrDiagnosticListener::translateMessage(
const std::string& rawMsg,
antlr4::Token* offendingSymbol,
antlr4::Recognizer* /*recognizer*/) const
{
// ANTLR4 produces messages like:
// "mismatched input 'foo' expecting {'bar', 'baz'}"
// "extraneous input 'x' expecting '}'"
// "no viable alternative at input 'y'"
// We rewrite these into clearer forms for language users.
if (rawMsg.find("mismatched input") != std::string::npos) {
if (offendingSymbol) {
std::string tokenText = offendingSymbol->getText();
if (tokenText == "<EOF>") {
return "unexpected end of file; "
"the source file appears to be incomplete";
}
return "unexpected token '" + tokenText
+ "'; the syntax is not valid here";
}
}
if (rawMsg.find("extraneous input") != std::string::npos) {
if (offendingSymbol) {
return "unexpected extra token '"
+ offendingSymbol->getText()
+ "'; remove it or check for a missing "
"semicolon or bracket before this point";
}
}
if (rawMsg.find("no viable alternative") != std::string::npos) {
if (offendingSymbol) {
std::string tokenText = offendingSymbol->getText();
if (tokenText == "<EOF>") {
return "unexpected end of file";
}
return "the compiler cannot determine what you meant "
"here; check for a typo or a missing keyword";
}
}
if (rawMsg.find("missing") != std::string::npos) {
return "syntax error: " + rawMsg;
}
// Fall back to the raw message if we cannot improve it.
return rawMsg;
}
diag::SourceRange AntlrDiagnosticListener::tokenToRange(
antlr4::Token* token) const
{
if (!token) return {};
uint32_t startLine =
static_cast<uint32_t>(token->getLine());
uint32_t startCol =
static_cast<uint32_t>(
token->getCharPositionInLine()) + 1;
uint32_t startOff =
static_cast<uint32_t>(token->getStartIndex());
std::string text = token->getText();
uint32_t endCol = startCol
+ static_cast<uint32_t>(text.size());
uint32_t endOff =
static_cast<uint32_t>(token->getStopIndex()) + 1;
diag::SourceLocation begin(filePath_, startLine,
startCol, startOff);
diag::SourceLocation end(filePath_, startLine,
endCol, endOff);
return { begin, end };
}
} // namespace frontend
Now let us look at the custom error strategy. ANTLR4's default error recovery sometimes produces confusing cascading errors for complex mismatches. Our custom strategy uses panic-mode recovery with synchronization on statement boundaries, which is the right approach for a statement-based language like MotorLang.
// src/frontend/CustomErrorStrategy.h
//
// A customized ANTLR4 error recovery strategy that produces
// better error messages and more intelligent recovery behavior
// for MotorLang's specific syntax.
#pragma once
#include <antlr4-runtime.h>
#include "diag/DiagnosticEngine.h"
// Include the generated parser header for token type constants.
#include "MotorLangParser.h"
namespace frontend {
class CustomErrorStrategy
: public antlr4::DefaultErrorStrategy
{
public:
explicit CustomErrorStrategy(
diag::DiagnosticEngine& engine)
: engine_(engine)
{}
protected:
// Override the recovery mechanism. When we encounter an error
// in a statement, we skip tokens until we find a synchronization
// point: ';', '}', or a statement-starting keyword.
void recover(antlr4::Parser* recognizer,
std::exception_ptr e) override
{
beginErrorCondition(recognizer);
// Skip tokens until we reach a synchronization point.
antlr4::Token* t = recognizer->getCurrentToken();
while (t->getType() != antlr4::Token::EOF) {
size_t type = t->getType();
if (type == MotorLangParser::T__4 || // ';'
type == MotorLangParser::T__6 || // '}'
type == MotorLangParser::KW_IF ||
type == MotorLangParser::KW_WHILE ||
type == MotorLangParser::KW_RETURN ||
type == MotorLangParser::KW_INT ||
type == MotorLangParser::KW_FLOAT) {
// Consume the semicolon so the parser resumes
// cleanly after the statement boundary.
if (type == MotorLangParser::T__4) {
recognizer->consume();
}
break;
}
recognizer->consume();
t = recognizer->getCurrentToken();
}
endErrorCondition(recognizer);
}
// Override single-token insertion to avoid inserting tokens
// that would produce confusing parse trees.
antlr4::Token* recoverInline(
antlr4::Parser* recognizer) override
{
// For missing semicolons we can safely insert them.
antlr4::IntervalSet expected =
recognizer->getExpectedTokens();
if (expected.contains(MotorLangParser::T__4)) {
return getMissingSymbol(recognizer);
}
return DefaultErrorStrategy::recoverInline(recognizer);
}
private:
diag::DiagnosticEngine& engine_;
};
} // namespace frontend
CHAPTER NINE: BUILDING THE AST WITH EMBEDDED LOCATION INFORMATION
The parse tree that ANTLR4 produces is a concrete syntax tree, meaning it contains every token including punctuation. For semantic analysis and code generation, we want an abstract syntax tree that retains only the semantically meaningful structure. The crucial requirement is that every AST node must carry its source location, because semantic errors are reported against AST nodes. I make location information impossible to forget by requiring it in every AST node constructor.
// src/ast/AST.h
//
// Abstract Syntax Tree node hierarchy for the MotorLang compiler.
// Every node carries a SourceRange covering its full source extent,
// enabling precise error reporting at all subsequent compiler phases.
#pragma once
#include "diag/SourceLocation.h"
#include <string>
#include <vector>
#include <memory>
namespace ast {
// Forward declarations for the visitor pattern.
class IntLiteralExpr;
class FloatLiteralExpr;
class VariableExpr;
class BinaryExpr;
class AssignStmt;
class VarDeclStmt;
class IfStmt;
class WhileStmt;
class ReturnStmt;
class BlockStmt;
class FunctionDecl;
// The visitor interface allows compiler phases to traverse the AST
// without modifying the node classes themselves.
class ASTVisitor {
public:
virtual ~ASTVisitor() = default;
virtual void visit(IntLiteralExpr&) = 0;
virtual void visit(FloatLiteralExpr&) = 0;
virtual void visit(VariableExpr&) = 0;
virtual void visit(BinaryExpr&) = 0;
virtual void visit(AssignStmt&) = 0;
virtual void visit(VarDeclStmt&) = 0;
virtual void visit(IfStmt&) = 0;
virtual void visit(WhileStmt&) = 0;
virtual void visit(ReturnStmt&) = 0;
virtual void visit(BlockStmt&) = 0;
virtual void visit(FunctionDecl&) = 0;
};
// The base class for all AST nodes. The SourceRange is mandatory
// in the constructor, making it structurally impossible to create
// an AST node without location information.
class Node {
public:
explicit Node(diag::SourceRange range)
: range_(std::move(range)) {}
virtual ~Node() = default;
const diag::SourceRange& range() const { return range_; }
// Convenience accessor for the start location, which is the
// most commonly needed location for error reporting.
const diag::SourceLocation& loc() const { return range_.begin; }
private:
diag::SourceRange range_;
};
// Base class for all expression nodes.
class Expr : public Node {
public:
using Node::Node;
virtual void accept(ASTVisitor& v) = 0;
// The resolved type is filled in by the type checker.
// Before type checking, this is empty. After a type error,
// it is set to the sentinel value "<error>" to suppress
// cascading errors in parent expressions.
std::string resolvedType;
};
// Base class for all statement nodes.
class Stmt : public Node {
public:
using Node::Node;
virtual void accept(ASTVisitor& v) = 0;
};
// An integer literal, e.g., 42 or 0xFF.
class IntLiteralExpr : public Expr {
public:
IntLiteralExpr(diag::SourceRange range, int64_t value)
: Expr(std::move(range)), value_(value) {}
void accept(ASTVisitor& v) override { v.visit(*this); }
int64_t value() const { return value_; }
private:
int64_t value_;
};
// A floating-point literal, e.g., 3.14 or 1.5e-3.
class FloatLiteralExpr : public Expr {
public:
FloatLiteralExpr(diag::SourceRange range, double value)
: Expr(std::move(range)), value_(value) {}
void accept(ASTVisitor& v) override { v.visit(*this); }
double value() const { return value_; }
private:
double value_;
};
// A reference to a named variable, e.g., 'rpm' or 'speed'.
class VariableExpr : public Expr {
public:
VariableExpr(diag::SourceRange range, std::string name)
: Expr(std::move(range)), name_(std::move(name)) {}
void accept(ASTVisitor& v) override { v.visit(*this); }
const std::string& name() const { return name_; }
private:
std::string name_;
};
// A binary operation, e.g., 'a + b' or 'x * 1.5'.
class BinaryExpr : public Expr {
public:
BinaryExpr(diag::SourceRange range,
std::string op,
std::unique_ptr<Expr> left,
std::unique_ptr<Expr> right)
: Expr(std::move(range))
, op_(std::move(op))
, left_(std::move(left))
, right_(std::move(right))
{}
void accept(ASTVisitor& v) override { v.visit(*this); }
const std::string& op() const { return op_; }
Expr& left() const { return *left_; }
Expr& right() const { return *right_; }
private:
std::string op_;
std::unique_ptr<Expr> left_;
std::unique_ptr<Expr> right_;
};
// A variable declaration, e.g., 'int rpm = 0;'
class VarDeclStmt : public Stmt {
public:
VarDeclStmt(diag::SourceRange range,
std::string type,
std::string name,
std::unique_ptr<Expr> initializer)
: Stmt(std::move(range))
, type_(std::move(type))
, name_(std::move(name))
, initializer_(std::move(initializer))
{}
void accept(ASTVisitor& v) override { v.visit(*this); }
const std::string& type() const { return type_; }
const std::string& name() const { return name_; }
Expr* initializer() const { return initializer_.get(); }
private:
std::string type_;
std::string name_;
std::unique_ptr<Expr> initializer_;
};
// An assignment statement, e.g., 'rpm = speed * 1.5;'
class AssignStmt : public Stmt {
public:
AssignStmt(diag::SourceRange range,
std::string target,
std::unique_ptr<Expr> value)
: Stmt(std::move(range))
, target_(std::move(target))
, value_(std::move(value))
{}
void accept(ASTVisitor& v) override { v.visit(*this); }
const std::string& target() const { return target_; }
Expr& value() const { return *value_; }
private:
std::string target_;
std::unique_ptr<Expr> value_;
};
// A return statement, e.g., 'return rpm;'
class ReturnStmt : public Stmt {
public:
ReturnStmt(diag::SourceRange range,
std::unique_ptr<Expr> value)
: Stmt(std::move(range))
, value_(std::move(value))
{}
void accept(ASTVisitor& v) override { v.visit(*this); }
Expr* value() const { return value_.get(); }
private:
std::unique_ptr<Expr> value_;
};
// A block of statements enclosed in braces.
class BlockStmt : public Stmt {
public:
explicit BlockStmt(diag::SourceRange range)
: Stmt(std::move(range)) {}
void accept(ASTVisitor& v) override { v.visit(*this); }
void addStatement(std::unique_ptr<Stmt> stmt) {
stmts_.push_back(std::move(stmt));
}
const std::vector<std::unique_ptr<Stmt>>&
statements() const { return stmts_; }
private:
std::vector<std::unique_ptr<Stmt>> stmts_;
};
// An if statement with an optional else branch.
class IfStmt : public Stmt {
public:
IfStmt(diag::SourceRange range,
std::unique_ptr<Expr> condition,
std::unique_ptr<Stmt> thenBranch,
std::unique_ptr<Stmt> elseBranch)
: Stmt(std::move(range))
, condition_(std::move(condition))
, thenBranch_(std::move(thenBranch))
, elseBranch_(std::move(elseBranch))
{}
void accept(ASTVisitor& v) override { v.visit(*this); }
Expr& condition() const { return *condition_; }
Stmt& thenBranch() const { return *thenBranch_; }
Stmt* elseBranch() const { return elseBranch_.get(); }
private:
std::unique_ptr<Expr> condition_;
std::unique_ptr<Stmt> thenBranch_;
std::unique_ptr<Stmt> elseBranch_;
};
// A while loop.
class WhileStmt : public Stmt {
public:
WhileStmt(diag::SourceRange range,
std::unique_ptr<Expr> condition,
std::unique_ptr<Stmt> body)
: Stmt(std::move(range))
, condition_(std::move(condition))
, body_(std::move(body))
{}
void accept(ASTVisitor& v) override { v.visit(*this); }
Expr& condition() const { return *condition_; }
Stmt& body() const { return *body_; }
private:
std::unique_ptr<Expr> condition_;
std::unique_ptr<Stmt> body_;
};
// A function declaration with a parameter list and a body block.
class FunctionDecl : public Node {
public:
FunctionDecl(diag::SourceRange range,
std::string returnType,
std::string name)
: Node(std::move(range))
, returnType_(std::move(returnType))
, name_(std::move(name))
{}
void accept(ASTVisitor& v) { v.visit(*this); }
const std::string& returnType() const { return returnType_; }
const std::string& name() const { return name_; }
void addParam(std::string type, std::string name,
diag::SourceLocation loc) {
params_.push_back({std::move(type),
std::move(name),
std::move(loc)});
}
struct Param {
std::string type;
std::string name;
diag::SourceLocation loc;
};
const std::vector<Param>& params() const { return params_; }
void setBody(std::unique_ptr<BlockStmt> body) {
body_ = std::move(body);
}
BlockStmt* body() const { return body_.get(); }
private:
std::string returnType_;
std::string name_;
std::vector<Param> params_;
std::unique_ptr<BlockStmt> body_;
};
// The top-level program is a list of function declarations.
struct Program {
std::vector<std::unique_ptr<FunctionDecl>> functions;
};
} // namespace ast
The ASTBuilder visits the ANTLR4 parse tree and constructs AST nodes. The rangeFromContext function is the linchpin of location tracking in the frontend: it uses ANTLR4's start and stop tokens of a parser rule context to compute the full range of the syntactic construct.
// src/frontend/ASTBuilder.h
//
// Converts an ANTLR4 parse tree (CST) into our AST.
// Implemented as an ANTLR4 visitor that traverses the parse tree
// and constructs AST nodes with full location information.
#pragma once
#include "ast/AST.h"
#include "diag/DiagnosticEngine.h"
#include "MotorLangBaseVisitor.h"
#include <memory>
namespace frontend {
class ASTBuilder : public MotorLangBaseVisitor {
public:
ASTBuilder(diag::DiagnosticEngine& engine,
std::shared_ptr<const std::string> filePath)
: engine_(engine)
, filePath_(std::move(filePath))
{}
std::unique_ptr<ast::Program>
buildProgram(MotorLangParser::ProgramContext* ctx);
private:
// Extracts a SourceRange from an ANTLR4 parser rule context.
// This is the critical bridge between ANTLR4's location model
// and our SourceLocation model.
diag::SourceRange rangeFromContext(
antlr4::ParserRuleContext* ctx) const;
// Extracts a SourceLocation from a single ANTLR4 token.
diag::SourceLocation locFromToken(
antlr4::Token* tok) const;
// Visitor method overrides.
std::any visitProgram(
MotorLangParser::ProgramContext* ctx) override;
std::any visitFunctionDecl(
MotorLangParser::FunctionDeclContext* ctx) override;
std::any visitVarDeclStmt(
MotorLangParser::VarDeclStmtContext* ctx) override;
std::any visitAssignStmt(
MotorLangParser::AssignStmtContext* ctx) override;
std::any visitIfStmt(
MotorLangParser::IfStmtContext* ctx) override;
std::any visitWhileStmt(
MotorLangParser::WhileStmtContext* ctx) override;
std::any visitReturnStmt(
MotorLangParser::ReturnStmtContext* ctx) override;
std::any visitBlock(
MotorLangParser::BlockContext* ctx) override;
std::any visitMulDivExpr(
MotorLangParser::MulDivExprContext* ctx) override;
std::any visitAddSubExpr(
MotorLangParser::AddSubExprContext* ctx) override;
std::any visitCompareExpr(
MotorLangParser::CompareExprContext* ctx) override;
std::any visitParenExpr(
MotorLangParser::ParenExprContext* ctx) override;
std::any visitVarExpr(
MotorLangParser::VarExprContext* ctx) override;
std::any visitIntLiteralExpr(
MotorLangParser::IntLiteralExprContext* ctx) override;
std::any visitFloatLiteralExpr(
MotorLangParser::FloatLiteralExprContext* ctx) override;
diag::DiagnosticEngine& engine_;
std::shared_ptr<const std::string> filePath_;
};
} // namespace frontend
// src/frontend/ASTBuilder.cpp
#include "ASTBuilder.h"
#include "MotorLangParser.h"
#include <stdexcept>
namespace frontend {
diag::SourceRange ASTBuilder::rangeFromContext(
antlr4::ParserRuleContext* ctx) const
{
if (!ctx || !ctx->start || !ctx->stop) return {};
antlr4::Token* startTok = ctx->start;
antlr4::Token* stopTok = ctx->stop;
uint32_t startLine =
static_cast<uint32_t>(startTok->getLine());
uint32_t startCol =
static_cast<uint32_t>(
startTok->getCharPositionInLine()) + 1;
uint32_t startOff =
static_cast<uint32_t>(startTok->getStartIndex());
uint32_t stopLine =
static_cast<uint32_t>(stopTok->getLine());
uint32_t stopCol =
static_cast<uint32_t>(
stopTok->getCharPositionInLine())
+ static_cast<uint32_t>(stopTok->getText().size()) + 1;
uint32_t stopOff =
static_cast<uint32_t>(stopTok->getStopIndex()) + 1;
diag::SourceLocation begin(filePath_, startLine,
startCol, startOff);
diag::SourceLocation end(filePath_, stopLine,
stopCol, stopOff);
return { begin, end };
}
diag::SourceLocation ASTBuilder::locFromToken(
antlr4::Token* tok) const
{
if (!tok) return {};
uint32_t line =
static_cast<uint32_t>(tok->getLine());
uint32_t col =
static_cast<uint32_t>(
tok->getCharPositionInLine()) + 1;
uint32_t off =
static_cast<uint32_t>(tok->getStartIndex());
return diag::SourceLocation(filePath_, line, col, off);
}
std::unique_ptr<ast::Program>
ASTBuilder::buildProgram(
MotorLangParser::ProgramContext* ctx)
{
auto result = std::any_cast<std::unique_ptr<ast::Program>>(
visit(ctx));
return result;
}
std::any ASTBuilder::visitProgram(
MotorLangParser::ProgramContext* ctx)
{
auto program = std::make_unique<ast::Program>();
for (auto* decl : ctx->declaration()) {
auto result = visit(decl);
// Each declaration visitor returns a FunctionDecl or
// a VarDeclStmt; we handle both cases here.
if (result.has_value()) {
try {
auto fn = std::any_cast<
std::unique_ptr<ast::FunctionDecl>>(
std::move(result));
if (fn) program->functions.push_back(
std::move(fn));
} catch (const std::bad_any_cast&) {
// Not a function decl; other declaration types
// would be handled here in a full implementation.
}
}
}
return std::make_any<std::unique_ptr<ast::Program>>(
std::move(program));
}
std::any ASTBuilder::visitFunctionDecl(
MotorLangParser::FunctionDeclContext* ctx)
{
diag::SourceRange range = rangeFromContext(ctx);
std::string retType = ctx->type()->getText();
std::string name = ctx->IDENTIFIER()->getText();
auto fn = std::make_unique<ast::FunctionDecl>(
range, retType, name);
if (ctx->paramList()) {
for (auto* param : ctx->paramList()->param()) {
std::string pType = param->type()->getText();
std::string pName = param->IDENTIFIER()->getText();
diag::SourceLocation pLoc =
locFromToken(param->IDENTIFIER()->getSymbol());
fn->addParam(pType, pName, pLoc);
}
}
auto blockResult = visit(ctx->block());
auto block = std::any_cast<std::unique_ptr<ast::BlockStmt>>(
std::move(blockResult));
fn->setBody(std::move(block));
return std::make_any<std::unique_ptr<ast::FunctionDecl>>(
std::move(fn));
}
std::any ASTBuilder::visitBlock(
MotorLangParser::BlockContext* ctx)
{
diag::SourceRange range = rangeFromContext(ctx);
auto block = std::make_unique<ast::BlockStmt>(range);
for (auto* stmt : ctx->statement()) {
auto result = visit(stmt);
if (result.has_value()) {
try {
auto s = std::any_cast<
std::unique_ptr<ast::Stmt>>(
std::move(result));
if (s) block->addStatement(std::move(s));
} catch (const std::bad_any_cast&) {}
}
}
return std::make_any<std::unique_ptr<ast::BlockStmt>>(
std::move(block));
}
std::any ASTBuilder::visitVarDeclStmt(
MotorLangParser::VarDeclStmtContext* ctx)
{
diag::SourceRange range = rangeFromContext(ctx);
std::string type = ctx->type()->getText();
std::string name = ctx->IDENTIFIER()->getText();
std::unique_ptr<ast::Expr> init;
if (ctx->expr()) {
auto result = visit(ctx->expr());
init = std::any_cast<std::unique_ptr<ast::Expr>>(
std::move(result));
}
auto stmt = std::make_unique<ast::VarDeclStmt>(
range, type, name, std::move(init));
return std::make_any<std::unique_ptr<ast::Stmt>>(
std::move(stmt));
}
std::any ASTBuilder::visitAssignStmt(
MotorLangParser::AssignStmtContext* ctx)
{
diag::SourceRange range = rangeFromContext(ctx);
std::string target = ctx->IDENTIFIER()->getText();
auto valResult = visit(ctx->expr());
auto val = std::any_cast<std::unique_ptr<ast::Expr>>(
std::move(valResult));
auto stmt = std::make_unique<ast::AssignStmt>(
range, target, std::move(val));
return std::make_any<std::unique_ptr<ast::Stmt>>(
std::move(stmt));
}
std::any ASTBuilder::visitReturnStmt(
MotorLangParser::ReturnStmtContext* ctx)
{
diag::SourceRange range = rangeFromContext(ctx);
std::unique_ptr<ast::Expr> val;
if (ctx->expr()) {
auto result = visit(ctx->expr());
val = std::any_cast<std::unique_ptr<ast::Expr>>(
std::move(result));
}
auto stmt = std::make_unique<ast::ReturnStmt>(
range, std::move(val));
return std::make_any<std::unique_ptr<ast::Stmt>>(
std::move(stmt));
}
std::any ASTBuilder::visitIfStmt(
MotorLangParser::IfStmtContext* ctx)
{
diag::SourceRange range = rangeFromContext(ctx);
auto condResult = visit(ctx->expr());
auto cond = std::any_cast<std::unique_ptr<ast::Expr>>(
std::move(condResult));
auto thenResult = visit(ctx->block(0));
auto thenBranch = std::any_cast<std::unique_ptr<ast::BlockStmt>>(
std::move(thenResult));
std::unique_ptr<ast::Stmt> elseBranch;
if (ctx->block().size() > 1) {
auto elseResult = visit(ctx->block(1));
elseBranch = std::any_cast<std::unique_ptr<ast::BlockStmt>>(
std::move(elseResult));
}
auto stmt = std::make_unique<ast::IfStmt>(
range, std::move(cond),
std::move(thenBranch), std::move(elseBranch));
return std::make_any<std::unique_ptr<ast::Stmt>>(
std::move(stmt));
}
std::any ASTBuilder::visitWhileStmt(
MotorLangParser::WhileStmtContext* ctx)
{
diag::SourceRange range = rangeFromContext(ctx);
auto condResult = visit(ctx->expr());
auto cond = std::any_cast<std::unique_ptr<ast::Expr>>(
std::move(condResult));
auto bodyResult = visit(ctx->block());
auto body = std::any_cast<std::unique_ptr<ast::BlockStmt>>(
std::move(bodyResult));
auto stmt = std::make_unique<ast::WhileStmt>(
range, std::move(cond), std::move(body));
return std::make_any<std::unique_ptr<ast::Stmt>>(
std::move(stmt));
}
std::any ASTBuilder::visitMulDivExpr(
MotorLangParser::MulDivExprContext* ctx)
{
diag::SourceRange range = rangeFromContext(ctx);
std::string op = ctx->op->getText();
auto leftResult = visit(ctx->expr(0));
auto rightResult = visit(ctx->expr(1));
auto left = std::any_cast<std::unique_ptr<ast::Expr>>(
std::move(leftResult));
auto right = std::any_cast<std::unique_ptr<ast::Expr>>(
std::move(rightResult));
auto expr = std::make_unique<ast::BinaryExpr>(
range, op, std::move(left), std::move(right));
return std::make_any<std::unique_ptr<ast::Expr>>(
std::move(expr));
}
std::any ASTBuilder::visitAddSubExpr(
MotorLangParser::AddSubExprContext* ctx)
{
diag::SourceRange range = rangeFromContext(ctx);
std::string op = ctx->op->getText();
auto left = std::any_cast<std::unique_ptr<ast::Expr>>(
visit(ctx->expr(0)));
auto right = std::any_cast<std::unique_ptr<ast::Expr>>(
visit(ctx->expr(1)));
auto expr = std::make_unique<ast::BinaryExpr>(
range, op, std::move(left), std::move(right));
return std::make_any<std::unique_ptr<ast::Expr>>(
std::move(expr));
}
std::any ASTBuilder::visitCompareExpr(
MotorLangParser::CompareExprContext* ctx)
{
diag::SourceRange range = rangeFromContext(ctx);
std::string op = ctx->op->getText();
auto left = std::any_cast<std::unique_ptr<ast::Expr>>(
visit(ctx->expr(0)));
auto right = std::any_cast<std::unique_ptr<ast::Expr>>(
visit(ctx->expr(1)));
auto expr = std::make_unique<ast::BinaryExpr>(
range, op, std::move(left), std::move(right));
return std::make_any<std::unique_ptr<ast::Expr>>(
std::move(expr));
}
std::any ASTBuilder::visitParenExpr(
MotorLangParser::ParenExprContext* ctx)
{
// Parentheses are transparent: we just visit the inner expression.
return visit(ctx->expr());
}
std::any ASTBuilder::visitVarExpr(
MotorLangParser::VarExprContext* ctx)
{
diag::SourceRange range = rangeFromContext(ctx);
std::string name = ctx->IDENTIFIER()->getText();
auto expr = std::make_unique<ast::VariableExpr>(range, name);
return std::make_any<std::unique_ptr<ast::Expr>>(
std::move(expr));
}
std::any ASTBuilder::visitIntLiteralExpr(
MotorLangParser::IntLiteralExprContext* ctx)
{
diag::SourceRange range = rangeFromContext(ctx);
int64_t value = 0;
try {
std::string text = ctx->INT_LITERAL()->getText();
if (text.size() > 2 && text[0] == '0' && text[1] == 'x') {
value = static_cast<int64_t>(
std::stoull(text.substr(2), nullptr, 16));
} else if (text.size() > 2 &&
text[0] == '0' && text[1] == 'b') {
value = static_cast<int64_t>(
std::stoull(text.substr(2), nullptr, 2));
} else {
value = std::stoll(text);
}
} catch (const std::out_of_range&) {
diag::Diagnostic d;
d.code = diag::codes::LEX_INTEGER_OVERFLOW;
d.severity = diag::Severity::Error;
d.location = range.begin;
d.range = range;
d.message = "integer literal '"
+ ctx->INT_LITERAL()->getText()
+ "' is too large for type 'int'";
d.phase = "ASTBuilder";
engine_.emit(std::move(d));
}
auto expr = std::make_unique<ast::IntLiteralExpr>(range, value);
return std::make_any<std::unique_ptr<ast::Expr>>(
std::move(expr));
}
std::any ASTBuilder::visitFloatLiteralExpr(
MotorLangParser::FloatLiteralExprContext* ctx)
{
diag::SourceRange range = rangeFromContext(ctx);
double value = std::stod(ctx->FLOAT_LITERAL()->getText());
auto expr = std::make_unique<ast::FloatLiteralExpr>(range, value);
return std::make_any<std::unique_ptr<ast::Expr>>(
std::move(expr));
}
} // namespace frontend
CHAPTER TEN: SEMANTIC ANALYSIS AND TYPE CHECKING
Semantic analysis is where the most interesting and most numerous errors live. The parser can only catch syntactic problems: missing semicolons, unmatched braces, tokens in the wrong order. Semantic analysis catches the deeper problems: using a variable before declaring it, calling a function with the wrong number of arguments, assigning a value of the wrong type. Each of these errors requires not just a location but context, and building that context into the error message is where you earn the trust of your users.
Let us start with the symbol table, which is the data structure that tracks what names are in scope and what we know about them.
// src/sema/SymbolTable.h
//
// A scoped symbol table that records declared names along with
// their types and declaration locations. The declaration location
// is essential for generating "declared here" notes in diagnostics.
#pragma once
#include "diag/SourceLocation.h"
#include <string>
#include <vector>
#include <unordered_map>
#include <optional>
namespace sema {
// Everything we know about a declared name.
struct SymbolInfo {
std::string type;
diag::SourceLocation declarationLoc;
diag::SourceRange declarationRange;
bool isInitialized = false;
bool isParameter = false;
};
class SymbolTable {
public:
void enterScope();
void exitScope();
// Declare a name in the current scope.
// Returns false if the name is already declared in this scope.
bool declare(const std::string& name, SymbolInfo info);
// Look up a name, searching from innermost to outermost scope.
std::optional<SymbolInfo> lookup(
const std::string& name) const;
// Check if a name is declared in the current (innermost) scope.
bool isDeclaredInCurrentScope(
const std::string& name) const;
// Mark a variable as initialized (called after assignment).
void markInitialized(const std::string& name);
// Find the declared name most similar to the given name,
// using Levenshtein edit distance. Returns nullopt if no
// sufficiently similar name exists.
std::optional<std::pair<std::string, SymbolInfo>>
findSimilar(const std::string& name) const;
private:
static int editDistance(const std::string& a,
const std::string& b);
std::vector<
std::unordered_map<std::string, SymbolInfo>> scopes_;
};
} // namespace sema
// src/sema/SymbolTable.cpp
#include "SymbolTable.h"
#include <algorithm>
#include <climits>
namespace sema {
void SymbolTable::enterScope() {
scopes_.push_back({});
}
void SymbolTable::exitScope() {
if (!scopes_.empty()) scopes_.pop_back();
}
bool SymbolTable::declare(const std::string& name,
SymbolInfo info) {
if (scopes_.empty()) return false;
auto& current = scopes_.back();
if (current.count(name)) return false;
current[name] = std::move(info);
return true;
}
std::optional<SymbolInfo>
SymbolTable::lookup(const std::string& name) const {
for (auto it = scopes_.rbegin();
it != scopes_.rend(); ++it) {
auto found = it->find(name);
if (found != it->end()) return found->second;
}
return std::nullopt;
}
bool SymbolTable::isDeclaredInCurrentScope(
const std::string& name) const {
if (scopes_.empty()) return false;
return scopes_.back().count(name) > 0;
}
void SymbolTable::markInitialized(const std::string& name) {
for (auto it = scopes_.rbegin();
it != scopes_.rend(); ++it) {
auto found = it->find(name);
if (found != it->end()) {
found->second.isInitialized = true;
return;
}
}
}
std::optional<std::pair<std::string, SymbolInfo>>
SymbolTable::findSimilar(const std::string& name) const {
// Only suggest names within an edit distance of 3.
// Beyond that, suggestions become more confusing than helpful.
const int maxDistance = 3;
std::string bestName;
SymbolInfo bestInfo;
int bestDist = INT_MAX;
for (const auto& scope : scopes_) {
for (const auto& [symName, symInfo] : scope) {
int dist = editDistance(name, symName);
if (dist < bestDist && dist <= maxDistance) {
bestDist = dist;
bestName = symName;
bestInfo = symInfo;
}
}
}
if (bestDist == INT_MAX) return std::nullopt;
return std::make_pair(bestName, bestInfo);
}
int SymbolTable::editDistance(const std::string& a,
const std::string& b) {
// Standard Levenshtein distance with a single-row DP table.
const int m = static_cast<int>(a.size());
const int n = static_cast<int>(b.size());
std::vector<int> dp(n + 1);
for (int j = 0; j <= n; ++j) dp[j] = j;
for (int i = 1; i <= m; ++i) {
int prev = dp[0];
dp[0] = i;
for (int j = 1; j <= n; ++j) {
int temp = dp[j];
if (a[static_cast<size_t>(i - 1)] ==
b[static_cast<size_t>(j - 1)]) {
dp[j] = prev;
} else {
dp[j] = 1 + std::min({prev, dp[j], dp[j - 1]});
}
prev = temp;
}
}
return dp[n];
}
} // namespace sema
Now the type checker itself. This is the most error-rich phase of the compiler, and the quality of its error messages will define the user experience more than any other single component.
// src/sema/TypeChecker.h
//
// Performs type checking on the AST and reports semantic errors
// through the DiagnosticEngine. This phase is the primary source
// of semantic error diagnostics in the compiler.
#pragma once
#include "ast/AST.h"
#include "diag/DiagnosticEngine.h"
#include "SymbolTable.h"
#include <string>
namespace sema {
class TypeChecker : public ast::ASTVisitor {
public:
explicit TypeChecker(diag::DiagnosticEngine& engine);
void visit(ast::IntLiteralExpr&) override;
void visit(ast::FloatLiteralExpr&) override;
void visit(ast::VariableExpr&) override;
void visit(ast::BinaryExpr&) override;
void visit(ast::AssignStmt&) override;
void visit(ast::VarDeclStmt&) override;
void visit(ast::IfStmt&) override;
void visit(ast::WhileStmt&) override;
void visit(ast::ReturnStmt&) override;
void visit(ast::BlockStmt&) override;
void visit(ast::FunctionDecl&) override;
private:
bool isAssignable(const std::string& target,
const std::string& source) const;
bool areCompatible(const std::string& a,
const std::string& b) const;
bool isNumericType(const std::string& type) const;
std::string promoteTypes(const std::string& a,
const std::string& b) const;
diag::DiagnosticEngine& engine_;
SymbolTable symbols_;
std::string currentReturnType_;
};
} // namespace sema
// src/sema/TypeChecker.cpp
#include "TypeChecker.h"
#include "diag/DiagnosticCodes.h"
namespace sema {
TypeChecker::TypeChecker(diag::DiagnosticEngine& engine)
: engine_(engine)
{
symbols_.enterScope(); // Global scope.
}
void TypeChecker::visit(ast::IntLiteralExpr& node) {
node.resolvedType = "int";
}
void TypeChecker::visit(ast::FloatLiteralExpr& node) {
node.resolvedType = "float";
}
void TypeChecker::visit(ast::VariableExpr& node) {
auto info = symbols_.lookup(node.name());
if (!info.has_value()) {
diag::Diagnostic d;
d.code = diag::codes::SEM_UNDECLARED_VARIABLE;
d.severity = diag::Severity::Error;
d.location = node.loc();
d.range = node.range();
d.message = "use of undeclared variable '"
+ node.name() + "'";
d.phase = "TypeChecker";
// Attempt a "did you mean?" suggestion using edit distance.
auto suggestion = symbols_.findSimilar(node.name());
if (suggestion.has_value()) {
d.message += "; did you mean '"
+ suggestion->first + "'?";
diag::Diagnostic note;
note.code = diag::codes::SEM_DECLARATION_NOTE;
note.severity = diag::Severity::Note;
note.location = suggestion->second.declarationLoc;
note.range = suggestion->second.declarationRange;
note.message = "'" + suggestion->first
+ "' was declared here";
note.phase = "TypeChecker";
d.notes.push_back(std::move(note));
}
engine_.emit(std::move(d));
node.resolvedType = "<error>";
return;
}
// Warn about use before initialization.
if (!info->isInitialized && !info->isParameter) {
diag::Diagnostic d;
d.code = diag::codes::SEM_UNINITIALIZED_VAR;
d.severity = diag::Severity::Warning;
d.location = node.loc();
d.range = node.range();
d.message = "variable '" + node.name()
+ "' is used before it has been initialized";
d.phase = "TypeChecker";
diag::Diagnostic note;
note.code = diag::codes::SEM_DECLARATION_NOTE;
note.severity = diag::Severity::Note;
note.location = info->declarationLoc;
note.range = info->declarationRange;
note.message = "'" + node.name()
+ "' was declared here without initialization";
note.phase = "TypeChecker";
d.notes.push_back(std::move(note));
engine_.emit(std::move(d));
}
node.resolvedType = info->type;
}
void TypeChecker::visit(ast::BinaryExpr& node) {
node.left().accept(*this);
node.right().accept(*this);
const std::string& leftType = node.left().resolvedType;
const std::string& rightType = node.right().resolvedType;
// Skip type checking if either operand already has an error.
// This prevents cascading errors from a single mistake.
if (leftType == "<error>" || rightType == "<error>") {
node.resolvedType = "<error>";
return;
}
if (!areCompatible(leftType, rightType)) {
diag::Diagnostic d;
d.code = diag::codes::SEM_INCOMPATIBLE_OPERANDS;
d.severity = diag::Severity::Error;
d.location = node.loc();
d.range = node.range();
d.message = "binary operator '" + node.op()
+ "' cannot be applied to operands of type '"
+ leftType + "' and '" + rightType + "'";
d.phase = "TypeChecker";
// Point to each operand individually with notes.
// This is the multi-location diagnostic pattern that makes
// type errors so much easier to understand.
diag::Diagnostic leftNote;
leftNote.code = diag::codes::SEM_LEFT_OPERAND_NOTE;
leftNote.severity = diag::Severity::Note;
leftNote.location = node.left().loc();
leftNote.range = node.left().range();
leftNote.message = "left operand has type '"
+ leftType + "'";
leftNote.phase = "TypeChecker";
d.notes.push_back(std::move(leftNote));
diag::Diagnostic rightNote;
rightNote.code = diag::codes::SEM_RIGHT_OPERAND_NOTE;
rightNote.severity = diag::Severity::Note;
rightNote.location = node.right().loc();
rightNote.range = node.right().range();
rightNote.message = "right operand has type '"
+ rightType + "'";
rightNote.phase = "TypeChecker";
d.notes.push_back(std::move(rightNote));
engine_.emit(std::move(d));
node.resolvedType = "<error>";
return;
}
node.resolvedType = promoteTypes(leftType, rightType);
}
void TypeChecker::visit(ast::VarDeclStmt& node) {
const std::string& name = node.name();
const std::string& type = node.type();
if (symbols_.isDeclaredInCurrentScope(name)) {
auto existing = symbols_.lookup(name);
diag::Diagnostic d;
d.code = diag::codes::SEM_REDECLARATION;
d.severity = diag::Severity::Error;
d.location = node.loc();
d.range = node.range();
d.message = "redeclaration of variable '" + name + "'";
d.phase = "TypeChecker";
if (existing.has_value()) {
diag::Diagnostic note;
note.code = diag::codes::SEM_DECLARATION_NOTE;
note.severity = diag::Severity::Note;
note.location = existing->declarationLoc;
note.range = existing->declarationRange;
note.message = "'" + name
+ "' was previously declared here";
note.phase = "TypeChecker";
d.notes.push_back(std::move(note));
}
engine_.emit(std::move(d));
return;
}
if (node.initializer()) {
node.initializer()->accept(*this);
const std::string& initType =
node.initializer()->resolvedType;
if (initType != "<error>" &&
!isAssignable(type, initType)) {
diag::Diagnostic d;
d.code = diag::codes::SEM_TYPE_MISMATCH_INIT;
d.severity = diag::Severity::Error;
d.location = node.initializer()->loc();
d.range = node.initializer()->range();
d.message = "cannot initialize variable '"
+ name + "' of type '" + type
+ "' with a value of type '"
+ initType + "'";
d.phase = "TypeChecker";
if (isNumericType(type) && isNumericType(initType)) {
d.message += "; consider an explicit cast";
d.fixItHints.push_back(
diag::FixItHint::makeInsertion(
node.initializer()->loc(),
"(" + type + ")"));
}
engine_.emit(std::move(d));
}
}
SymbolInfo info;
info.type = type;
info.declarationLoc = node.loc();
info.declarationRange = node.range();
info.isInitialized = (node.initializer() != nullptr);
symbols_.declare(name, std::move(info));
}
void TypeChecker::visit(ast::AssignStmt& node) {
auto info = symbols_.lookup(node.target());
if (!info.has_value()) {
diag::Diagnostic d;
d.code = diag::codes::SEM_UNDECLARED_VARIABLE;
d.severity = diag::Severity::Error;
d.location = node.loc();
d.range = node.range();
d.message = "assignment to undeclared variable '"
+ node.target() + "'";
d.phase = "TypeChecker";
auto suggestion = symbols_.findSimilar(node.target());
if (suggestion.has_value()) {
d.message += "; did you mean '"
+ suggestion->first + "'?";
}
engine_.emit(std::move(d));
return;
}
node.value().accept(*this);
const std::string& valueType = node.value().resolvedType;
if (valueType != "<error>" &&
!isAssignable(info->type, valueType)) {
diag::Diagnostic d;
d.code = diag::codes::SEM_TYPE_MISMATCH_INIT;
d.severity = diag::Severity::Error;
d.location = node.value().loc();
d.range = node.value().range();
d.message = "cannot assign value of type '"
+ valueType + "' to variable '"
+ node.target() + "' of type '"
+ info->type + "'";
d.phase = "TypeChecker";
diag::Diagnostic note;
note.code = diag::codes::SEM_DECLARATION_NOTE;
note.severity = diag::Severity::Note;
note.location = info->declarationLoc;
note.range = info->declarationRange;
note.message = "'" + node.target()
+ "' was declared here as '"
+ info->type + "'";
note.phase = "TypeChecker";
d.notes.push_back(std::move(note));
if (isNumericType(info->type) &&
isNumericType(valueType)) {
d.fixItHints.push_back(
diag::FixItHint::makeInsertion(
node.value().loc(),
"(" + info->type + ")"));
}
engine_.emit(std::move(d));
return;
}
symbols_.markInitialized(node.target());
}
void TypeChecker::visit(ast::IfStmt& node) {
node.condition().accept(*this);
symbols_.enterScope();
node.thenBranch().accept(*this);
symbols_.exitScope();
if (node.elseBranch()) {
symbols_.enterScope();
node.elseBranch()->accept(*this);
symbols_.exitScope();
}
}
void TypeChecker::visit(ast::WhileStmt& node) {
node.condition().accept(*this);
symbols_.enterScope();
node.body().accept(*this);
symbols_.exitScope();
}
void TypeChecker::visit(ast::ReturnStmt& node) {
if (node.value()) {
node.value()->accept(*this);
}
}
void TypeChecker::visit(ast::BlockStmt& node) {
for (const auto& stmt : node.statements()) {
stmt->accept(*this);
}
}
void TypeChecker::visit(ast::FunctionDecl& node) {
currentReturnType_ = node.returnType();
symbols_.enterScope();
for (const auto& param : node.params()) {
SymbolInfo info;
info.type = param.type;
info.declarationLoc = param.loc;
info.declarationRange = { param.loc, param.loc };
info.isInitialized = true;
info.isParameter = true;
symbols_.declare(param.name, std::move(info));
}
if (node.body()) {
node.body()->accept(*this);
}
symbols_.exitScope();
}
bool TypeChecker::isAssignable(const std::string& target,
const std::string& source) const {
if (target == source) return true;
// Allow implicit int-to-float and int-to-double promotion.
if ((target == "float" || target == "double") &&
source == "int") return true;
if (target == "double" && source == "float") return true;
return false;
}
bool TypeChecker::areCompatible(const std::string& a,
const std::string& b) const {
return a == b ||
(isNumericType(a) && isNumericType(b));
}
bool TypeChecker::isNumericType(const std::string& type) const {
return type == "int" || type == "float" ||
type == "double"|| type == "long";
}
std::string TypeChecker::promoteTypes(const std::string& a,
const std::string& b) const {
if (a == "double" || b == "double") return "double";
if (a == "float" || b == "float") return "float";
return "int";
}
} // namespace sema
CHAPTER ELEVEN: PROPAGATING LOCATION INFORMATION INTO LLVM IR
When we move from the AST to LLVM IR, we face a new challenge. LLVM IR is a low-level, typed, SSA-form intermediate representation that has its own location system based on the DWARF debugging standard. The key insight is that we should attach debug locations to every IR instruction we generate, even if the user did not request a debug build. This ensures that backend errors can always be traced back to the source, and it enables source-level debugging as a bonus.
// src/codegen/LLVMDiagnosticBridge.h
//
// Bridges LLVM's internal diagnostic system to our DiagnosticEngine.
// This ensures that LLVM backend errors are reported with the same
// quality and format as frontend errors.
#pragma once
#include "diag/DiagnosticEngine.h"
#include <llvm/IR/LLVMContext.h>
namespace codegen {
// Installs our diagnostic handler into an LLVMContext.
// Call this immediately after creating the LLVMContext, before
// any IR generation begins.
void installLLVMDiagnosticHandler(
llvm::LLVMContext& ctx,
diag::DiagnosticEngine& engine);
} // namespace codegen
// src/codegen/LLVMDiagnosticBridge.cpp
#include "LLVMDiagnosticBridge.h"
#include "diag/DiagnosticCodes.h"
#include <llvm/IR/DiagnosticInfo.h>
#include <llvm/IR/DiagnosticPrinter.h>
#include <sstream>
#include <memory>
namespace codegen {
// Context structure passed through the LLVM callback mechanism.
struct HandlerContext {
diag::DiagnosticEngine* engine;
};
static void llvmDiagnosticHandler(
const llvm::DiagnosticInfo& diagInfo,
void* opaqueContext)
{
auto* ctx =
static_cast<HandlerContext*>(opaqueContext);
if (!ctx || !ctx->engine) return;
diag::DiagnosticEngine& engine = *ctx->engine;
// Extract the message text from LLVM's diagnostic.
std::string message;
{
llvm::raw_string_ostream stream(message);
llvm::DiagnosticPrinterRawOStream printer(stream);
diagInfo.print(printer);
}
// Map LLVM's severity to ours.
diag::Severity severity;
switch (diagInfo.getSeverity()) {
case llvm::DS_Error:
severity = diag::Severity::Error;
break;
case llvm::DS_Warning:
severity = diag::Severity::Warning;
break;
case llvm::DS_Remark:
severity = diag::Severity::Remark;
break;
case llvm::DS_Note:
severity = diag::Severity::Note;
break;
default:
severity = diag::Severity::Error;
break;
}
diag::Diagnostic d;
d.severity = severity;
d.phase = "LLVMBackend";
d.message = "LLVM backend: " + message;
d.code = diag::codes::GEN_LLVM_BACKEND
+ static_cast<uint32_t>(
diagInfo.getKind() % 1000);
// Attempt to extract a source location from the diagnostic.
if (const auto* withLoc =
llvm::dyn_cast<
llvm::DiagnosticInfoWithLocationBase>(&diagInfo)) {
std::string file =
withLoc->getAbsolutePath().str();
uint32_t line =
static_cast<uint32_t>(withLoc->getLine());
uint32_t col =
static_cast<uint32_t>(withLoc->getColumn());
auto filePath =
std::make_shared<const std::string>(file);
d.location =
diag::SourceLocation(filePath, line, col, 0);
}
engine.emit(std::move(d));
}
// We store the HandlerContext as a static so it outlives the
// LLVMContext's use of the pointer. In a multi-threaded compiler
// this would need to be per-context storage.
static HandlerContext gHandlerContext;
void installLLVMDiagnosticHandler(
llvm::LLVMContext& ctx,
diag::DiagnosticEngine& engine)
{
gHandlerContext.engine = &engine;
ctx.setDiagnosticHandlerCallBack(
llvmDiagnosticHandler,
&gHandlerContext);
}
} // namespace codegen
Now the IR generator itself. Every instruction is generated with a debug location attached, which is the bridge between our source location system and LLVM's debug information system.
// src/codegen/IRGenerator.h
//
// Generates LLVM IR from the AST, attaching debug location
// information to every instruction for precise backend error
// reporting and source-level debugging support.
#pragma once
#include "ast/AST.h"
#include "diag/DiagnosticEngine.h"
#include "diag/SourceLocation.h"
#include <llvm/IR/IRBuilder.h>
#include <llvm/IR/LLVMContext.h>
#include <llvm/IR/Module.h>
#include <llvm/IR/DIBuilder.h>
#include <llvm/IR/DebugInfoMetadata.h>
#include <memory>
#include <string>
#include <unordered_map>
namespace codegen {
class IRGenerator : public ast::ASTVisitor {
public:
IRGenerator(
llvm::LLVMContext& ctx,
llvm::Module& module,
diag::DiagnosticEngine& engine,
const std::string& sourceFilePath);
// Call after visiting all AST nodes to finalize debug info.
void finalize();
// Visitor methods.
void visit(ast::IntLiteralExpr&) override;
void visit(ast::FloatLiteralExpr&) override;
void visit(ast::VariableExpr&) override;
void visit(ast::BinaryExpr&) override;
void visit(ast::AssignStmt&) override;
void visit(ast::VarDeclStmt&) override;
void visit(ast::IfStmt&) override;
void visit(ast::WhileStmt&) override;
void visit(ast::ReturnStmt&) override;
void visit(ast::BlockStmt&) override;
void visit(ast::FunctionDecl&) override;
private:
void setDebugLoc(const diag::SourceLocation& loc);
llvm::Type* llvmTypeFor(const std::string& typeName,
const diag::SourceLocation& loc);
llvm::DIType* diTypeFor(const std::string& typeName);
// Emits an internal compiler error diagnostic. Used when the
// IR generator encounters a state that should have been caught
// by an earlier phase, indicating a bug in the compiler itself.
void emitInternalError(
const diag::SourceLocation& loc,
const std::string& message,
const std::string& compilerFile,
int compilerLine);
llvm::LLVMContext& ctx_;
llvm::Module& module_;
diag::DiagnosticEngine& engine_;
llvm::IRBuilder<> builder_;
llvm::DIBuilder diBuilder_;
llvm::DICompileUnit* diCompileUnit_ = nullptr;
llvm::DIFile* diFile_ = nullptr;
llvm::DIScope* diScope_ = nullptr;
llvm::Function* currentFunction_ = nullptr;
// The last computed value, used to pass results between
// visitor calls (since the visitor interface returns void).
llvm::Value* lastValue_ = nullptr;
std::unordered_map<std::string, llvm::AllocaInst*> namedValues_;
};
} // namespace codegen
// src/codegen/IRGenerator.cpp
#include "IRGenerator.h"
#include "diag/DiagnosticCodes.h"
#include <llvm/IR/Verifier.h>
#include <llvm/IR/BasicBlock.h>
#include <llvm/IR/Constants.h>
#include <llvm/IR/Function.h>
#include <llvm/IR/Type.h>
#include <llvm/Support/Path.h>
namespace codegen {
IRGenerator::IRGenerator(
llvm::LLVMContext& ctx,
llvm::Module& module,
diag::DiagnosticEngine& engine,
const std::string& sourceFilePath)
: ctx_(ctx)
, module_(module)
, engine_(engine)
, builder_(ctx)
, diBuilder_(module)
{
// Set up the debug info compile unit. Even when not generating
// a full debug build, we need this to attach DILocations to
// instructions for backend error reporting.
llvm::StringRef filename =
llvm::sys::path::filename(sourceFilePath);
llvm::StringRef directory =
llvm::sys::path::parent_path(sourceFilePath);
diFile_ = diBuilder_.createFile(filename, directory);
diCompileUnit_ = diBuilder_.createCompileUnit(
llvm::dwarf::DW_LANG_C,
diFile_,
"MotorLang Compiler 1.0",
/*isOptimized=*/false,
/*Flags=*/"",
/*RuntimeVersion=*/0);
}
void IRGenerator::finalize() {
diBuilder_.finalize();
}
void IRGenerator::setDebugLoc(
const diag::SourceLocation& loc)
{
if (!loc.isValid() || !diScope_) return;
llvm::DebugLoc dl = llvm::DILocation::get(
ctx_,
loc.line,
loc.column,
diScope_);
builder_.SetCurrentDebugLocation(dl);
}
void IRGenerator::visit(ast::IntLiteralExpr& node) {
setDebugLoc(node.loc());
lastValue_ = llvm::ConstantInt::get(
llvm::Type::getInt64Ty(ctx_),
static_cast<uint64_t>(node.value()),
/*isSigned=*/true);
}
void IRGenerator::visit(ast::FloatLiteralExpr& node) {
setDebugLoc(node.loc());
lastValue_ = llvm::ConstantFP::get(
llvm::Type::getDoubleTy(ctx_),
node.value());
}
void IRGenerator::visit(ast::VariableExpr& node) {
setDebugLoc(node.loc());
auto it = namedValues_.find(node.name());
if (it == namedValues_.end()) {
emitInternalError(
node.loc(),
"IRGenerator: variable '" + node.name()
+ "' not found; type checker should have caught this",
__FILE__, __LINE__);
lastValue_ = nullptr;
return;
}
llvm::AllocaInst* alloca = it->second;
llvm::Type* varType = alloca->getAllocatedType();
lastValue_ = builder_.CreateLoad(
varType, alloca, node.name() + ".load");
}
void IRGenerator::visit(ast::BinaryExpr& node) {
setDebugLoc(node.loc());
node.left().accept(*this);
llvm::Value* leftVal = lastValue_;
node.right().accept(*this);
llvm::Value* rightVal = lastValue_;
if (!leftVal || !rightVal) {
lastValue_ = nullptr;
return;
}
const std::string& op = node.op();
const std::string& type = node.resolvedType;
bool isFloat = (type == "float" || type == "double");
if (op == "+") {
lastValue_ = isFloat
? builder_.CreateFAdd(leftVal, rightVal, "fadd")
: builder_.CreateAdd(leftVal, rightVal, "add");
} else if (op == "-") {
lastValue_ = isFloat
? builder_.CreateFSub(leftVal, rightVal, "fsub")
: builder_.CreateSub(leftVal, rightVal, "sub");
} else if (op == "*") {
lastValue_ = isFloat
? builder_.CreateFMul(leftVal, rightVal, "fmul")
: builder_.CreateMul(leftVal, rightVal, "mul");
} else if (op == "/") {
lastValue_ = isFloat
? builder_.CreateFDiv(leftVal, rightVal, "fdiv")
: builder_.CreateSDiv(leftVal, rightVal, "sdiv");
} else if (op == "<") {
lastValue_ = isFloat
? builder_.CreateFCmpOLT(leftVal, rightVal, "fcmp.lt")
: builder_.CreateICmpSLT(leftVal, rightVal, "icmp.lt");
} else if (op == ">") {
lastValue_ = isFloat
? builder_.CreateFCmpOGT(leftVal, rightVal, "fcmp.gt")
: builder_.CreateICmpSGT(leftVal, rightVal, "icmp.gt");
} else if (op == "<=") {
lastValue_ = isFloat
? builder_.CreateFCmpOLE(leftVal, rightVal, "fcmp.le")
: builder_.CreateICmpSLE(leftVal, rightVal, "icmp.le");
} else if (op == ">=") {
lastValue_ = isFloat
? builder_.CreateFCmpOGE(leftVal, rightVal, "fcmp.ge")
: builder_.CreateICmpSGE(leftVal, rightVal, "icmp.ge");
} else if (op == "==") {
lastValue_ = isFloat
? builder_.CreateFCmpOEQ(leftVal, rightVal, "fcmp.eq")
: builder_.CreateICmpEQ(leftVal, rightVal, "icmp.eq");
} else if (op == "!=") {
lastValue_ = isFloat
? builder_.CreateFCmpONE(leftVal, rightVal, "fcmp.ne")
: builder_.CreateICmpNE(leftVal, rightVal, "icmp.ne");
} else {
emitInternalError(
node.loc(),
"IRGenerator: unknown binary operator '" + op + "'",
__FILE__, __LINE__);
lastValue_ = nullptr;
}
}
void IRGenerator::visit(ast::VarDeclStmt& node) {
setDebugLoc(node.loc());
llvm::Type* varType =
llvmTypeFor(node.type(), node.loc());
if (!varType) return;
// Allocate stack space at the entry block of the current
// function. This is the standard LLVM pattern for local
// variables; the mem2reg pass promotes them to SSA registers.
if (!currentFunction_) {
emitInternalError(
node.loc(),
"IRGenerator: VarDeclStmt outside of a function",
__FILE__, __LINE__);
return;
}
llvm::IRBuilder<> entryBuilder(
¤tFunction_->getEntryBlock(),
currentFunction_->getEntryBlock().begin());
llvm::AllocaInst* alloca =
entryBuilder.CreateAlloca(
varType, nullptr, node.name());
namedValues_[node.name()] = alloca;
// Attach debug variable info for source-level debugging.
llvm::DIType* diType = diTypeFor(node.type());
if (diType && diScope_) {
llvm::DILocalVariable* diVar =
diBuilder_.createAutoVariable(
diScope_,
node.name(),
diFile_,
node.loc().line,
diType);
diBuilder_.insertDeclare(
alloca,
diVar,
diBuilder_.createExpression(),
llvm::DILocation::get(
ctx_,
node.loc().line,
node.loc().column,
diScope_),
builder_.GetInsertBlock());
}
if (node.initializer()) {
node.initializer()->accept(*this);
if (lastValue_) {
builder_.CreateStore(lastValue_, alloca);
}
}
}
void IRGenerator::visit(ast::AssignStmt& node) {
setDebugLoc(node.loc());
auto it = namedValues_.find(node.target());
if (it == namedValues_.end()) {
emitInternalError(
node.loc(),
"IRGenerator: assignment target '" + node.target()
+ "' not found in namedValues_",
__FILE__, __LINE__);
return;
}
node.value().accept(*this);
if (lastValue_) {
builder_.CreateStore(lastValue_, it->second);
}
}
void IRGenerator::visit(ast::ReturnStmt& node) {
setDebugLoc(node.loc());
if (node.value()) {
node.value()->accept(*this);
if (lastValue_) {
builder_.CreateRet(lastValue_);
}
} else {
builder_.CreateRetVoid();
}
}
void IRGenerator::visit(ast::BlockStmt& node) {
for (const auto& stmt : node.statements()) {
stmt->accept(*this);
// Stop generating code after a return statement.
if (builder_.GetInsertBlock()->getTerminator()) break;
}
}
void IRGenerator::visit(ast::IfStmt& node) {
setDebugLoc(node.loc());
node.condition().accept(*this);
llvm::Value* condVal = lastValue_;
if (!condVal) return;
// Convert condition to i1 (boolean).
llvm::Value* condBool = builder_.CreateICmpNE(
condVal,
llvm::ConstantInt::get(condVal->getType(), 0),
"if.cond");
llvm::Function* fn = builder_.GetInsertBlock()->getParent();
llvm::BasicBlock* thenBB =
llvm::BasicBlock::Create(ctx_, "if.then", fn);
llvm::BasicBlock* elseBB =
llvm::BasicBlock::Create(ctx_, "if.else", fn);
llvm::BasicBlock* mergeBB =
llvm::BasicBlock::Create(ctx_, "if.merge", fn);
builder_.CreateCondBr(condBool, thenBB, elseBB);
builder_.SetInsertPoint(thenBB);
node.thenBranch().accept(*this);
if (!builder_.GetInsertBlock()->getTerminator()) {
builder_.CreateBr(mergeBB);
}
builder_.SetInsertPoint(elseBB);
if (node.elseBranch()) {
node.elseBranch()->accept(*this);
}
if (!builder_.GetInsertBlock()->getTerminator()) {
builder_.CreateBr(mergeBB);
}
builder_.SetInsertPoint(mergeBB);
}
void IRGenerator::visit(ast::WhileStmt& node) {
setDebugLoc(node.loc());
llvm::Function* fn = builder_.GetInsertBlock()->getParent();
llvm::BasicBlock* condBB =
llvm::BasicBlock::Create(ctx_, "while.cond", fn);
llvm::BasicBlock* bodyBB =
llvm::BasicBlock::Create(ctx_, "while.body", fn);
llvm::BasicBlock* exitBB =
llvm::BasicBlock::Create(ctx_, "while.exit", fn);
builder_.CreateBr(condBB);
builder_.SetInsertPoint(condBB);
node.condition().accept(*this);
llvm::Value* condVal = lastValue_;
if (!condVal) return;
llvm::Value* condBool = builder_.CreateICmpNE(
condVal,
llvm::ConstantInt::get(condVal->getType(), 0),
"while.cond.bool");
builder_.CreateCondBr(condBool, bodyBB, exitBB);
builder_.SetInsertPoint(bodyBB);
node.body().accept(*this);
if (!builder_.GetInsertBlock()->getTerminator()) {
builder_.CreateBr(condBB);
}
builder_.SetInsertPoint(exitBB);
}
void IRGenerator::visit(ast::FunctionDecl& node) {
// Build the LLVM function type from the parameter list.
llvm::Type* retType =
llvmTypeFor(node.returnType(), node.loc());
if (!retType) return;
std::vector<llvm::Type*> paramTypes;
for (const auto& param : node.params()) {
llvm::Type* pt = llvmTypeFor(param.type, param.loc);
if (!pt) return;
paramTypes.push_back(pt);
}
llvm::FunctionType* funcType =
llvm::FunctionType::get(retType, paramTypes, false);
llvm::Function* fn = llvm::Function::Create(
funcType,
llvm::Function::ExternalLinkage,
node.name(),
&module_);
// Set up debug information for this function.
llvm::DISubroutineType* diSubType =
diBuilder_.createSubroutineType(
diBuilder_.getOrCreateTypeArray({}));
llvm::DISubprogram* diSub = diBuilder_.createFunction(
diFile_,
node.name(),
llvm::StringRef(),
diFile_,
node.loc().line,
diSubType,
node.loc().line,
llvm::DINode::FlagPrototyped,
llvm::DISubprogram::SPFlagDefinition);
fn->setSubprogram(diSub);
diScope_ = diSub;
// Create the entry basic block.
llvm::BasicBlock* entryBB =
llvm::BasicBlock::Create(ctx_, "entry", fn);
builder_.SetInsertPoint(entryBB);
setDebugLoc(node.loc());
currentFunction_ = fn;
namedValues_.clear();
// Allocate stack slots for parameters and store the
// incoming argument values into them. This allows parameters
// to be treated like local variables by the rest of the code.
size_t argIdx = 0;
for (auto& arg : fn->args()) {
if (argIdx >= node.params().size()) break;
const auto& param = node.params()[argIdx];
arg.setName(param.name);
llvm::AllocaInst* alloca =
builder_.CreateAlloca(
arg.getType(), nullptr, param.name);
builder_.CreateStore(&arg, alloca);
namedValues_[param.name] = alloca;
++argIdx;
}
if (node.body()) {
node.body()->accept(*this);
}
// Ensure the function has a terminator in the last block.
if (!builder_.GetInsertBlock()->getTerminator()) {
if (retType->isVoidTy()) {
builder_.CreateRetVoid();
} else {
// Return a zero value as a fallback. The type checker
// should have caught missing return statements.
builder_.CreateRet(
llvm::Constant::getNullValue(retType));
}
}
}
llvm::Type* IRGenerator::llvmTypeFor(
const std::string& typeName,
const diag::SourceLocation& loc)
{
if (typeName == "int") return llvm::Type::getInt64Ty(ctx_);
if (typeName == "float") return llvm::Type::getFloatTy(ctx_);
if (typeName == "double") return llvm::Type::getDoubleTy(ctx_);
if (typeName == "bool") return llvm::Type::getInt1Ty(ctx_);
if (typeName == "void") return llvm::Type::getVoidTy(ctx_);
emitInternalError(loc,
"IRGenerator: unknown type '" + typeName + "'",
__FILE__, __LINE__);
return nullptr;
}
llvm::DIType* IRGenerator::diTypeFor(
const std::string& typeName)
{
if (typeName == "int")
return diBuilder_.createBasicType(
"int", 64, llvm::dwarf::DW_ATE_signed);
if (typeName == "float")
return diBuilder_.createBasicType(
"float", 32, llvm::dwarf::DW_ATE_float);
if (typeName == "double")
return diBuilder_.createBasicType(
"double", 64, llvm::dwarf::DW_ATE_float);
if (typeName == "bool")
return diBuilder_.createBasicType(
"bool", 1, llvm::dwarf::DW_ATE_boolean);
return nullptr;
}
void IRGenerator::emitInternalError(
const diag::SourceLocation& loc,
const std::string& message,
const std::string& compilerFile,
int compilerLine)
{
diag::Diagnostic d;
d.code = diag::codes::ICE_UNEXPECTED_STATE;
d.severity = diag::Severity::InternalError;
d.location = loc;
d.message = message;
d.phase = "IRGenerator";
d.compilerSourceLocation =
compilerFile + ":" + std::to_string(compilerLine);
engine_.emit(std::move(d));
}
} // namespace codegen
CHAPTER TWELVE: IR VERIFICATION
LLVM's IR verifier checks the generated IR for correctness before passing it to the optimization pipeline. Running it after IR generation is an essential practice, because it catches bugs in the IR generator that would otherwise manifest as mysterious crashes later. When the verifier finds a problem, we report it as an internal compiler error, because a verification failure always means a bug in our compiler, not in the user's source code.
// src/codegen/IRVerification.h
#pragma once
#include "diag/DiagnosticEngine.h"
#include <llvm/IR/Module.h>
namespace codegen {
// Verifies the LLVM module and reports any errors through the
// DiagnosticEngine as internal compiler errors.
// Returns true if the module is valid, false otherwise.
bool verifyGeneratedModule(
llvm::Module& module,
diag::DiagnosticEngine& engine);
} // namespace codegen
// src/codegen/IRVerification.cpp
#include "IRVerification.h"
#include "diag/DiagnosticCodes.h"
#include <llvm/IR/Verifier.h>
#include <llvm/Support/raw_ostream.h>
#include <sstream>
namespace codegen {
bool verifyGeneratedModule(
llvm::Module& module,
diag::DiagnosticEngine& engine)
{
std::string errorOutput;
llvm::raw_string_ostream errorStream(errorOutput);
// llvm::verifyModule returns true if the module is broken.
bool broken = llvm::verifyModule(module, &errorStream);
errorStream.flush();
if (!broken) return true;
// Parse the verifier output line by line and emit one
// diagnostic per failure. Each failure is a compiler bug.
std::istringstream lines(errorOutput);
std::string line;
while (std::getline(lines, line)) {
if (line.empty()) continue;
diag::Diagnostic d;
d.code = diag::codes::GEN_IR_VERIFICATION;
d.severity = diag::Severity::InternalError;
d.phase = "IRVerifier";
d.message = "LLVM IR verification failure: " + line
+ "; this is a bug in the compiler's code "
"generator, not in your source code";
engine.emit(std::move(d));
}
return false;
}
} // namespace codegen
CHAPTER THIRTEEN: THE COMPILER DRIVER
The compiler driver is the top-level orchestrator that connects all the pieces. It parses command-line arguments, sets up the SourceManager and DiagnosticEngine, invokes each compilation phase in sequence, and handles the transitions between phases gracefully when errors occur. Getting the phase sequencing right is important: we want to detect as many errors as possible in a single run, but we also want to stop early when further analysis would only produce noise.
// src/CompilerDriver.h
#pragma once
#include <string>
#include <cstdint>
namespace driver {
struct CompilerOptions {
std::string inputFile;
std::string outputFile = "a.out";
bool useColor = true;
bool developerMode = false;
bool jsonOutput = false;
std::string jsonOutputFile = "diagnostics.json";
bool treatWarningsAsErrors = false;
uint32_t maxErrors = 20;
bool emitLLVMIR = false;
bool verifyIR = true;
};
// Parse command-line arguments into a CompilerOptions struct.
// Returns false and prints usage if the arguments are invalid.
bool parseArguments(int argc, char* argv[],
CompilerOptions& opts);
// Run the full compilation pipeline.
// Returns 0 on success, 1 on error.
int runCompiler(const CompilerOptions& opts);
} // namespace driver
// src/CompilerDriver.cpp
#include "CompilerDriver.h"
#include "diag/SourceManager.h"
#include "diag/DiagnosticEngine.h"
#include "diag/TerminalDiagnosticConsumer.h"
#include "diag/JsonDiagnosticConsumer.h"
#include "frontend/AntlrDiagnosticListener.h"
#include "frontend/ASTBuilder.h"
#include "frontend/CustomErrorStrategy.h"
#include "sema/TypeChecker.h"
#include "codegen/IRGenerator.h"
#include "codegen/IRVerification.h"
#include "codegen/LLVMDiagnosticBridge.h"
#include "MotorLangLexer.h"
#include "MotorLangParser.h"
#include <antlr4-runtime.h>
#include <llvm/IR/LLVMContext.h>
#include <llvm/IR/Module.h>
#include <llvm/Support/InitLLVM.h>
#include <llvm/Support/TargetSelect.h>
#include <llvm/Support/raw_ostream.h>
#include <iostream>
#include <fstream>
#include <string>
namespace driver {
bool parseArguments(int argc, char* argv[],
CompilerOptions& opts)
{
if (argc < 2) {
std::cerr << "Usage: motorlangc [options] <source.ml>\n\n"
<< "Options:\n"
<< " -o <file> Output file (default: a.out)\n"
<< " -Werror Treat warnings as errors\n"
<< " -ferror-limit=N Stop after N errors (default: 20)\n"
<< " -fno-color Disable colored output\n"
<< " --developer Show compiler-internal info\n"
<< " --json-output=<f> Write JSON diagnostics to file\n"
<< " --emit-llvm Emit LLVM IR to <output>.ll\n"
<< " --no-verify Skip LLVM IR verification\n";
return false;
}
for (int i = 1; i < argc; ++i) {
std::string arg = argv[i];
if (arg == "-Werror") {
opts.treatWarningsAsErrors = true;
} else if (arg.substr(0, 14) == "-ferror-limit=") {
opts.maxErrors =
static_cast<uint32_t>(
std::stoul(arg.substr(14)));
} else if (arg == "-fno-color") {
opts.useColor = false;
} else if (arg == "--developer") {
opts.developerMode = true;
} else if (arg.substr(0, 14) == "--json-output=") {
opts.jsonOutput = true;
opts.jsonOutputFile = arg.substr(14);
} else if (arg == "--emit-llvm") {
opts.emitLLVMIR = true;
} else if (arg == "--no-verify") {
opts.verifyIR = false;
} else if (arg.substr(0, 2) == "-o") {
if (arg.size() > 2) {
opts.outputFile = arg.substr(2);
} else if (i + 1 < argc) {
opts.outputFile = argv[++i];
}
} else if (arg[0] != '-') {
opts.inputFile = arg;
} else {
std::cerr << "motorlangc: unknown option '"
<< arg << "'\n";
return false;
}
}
if (opts.inputFile.empty()) {
std::cerr << "motorlangc: no input file specified\n";
return false;
}
return true;
}
static void printSummary(
const diag::DiagnosticEngine& engine)
{
uint32_t errs = engine.errorCount();
uint32_t warns = engine.warningCount();
if (errs > 0 || warns > 0) {
std::cerr << "\nCompilation finished with "
<< errs << " error"
<< (errs != 1 ? "s" : "") << " and "
<< warns << " warning"
<< (warns != 1 ? "s" : "") << ".\n";
}
}
int runCompiler(const CompilerOptions& opts) {
// ---- Step 1: Diagnostic infrastructure ----
diag::SourceManager srcMgr;
diag::DiagnosticEngine engine(opts.maxErrors);
engine.setTreatWarningsAsErrors(opts.treatWarningsAsErrors);
auto termConsumer =
std::make_unique<diag::TerminalDiagnosticConsumer>(
srcMgr,
opts.useColor,
opts.developerMode,
std::cerr);
engine.addConsumer(std::move(termConsumer));
std::ofstream jsonStream;
if (opts.jsonOutput) {
jsonStream.open(opts.jsonOutputFile);
if (jsonStream.is_open()) {
engine.addConsumer(
std::make_unique<diag::JsonDiagnosticConsumer>(
jsonStream));
} else {
std::cerr << "motorlangc: warning: cannot open '"
<< opts.jsonOutputFile
<< "' for JSON output\n";
}
}
// ---- Step 2: LLVM initialization ----
llvm::InitializeAllTargetInfos();
llvm::InitializeAllTargets();
llvm::InitializeAllTargetMCs();
llvm::InitializeAllAsmParsers();
llvm::InitializeAllAsmPrinters();
llvm::LLVMContext llvmCtx;
codegen::installLLVMDiagnosticHandler(llvmCtx, engine);
auto llvmModule = std::make_unique<llvm::Module>(
opts.inputFile, llvmCtx);
// ---- Step 3: Load the source file ----
auto canonicalPath = srcMgr.loadFile(opts.inputFile);
if (!canonicalPath) {
std::cerr << "motorlangc: cannot open '"
<< opts.inputFile << "'\n";
return 1;
}
// ---- Step 4: Lex and parse ----
antlr4::ANTLRFileStream inputStream;
inputStream.loadFromFile(opts.inputFile);
MotorLangLexer lexer(&inputStream);
lexer.removeErrorListeners();
auto lexerListener =
std::make_unique<frontend::AntlrDiagnosticListener>(
engine, canonicalPath, /*isLexer=*/true);
lexer.addErrorListener(lexerListener.get());
antlr4::CommonTokenStream tokenStream(&lexer);
MotorLangParser parser(&tokenStream);
parser.removeErrorListeners();
auto parserListener =
std::make_unique<frontend::AntlrDiagnosticListener>(
engine, canonicalPath, /*isLexer=*/false);
parser.addErrorListener(parserListener.get());
auto errorStrategy =
std::make_shared<frontend::CustomErrorStrategy>(engine);
parser.setErrorHandler(errorStrategy);
MotorLangParser::ProgramContext* parseTree =
parser.program();
// After parsing, stop if there are errors. Semantic analysis
// on a broken parse tree produces mostly noise.
if (engine.hasErrors()) {
printSummary(engine);
engine.finalize();
return 1;
}
// ---- Step 5: Build the AST ----
frontend::ASTBuilder astBuilder(engine, canonicalPath);
auto program = astBuilder.buildProgram(parseTree);
if (!program || engine.hasErrors()) {
printSummary(engine);
engine.finalize();
return 1;
}
// ---- Step 6: Type checking ----
sema::TypeChecker typeChecker(engine);
for (auto& fn : program->functions) {
fn->accept(typeChecker);
}
// Stop on errors but continue on warnings.
if (engine.hasErrors()) {
printSummary(engine);
engine.finalize();
return 1;
}
// ---- Step 7: IR generation ----
codegen::IRGenerator irGen(
llvmCtx, *llvmModule, engine, opts.inputFile);
for (auto& fn : program->functions) {
fn->accept(irGen);
}
irGen.finalize();
if (engine.hasErrors()) {
printSummary(engine);
engine.finalize();
return 1;
}
// ---- Step 8: IR verification ----
if (opts.verifyIR) {
if (!codegen::verifyGeneratedModule(*llvmModule, engine)) {
printSummary(engine);
engine.finalize();
return 1;
}
}
// ---- Step 9: Emit LLVM IR if requested ----
if (opts.emitLLVMIR) {
std::string irFile = opts.outputFile + ".ll";
std::error_code ec;
llvm::raw_fd_ostream irOut(irFile, ec);
if (!ec) {
llvmModule->print(irOut, nullptr);
std::cerr << "Emitted LLVM IR to " << irFile << "\n";
} else {
std::cerr << "motorlangc: warning: cannot write '"
<< irFile << "': " << ec.message() << "\n";
}
}
// ---- Step 10: Summary ----
printSummary(engine);
engine.finalize();
return engine.hasErrors() ? 1 : 0;
}
} // namespace driver
// src/main.cpp
//
// Entry point for the MotorLang compiler.
// Parses command-line arguments and delegates to the driver.
#include "CompilerDriver.h"
#include <llvm/Support/InitLLVM.h>
int main(int argc, char* argv[]) {
// InitLLVM handles signal handling and pretty-printing of
// stack traces on crashes, which is invaluable during development.
llvm::InitLLVM x(argc, argv);
driver::CompilerOptions opts;
if (!driver::parseArguments(argc, argv, opts)) {
return 1;
}
return driver::runCompiler(opts);
}
CHAPTER FOURTEEN: TESTING THE ERROR REPORTING SYSTEM
A diagnostic system that is not tested is a diagnostic system that will silently break. I have seen this happen on real projects: someone refactors a message string, the tests do not catch it because they only check that an error was emitted (not what it said), and six months later a user files a bug report saying the error messages are confusing. The testing strategy I use here verifies not just that an error is reported, but that it is reported at the correct location, with the correct code, and with the correct message content.
Here is a sample test source file that exercises the type checker:
// tests/test_type_errors.ml
// Tests for type mismatch diagnostics in MotorLang.
// This file is intentionally full of errors.
int computeSpeed(int maxRpm, float factor) {
int rpm = 0;
float result = maxRpm * factor;
rpm = result; // SEM-3042: cannot assign float to int
int rpm = 10; // SEM-3010: redeclaration of 'rpm'
return lenght; // SEM-3020: undeclared 'lenght' (typo)
}
And here is the test runner:
// tests/ErrorReportingTests.cpp
//
// Unit tests for the error reporting infrastructure.
// Verifies that diagnostics are emitted with the correct location,
// code, and message content.
#include "diag/DiagnosticEngine.h"
#include "diag/SourceManager.h"
#include "diag/DiagnosticCodes.h"
#include <cassert>
#include <vector>
#include <string>
#include <iostream>
// A test consumer that captures all diagnostics for inspection.
class CapturingConsumer : public diag::DiagnosticConsumer {
public:
void handleDiagnostic(
const diag::Diagnostic& d) override
{
captured_.push_back(d);
}
const std::vector<diag::Diagnostic>& captured() const {
return captured_;
}
// Check whether a diagnostic with the given code was emitted
// at the given line and column.
bool hasErrorAt(uint32_t code,
uint32_t line,
uint32_t col) const
{
for (const auto& d : captured_) {
if (d.code == code &&
d.location.line == line &&
d.location.column == col) {
return true;
}
}
return false;
}
// Check whether any captured diagnostic's message contains
// the given substring.
bool hasMessageContaining(
const std::string& substr) const
{
for (const auto& d : captured_) {
if (d.message.find(substr) != std::string::npos) {
return true;
}
}
return false;
}
bool hasCode(uint32_t code) const {
for (const auto& d : captured_) {
if (d.code == code) return true;
}
return false;
}
private:
std::vector<diag::Diagnostic> captured_;
};
// ---------------------------------------------------------------
// Test: undeclared variable produces SEM-3020 at the right location
// ---------------------------------------------------------------
static void testUndeclaredVariable() {
diag::DiagnosticEngine engine;
auto consumer = std::make_unique<CapturingConsumer>();
CapturingConsumer* raw = consumer.get();
engine.addConsumer(std::move(consumer));
diag::SourceManager srcMgr;
auto filePath = srcMgr.registerInMemorySource(
"test.ml",
"int x = 0;\n"
"int y = z + 1;\n"
);
// Simulate the type checker emitting the diagnostic.
// In a full integration test, we would run the whole pipeline.
diag::Diagnostic d;
d.code = diag::codes::SEM_UNDECLARED_VARIABLE;
d.severity = diag::Severity::Error;
d.location = diag::SourceLocation(filePath, 2, 9, 0);
d.message = "use of undeclared variable 'z'";
d.phase = "TypeChecker";
engine.emit(std::move(d));
assert(raw->hasErrorAt(
diag::codes::SEM_UNDECLARED_VARIABLE, 2, 9));
assert(raw->hasMessageContaining("undeclared variable 'z'"));
assert(engine.errorCount() == 1);
std::cout << "PASS: testUndeclaredVariable\n";
}
// ---------------------------------------------------------------
// Test: error limit stops emission after maxErrors errors
// ---------------------------------------------------------------
static void testErrorLimitEnforcement() {
diag::DiagnosticEngine engine(/*maxErrors=*/3);
auto consumer = std::make_unique<CapturingConsumer>();
CapturingConsumer* raw = consumer.get();
engine.addConsumer(std::move(consumer));
auto filePath =
std::make_shared<const std::string>("test.ml");
for (int i = 0; i < 10; ++i) {
if (engine.shouldAbortDueToErrorLimit()) break;
diag::Diagnostic d;
d.code = diag::codes::SEM_UNDECLARED_VARIABLE;
d.severity = diag::Severity::Error;
d.location = diag::SourceLocation(
filePath,
static_cast<uint32_t>(i + 1), 1, 0);
d.message = "error number " + std::to_string(i + 1);
d.phase = "TypeChecker";
engine.emit(std::move(d));
}
assert(engine.errorCount() == 3);
assert(raw->captured().size() == 3);
std::cout << "PASS: testErrorLimitEnforcement\n";
}
// ---------------------------------------------------------------
// Test: warnings are promoted to errors with -Werror
// ---------------------------------------------------------------
static void testWarningsAsErrors() {
diag::DiagnosticEngine engine;
engine.setTreatWarningsAsErrors(true);
auto consumer = std::make_unique<CapturingConsumer>();
CapturingConsumer* raw = consumer.get();
engine.addConsumer(std::move(consumer));
auto filePath =
std::make_shared<const std::string>("test.ml");
diag::Diagnostic d;
d.code = diag::codes::SEM_UNINITIALIZED_VAR;
d.severity = diag::Severity::Warning;
d.location = diag::SourceLocation(filePath, 5, 3, 0);
d.message = "variable 'x' used before initialization";
d.phase = "TypeChecker";
engine.emit(std::move(d));
// The warning should have been promoted to an error.
assert(engine.errorCount() == 1);
assert(engine.warningCount() == 0);
// A note should have been attached explaining the promotion.
assert(!raw->captured().empty());
assert(!raw->captured()[0].notes.empty());
std::cout << "PASS: testWarningsAsErrors\n";
}
// ---------------------------------------------------------------
// Test: fix-it hints are attached correctly
// ---------------------------------------------------------------
static void testFixItHints() {
diag::DiagnosticEngine engine;
auto consumer = std::make_unique<CapturingConsumer>();
CapturingConsumer* raw = consumer.get();
engine.addConsumer(std::move(consumer));
auto filePath =
std::make_shared<const std::string>("test.ml");
diag::SourceLocation loc(filePath, 7, 12, 0);
diag::Diagnostic d;
d.code = diag::codes::SEM_TYPE_MISMATCH_INIT;
d.severity = diag::Severity::Error;
d.location = loc;
d.message = "cannot assign float to int";
d.phase = "TypeChecker";
d.fixItHints.push_back(
diag::FixItHint::makeInsertion(loc, "(int)"));
engine.emit(std::move(d));
assert(!raw->captured().empty());
assert(!raw->captured()[0].fixItHints.empty());
assert(raw->captured()[0].fixItHints[0].replacementText
== "(int)");
std::cout << "PASS: testFixItHints\n";
}
// ---------------------------------------------------------------
// Test: internal compiler errors have the right severity
// ---------------------------------------------------------------
static void testInternalError() {
diag::DiagnosticEngine engine;
auto consumer = std::make_unique<CapturingConsumer>();
CapturingConsumer* raw = consumer.get();
engine.addConsumer(std::move(consumer));
diag::Diagnostic d;
d.code = diag::codes::ICE_UNEXPECTED_STATE;
d.severity = diag::Severity::InternalError;
d.message = "unexpected null pointer in IRGenerator";
d.phase = "IRGenerator";
d.compilerSourceLocation = "IRGenerator.cpp:142";
engine.emit(std::move(d));
assert(engine.errorCount() == 1);
assert(engine.hasFatal());
assert(raw->hasCode(diag::codes::ICE_UNEXPECTED_STATE));
std::cout << "PASS: testInternalError\n";
}
int main() {
std::cout << "Running error reporting tests...\n\n";
testUndeclaredVariable();
testErrorLimitEnforcement();
testWarningsAsErrors();
testFixItHints();
testInternalError();
std::cout << "\nAll tests passed.\n";
return 0;
}
CHAPTER FIFTEEN: ADVANCED TOPICS AND BEST PRACTICES
Having built the complete infrastructure, I want to share some hard-won lessons about what separates good error reporting from great error reporting. These are things I have learned from watching real users struggle with compiler messages, from reading the Clang and Rust compiler source code, and from making mistakes in my own compiler projects.
The first lesson is about error deduplication. When the compiler processes a template instantiation or an inlined function, it may encounter the same logical error multiple times. Without deduplication, the user sees the same message five or ten times, which is confusing and makes the output feel broken. The DiagnosticEngine should track recently emitted diagnostics and suppress exact duplicates. A diagnostic is a duplicate if it has the same code, the same location, and the same message text as a recently emitted diagnostic. Here is a simple deduplication mechanism you can add to the engine:
// Add to DiagnosticEngine's private section:
struct DiagKey {
uint32_t code;
uint32_t line;
uint32_t col;
std::string message;
bool operator==(const DiagKey& o) const {
return code == o.code && line == o.line
&& col == o.col && message == o.message;
}
};
struct DiagKeyHash {
size_t operator()(const DiagKey& k) const {
size_t h = std::hash<uint32_t>{}(k.code);
h ^= std::hash<uint32_t>{}(k.line) + 0x9e3779b9 + (h<<6) + (h>>2);
h ^= std::hash<uint32_t>{}(k.col) + 0x9e3779b9 + (h<<6) + (h>>2);
h ^= std::hash<std::string>{}(k.message);
return h;
}
};
std::unordered_set<DiagKey, DiagKeyHash> emittedDiags_;
// At the start of emit(), before dispatching:
DiagKey key{
diag.code,
diag.location.line,
diag.location.column,
diag.message
};
if (emittedDiags_.count(key)) return true; // Suppress duplicate
emittedDiags_.insert(key);
The second lesson is about the "did you mean?" suggestion, which we implemented using Levenshtein edit distance. This feature is enormously valuable, but it needs to be tuned carefully. A threshold that is too low produces no suggestions at all. A threshold that is too high produces absurd suggestions that make the compiler look foolish. In my experience, a threshold of 3 edits works well for typical identifier names in domain-specific languages, where names tend to be longer and more distinctive than in general-purpose languages.
The third lesson is about error recovery in the parser. The custom error strategy we built uses panic-mode recovery, which is the right approach for a statement-based language. However, there is a subtlety: after recovering from an error, the parser sometimes produces parse tree nodes that are structurally valid but semantically nonsensical. The AST builder and type checker need to be robust to these cases. The most reliable approach is to check for null pointers and empty strings at every step of the AST building process, and to emit internal errors (not user-facing errors) when they are encountered.
The fourth lesson is about the relationship between error messages and documentation. Every diagnostic code in our system corresponds to a documentation entry. When a user sees "SEM-3042", they should be able to look it up in the compiler's documentation and find a detailed explanation with examples. This documentation is most valuable when it includes not just the definition of the error but also common causes and concrete fix examples. The DiagnosticCodes.h file we built is the seed of this documentation.
The fifth and most philosophical lesson is about what makes an error message "good." Research in programming language usability has identified several principles that I have found to be consistently true in practice. The message should describe what the compiler expected, not just what it found. The message should be written from the user's perspective, not the compiler's internal perspective. The message should suggest a fix when the fix is unambiguous. The message should not blame the user. And the message should be consistent: the same kind of error should always produce the same kind of message, so users can develop intuitions about what messages mean.
CHAPTER SIXTEEN: A COMPLETE WORKED EXAMPLE
Let us bring everything together with a complete example. Here is a MotorLang source file with several deliberate errors:
// tests/motor_control_errors.ml
// A motor control program with several deliberate errors.
// Used to demonstrate the quality of the compiler's diagnostics.
int computeTorque(int rpm, float efficiency) {
float torque = rpm * efficiency;
int result = torque;
int result = 0;
return lenght;
}
float calibrate(float setpoint) {
float output = 0;
output = setpoint * 1.5;
return output;
}
When we compile this file with our compiler, the output looks like this:
tests/motor_control_errors.ml:6:17: error [SEM-3042]: cannot initialize
variable 'result' of type 'int' with a value of type 'float'; consider
an explicit cast
6 | int result = torque;
| ^~~~~~
fix-it: replace with '(int)torque' at tests/motor_control_errors.ml:6:17
tests/motor_control_errors.ml:5:5: note [SEM-3001]: 'torque' was declared
here as 'float'
5 | float torque = rpm * efficiency;
| ^~~~~~
[compiler phase: TypeChecker]
tests/motor_control_errors.ml:7:9: error [SEM-3010]: redeclaration of
variable 'result'
7 | int result = 0;
| ^~~~~~
tests/motor_control_errors.ml:6:9: note [SEM-3001]: 'result' was previously
declared here
6 | int result = torque;
| ^~~~~~
[compiler phase: TypeChecker]
tests/motor_control_errors.ml:8:12: error [SEM-3020]: use of undeclared
variable 'lenght'; did you mean 'result'?
8 | return lenght;
| ^~~~~~
tests/motor_control_errors.ml:6:9: note [SEM-3001]: 'result' was declared here
6 | int result = torque;
| ^~~~~~
[compiler phase: TypeChecker]
Compilation finished with 3 errors and 0 warnings.
This is exactly the kind of output that makes a compiler feel like a helpful colleague rather than an indifferent gatekeeper. Each error points precisely to the problem, shows the relevant source line, and provides context through notes. The "did you mean?" suggestion for the typo "lenght" is the kind of small touch that users remember and appreciate.
EPILOGUE: THE COMPILER AS A TEACHER
The best compilers are teachers. They do not just reject incorrect programs; they explain why the program is incorrect, point precisely to the problem, suggest how to fix it, and do all of this in language that the programmer can understand without needing to know the internals of the compiler. Building a compiler that achieves this standard requires treating error reporting as a first-class engineering concern, not an afterthought.
The infrastructure we have built in this article -- the SourceLocation and SourceRange types, the Diagnostic structure, the DiagnosticEngine, the TerminalDiagnosticConsumer with its caret-and-underline display, the AntlrDiagnosticListener that bridges ANTLR4's error model to ours, the AST nodes with mandatory location information, the type checker with its multi-location diagnostics and error suppression, the IRGenerator that attaches debug locations to every instruction, the LLVM diagnostic bridge, and the JSON consumer for tooling integration -- all of these components work together to create a compiler that is not just correct, but genuinely helpful.
The investment in this infrastructure pays dividends throughout the life of the compiler. Users spend less time debugging their programs. Compiler developers spend less time debugging the compiler. IDEs can provide richer integration. CI systems can produce more actionable reports. And the compiler itself becomes a more trustworthy tool, one that engineers can rely on to tell them the truth about their code in a way they can actually use.
In the world of industrial software, where the programs we compile control real physical systems, this trustworthiness is not just a convenience. It is an engineering imperative. Build compilers that talk to people, not at them.
REFERENCES AND FURTHER READING
Parr, Terence. "The Definitive ANTLR 4 Reference." Pragmatic Bookshelf, 2013. This is the primary reference for ANTLR4, covering grammar design, error handling, and the visitor pattern in depth. If you are going to build a serious compiler on ANTLR4, this book belongs on your desk.
Lattner, Chris, and Vikram Adve. "LLVM: A Compilation Framework for Lifelong Program Analysis and Transformation." CGO 2004. The original LLVM paper describes the design philosophy that underlies the entire LLVM infrastructure including its diagnostic system.
The LLVM Project. "LLVM Language Reference Manual." Available at llvm.org/docs/LangRef.html. The authoritative reference for LLVM IR, including the debug information metadata format used for attaching source locations to instructions.
The Clang Project. "Clang Diagnostics Reference." Available at clang.llvm.org/docs/DiagnosticsReference.html. An excellent example of how a mature compiler organizes and presents its diagnostic messages. Reading through Clang's diagnostic definitions is one of the best ways to develop an intuition for what makes a good error message.
Marceau, Guillaume, Kathi Fisler, and Shriram Krishnamurthi. "Measuring the Effectiveness of Error Messages Designed for Novice Programmers." SIGCSE 2011. A research paper that empirically measures which properties of error messages help programmers most. The results are sometimes surprising and always worth knowing.
Cooper, Keith, and Linda Torczon. "Engineering a Compiler." Morgan Kaufmann, 2011. A comprehensive textbook on compiler construction that covers error recovery strategies in detail, including the synchronization-set approach we implemented in our custom error strategy.
Levenshtein, Vladimir I. "Binary codes capable of correcting deletions, insertions, and reversals." Soviet Physics Doklady, 1966. The original paper describing the edit distance metric we use for "did you mean?" suggestions. Worth reading if you want to understand why the algorithm works the way it does.
No comments:
Post a Comment