PREFACE: WHY CONFIGURABILITY IS THE INVISIBLE BACKBONE OF GREAT SOFTWARE
There is a quiet superpower that separates merely functional software from truly excellent software, and most developers only discover it after they have suffered enough. That superpower is configurability — the deliberate, principled design of a system so that its behavior, structure, and capabilities can be changed without rewriting it from scratch.
Think about the last time you changed a database connection string in a config file and restarted a service. Or the last time a feature flag let you roll out a new UI to five percent of your users before anyone else saw it. Or the last time a plugin you installed transformed a plain text editor into a full-blown IDE. All of these are manifestations of configurability, and they exist on a rich spectrum that spans from the moment source code is written all the way to the moment a user clicks a button in a running application.
This article is your guided tour through that entire spectrum. We will start from first principles, travel through every phase of the software lifecycle where configuration decisions can be made, explore the architectural patterns that make those decisions clean and maintainable, and land on a fully worked example involving a real-world Agentic AI system that can switch between local and remote Large Language Models at runtime. Along the way we will call out the anti-patterns that will ruin your day if you let them.
CHAPTER ONE: THE CONFIGURABILITY TIMELINE
Before we look at any code, we need to build a mental model. Configurability does not happen at a single moment. It happens at many different points in the lifecycle of a software artifact, and the right choice of moment has profound consequences for flexibility, performance, safety, and operational complexity.
Think of the lifecycle of a piece of software as a pipeline with several distinct stations. At each station, decisions can be frozen — baked into the artifact — or left open for a later station to decide. The earlier you freeze a decision, the faster and more optimized the result, but the less flexible it becomes. The later you freeze a decision, the more flexible the system, but the more complexity you introduce into the runtime environment.
The stations, in order from earliest to latest, are as follows.
Design time is the moment when a software architect or developer makes structural choices about how a system will be organized. These choices are encoded in interfaces, abstract classes, module boundaries, and dependency graphs. A decision made at design time — for example, "we will use the Strategy pattern so that the sorting algorithm can be swapped" — enables configurability at later stages without itself being a runtime mechanism.
Compile time is when the compiler processes source code and produces object files or bytecode. In languages like C and C++, the preprocessor runs before the compiler proper, and it can include or exclude entire blocks of code based on preprocessor macros. This is one of the oldest forms of configurability.
Build time is when a build system (CMake, Gradle, Bazel, Make, or a CI/CD pipeline) assembles compiled artifacts into a deployable package. Build systems can parameterize the build with variables that select features, target platforms, optimization levels, or bundled assets.
Link time is when the linker combines object files and libraries into a final executable. Dynamic linking defers some decisions to load time, while static linking bakes everything in at this stage.
Package and deployment time is when the compiled artifact is wrapped into a container image, an installer, or a deployment bundle. Tools like Docker allow environment variables and configuration files to be injected at this stage.
Startup time is when the application process begins executing but before it starts serving requests. Many frameworks read configuration files, parse environment variables, and wire up dependency injection containers during startup. This is sometimes called "initialization time" or "bootstrap time."
Runtime is when the application is actively processing work. Configuration changes at runtime — without restarting the process — represent the highest degree of flexibility and the highest degree of engineering complexity.
The full landscape, rendered as a diagram:
CONFIGURABILITY TIMELINE
========================
DESIGN TIME
| Interfaces, abstract classes, Strategy pattern, Plugin contracts.
| Decisions: What CAN be configured? What are the extension points?
|
COMPILE TIME
| Preprocessor macros (#ifdef), C++ templates (if constexpr).
| Decisions: Which platform? Which optional subsystems?
| Tradeoff: Maximum performance, zero flexibility after compilation.
|
BUILD TIME
| CMake options, environment variables passed to pip/gradle/bazel.
| Decisions: Which GPU backend? Which feature set? Which target OS?
| Tradeoff: Flexible per-build, but requires a rebuild to change.
|
LINK TIME
| Static vs. dynamic linking. Plugin DLLs / .so files.
| Decisions: Which libraries are bundled vs. loaded at runtime?
|
PACKAGE / DEPLOY TIME
| Docker ENV, Kubernetes ConfigMaps and Secrets, Helm values.
| Decisions: Which environment? Which secrets? Which replicas?
|
STARTUP TIME
| Environment variables, config files, dependency injection containers.
| Decisions: Which provider? Which model? Which log level?
| Pattern: Settings class with eager validation and fail-fast behavior.
|
RUNTIME
Feature flags (release, experiment, ops, permission).
Admin APIs, hot-reload of config files, dynamic provider switching.
Decisions: Which users see which features? Which model for this request?
Pattern: Gateway with Strategy, Factory, and FeatureFlags.
Tradeoff: Maximum flexibility, highest engineering complexity.
Let us now visit each of these stations in depth.
STATION ONE: COMPILE-TIME CONFIGURATION
Compile-time configuration is the oldest trick in the book. The C preprocessor has been doing it since the 1970s. The idea is simple: before the compiler sees your code, a preprocessor scans it and makes substitutions or inclusions based on macro definitions that you supply on the command line or in header files.
The following C example demonstrates a classic use of compile-time configuration to select between a debug logging mode and a production silent mode. Notice how the entire debug logging infrastructure simply does not exist in the compiled binary when NDEBUG is defined — it is as if those lines were never written.
/*
* logger.h
*
* Compile-time selection of logging behavior.
*
* To build in release mode (no logging overhead):
* gcc -DNDEBUG -o myapp main.c
*
* To build in debug mode (full logging):
* gcc -o myapp main.c
*
* Note: LOG_DEBUG uses the GCC/Clang extension ##__VA_ARGS__ to handle
* the case where no variadic arguments are supplied. This is supported
* by GCC, Clang, and MSVC and is the de-facto standard for C logging
* macros, though it is not strictly ISO C99.
*/
#ifndef LOGGER_H
#define LOGGER_H
#include <stdio.h>
#ifdef NDEBUG
/*
* In release builds, LOG_DEBUG expands to a void cast of nothing.
* The compiler sees no code here at all -- zero overhead, and no
* "unused variable" warnings for arguments that only appear in the
* format string.
*/
#define LOG_DEBUG(fmt, ...) ((void)0)
#else
/*
* In debug builds, LOG_DEBUG prints a timestamped message to stderr
* with the source file name and line number included, which makes
* tracking down the origin of a log message trivial.
*/
#define LOG_DEBUG(fmt, ...) \
fprintf(stderr, "[DEBUG %s:%d] " fmt "\n", \
__FILE__, __LINE__, ##__VA_ARGS__)
#endif
/*
* LOG_ERROR is always active regardless of build mode.
* Errors must never be silently swallowed, even in production.
*/
#define LOG_ERROR(fmt, ...) \
fprintf(stderr, "[ERROR %s:%d] " fmt "\n", \
__FILE__, __LINE__, ##__VA_ARGS__)
#endif /* LOGGER_H */
This pattern is powerful but it has a serious limitation: once the binary is compiled, you cannot change the behavior without recompiling. If a customer reports a bug in production and you need debug logging, you must ship a new binary. That is why compile-time configuration is best reserved for things that genuinely never change between environments: platform-specific code paths, hardware-specific optimizations, and fundamental architectural choices like whether to include a GUI subsystem at all.
In the C++ world, compile-time configuration has evolved dramatically with templates and the if constexpr construct introduced in C++17. The following snippet shows how a modern C++ system can select a GPU backend at compile time using template specialization — a technique that is directly relevant to our LLM example later in this article.
/*
* gpu_backend.hpp
*
* Compile-time GPU backend selection using C++17 if constexpr.
*
* The backend is selected by defining exactly one of the following
* macros at compile time via the build system (e.g., CMake):
*
* -DUSE_CUDA -- NVIDIA CUDA (requires CUDA Toolkit)
* -DUSE_METAL -- Apple Metal (macOS / Apple Silicon only)
* -DUSE_VULKAN -- Vulkan (cross-platform, requires Vulkan SDK)
*
* If none of the above is defined, the CPU-only fallback is used.
*/
#pragma once
#include <string>
#include <type_traits>
/* ------------------------------------------------------------------ */
/* Backend tag types */
/* Each tag is an empty struct used purely as a compile-time label. */
/* ------------------------------------------------------------------ */
struct CudaBackend {};
struct MetalBackend {};
struct VulkanBackend {};
struct CpuBackend {};
/* ------------------------------------------------------------------ */
/* Active backend selection */
/* Exactly one of these branches will be compiled. */
/* ------------------------------------------------------------------ */
#if defined(USE_CUDA)
using ActiveBackend = CudaBackend;
#elif defined(USE_METAL)
using ActiveBackend = MetalBackend;
#elif defined(USE_VULKAN)
using ActiveBackend = VulkanBackend;
#else
using ActiveBackend = CpuBackend;
#endif
/* ------------------------------------------------------------------ */
/* GpuContext<Backend> */
/* */
/* A template class that provides a unified interface for initializing */
/* and querying the GPU backend. Each template instantiation handles */
/* exactly one backend. The compiler generates only the code path that */
/* matches the active backend and eliminates all others entirely -- */
/* there is no runtime branching and no virtual dispatch overhead. */
/* ------------------------------------------------------------------ */
template<typename Backend>
class GpuContext {
public:
/*
* Returns the human-readable name of the active backend.
* The dead branches are eliminated by the compiler at compile time
* via if constexpr, so this function is a single return statement
* in the generated machine code.
*/
std::string name() const {
if constexpr (std::is_same_v<Backend, CudaBackend>) {
return "NVIDIA CUDA";
} else if constexpr (std::is_same_v<Backend, MetalBackend>) {
return "Apple Metal";
} else if constexpr (std::is_same_v<Backend, VulkanBackend>) {
return "Vulkan (cross-platform)";
} else {
return "CPU (no GPU acceleration)";
}
}
/*
* Performs backend-specific initialization.
* In a real implementation, each branch would call the appropriate
* platform SDK initialization function:
* CUDA: cudaSetDevice(), cublasCreate(), etc.
* Metal: MTLCreateSystemDefaultDevice()
* Vulkan: vkCreateInstance(), vkCreateDevice(), etc.
* CPU: No initialization required.
*/
void initialize() const {
if constexpr (std::is_same_v<Backend, CudaBackend>) {
/* cudaSetDevice(0); */
} else if constexpr (std::is_same_v<Backend, MetalBackend>) {
/* id<MTLDevice> device = MTLCreateSystemDefaultDevice(); */
} else if constexpr (std::is_same_v<Backend, VulkanBackend>) {
/* vkCreateInstance(&createInfo, nullptr, &instance); */
}
/* CpuBackend: no initialization needed. */
}
};
/* ------------------------------------------------------------------ */
/* AppGpuContext */
/* */
/* The application-wide GPU context type alias, resolved entirely at */
/* compile time. Use this alias throughout the application to refer to */
/* the GPU context -- never use GpuContext<SomeBackend> directly. */
/* ------------------------------------------------------------------ */
using AppGpuContext = GpuContext<ActiveBackend>;
This is elegant because the compiler generates exactly one code path and eliminates all the others. There is no runtime branching, no virtual dispatch, and no overhead whatsoever. The tradeoff is that you must recompile to switch backends, which is perfectly acceptable for an embedded system or a performance-critical inference engine but would be unacceptable for a cloud-hosted service that needs to support multiple GPU types simultaneously.
STATION TWO: BUILD-TIME CONFIGURATION
Build-time configuration sits one level above compile-time configuration. While compile-time configuration is expressed in source code (macros, templates), build-time configuration is expressed in the build system itself. The build system reads parameters — from environment variables, from command-line arguments, or from configuration files — and uses them to control what gets compiled, how it gets compiled, and what gets bundled into the final artifact.
CMake is the dominant build system for C and C++ projects, and it has a rich vocabulary for expressing build-time configuration. The following CMakeLists.txt demonstrates how to expose GPU backend selection as a CMake option, so that a developer or CI/CD pipeline can choose the backend without touching any source file.
# CMakeLists.txt
#
# Build-time GPU backend selection for the LLM Inference Engine.
#
# Usage examples:
# cmake -DGPU_BACKEND=CUDA -B build && cmake --build build
# cmake -DGPU_BACKEND=METAL -B build && cmake --build build
# cmake -DGPU_BACKEND=VULKAN -B build && cmake --build build
# cmake -DGPU_BACKEND=CPU -B build && cmake --build build
#
# The GPU_BACKEND value is translated into a preprocessor define
# (USE_CUDA, USE_METAL, USE_VULKAN) that gpu_backend.hpp reads.
# If GPU_BACKEND=CPU, no define is emitted and the CPU fallback is used.
cmake_minimum_required(VERSION 3.20)
project(LLMInferenceEngine VERSION 1.0.0 LANGUAGES CXX)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF) # Disable compiler-specific extensions
# for maximum portability.
# ---------------------------------------------------------------------------
# GPU_BACKEND option
# Declare with a default of "CPU" so that a plain cmake invocation with no
# arguments produces a working, if unaccelerated, build.
# ---------------------------------------------------------------------------
set(GPU_BACKEND "CPU" CACHE STRING
"Select GPU backend: CUDA, METAL, VULKAN, or CPU")
set_property(CACHE GPU_BACKEND PROPERTY STRINGS CUDA METAL VULKAN CPU)
# ---------------------------------------------------------------------------
# Validate the choice and configure the build accordingly.
# ---------------------------------------------------------------------------
if(GPU_BACKEND STREQUAL "CUDA")
message(STATUS "[LLMEngine] Building with NVIDIA CUDA backend")
find_package(CUDAToolkit REQUIRED) # Requires CMake 3.17+
add_compile_definitions(USE_CUDA)
elseif(GPU_BACKEND STREQUAL "METAL")
message(STATUS "[LLMEngine] Building with Apple Metal backend")
if(NOT APPLE)
message(FATAL_ERROR
"GPU_BACKEND=METAL requires macOS or iOS. "
"Current platform is not Apple.")
endif()
add_compile_definitions(USE_METAL)
elseif(GPU_BACKEND STREQUAL "VULKAN")
message(STATUS "[LLMEngine] Building with Vulkan backend")
find_package(Vulkan REQUIRED)
add_compile_definitions(USE_VULKAN)
elseif(GPU_BACKEND STREQUAL "CPU")
message(STATUS "[LLMEngine] Building CPU-only (no GPU acceleration)")
# No preprocessor define is emitted; gpu_backend.hpp falls through
# to the CpuBackend alias automatically.
else()
message(FATAL_ERROR
"Unknown GPU_BACKEND value: '${GPU_BACKEND}'. "
"Valid options are: CUDA, METAL, VULKAN, CPU")
endif()
# ---------------------------------------------------------------------------
# Executable target
# ---------------------------------------------------------------------------
add_executable(llm_engine
src/main.cpp
src/inference_engine.cpp
)
target_include_directories(llm_engine PRIVATE include)
target_compile_features(llm_engine PRIVATE cxx_std_17)
# ---------------------------------------------------------------------------
# Link GPU-specific libraries.
# Only the libraries for the selected backend are linked; all others are
# completely absent from the final binary.
# ---------------------------------------------------------------------------
if(GPU_BACKEND STREQUAL "CUDA")
target_link_libraries(llm_engine PRIVATE
CUDA::cudart
CUDA::cublas
)
elseif(GPU_BACKEND STREQUAL "VULKAN")
target_link_libraries(llm_engine PRIVATE
Vulkan::Vulkan
)
endif()
# Metal does not require explicit linking on Apple platforms;
# the framework is linked automatically by the system linker
# when Metal API headers are included.
message(STATUS "[LLMEngine] Configuration complete. "
"GPU_BACKEND=${GPU_BACKEND}")
Build-time configuration is particularly powerful in large organizations where different teams or products share the same codebase but need different feature sets. An industrial automation product, for example, might share a core communication library with a medical device product, but the two products would be built with entirely different feature sets enabled. The build system is the gatekeeper that ensures each product variant is assembled correctly.
In the Python ecosystem, build-time configuration appears in the form of setup.py or pyproject.toml files that control which C extensions get compiled, which optional dependencies get included, and which platform-specific wheels get built. When you install llama-cpp-python with CUDA support, you are performing build-time configuration by passing CMAKE_ARGS as an environment variable to the pip build process:
# Install llama-cpp-python with NVIDIA CUDA support
# (Linux or Windows with CUDA Toolkit 11.8+ installed)
CMAKE_ARGS="-DGGML_CUDA=on" \
pip install --upgrade --force-reinstall --no-cache-dir \
llama-cpp-python
# Install llama-cpp-python with Apple Metal support
# (macOS with Apple Silicon: M1, M2, M3, M4)
CMAKE_ARGS="-DGGML_METAL=on" \
pip install --upgrade --force-reinstall --no-cache-dir \
llama-cpp-python
# Install llama-cpp-python with Vulkan support
# (cross-platform: Linux, Windows, Android with Vulkan SDK installed)
CMAKE_ARGS="-DGGML_VULKAN=on" \
pip install --upgrade --force-reinstall --no-cache-dir \
llama-cpp-python
# Install llama-cpp-python CPU-only (no GPU required)
pip install llama-cpp-python
Once llama-cpp-python is installed with a specific backend, that choice is baked into the compiled .so or .pydextension file. You cannot switch from CUDA to Vulkan at Python runtime — you must reinstall the package. This is a perfect illustration of the tradeoff: maximum performance, zero runtime overhead, but zero runtime flexibility for the backend choice itself.
STATION THREE: STARTUP-TIME CONFIGURATION
Startup-time configuration is where most application developers spend the majority of their configuration energy, and for good reason. It strikes a comfortable balance: the application reads its configuration once when it starts, wires up all its dependencies, and then runs with those settings for its entire lifetime. This is fast enough for most purposes and flexible enough to support different environments (development, staging, production) without recompiling anything.
The most important principle for startup-time configuration is the Twelve-Factor App methodology, which states that configuration should be stored in the environment, not in the code. This means reading from environment variables, not from hardcoded values or files committed to version control.
A well-designed startup configuration system has several layers. First, it defines a schema for all configuration values — their names, types, default values, and whether they are required. Second, it reads values from multiple sources in a defined priority order (environment variables override config files, which override defaults). Third, it validates all values before the application starts serving requests, so that a misconfiguration fails fast and loudly rather than causing mysterious errors hours later.
The following Python module implements exactly this kind of layered, validated startup configuration system. It is the foundation upon which our LLM switching example will be built.
# config/settings.py
#
# Startup-time configuration for the LLM Gateway application.
#
# Priority order (highest to lowest):
# 1. Environment variables (e.g., export LLM_PROVIDER=ollama)
# 2. A .env file in the project root (for local development only)
# 3. Hardcoded defaults in this module
#
# The Settings dataclass uses frozen=True to prevent accidental mutation
# of configuration values after the application has started. Note that
# frozen=True prevents reassignment of attributes but does not deeply
# freeze mutable values stored inside dict fields. The intent is to
# signal immutability and catch accidental reassignment bugs at the
# attribute level; the dict contents should be treated as read-only
# by convention. Because Settings is never used as a dict key or set
# member, the lack of __hash__ on frozen dataclasses with dict fields
# is not a concern in practice.
#
# Usage:
# from config.settings import Settings
# settings = Settings.load()
# print(settings.llm_provider)
from __future__ import annotations
import json
import logging
import os
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Dict
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class LLMProviderSettings:
"""
Immutable settings for a single LLM provider endpoint.
Each field corresponds to one aspect of the provider's connection
and retry configuration. All fields are set at construction time
and cannot be reassigned afterwards (frozen=True).
"""
name: str # Human-readable provider name
base_url: str # OpenAI-compatible API base URL
api_key: str # API key or placeholder string
default_model: str # Default model identifier for this provider
timeout_seconds: float # HTTP request timeout in seconds
max_retries: int # Number of retries on transient failure
@dataclass(frozen=True)
class Settings:
"""
Top-level application settings, assembled once at startup.
All fields are immutable after construction (frozen=True).
The 'providers' and 'feature_flags' fields hold dicts whose
contents should be treated as read-only by convention.
"""
llm_provider: str # Active provider name
providers: Dict[str, LLMProviderSettings] # All configured providers
log_level: str # Logging verbosity level
feature_flags: Dict[str, Any] # Feature toggle state map
gpu_layer_count: int # Layers to offload to GPU
@classmethod
def load(cls) -> "Settings":
"""
Load and validate settings from environment variables and an
optional .env file. Raises ValueError if any required setting
is missing or invalid. This method is designed to be called
exactly once at application startup.
"""
# Step 1: Load .env file if present (for local development).
# In production, environment variables are set by the platform
# (Kubernetes, Docker, systemd, etc.) and no .env file is used.
cls._load_dotenv()
# Step 2: Read and validate each setting with clear error messages.
llm_provider = cls._require_one_of(
key="LLM_PROVIDER",
valid_values=["openai", "ollama", "llamacpp"],
default="ollama"
)
gpu_layer_count = cls._read_int(
key="GPU_LAYER_COUNT",
default=0, # 0 = CPU-only; -1 = all layers to GPU
min_value=-1,
max_value=200
)
log_level = cls._require_one_of(
key="LOG_LEVEL",
valid_values=["DEBUG", "INFO", "WARNING", "ERROR"],
default="INFO"
)
flags_path = os.environ.get(
"FEATURE_FLAGS_PATH", "features/flags.json"
)
# Step 3: Build the provider configuration map.
# Each provider's settings are read from environment variables
# with sensible defaults for local development.
providers: Dict[str, LLMProviderSettings] = {
"openai": LLMProviderSettings(
name="OpenAI (remote)",
base_url="https://api.openai.com/v1",
api_key=os.environ.get("OPENAI_API_KEY", ""),
default_model=os.environ.get(
"OPENAI_DEFAULT_MODEL", "gpt-4o-mini"),
timeout_seconds=float(
os.environ.get("OPENAI_TIMEOUT", "30.0")),
max_retries=int(
os.environ.get("OPENAI_MAX_RETRIES", "3"))
),
"ollama": LLMProviderSettings(
name="Ollama (local)",
base_url=os.environ.get(
"OLLAMA_BASE_URL", "http://localhost:11434/v1"),
api_key="ollama", # Ollama does not require a real key
default_model=os.environ.get(
"OLLAMA_DEFAULT_MODEL", "llama3.2"),
timeout_seconds=float(
os.environ.get("OLLAMA_TIMEOUT", "120.0")),
max_retries=int(
os.environ.get("OLLAMA_MAX_RETRIES", "2"))
),
"llamacpp": LLMProviderSettings(
name="llama.cpp (in-process)",
base_url="", # In-process: no HTTP server needed
api_key="", # In-process: no API key needed
default_model=os.environ.get(
"LLAMACPP_DEFAULT_MODEL", "local-model"),
timeout_seconds=float(
os.environ.get("LLAMACPP_TIMEOUT", "180.0")),
max_retries=int(
os.environ.get("LLAMACPP_MAX_RETRIES", "1"))
),
}
# Step 4: Validate that the active provider has required secrets.
if llm_provider == "openai" and not providers["openai"].api_key:
raise ValueError(
"LLM_PROVIDER is set to 'openai' but OPENAI_API_KEY "
"is not set. Please export your OpenAI API key:\n"
" export OPENAI_API_KEY=sk-..."
)
if llm_provider == "llamacpp":
model_path = os.environ.get("LLAMACPP_MODEL_PATH", "")
if not model_path:
raise ValueError(
"LLM_PROVIDER is set to 'llamacpp' but "
"LLAMACPP_MODEL_PATH is not set. "
"Please set it to the path of your .gguf model file:\n"
" export LLAMACPP_MODEL_PATH=/path/to/model.gguf"
)
# Step 5: Read feature flags from a JSON string in the environment,
# falling back to a safe set of defaults if not specified.
feature_flags: Dict[str, Any] = cls._read_json_dict(
key="FEATURE_FLAGS",
default={
"streaming_enabled": False,
"fallback_on_error": True,
"log_prompts": False,
}
)
logger.info(
"Configuration loaded: provider=%s, gpu_layers=%d, "
"log_level=%s, flags_path=%s",
llm_provider, gpu_layer_count, log_level, flags_path
)
return cls(
llm_provider=llm_provider,
providers=providers,
log_level=log_level,
feature_flags=feature_flags,
gpu_layer_count=gpu_layer_count,
)
@staticmethod
def _load_dotenv() -> None:
"""
Manually parse a .env file and populate os.environ.
Environment variables already set in the shell take precedence
over values in the .env file, which is the standard behaviour
expected by developers using tools like direnv or dotenv-vault.
Blank lines and lines beginning with '#' are ignored.
Inline comments (text after '#' on a value line) are NOT
supported by this minimal parser; use a dedicated library such
as python-dotenv if inline comment support is required.
"""
env_path = Path(".env")
if not env_path.exists():
return
with env_path.open(encoding="utf-8") as f:
for line_number, line in enumerate(f, start=1):
line = line.strip()
if not line or line.startswith("#"):
continue
if "=" not in line:
logger.warning(
".env line %d has no '=' separator, skipping: %r",
line_number, line
)
continue
key, _, value = line.partition("=")
key = key.strip()
value = value.strip().strip('"').strip("'")
if not key:
continue
# Do not override variables already set in the environment.
if key not in os.environ:
os.environ[key] = value
@staticmethod
def _require_one_of(
key: str,
valid_values: list,
default: str,
) -> str:
"""
Read an environment variable and validate it against an allowed
list. Returns the default if the variable is not set. Raises
ValueError if the variable is set to an unrecognised value.
"""
value = os.environ.get(key, default)
if value not in valid_values:
raise ValueError(
f"Environment variable {key}='{value}' is not valid. "
f"Must be one of: {valid_values}"
)
return value
@staticmethod
def _read_int(
key: str,
default: int,
min_value: int,
max_value: int,
) -> int:
"""
Read an integer environment variable with range validation.
Raises ValueError if the value is not an integer or is outside
the allowed range.
"""
raw = os.environ.get(key, str(default))
try:
value = int(raw)
except ValueError:
raise ValueError(
f"Environment variable {key}='{raw}' must be an integer."
)
if not (min_value <= value <= max_value):
raise ValueError(
f"Environment variable {key}={value} is out of the "
f"allowed range [{min_value}, {max_value}]."
)
return value
@staticmethod
def _read_json_dict(key: str, default: Dict[str, Any]) -> Dict[str, Any]:
"""
Read a JSON-encoded dictionary from an environment variable.
Returns the default if the variable is not set.
Raises ValueError if the variable is set but contains invalid JSON
or does not decode to a dict.
"""
raw = os.environ.get(key)
if raw is None:
return default
try:
parsed = json.loads(raw)
if not isinstance(parsed, dict):
raise ValueError(
f"Expected a JSON object (dict), got {type(parsed).__name__}."
)
return parsed
except json.JSONDecodeError as e:
raise ValueError(
f"Environment variable {key} contains invalid JSON: {e}"
) from e
This Settings class embodies several important principles. It is immutable (frozen=True), which means that once the application has started, no part of the code can accidentally reassign a setting and cause a subtle, hard-to-debug behavior change. It validates eagerly, which means misconfiguration is caught at startup rather than hours later when a specific code path is first exercised. And it reads from a layered hierarchy of sources, which makes it work equally well in a developer's local environment (using a .env file) and in a production Kubernetes cluster (using environment variables injected by the platform).
STATION FOUR: RUNTIME CONFIGURATION — FEATURE FLAGS
Runtime configuration is where things get genuinely exciting — and genuinely complex. The ability to change an application's behavior while it is running, without restarting it, is enormously powerful. It enables A/B testing, gradual rollouts, instant kill switches for problematic features, and dynamic adaptation to changing conditions.
The most widely used mechanism for runtime configuration is the feature flag (also called a feature toggle). A feature flag is a named boolean (or sometimes a richer value) that the application checks at key decision points to determine which code path to follow. The flag's value can be changed in a configuration store — a database, a file, a dedicated service like LaunchDarkly or Unleash — and the application picks up the change without restarting.
Feature flags come in several distinct flavors, and confusing them leads to the "flag spaghetti" anti-pattern:
- Release flags are short-lived flags that control whether a new feature is visible to users; they exist to decouple deployment from release and should be removed once the feature is fully rolled out.
- Experiment flags (A/B test flags) control which variant of an experience a user sees; they are owned by product managers and data scientists and may live longer.
- Operations flags (ops flags or kill switches) allow the operations team to disable a feature or an external dependency instantly in response to an incident; these may be permanent.
- Permission flags control access to features based on user identity, subscription tier, or organization membership.
The following Python implementation shows a clean, extensible feature flag system that supports all of these categories and can reload its state from a JSON file without restarting the application.
# features/feature_flags.py
#
# A lightweight, file-backed feature flag system with hot-reload support.
#
# Flag definitions live in a JSON file (default: features/flags.json).
# The path is configurable via the FEATURE_FLAGS_PATH environment variable.
# The application checks flags via the FeatureFlags class.
# An operator can update the JSON file and call reload() to apply changes
# without restarting the application -- this is the hot-reload mechanism.
#
# Thread safety: all public methods acquire a reentrant lock before
# reading or writing the internal flag registry, so FeatureFlags is
# safe to use from multiple threads simultaneously.
from __future__ import annotations
import json
import logging
import threading
from dataclasses import dataclass
from enum import Enum, auto
from pathlib import Path
from typing import Any, Dict, Optional
logger = logging.getLogger(__name__)
class FlagCategory(Enum):
"""
Semantic category of a feature flag.
This is metadata only -- it does not affect flag evaluation --
but it is invaluable for lifecycle management and auditing.
Knowing a flag's category tells you who owns it, how long it
should live, and what process governs its removal.
"""
RELEASE = auto() # Short-lived: decouple deploy from release
EXPERIMENT = auto() # A/B test or multivariate experiment
OPS = auto() # Kill switch or operational control
PERMISSION = auto() # Access control by user / tenant / tier
@dataclass
class FlagDefinition:
"""
The complete definition of a single feature flag.
Includes both the runtime value (enabled, rollout_percentage) and
the lifecycle metadata (category, owner, expires_on) needed to
manage the flag over its lifetime.
"""
name: str
category: FlagCategory
description: str
enabled: bool
owner: str # Team or person responsible for this flag
expires_on: Optional[str] # ISO 8601 date string; None = permanent
rollout_percentage: float # 0.0 (no users) to 100.0 (all users)
class FeatureFlags:
"""
Thread-safe, hot-reloadable feature flag registry.
Usage:
flags = FeatureFlags.load("features/flags.json")
if flags.is_enabled("streaming_responses"):
# use streaming path
else:
# use batch response path
Hot reload (call from a background thread, signal handler, or
admin API endpoint to pick up changes without restarting):
flags.reload()
"""
def __init__(self, flags_path: str) -> None:
self._path = Path(flags_path)
self._flags: Dict[str, FlagDefinition] = {}
self._lock = threading.RLock()
self._load()
@classmethod
def load(cls, flags_path: str) -> "FeatureFlags":
"""Factory method: construct and return a loaded FeatureFlags."""
return cls(flags_path)
def is_enabled(self, flag_name: str, default: bool = False) -> bool:
"""
Return True if the named flag is enabled.
Returns 'default' if the flag is not defined in the registry.
This allows new flags to be added to the application code before
they appear in the flags file; they default to off (False) until
explicitly enabled, which is the safe direction for new features.
"""
with self._lock:
flag = self._flags.get(flag_name)
if flag is None:
logger.warning(
"Feature flag '%s' is not defined in %s. "
"Returning default=%s.",
flag_name, self._path, default
)
return default
return flag.enabled
def get_rollout_percentage(self, flag_name: str) -> float:
"""
Return the rollout percentage for a named flag (0.0 to 100.0).
Returns 0.0 if the flag is not defined.
"""
with self._lock:
flag = self._flags.get(flag_name)
return flag.rollout_percentage if flag else 0.0
def reload(self) -> None:
"""
Reload flag definitions from disk and atomically replace the
internal registry. Thread-safe: concurrent flag checks will see
either the old registry or the new one, never a partial state.
"""
logger.info("Reloading feature flags from %s", self._path)
self._load()
def _load(self) -> None:
"""
Parse the flags JSON file and atomically replace the registry.
Invalid flag entries are logged and skipped rather than causing
the entire reload to fail. This means a partially broken flags
file still loads the valid flags, which is the safer behaviour
for a production system. The operator is notified via ERROR logs.
"""
if not self._path.exists():
logger.error(
"Feature flags file not found: %s. "
"All flags will be disabled until the file is created.",
self._path
)
return
try:
with self._path.open(encoding="utf-8") as f:
raw = json.load(f)
except json.JSONDecodeError as e:
logger.error(
"Feature flags file %s contains invalid JSON: %s. "
"Keeping existing flags unchanged.",
self._path, e
)
return
new_flags: Dict[str, FlagDefinition] = {}
for entry in raw.get("flags", []):
try:
flag = FlagDefinition(
name=entry["name"],
category=FlagCategory[entry["category"].upper()],
description=entry.get("description", ""),
enabled=bool(entry["enabled"]),
owner=entry.get("owner", "unknown"),
expires_on=entry.get("expires_on"),
rollout_percentage=float(
entry.get("rollout_percentage", 100.0)
),
)
new_flags[flag.name] = flag
except (KeyError, ValueError) as e:
logger.error(
"Invalid flag entry in %s (skipping): %s -- error: %s",
self._path, entry, e
)
with self._lock:
self._flags = new_flags
logger.info(
"Loaded %d feature flag(s) from %s",
len(new_flags), self._path
)
The corresponding features/flags.json file that this system reads:
{
"flags": [
{
"name": "streaming_responses",
"category": "RELEASE",
"description": "Stream LLM tokens to the client as they are generated",
"enabled": false,
"owner": "platform-team",
"expires_on": "2025-03-01",
"rollout_percentage": 0.0
},
{
"name": "fallback_on_error",
"category": "OPS",
"description": "Fall back to secondary LLM provider on primary failure",
"enabled": true,
"owner": "sre-team",
"expires_on": null,
"rollout_percentage": 100.0
},
{
"name": "use_experimental_model",
"category": "EXPERIMENT",
"description": "Route 10% of traffic to the new experimental model",
"enabled": true,
"owner": "ml-team",
"expires_on": "2025-06-01",
"rollout_percentage": 10.0
},
{
"name": "premium_context_window",
"category": "PERMISSION",
"description": "Allow premium users to use the 128k context window",
"enabled": true,
"owner": "product-team",
"expires_on": null,
"rollout_percentage": 100.0
},
{
"name": "log_prompts",
"category": "OPS",
"description": "Log full prompt content for debugging (disable in production for privacy)",
"enabled": false,
"owner": "sre-team",
"expires_on": null,
"rollout_percentage": 100.0
}
]
}
CHAPTER TWO: ARCHITECTURAL PATTERNS FOR CONFIGURABILITY
Having surveyed the timeline of configuration decisions, we now turn to the architectural patterns that make those decisions clean, maintainable, and testable. These patterns are the structural vocabulary of configurability.
THE STRATEGY PATTERN: SWAPPABLE ALGORITHMS AND BEHAVIORS
The Strategy pattern is the single most important design pattern for configurability. Its premise is simple: if you have a family of algorithms or behaviors that you want to be able to swap, define a common interface for them, implement each variant as a separate class that satisfies that interface, and inject the desired variant into the context object that uses it.
The power of this pattern is that the context object — the object that does the work — does not know or care which specific strategy it is using. It only knows the interface. This means you can swap strategies at startup time (by choosing which concrete class to inject) or even at runtime (by replacing the strategy object while the application is running).
For our LLM gateway, the Strategy pattern is the natural fit for representing different LLM providers. Each provider (OpenAI, Ollama, llama.cpp) is a concrete strategy that implements a common LLMProvider interface.
# llm/provider_interface.py
#
# The abstract base class (interface) that all LLM providers must implement.
#
# This is the heart of the Strategy pattern for LLM configurability.
# Adding a new provider means implementing this interface -- nothing else
# in the application needs to change. All business logic depends only on
# LLMProvider, never on a concrete provider class.
from __future__ import annotations
import abc
from dataclasses import dataclass
@dataclass
class ChatMessage:
"""A single message in a multi-turn conversation."""
role: str # "system", "user", or "assistant"
content: str
@dataclass
class CompletionResult:
"""
The structured result of a single completion request.
Carrying provider and model information in the result (rather than
relying on the caller to know which provider was used) is important
for observability: every log entry and metric can be attributed to
the exact provider and model that produced it, even after a runtime
provider switch or a fallback event.
"""
content: str # The generated text content
model: str # Model identifier that produced this result
provider: str # Provider name that served this result
prompt_tokens: int # Tokens consumed by the input prompt
completion_tokens: int # Tokens generated in the response
latency_ms: float # Wall-clock time for the full request (ms)
class LLMProvider(abc.ABC):
"""
Abstract interface for all LLM providers.
Concrete implementations:
- OpenAIProvider -- remote OpenAI API (or any compatible endpoint)
- OllamaProvider -- local Ollama server (OpenAI-compatible API)
- LlamaCppProvider -- local in-process inference via llama-cpp-python
The application code interacts exclusively with this interface.
The concrete provider is selected at startup time (via the factory)
and can be swapped at runtime (via the gateway's set_primary_provider).
"""
@property
@abc.abstractmethod
def provider_name(self) -> str:
"""Human-readable name of this provider, used in logs and metrics."""
@property
@abc.abstractmethod
def active_model(self) -> str:
"""Identifier of the model currently in use by this provider."""
@abc.abstractmethod
def complete(
self,
messages: list[ChatMessage],
temperature: float = 0.7,
max_tokens: int = 1024,
) -> CompletionResult:
"""
Send a list of chat messages and return the completion result.
Implementations are responsible for:
- Formatting messages according to the provider's API contract.
- Handling retries on transient network or rate-limit errors.
- Raising an exception (not returning None) on unrecoverable failure.
"""
@abc.abstractmethod
def health_check(self) -> bool:
"""
Return True if the provider is reachable and ready to serve requests.
Used by the gateway's fallback mechanism and by external monitoring
systems. Implementations should be fast (< 2 seconds) and should
not consume significant quota or resources.
"""
Now let us implement the concrete strategies. The first is the OpenAI provider, which connects to the remote OpenAI API. The second is the Ollama provider, which connects to a locally running Ollama server. The third is the most interesting one: the llama.cpp in-process provider, which loads a model directly into the Python process using llama-cpp-python, bypassing any HTTP server entirely and giving us direct control over GPU layer offloading.
# llm/openai_provider.py
#
# Concrete LLM provider strategy for the remote OpenAI API.
#
# This provider also works with any OpenAI-compatible endpoint by
# changing the base_url in the provider settings. Compatible endpoints
# include Azure OpenAI, Together AI, Groq, Fireworks AI, and others.
from __future__ import annotations
import logging
import time
import openai
from config.settings import LLMProviderSettings
from llm.provider_interface import ChatMessage, CompletionResult, LLMProvider
logger = logging.getLogger(__name__)
class OpenAIProvider(LLMProvider):
"""
LLM provider that connects to the remote OpenAI API.
The openai.OpenAI client is thread-safe and connection-pooling is
handled internally by the httpx library that the openai package uses.
A single OpenAIProvider instance can safely serve concurrent requests
from multiple threads.
"""
def __init__(self, settings: LLMProviderSettings) -> None:
self._settings = settings
self._model = settings.default_model
self._client = openai.OpenAI(
api_key=settings.api_key,
base_url=settings.base_url,
timeout=settings.timeout_seconds,
max_retries=settings.max_retries,
)
logger.info(
"OpenAIProvider initialized: model=%s, base_url=%s",
self._model, settings.base_url
)
@property
def provider_name(self) -> str:
return self._settings.name
@property
def active_model(self) -> str:
return self._model
def complete(
self,
messages: list[ChatMessage],
temperature: float = 0.7,
max_tokens: int = 1024,
) -> CompletionResult:
"""Send messages to the OpenAI API and return a structured result."""
api_messages = [
{"role": m.role, "content": m.content}
for m in messages
]
start_time = time.monotonic()
try:
response = self._client.chat.completions.create(
model=self._model,
messages=api_messages,
temperature=temperature,
max_tokens=max_tokens,
)
except openai.OpenAIError as e:
logger.error(
"OpenAI API error (model=%s): %s", self._model, e
)
raise
latency_ms = (time.monotonic() - start_time) * 1000.0
choice = response.choices[0]
usage = response.usage
return CompletionResult(
content=choice.message.content or "",
model=response.model,
provider=self.provider_name,
prompt_tokens=usage.prompt_tokens if usage else 0,
completion_tokens=usage.completion_tokens if usage else 0,
latency_ms=latency_ms,
)
def health_check(self) -> bool:
"""Verify connectivity by listing available models from the API."""
try:
self._client.models.list()
return True
except Exception as e:
logger.warning("OpenAI health check failed: %s", e)
return False
# llm/ollama_provider.py
#
# Concrete LLM provider strategy for a locally running Ollama server.
#
# Ollama exposes an OpenAI-compatible REST API at port 11434, which means
# we can use the standard openai Python client library to talk to it.
# The only differences from the OpenAI provider are the base_url and the
# api_key placeholder (Ollama does not require authentication).
#
# Prerequisites:
# 1. Install Ollama: https://ollama.com/download
# 2. Start the server: ollama serve
# 3. Pull a model: ollama pull llama3.2
# 4. Verify it is up: curl http://localhost:11434/v1/models
from __future__ import annotations
import logging
import time
import openai
from config.settings import LLMProviderSettings
from llm.provider_interface import ChatMessage, CompletionResult, LLMProvider
logger = logging.getLogger(__name__)
class OllamaProvider(LLMProvider):
"""
LLM provider that connects to a locally running Ollama server.
Because Ollama exposes an OpenAI-compatible API, this implementation
is structurally identical to OpenAIProvider. The key differences are:
- base_url points to localhost:11434 instead of api.openai.com
- api_key is a placeholder string ("ollama"), not a real secret
- Timeouts are longer because local inference is slower than a
cloud API backed by dedicated GPU clusters
"""
def __init__(self, settings: LLMProviderSettings) -> None:
self._settings = settings
self._model = settings.default_model
self._client = openai.OpenAI(
api_key=settings.api_key,
base_url=settings.base_url,
timeout=settings.timeout_seconds,
max_retries=settings.max_retries,
)
logger.info(
"OllamaProvider initialized: model=%s, url=%s",
self._model, settings.base_url
)
@property
def provider_name(self) -> str:
return self._settings.name
@property
def active_model(self) -> str:
return self._model
def complete(
self,
messages: list[ChatMessage],
temperature: float = 0.7,
max_tokens: int = 1024,
) -> CompletionResult:
"""Send messages to the local Ollama server and return the result."""
api_messages = [
{"role": m.role, "content": m.content}
for m in messages
]
start_time = time.monotonic()
try:
response = self._client.chat.completions.create(
model=self._model,
messages=api_messages,
temperature=temperature,
max_tokens=max_tokens,
)
except openai.OpenAIError as e:
logger.error(
"Ollama API error (model=%s): %s", self._model, e
)
raise
latency_ms = (time.monotonic() - start_time) * 1000.0
choice = response.choices[0]
usage = response.usage
return CompletionResult(
content=choice.message.content or "",
model=self._model,
provider=self.provider_name,
prompt_tokens=usage.prompt_tokens if usage else 0,
completion_tokens=usage.completion_tokens if usage else 0,
latency_ms=latency_ms,
)
def health_check(self) -> bool:
"""
Verify that Ollama is running by listing available local models.
The /v1/models endpoint returns the models currently pulled and
available for inference on this machine.
"""
try:
models = self._client.models.list()
logger.debug(
"Ollama health check OK. Available models: %s",
[m.id for m in models.data]
)
return True
except Exception as e:
logger.warning("Ollama health check failed: %s", e)
return False
The most technically interesting provider is the llama.cpp in-process provider. Instead of making HTTP calls to a server, it loads the model directly into the Python process using the llama-cpp-python library. This gives us direct control over how many transformer layers are offloaded to the GPU — the n_gpu_layers parameter — which is the key to leveraging CUDA, Metal, or Vulkan acceleration from Python code. The GPU backend itself was selected at build time (when llama-cpp-python was compiled), but the degree of GPU utilization is configured at startup time through n_gpu_layers.
# llm/llamacpp_provider.py
#
# Concrete LLM provider strategy using llama-cpp-python in-process.
#
# This provider loads a GGUF model file directly into the Python process,
# bypassing any HTTP server. GPU acceleration (CUDA, Metal, Vulkan) is
# controlled by the n_gpu_layers parameter at startup time. The actual
# GPU backend is determined by how llama-cpp-python was compiled (build
# time); this class does not need to know which backend is active.
#
# Build-time installation (choose ONE based on your hardware):
#
# NVIDIA CUDA (Linux / Windows, requires CUDA Toolkit 11.8+):
# CMAKE_ARGS="-DGGML_CUDA=on" \
# pip install --upgrade --force-reinstall --no-cache-dir \
# llama-cpp-python
#
# Apple Metal (macOS, Apple Silicon M1/M2/M3/M4):
# CMAKE_ARGS="-DGGML_METAL=on" \
# pip install --upgrade --force-reinstall --no-cache-dir \
# llama-cpp-python
#
# Vulkan (cross-platform, requires Vulkan SDK):
# CMAKE_ARGS="-DGGML_VULKAN=on" \
# pip install --upgrade --force-reinstall --no-cache-dir \
# llama-cpp-python
#
# CPU only (no GPU required):
# pip install llama-cpp-python
#
# Model files:
# Download GGUF models from https://huggingface.co
# Recommended starting point: bartowski/Llama-3.2-3B-Instruct-GGUF
# Set LLAMACPP_MODEL_PATH to the downloaded .gguf file path.
from __future__ import annotations
import logging
import time
from pathlib import Path
from llm.provider_interface import ChatMessage, CompletionResult, LLMProvider
logger = logging.getLogger(__name__)
class LlamaCppProvider(LLMProvider):
"""
LLM provider that runs inference in-process using llama-cpp-python.
The GGUF model is loaded once at construction time and kept in memory
(and GPU VRAM if n_gpu_layers > 0) for the lifetime of the provider.
GPU layer offloading via n_gpu_layers:
n_gpu_layers = 0 -- CPU-only inference (slowest; no GPU required)
n_gpu_layers > 0 -- that many transformer layers run on the GPU
n_gpu_layers = -1 -- all layers run on the GPU (fastest; needs VRAM)
The actual GPU backend (CUDA, Metal, Vulkan) is determined by the
build flags used when llama-cpp-python was installed. This class
is backend-agnostic; llama.cpp handles the dispatch transparently.
"""
def __init__(
self,
model_path: str,
n_gpu_layers: int = 0,
context_length: int = 4096,
verbose: bool = False,
) -> None:
"""
Load the GGUF model and initialize the inference engine.
Args:
model_path: Absolute or relative path to the .gguf file.
n_gpu_layers: Transformer layers to offload to GPU.
Use -1 to offload all layers.
context_length: Maximum context window in tokens.
verbose: If True, llama.cpp prints diagnostic output
to stderr (useful for debugging GPU offload).
Raises:
FileNotFoundError: If the model file does not exist.
ImportError: If llama-cpp-python is not installed.
"""
model_file = Path(model_path)
if not model_file.exists():
raise FileNotFoundError(
f"GGUF model file not found: {model_path}\n"
f"Download a GGUF model from https://huggingface.co and "
f"set the LLAMACPP_MODEL_PATH environment variable to its "
f"absolute path."
)
# Lazy import: the rest of the application can import this module
# even if llama-cpp-python is not installed. The ImportError is
# raised only when a LlamaCppProvider is actually instantiated.
try:
from llama_cpp import Llama
except ImportError as e:
raise ImportError(
"llama-cpp-python is not installed. Install it with:\n"
" pip install llama-cpp-python\n"
"For GPU support, see the build-time installation "
"instructions at the top of this file."
) from e
logger.info(
"Loading GGUF model: path=%s, n_gpu_layers=%d, "
"context_length=%d, verbose=%s",
model_path, n_gpu_layers, context_length, verbose
)
# Model loading is the expensive operation: it reads the weight
# file from disk and, if n_gpu_layers > 0, transfers the specified
# layers to GPU VRAM. This may take several seconds for large models.
self._llm = Llama(
model_path=str(model_file),
n_gpu_layers=n_gpu_layers,
n_ctx=context_length,
verbose=verbose,
)
self._model_path = model_path
self._n_gpu_layers = n_gpu_layers
logger.info(
"GGUF model loaded. GPU layers offloaded: %d. "
"Active backend determined by build flags.",
n_gpu_layers
)
@property
def provider_name(self) -> str:
return "llama.cpp (in-process)"
@property
def active_model(self) -> str:
return Path(self._model_path).stem
def complete(
self,
messages: list[ChatMessage],
temperature: float = 0.7,
max_tokens: int = 1024,
) -> CompletionResult:
"""
Run inference using the in-process llama.cpp engine.
create_chat_completion() applies the model's chat template
(e.g., ChatML, Llama-3 instruct format) automatically based on
the metadata embedded in the GGUF file, so callers do not need
to format prompts manually.
"""
api_messages = [
{"role": m.role, "content": m.content}
for m in messages
]
start_time = time.monotonic()
response = self._llm.create_chat_completion(
messages=api_messages,
temperature=temperature,
max_tokens=max_tokens,
)
latency_ms = (time.monotonic() - start_time) * 1000.0
choice = response["choices"][0]
usage = response.get("usage", {})
return CompletionResult(
content=choice["message"]["content"] or "",
model=self.active_model,
provider=self.provider_name,
prompt_tokens=usage.get("prompt_tokens", 0),
completion_tokens=usage.get("completion_tokens", 0),
latency_ms=latency_ms,
)
def health_check(self) -> bool:
"""
Return True if the model is loaded and ready for inference.
The in-process provider is considered healthy if the Llama object
was successfully constructed (model loaded into memory/VRAM).
Unlike network providers, there is no connectivity to check.
"""
return self._llm is not None
THE FACTORY PATTERN: CREATING THE RIGHT STRATEGY
The Strategy pattern tells us how to define and use swappable behaviors. The Factory pattern tells us how to create the right concrete strategy based on configuration. Together, they form the backbone of a configurable system.
A provider factory reads the application settings and constructs the appropriate concrete LLMProvider instance. The rest of the application never calls a constructor directly — it always goes through the factory. This means that adding a new provider only requires adding a new branch in the factory and a new concrete class; no other code changes are needed.
# llm/provider_factory.py
#
# Factory for constructing the correct LLMProvider based on settings.
#
# This is the single place in the application that knows about all
# concrete provider implementations. Everything else depends only on
# the LLMProvider abstract interface, which keeps the dependency graph
# clean and makes adding new providers a localised change.
from __future__ import annotations
import logging
import os
from config.settings import Settings
from llm.provider_interface import LLMProvider
from llm.openai_provider import OpenAIProvider
from llm.ollama_provider import OllamaProvider
from llm.llamacpp_provider import LlamaCppProvider
logger = logging.getLogger(__name__)
class LLMProviderFactory:
"""
Creates LLMProvider instances based on application settings.
Design note: this factory imports all concrete provider classes.
All other application code (gateway, business logic, tests) imports
only the LLMProvider abstract interface, keeping them decoupled from
any specific provider implementation.
"""
@staticmethod
def create(settings: Settings) -> LLMProvider:
"""
Construct and return the configured LLM provider.
Args:
settings: The application settings loaded at startup.
Returns:
A fully initialized LLMProvider ready to serve requests.
Raises:
ValueError: If the configured provider name is unknown.
FileNotFoundError: If the llamacpp model file is not found.
ImportError: If a required provider library is not installed.
"""
provider_name = settings.llm_provider
provider_settings = settings.providers.get(provider_name)
if provider_settings is None:
raise ValueError(
f"No settings found for provider '{provider_name}'. "
f"Available providers: {list(settings.providers.keys())}"
)
logger.info("Creating LLM provider: %s", provider_name)
if provider_name == "openai":
return OpenAIProvider(provider_settings)
elif provider_name == "ollama":
return OllamaProvider(provider_settings)
elif provider_name == "llamacpp":
# The llama.cpp in-process provider needs the model file path
# and the GPU layer count from the environment and settings.
model_path = os.environ.get("LLAMACPP_MODEL_PATH", "")
context_length = int(
os.environ.get("LLAMACPP_CONTEXT_LENGTH", "4096")
)
return LlamaCppProvider(
model_path=model_path,
n_gpu_layers=settings.gpu_layer_count,
context_length=context_length,
verbose=(settings.log_level == "DEBUG"),
)
else:
raise ValueError(
f"Unknown LLM provider: '{provider_name}'. "
f"Supported providers: openai, ollama, llamacpp"
)
THE GATEWAY PATTERN WITH RUNTIME FALLBACK
Now we assemble the pieces into an LLM Gateway — the component that the rest of the application actually talks to. The gateway wraps the active provider, applies feature flags, implements the fallback strategy when the primary provider fails, and collects observability data. This is where runtime configurability truly shines: the gateway can switch providers on the fly if the feature flag system or a health check demands it.
# llm/gateway.py
#
# The LLM Gateway: the single entry point for all LLM interactions.
#
# The gateway applies the following behaviours in order:
# 1. Attempt the request with the primary provider.
# 2. If the primary fails and the 'fallback_on_error' flag is enabled,
# attempt the request with the fallback provider.
# 3. Log structured observability data for every completed request,
# including whether a fallback was used and the full prompt content
# if the 'log_prompts' flag is enabled.
#
# The primary and fallback providers can be swapped at runtime without
# restarting the application by calling set_primary_provider() or
# set_fallback_provider(). This is the key runtime configurability
# mechanism of the entire system.
from __future__ import annotations
import logging
from typing import Optional
import time
from features.feature_flags import FeatureFlags
from llm.provider_interface import ChatMessage, CompletionResult, LLMProvider
logger = logging.getLogger(__name__)
class LLMGateway:
"""
Application-facing facade for all LLM interactions.
The gateway hides provider selection, fallback logic, and observability
from the business logic layer. Application code calls complete() and
receives a CompletionResult; it never interacts with a provider directly.
Thread safety: set_primary_provider() and set_fallback_provider() are
not atomic with respect to in-flight complete() calls. In a production
system with concurrent requests, provider switches should be coordinated
with a read-write lock. For the purposes of this article, the gateway
is used from a single thread and this simplification is acceptable.
"""
def __init__(
self,
primary_provider: LLMProvider,
feature_flags: FeatureFlags,
fallback_provider: Optional[LLMProvider] = None,
) -> None:
self._primary = primary_provider
self._fallback = fallback_provider
self._flags = feature_flags
logger.info(
"LLMGateway initialized. Primary: '%s'. Fallback: '%s'.",
primary_provider.provider_name,
fallback_provider.provider_name if fallback_provider else "none",
)
def set_primary_provider(self, provider: LLMProvider) -> None:
"""
Swap the primary provider at runtime.
This is the key runtime configurability mechanism. An operator
or an automated system (e.g., a health-check loop, a cost-control
daemon, or an admin API handler) can call this method to switch
the active LLM without restarting the application.
"""
logger.info(
"Runtime provider switch: '%s' -> '%s'",
self._primary.provider_name,
provider.provider_name,
)
self._primary = provider
def set_fallback_provider(
self, provider: Optional[LLMProvider]
) -> None:
"""Set or clear the fallback provider at runtime."""
self._fallback = provider
logger.info(
"Fallback provider updated: '%s'",
provider.provider_name if provider else "none",
)
def complete(
self,
messages: list[ChatMessage],
temperature: float = 0.7,
max_tokens: int = 1024,
) -> CompletionResult:
"""
Execute a completion request with automatic fallback support.
The fallback mechanism is controlled by the 'fallback_on_error'
feature flag. If the flag is disabled, errors from the primary
provider propagate directly to the caller with no retry.
"""
fallback_enabled = self._flags.is_enabled(
"fallback_on_error", default=True
)
try:
result = self._primary.complete(messages, temperature, max_tokens)
self._log_result(result, messages, is_fallback=False)
return result
except Exception as primary_error:
logger.error(
"Primary provider '%s' failed: %s",
self._primary.provider_name, primary_error,
)
if fallback_enabled and self._fallback is not None:
logger.warning(
"Falling back to '%s'.",
self._fallback.provider_name,
)
try:
result = self._fallback.complete(
messages, temperature, max_tokens
)
self._log_result(result, messages, is_fallback=True)
return result
except Exception as fallback_error:
logger.error(
"Fallback provider '%s' also failed: %s",
self._fallback.provider_name, fallback_error,
)
# Chain the exceptions so the caller can see both
# the primary and fallback failure reasons.
raise fallback_error from primary_error
else:
raise
def _log_result(
self,
result: CompletionResult,
messages: list[ChatMessage],
is_fallback: bool,
) -> None:
"""
Log structured observability data for a completed request.
The 'log_prompts' feature flag controls whether the full prompt
content is included in the log output. It should be disabled in
production environments where prompts may contain sensitive data.
"""
logger.info(
"Completion: provider=%s, model=%s, "
"prompt_tokens=%d, completion_tokens=%d, "
"latency_ms=%.1f, fallback=%s",
result.provider,
result.model,
result.prompt_tokens,
result.completion_tokens,
result.latency_ms,
is_fallback,
)
# Conditionally log the full prompt content based on the feature flag.
# This flag should be enabled only in development or debugging sessions,
# never in production, to protect user privacy.
if self._flags.is_enabled("log_prompts", default=False):
for i, msg in enumerate(messages):
logger.debug(
" [prompt msg %d] role=%s content=%r",
i, msg.role, msg.content,
)
logger.debug(
" [response] %r", result.content
)
WIRING IT ALL TOGETHER: THE APPLICATION ENTRY POINT
The final piece of the puzzle is the application entry point, which reads the settings, constructs the providers through the factory, wires up the gateway, and demonstrates runtime provider switching. This is where all the configuration layers we have discussed come together into a coherent, working system.
# main.py
#
# Application entry point for the LLM Gateway demonstration.
#
# This file demonstrates the full configurability lifecycle:
# - Startup-time configuration via Settings.load()
# - Provider construction via LLMProviderFactory
# - Runtime feature flag checking via FeatureFlags
# - Runtime provider switching via LLMGateway.set_primary_provider()
#
# Quick-start examples (see DEPLOYMENT GUIDE below for full details):
#
# # Run with Ollama (local, no API key needed):
# LLM_PROVIDER=ollama python main.py
#
# # Run with OpenAI (remote):
# LLM_PROVIDER=openai OPENAI_API_KEY=sk-... python main.py
#
# # Run with llama.cpp in-process (CPU):
# LLM_PROVIDER=llamacpp \
# LLAMACPP_MODEL_PATH=/path/to/model.gguf \
# python main.py
#
# # Run with llama.cpp in-process (all layers on GPU):
# LLM_PROVIDER=llamacpp \
# LLAMACPP_MODEL_PATH=/path/to/model.gguf \
# GPU_LAYER_COUNT=-1 \
# python main.py
from __future__ import annotations
import logging
import sys
from typing import Optional
from config.settings import Settings
from features.feature_flags import FeatureFlags
from llm.gateway import LLMGateway
from llm.ollama_provider import OllamaProvider
from llm.provider_factory import LLMProviderFactory
from llm.provider_interface import ChatMessage
def configure_logging(level: str) -> None:
"""Configure structured logging for the application."""
logging.basicConfig(
level=getattr(logging, level, logging.INFO),
format="%(asctime)s %(levelname)-8s %(name)-30s %(message)s",
datefmt="%Y-%m-%dT%H:%M:%S",
stream=sys.stderr,
)
def build_fallback_provider(
settings: Settings,
) -> Optional[OllamaProvider]:
"""
Attempt to construct an Ollama fallback provider.
Returns None if Ollama is not reachable or if the primary provider
is already Ollama (no point falling back to the same provider).
"""
if settings.llm_provider == "ollama":
return None # Primary is already Ollama; no fallback needed.
ollama_settings = settings.providers.get("ollama")
if ollama_settings is None:
return None
try:
candidate = OllamaProvider(ollama_settings)
if candidate.health_check():
logger = logging.getLogger(__name__)
logger.info(
"Ollama fallback provider is available at %s",
ollama_settings.base_url,
)
return candidate
else:
logging.getLogger(__name__).warning(
"Ollama fallback configured but not reachable at %s. "
"Fallback will be disabled.",
ollama_settings.base_url,
)
return None
except Exception as e:
logging.getLogger(__name__).warning(
"Could not initialize Ollama fallback provider: %s", e
)
return None
def main() -> int:
"""
Application entry point.
Returns:
0 on success, 1 on configuration or unrecoverable runtime error.
"""
# ------------------------------------------------------------------
# STARTUP-TIME CONFIGURATION
# Read and validate all settings before doing any real work.
# A misconfigured application should fail fast and loudly here,
# not silently hours later when a specific code path is exercised.
# ------------------------------------------------------------------
try:
settings = Settings.load()
except ValueError as e:
# Use print here because logging is not yet configured.
print(f"FATAL: Configuration error: {e}", file=sys.stderr)
return 1
configure_logging(settings.log_level)
logger = logging.getLogger(__name__)
logger.info("LLM Gateway starting up.")
# ------------------------------------------------------------------
# FEATURE FLAGS
# Load the feature flag registry. The path is configurable via the
# FEATURE_FLAGS_PATH environment variable (default: features/flags.json).
# ------------------------------------------------------------------
import os
flags_path = os.environ.get("FEATURE_FLAGS_PATH", "features/flags.json")
flags = FeatureFlags.load(flags_path)
# ------------------------------------------------------------------
# PRIMARY PROVIDER CONSTRUCTION
# The factory reads settings and constructs the right provider.
# ------------------------------------------------------------------
try:
primary_provider = LLMProviderFactory.create(settings)
except (ValueError, FileNotFoundError, ImportError) as e:
logger.error("Failed to create primary LLM provider: %s", e)
return 1
# ------------------------------------------------------------------
# FALLBACK PROVIDER CONSTRUCTION
# Attempt to set up Ollama as a fallback. If Ollama is not running
# or the primary is already Ollama, the fallback is disabled.
# ------------------------------------------------------------------
fallback_provider = build_fallback_provider(settings)
# ------------------------------------------------------------------
# GATEWAY ASSEMBLY
# Wire the providers and feature flags into the gateway.
# ------------------------------------------------------------------
gateway = LLMGateway(
primary_provider=primary_provider,
feature_flags=flags,
fallback_provider=fallback_provider,
)
# ------------------------------------------------------------------
# DEMONSTRATION: NORMAL COMPLETION REQUEST
# ------------------------------------------------------------------
logger.info("=== Demo: Normal completion request ===")
messages = [
ChatMessage(
role="system",
content=(
"You are a concise technical assistant. "
"Answer in at most two sentences."
),
),
ChatMessage(
role="user",
content=(
"What is the key benefit of the Strategy design pattern "
"for software configurability?"
),
),
]
try:
result = gateway.complete(messages, temperature=0.3, max_tokens=200)
print(f"\nProvider : {result.provider}")
print(f"Model : {result.model}")
print(f"Latency : {result.latency_ms:.0f} ms")
print(
f"Tokens : {result.prompt_tokens} prompt / "
f"{result.completion_tokens} completion"
)
print(f"\nResponse :\n{result.content}\n")
except Exception as e:
logger.error("Completion failed: %s", e)
return 1
# ------------------------------------------------------------------
# DEMONSTRATION: RUNTIME PROVIDER SWITCHING
#
# Switch the primary provider without restarting the application.
# In a real system this switch might be triggered by:
# - An operator command via an admin REST API endpoint
# - A health-check loop detecting that the primary is degraded
# - A cost-control daemon switching to a cheaper model at peak hours
# - A feature flag change routing traffic to a new model version
# ------------------------------------------------------------------
if fallback_provider is not None:
logger.info("=== Demo: Runtime provider switch ===")
gateway.set_primary_provider(fallback_provider)
messages2 = [
ChatMessage(role="system", content="You are helpful."),
ChatMessage(
role="user",
content="Name one advantage of running LLMs locally.",
),
]
try:
result2 = gateway.complete(
messages2, temperature=0.5, max_tokens=100
)
print(f"After runtime switch:")
print(f"Provider : {result2.provider}")
print(f"Response :\n{result2.content}\n")
except Exception as e:
logger.error("Post-switch completion failed: %s", e)
# Non-fatal: the demo continues.
logger.info("LLM Gateway demonstration complete.")
return 0
if __name__ == "__main__":
sys.exit(main())
DEPLOYMENT GUIDE: INSTALLING, CONFIGURING, AND RUNNING THE APPLICATION
This section consolidates everything you need to go from a clean machine to a running LLM Gateway.
Project Directory Structure
llm-gateway/
├── main.py
├── requirements.txt
├── .env.example
├── config/
│ ├── __init__.py
│ └── settings.py
├── features/
│ ├── __init__.py
│ ├── feature_flags.py
│ └── flags.json
└── llm/
├── __init__.py
├── provider_interface.py
├── openai_provider.py
├── ollama_provider.py
├── llamacpp_provider.py
├── provider_factory.py
└── gateway.py
Create the empty __init__.py files:
touch config/__init__.py features/__init__.py llm/__init__.py
Python Dependencies
# requirements.txt
#
# Core dependencies for the LLM Gateway application.
# Pin to minor versions for reproducible installs.
# Run: pip install -r requirements.txt
# OpenAI-compatible Python client (used for OpenAI and Ollama providers)
openai>=1.30.0,<2.0.0
# llama-cpp-python is NOT listed here because its installation requires
# build-time GPU backend selection via CMAKE_ARGS.
# Install it separately using one of the commands in the PROVIDER SETUP
# section below.
Environment Configuration
# .env.example
#
# Copy this file to .env and fill in your values for local development.
# NEVER commit .env to version control -- add it to .gitignore.
# In production, set these variables directly in the platform environment
# (Kubernetes Secret, Docker --env-file, systemd EnvironmentFile, etc.).
# -----------------------------------------------------------------------
# Core application settings
# -----------------------------------------------------------------------
# Which LLM provider to use as the primary.
# Valid values: openai | ollama | llamacpp
LLM_PROVIDER=ollama
# Logging verbosity.
# Valid values: DEBUG | INFO | WARNING | ERROR
LOG_LEVEL=INFO
# Path to the feature flags JSON file.
FEATURE_FLAGS_PATH=features/flags.json
# -----------------------------------------------------------------------
# GPU settings (for llamacpp provider only)
# -----------------------------------------------------------------------
# Number of transformer layers to offload to the GPU.
# 0 = CPU-only (default; works on any machine)
# -1 = all layers on GPU (fastest; requires enough VRAM)
# >0 = that many layers on GPU (tune to fit your VRAM budget)
GPU_LAYER_COUNT=0
# -----------------------------------------------------------------------
# OpenAI provider settings
# -----------------------------------------------------------------------
OPENAI_API_KEY=sk-your-key-here
OPENAI_DEFAULT_MODEL=gpt-4o-mini
OPENAI_TIMEOUT=30.0
OPENAI_MAX_RETRIES=3
# -----------------------------------------------------------------------
# Ollama provider settings
# -----------------------------------------------------------------------
OLLAMA_BASE_URL=http://localhost:11434/v1
OLLAMA_DEFAULT_MODEL=llama3.2
OLLAMA_TIMEOUT=120.0
OLLAMA_MAX_RETRIES=2
# -----------------------------------------------------------------------
# llama.cpp in-process provider settings
# -----------------------------------------------------------------------
# Absolute path to your downloaded .gguf model file.
LLAMACPP_MODEL_PATH=/path/to/your/model.gguf
LLAMACPP_DEFAULT_MODEL=local-model
LLAMACPP_CONTEXT_LENGTH=4096
LLAMACPP_TIMEOUT=180.0
LLAMACPP_MAX_RETRIES=1
Provider Setup
Option A — Ollama (recommended for first-time setup):
# 1. Install Ollama (Linux)
curl -fsSL https://ollama.com/install.sh | sh
# 1. Install Ollama (macOS)
brew install ollama
# 2. Start the Ollama server (runs in the background)
ollama serve &
# 3. Pull a model (this downloads the model weights, ~2 GB for llama3.2:3b)
ollama pull llama3.2
# 4. Verify the server is up and the model is available
curl http://localhost:11434/v1/models
Option B — OpenAI API:
# No local installation needed.
# Set your API key and run:
export LLM_PROVIDER=openai
export OPENAI_API_KEY=sk-your-key-here
Option C — llama.cpp in-process:
# Step 1: Install llama-cpp-python with the correct GPU backend.
# Choose ONE of the following commands based on your hardware:
# NVIDIA CUDA (Linux / Windows, CUDA Toolkit 11.8+ required):
CMAKE_ARGS="-DGGML_CUDA=on" \
pip install --upgrade --force-reinstall --no-cache-dir \
llama-cpp-python
# Apple Silicon Metal (macOS M1/M2/M3/M4):
CMAKE_ARGS="-DGGML_METAL=on" \
pip install --upgrade --force-reinstall --no-cache-dir \
llama-cpp-python
# Vulkan (cross-platform, Vulkan SDK required):
CMAKE_ARGS="-DGGML_VULKAN=on" \
pip install --upgrade --force-reinstall --no-cache-dir \
llama-cpp-python
# CPU only (any machine, no GPU required):
pip install llama-cpp-python
# Step 2: Download a GGUF model file.
# The following example uses huggingface-hub to download a small model:
pip install huggingface-hub
huggingface-cli download \
bartowski/Llama-3.2-3B-Instruct-GGUF \
Llama-3.2-3B-Instruct-Q4_K_M.gguf \
--local-dir ./models
# Step 3: Set the required environment variables:
export LLM_PROVIDER=llamacpp
export LLAMACPP_MODEL_PATH=./models/Llama-3.2-3B-Instruct-Q4_K_M.gguf
export GPU_LAYER_COUNT=-1 # -1 = all layers on GPU; 0 = CPU only
Installing Core Dependencies and Running
# 1. Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate # Linux / macOS
# .venv\Scripts\activate.bat # Windows CMD
# .venv\Scripts\Activate.ps1 # Windows PowerShell
# 2. Install core Python dependencies
pip install -r requirements.txt
# 3. Copy and edit the environment configuration
cp .env.example .env
# Edit .env with your preferred editor and fill in your values.
# 4. Run the application
python main.py
# --- Alternative: pass settings directly as environment variables ---
# Run with Ollama (local, no API key needed):
LLM_PROVIDER=ollama python main.py
# Run with OpenAI (remote):
LLM_PROVIDER=openai OPENAI_API_KEY=sk-... python main.py
# Run with llama.cpp in-process (CPU):
LLM_PROVIDER=llamacpp \
LLAMACPP_MODEL_PATH=./models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
python main.py
# Run with llama.cpp in-process (all layers on GPU):
LLM_PROVIDER=llamacpp \
LLAMACPP_MODEL_PATH=./models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
GPU_LAYER_COUNT=-1 \
python main.py
# Enable debug logging to see prompt content and llama.cpp diagnostics:
LLM_PROVIDER=ollama LOG_LEVEL=DEBUG python main.py
CHAPTER THREE: ANTI-PATTERNS TO AVOID
No article on configurability would be complete without a tour of the things that go wrong. The following anti-patterns are drawn from real-world experience and represent the most common ways that developers accidentally make their systems rigid, fragile, or unmaintainable.
Anti-pattern 1: Configuration Hardcoding. This is the practice of embedding configuration values — API keys, URLs, model names, timeouts — directly in source code. The tell-tale sign is a string literal or a numeric constant that should be a configuration value but is not. The consequence is that changing the value requires a code change, a code review, a build, and a deployment. In the worst cases, secrets like API keys end up committed to version control, where they can be exfiltrated by anyone with repository access. The fix is always the same: move the value to an environment variable or a configuration file, validate it at startup, and inject it as a dependency.
Anti-pattern 2: The God Configuration Object. This occurs when a single configuration class or dictionary accumulates every setting in the entire application, including settings for components that have nothing to do with each other. The result is a class with dozens or hundreds of fields, where changing any one field requires understanding the entire class. The fix is to decompose configuration into cohesive, focused configuration objects — one per major subsystem — and inject only the relevant configuration object into each subsystem.
Anti-pattern 3: Flag Spaghetti. This is what happens when feature flags are never cleaned up. Over time, the codebase accumulates dozens of flags, some of which are permanently enabled, some permanently disabled, and some whose purpose no one remembers. The code becomes littered with nested if-else branches that check flag combinations, and the number of possible execution paths grows exponentially. The fix is to treat every feature flag as having an explicit lifecycle: a creation date, an owner, and an expiry date. Flags that have been at 100% rollout for more than a few weeks should be removed, with their enabled code path becoming the only code path.
Anti-pattern 4: The Service Locator. Instead of having dependencies injected into a class through its constructor, the class reaches into a global registry to fetch its own dependencies. This hides the class's dependencies, makes it impossible to understand what a class needs just by looking at its constructor, and makes testing extremely difficult because you cannot easily substitute test doubles for the real dependencies. The fix is constructor injection: always pass dependencies explicitly.
Anti-pattern 5: Configuration Drift. In a system with many deployment environments, the configuration of each environment gradually diverges from the others as ad-hoc changes are made directly to running systems. The result is that behavior differs between environments in ways that are not captured in any version-controlled artifact, making bugs impossible to reproduce and deployments unpredictable. The fix is to treat configuration as code: store it in version control, apply changes through automated pipelines, and never make manual changes to running systems.
Anti-pattern 6: Premature Runtime Configurability. Not every configuration value needs to be changeable at runtime. Adding runtime configurability has real costs: you need thread-safe state management, you need a mechanism to propagate changes to all parts of the system that care about them, and you need to test all combinations of configuration values. If a value only changes when you deploy a new version of the application, startup-time configuration is the right choice. Reserve runtime configurability for values that genuinely need to change without a restart: feature flags, model routing decisions, rate limits, and kill switches.
CONCLUSION: THE FULL CONFIGURABILITY LANDSCAPE AT A GLANCE
To close, let us step back and see the entire landscape as a coherent whole. Every real-world system uses multiple levels of the configurability timeline simultaneously. An industrial IoT gateway might use compile-time macros to select the communication protocol stack for a specific hardware platform, build-time CMake options to include or exclude optional sensor drivers, startup-time environment variables to configure the cloud endpoint and authentication credentials, and runtime feature flags to enable experimental data compression algorithms for a subset of devices — all in the same product.
The art of software architecture lies in choosing the right level for each configuration decision. Choose too early and you lose flexibility. Choose too late and you pay in complexity and operational overhead. The patterns and examples in this article give you the vocabulary and the tools to make those choices deliberately, confidently, and with a clear understanding of the tradeoffs involved.
Build systems that know what they are, that can become what they need to be, and that fail loudly when they are misconfigured. Your future self — and your operations team — will thank you.
