INTRODUCTION AND PROBLEM STATEMENT
Software developers, architects, and designers frequently sketch diagrams on whiteboards, paper, or digital notebooks during brainstorming sessions and design meetings. These hand-drawn artifacts capture valuable insights about system architecture, class relationships, and component interactions. However, converting these sketches into professional, shareable digital diagrams traditionally requires manual redrawing using specialized tools, which is time-consuming and error-prone.
This tutorial presents a comprehensive solution: a web-based application that leverages Large Language Models and computer vision to automatically convert photographs of hand-drawn UML diagrams, system sketches, and CRC cards into beautified digital representations. The application analyzes uploaded images, interprets the diagrammatic elements, and generates clean, professional outputs in formats like PDF, PNG, and SVG using PlantUML and similar rendering engines.
The core challenge involves multiple technical domains. First, we must extract visual information from photographs that may contain varying lighting conditions, perspectives, and drawing styles. Second, we need to interpret the semantic meaning of shapes, arrows, text labels, and spatial relationships. Third, we must translate this understanding into formal diagram specifications that rendering tools can process. Finally, we provide an interactive refinement interface where users can guide the system toward their intended interpretation.
ARCHITECTURAL OVERVIEW AND TECHNOLOGY STACK
Our application follows a modular architecture with clear separation of concerns. The frontend provides an intuitive interface for image upload, preview, and interaction. The backend orchestrates the processing pipeline, coordinating between computer vision components, LLM-based interpretation, and diagram generation services. We prioritize open-source technologies to ensure accessibility and customizability.
The technology stack includes FastAPI for the backend web framework, which offers excellent performance and automatic API documentation. For the frontend, we use vanilla JavaScript with modern ES6+ features, avoiding heavy framework dependencies while maintaining clean, maintainable code. The computer vision pipeline combines OpenCV for image preprocessing with specialized object detection models. The LLM integration supports both local models through libraries like llama-cpp-python and remote APIs, ensuring flexibility across deployment scenarios.
GPU acceleration presents unique challenges across different hardware platforms. NVIDIA CUDA remains the most mature ecosystem, but we must also support AMD ROCm for AMD GPUs, Apple Metal Performance Shaders for M-series chips, and Intel oneAPI for Intel discrete graphics. Our implementation uses abstraction layers that detect available hardware and configure the appropriate backends automatically.
IMAGE PREPROCESSING AND FEATURE EXTRACTION
Before we can interpret diagram semantics, we must prepare the input image through a series of preprocessing steps. Photographs of whiteboards often suffer from perspective distortion, uneven lighting, shadows, and background noise. Our preprocessing pipeline addresses these issues systematically.
The first stage performs perspective correction. When users photograph a whiteboard at an angle, rectangular shapes appear as trapezoids. We detect the whiteboard boundaries using edge detection and Hough transforms, then apply a homography transformation to correct the perspective. This ensures that spatial relationships in the diagram are preserved accurately.
Here is a Python implementation of the perspective correction module:
import cv2
import numpy as np
class PerspectiveCorrector:
def __init__(self, min_area_ratio=0.3):
"""
Initialize the perspective corrector.
Args:
min_area_ratio: Minimum ratio of detected quadrilateral area
to image area to consider it a valid whiteboard
"""
self.min_area_ratio = min_area_ratio
def detect_whiteboard_corners(self, image):
"""
Detect the four corners of the whiteboard in the image.
This method uses edge detection and contour analysis to find
the largest quadrilateral in the image, which we assume is
the whiteboard boundary.
"""
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
blurred = cv2.GaussianBlur(gray, (5, 5), 0)
edges = cv2.Canny(blurred, 50, 150)
# Dilate edges to close gaps
kernel = np.ones((3, 3), np.uint8)
dilated = cv2.dilate(edges, kernel, iterations=2)
# Find contours
contours, _ = cv2.findContours(
dilated,
cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE
)
# Sort contours by area and find the largest quadrilateral
contours = sorted(contours, key=cv2.contourArea, reverse=True)
image_area = image.shape[0] * image.shape[1]
for contour in contours:
area = cv2.contourArea(contour)
if area < image_area * self.min_area_ratio:
continue
# Approximate the contour to a polygon
perimeter = cv2.arcLength(contour, True)
approx = cv2.approxPolyDP(contour, 0.02 * perimeter, True)
# Check if we have a quadrilateral
if len(approx) == 4:
return self._order_points(approx.reshape(4, 2))
# If no quadrilateral found, return image corners
h, w = image.shape[:2]
return np.array([
[0, 0],
[w - 1, 0],
[w - 1, h - 1],
[0, h - 1]
], dtype=np.float32)
def _order_points(self, points):
"""
Order points in consistent order: top-left, top-right,
bottom-right, bottom-left.
"""
rect = np.zeros((4, 2), dtype=np.float32)
# Sum and difference to find corners
s = points.sum(axis=1)
diff = np.diff(points, axis=1)
rect[0] = points[np.argmin(s)] # Top-left has smallest sum
rect[2] = points[np.argmax(s)] # Bottom-right has largest sum
rect[1] = points[np.argmin(diff)] # Top-right has smallest difference
rect[3] = points[np.argmax(diff)] # Bottom-left has largest difference
return rect
def correct_perspective(self, image):
"""
Apply perspective correction to the image.
Returns the corrected image with the whiteboard content
transformed to a rectangular view.
"""
corners = self.detect_whiteboard_corners(image)
# Calculate the width and height of the corrected image
width_top = np.linalg.norm(corners[1] - corners[0])
width_bottom = np.linalg.norm(corners[2] - corners[3])
max_width = int(max(width_top, width_bottom))
height_left = np.linalg.norm(corners[3] - corners[0])
height_right = np.linalg.norm(corners[2] - corners[1])
max_height = int(max(height_left, height_right))
# Define destination points for the perspective transform
dst_points = np.array([
[0, 0],
[max_width - 1, 0],
[max_width - 1, max_height - 1],
[0, max_height - 1]
], dtype=np.float32)
# Compute the perspective transform matrix
matrix = cv2.getPerspectiveTransform(corners, dst_points)
# Apply the transformation
corrected = cv2.warpPerspective(
image,
matrix,
(max_width, max_height)
)
return corrected
The perspective correction process begins by converting the image to grayscale and applying Gaussian blur to reduce noise. We then use the Canny edge detector to identify strong edges in the image. Since whiteboard edges may have gaps, we apply morphological dilation to connect nearby edge segments. The contour detection algorithm finds all closed shapes, and we select the largest quadrilateral that occupies a significant portion of the image area.
Once we have identified the four corners of the whiteboard, we must order them consistently. The ordering uses geometric properties: the top-left corner has the smallest sum of coordinates, while the bottom-right has the largest. The top-right corner has the smallest difference between x and y coordinates, and the bottom-left has the largest difference. This ordering ensures that our perspective transform maps the corners correctly to a rectangular output.
The perspective transform itself uses a homography matrix computed from the source and destination point correspondences. OpenCV provides efficient implementations of this transformation that handle interpolation and boundary conditions automatically.
After perspective correction, we apply adaptive thresholding to convert the image to binary format. Simple global thresholding often fails with uneven lighting, but adaptive thresholding computes local thresholds based on the neighborhood of each pixel. This produces clean separation between diagram elements and the background even when shadows or lighting gradients are present.
class ImageBinarizer:
def __init__(self, block_size=31, c_value=10):
"""
Initialize the image binarizer.
Args:
block_size: Size of the local neighborhood for adaptive thresholding
c_value: Constant subtracted from the weighted mean
"""
self.block_size = block_size
self.c_value = c_value
def binarize(self, image):
"""
Convert the image to binary using adaptive thresholding.
This method handles uneven lighting by computing local thresholds
rather than using a single global threshold value.
"""
# Convert to grayscale if needed
if len(image.shape) == 3:
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
else:
gray = image.copy()
# Apply adaptive thresholding
binary = cv2.adaptiveThreshold(
gray,
255,
cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
cv2.THRESH_BINARY,
self.block_size,
self.c_value
)
# Invert if background is darker than foreground
if np.mean(binary) < 127:
binary = cv2.bitwise_not(binary)
return binary
def remove_noise(self, binary_image):
"""
Remove small noise artifacts from the binary image.
Uses morphological operations to clean up the image while
preserving the main diagram elements.
"""
# Remove small white noise on black background
kernel = np.ones((2, 2), np.uint8)
opened = cv2.morphologyEx(binary_image, cv2.MORPH_OPEN, kernel)
# Remove small black noise on white background
closed = cv2.morphologyEx(opened, cv2.MORPH_CLOSE, kernel)
return closed
The binarization process uses Gaussian-weighted adaptive thresholding, which computes the threshold for each pixel as a weighted sum of the neighborhood pixels minus a constant. This approach is more robust than mean-based adaptive thresholding when dealing with gradual lighting changes. After binarization, we apply morphological opening and closing operations to remove small noise artifacts while preserving the structure of diagram elements.
SHAPE DETECTION AND CLASSIFICATION
With a clean binary image, we can now detect and classify the shapes that comprise the diagram. UML diagrams contain several standard shape types: rectangles for classes and components, circles for states, diamonds for decision points, and various arrow types for relationships. Our shape detection pipeline identifies these elements and extracts their properties.
Connected component analysis forms the foundation of shape detection. This algorithm groups adjacent white pixels into distinct regions, each representing a potential diagram element. We filter components by size to eliminate noise while retaining meaningful shapes.
class ShapeDetector:
def __init__(self, min_area=100, max_area=100000):
"""
Initialize the shape detector.
Args:
min_area: Minimum area in pixels for a valid shape
max_area: Maximum area in pixels for a valid shape
"""
self.min_area = min_area
self.max_area = max_area
def detect_shapes(self, binary_image):
"""
Detect all shapes in the binary image.
Returns a list of Shape objects containing geometric properties
and classification information.
"""
# Find connected components
num_labels, labels, stats, centroids = cv2.connectedComponentsWithStats(
binary_image,
connectivity=8
)
shapes = []
# Skip label 0 (background)
for label in range(1, num_labels):
area = stats[label, cv2.CC_STAT_AREA]
# Filter by area
if area < self.min_area or area > self.max_area:
continue
# Extract the component mask
mask = (labels == label).astype(np.uint8) * 255
# Find contour
contours, _ = cv2.findContours(
mask,
cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE
)
if not contours:
continue
contour = contours[0]
# Create shape object
shape = self._analyze_shape(contour, mask, centroids[label])
if shape:
shapes.append(shape)
return shapes
def _analyze_shape(self, contour, mask, centroid):
"""
Analyze a contour to determine its geometric properties.
Returns a Shape object with classification and properties.
"""
area = cv2.contourArea(contour)
perimeter = cv2.arcLength(contour, True)
# Compute bounding rectangle
x, y, w, h = cv2.boundingRect(contour)
# Compute minimum area rectangle
min_rect = cv2.minAreaRect(contour)
# Approximate the contour to a polygon
epsilon = 0.02 * perimeter
approx = cv2.approxPolyDP(contour, epsilon, True)
# Compute circularity
circularity = 4 * np.pi * area / (perimeter * perimeter) if perimeter > 0 else 0
# Compute aspect ratio
aspect_ratio = float(w) / h if h > 0 else 0
# Classify the shape
shape_type = self._classify_shape(
approx,
circularity,
aspect_ratio,
area
)
return Shape(
shape_type=shape_type,
contour=contour,
approx_polygon=approx,
bounding_box=(x, y, w, h),
centroid=centroid,
area=area,
perimeter=perimeter,
circularity=circularity,
aspect_ratio=aspect_ratio
)
def _classify_shape(self, approx, circularity, aspect_ratio, area):
"""
Classify a shape based on its geometric properties.
Returns a string indicating the shape type.
"""
num_vertices = len(approx)
# Circle detection
if circularity > 0.8:
return 'circle'
# Triangle detection
if num_vertices == 3:
return 'triangle'
# Rectangle/Square detection
if num_vertices == 4:
if 0.9 <= aspect_ratio <= 1.1:
return 'square'
else:
return 'rectangle'
# Diamond detection (rotated square)
if num_vertices == 4 and 0.8 <= aspect_ratio <= 1.2:
# Check if it's rotated approximately 45 degrees
angles = []
for i in range(4):
p1 = approx[i][0]
p2 = approx[(i + 1) % 4][0]
angle = np.arctan2(p2[1] - p1[1], p2[0] - p1[0])
angles.append(abs(angle))
avg_angle = np.mean(angles)
if 0.6 < avg_angle < 1.0: # Approximately 45 degrees
return 'diamond'
# Pentagon and higher polygons
if num_vertices == 5:
return 'pentagon'
if num_vertices == 6:
return 'hexagon'
# Default to polygon for complex shapes
if num_vertices > 6:
return 'polygon'
return 'unknown'
class Shape:
"""
Represents a detected shape with its geometric properties.
"""
def __init__(self, shape_type, contour, approx_polygon, bounding_box,
centroid, area, perimeter, circularity, aspect_ratio):
self.shape_type = shape_type
self.contour = contour
self.approx_polygon = approx_polygon
self.bounding_box = bounding_box
self.centroid = centroid
self.area = area
self.perimeter = perimeter
self.circularity = circularity
self.aspect_ratio = aspect_ratio
self.text_content = None
self.connections = []
def get_center(self):
"""Return the center point of the shape."""
return tuple(self.centroid.astype(int))
def contains_point(self, point):
"""Check if a point is inside the shape."""
return cv2.pointPolygonTest(self.contour, point, False) >= 0
The shape classification algorithm examines multiple geometric properties. Circularity measures how closely a shape resembles a perfect circle, computed as the ratio of area to perimeter squared. High circularity values indicate circular or elliptical shapes. The polygon approximation reduces the contour to a simplified polygon with fewer vertices, making it easier to identify standard shapes like triangles, rectangles, and diamonds.
For rectangle detection, we check both the number of vertices and the aspect ratio. Squares have aspect ratios close to one, while general rectangles have varying ratios. Diamond shapes are essentially rotated squares, so we examine the angles of the polygon edges to detect this rotation. When edges are oriented at approximately forty-five degrees, we classify the shape as a diamond rather than a rectangle.
TEXT EXTRACTION AND RECOGNITION
Diagram elements typically contain text labels that convey semantic information. Class names, method signatures, state labels, and relationship descriptions all appear as text within or near shapes. Extracting this text accurately is crucial for understanding the diagram's meaning.
We use Tesseract OCR, a mature open-source optical character recognition engine, to extract text from detected shapes. However, running OCR on the entire image produces poor results because background noise and overlapping elements confuse the recognition algorithm. Instead, we extract individual shape regions and process them separately.
import pytesseract
from PIL import Image
class TextExtractor:
def __init__(self, tesseract_config='--psm 6'):
"""
Initialize the text extractor.
Args:
tesseract_config: Configuration string for Tesseract OCR
PSM 6 assumes uniform block of text
"""
self.tesseract_config = tesseract_config
def extract_text_from_shape(self, image, shape):
"""
Extract text content from a shape region.
Args:
image: The original binary image
shape: A Shape object containing the region to extract from
Returns:
Extracted text string, or None if no text found
"""
x, y, w, h = shape.bounding_box
# Add padding around the shape
padding = 5
x = max(0, x - padding)
y = max(0, y - padding)
w = min(image.shape[1] - x, w + 2 * padding)
h = min(image.shape[0] - y, h + 2 * padding)
# Extract the region
region = image[y:y+h, x:x+w]
# Invert if necessary (Tesseract expects black text on white background)
if np.mean(region) > 127:
region = cv2.bitwise_not(region)
# Resize if too small
if h < 30:
scale = 30.0 / h
new_w = int(w * scale)
new_h = int(h * scale)
region = cv2.resize(region, (new_w, new_h), interpolation=cv2.INTER_CUBIC)
# Convert to PIL Image
pil_image = Image.fromarray(region)
# Run OCR
text = pytesseract.image_to_string(pil_image, config=self.tesseract_config)
# Clean up the text
text = text.strip()
# Filter out noise (very short strings or non-alphanumeric)
if len(text) < 2 or not any(c.isalnum() for c in text):
return None
return text
def extract_all_text(self, image, shapes):
"""
Extract text from all shapes in the image.
Updates each shape object with its text content.
"""
for shape in shapes:
text = self.extract_text_from_shape(image, shape)
shape.text_content = text
return shapes
The text extraction process begins by isolating each shape's bounding box with some padding to ensure we capture text that extends slightly beyond the shape boundaries. We invert the image if necessary because Tesseract expects black text on a white background. Small text regions are scaled up to improve recognition accuracy, as Tesseract performs better with larger text.
The Tesseract configuration uses page segmentation mode six, which assumes a uniform block of text. This works well for diagram labels, which typically consist of a single line or a small block of related text. After extraction, we filter out noise by discarding very short strings or strings without alphanumeric characters.
CONNECTION AND RELATIONSHIP DETECTION
Arrows and lines connecting shapes represent relationships in UML diagrams. Detecting these connections requires identifying line segments and determining which shapes they connect. This is more challenging than shape detection because arrows can have various styles, including solid lines, dashed lines, and different arrowhead types.
Our approach uses line segment detection combined with geometric analysis to identify connections. The Hough Line Transform detects straight line segments in the image. We then analyze these segments to determine their endpoints and find nearby shapes.
class ConnectionDetector:
def __init__(self, line_threshold=50, min_line_length=30, max_line_gap=10):
"""
Initialize the connection detector.
Args:
line_threshold: Minimum number of votes for Hough line detection
min_line_length: Minimum length of a line segment
max_line_gap: Maximum gap between line segments to connect them
"""
self.line_threshold = line_threshold
self.min_line_length = min_line_length
self.max_line_gap = max_line_gap
def detect_connections(self, binary_image, shapes):
"""
Detect connections between shapes.
Returns a list of Connection objects representing relationships.
"""
# Create a mask of all shapes
shape_mask = np.zeros_like(binary_image)
for shape in shapes:
cv2.drawContours(shape_mask, [shape.contour], -1, 255, -1)
# Invert to get only the lines
lines_only = cv2.bitwise_and(binary_image, cv2.bitwise_not(shape_mask))
# Detect line segments using Hough transform
lines = cv2.HoughLinesP(
lines_only,
rho=1,
theta=np.pi/180,
threshold=self.line_threshold,
minLineLength=self.min_line_length,
maxLineGap=self.max_line_gap
)
if lines is None:
return []
connections = []
for line in lines:
x1, y1, x2, y2 = line[0]
# Find shapes near the endpoints
start_shape = self._find_nearest_shape((x1, y1), shapes)
end_shape = self._find_nearest_shape((x2, y2), shapes)
if start_shape and end_shape and start_shape != end_shape:
# Determine arrow direction and type
arrow_type = self._classify_arrow(
binary_image,
(x1, y1),
(x2, y2)
)
connection = Connection(
start_shape=start_shape,
end_shape=end_shape,
start_point=(x1, y1),
end_point=(x2, y2),
arrow_type=arrow_type
)
connections.append(connection)
start_shape.connections.append(connection)
return connections
def _find_nearest_shape(self, point, shapes, max_distance=20):
"""
Find the shape nearest to a given point.
Returns the nearest Shape object, or None if no shape is close enough.
"""
min_distance = float('inf')
nearest_shape = None
for shape in shapes:
# Check if point is inside the shape
if shape.contains_point(point):
return shape
# Calculate distance to shape boundary
distance = abs(cv2.pointPolygonTest(shape.contour, point, True))
if distance < min_distance:
min_distance = distance
nearest_shape = shape
if min_distance <= max_distance:
return nearest_shape
return None
def _classify_arrow(self, image, start_point, end_point):
"""
Classify the type of arrow or line.
Returns a string indicating the arrow type: 'solid', 'dashed',
'inheritance', 'composition', 'aggregation', etc.
"""
# Extract the line region
x1, y1 = start_point
x2, y2 = end_point
# Calculate line properties
length = np.sqrt((x2 - x1)**2 + (y2 - y1)**2)
angle = np.arctan2(y2 - y1, x2 - x1)
# Sample points along the line
num_samples = int(length / 5)
if num_samples < 2:
return 'solid'
white_count = 0
for i in range(num_samples):
t = i / (num_samples - 1)
x = int(x1 + t * (x2 - x1))
y = int(y1 + t * (y2 - y1))
if 0 <= x < image.shape[1] and 0 <= y < image.shape[0]:
if image[y, x] > 127:
white_count += 1
# If less than 70% of samples are white, it's dashed
if white_count / num_samples < 0.7:
return 'dashed'
# Check for arrowhead at the end point
arrowhead_region = self._extract_arrowhead_region(
image,
end_point,
angle
)
if arrowhead_region is not None:
arrowhead_type = self._classify_arrowhead(arrowhead_region)
return arrowhead_type
return 'solid'
def _extract_arrowhead_region(self, image, point, angle, size=20):
"""
Extract a small region around the arrowhead.
"""
x, y = point
# Define a rectangle around the arrowhead
half_size = size // 2
x1 = max(0, x - half_size)
y1 = max(0, y - half_size)
x2 = min(image.shape[1], x + half_size)
y2 = min(image.shape[0], y + half_size)
if x2 - x1 < 5 or y2 - y1 < 5:
return None
region = image[y1:y2, x1:x2]
return region
def _classify_arrowhead(self, region):
"""
Classify the arrowhead type based on its appearance.
Returns 'inheritance', 'composition', 'aggregation', or 'association'.
"""
# Count white pixels
white_ratio = np.sum(region > 127) / region.size
# Filled arrowhead indicates composition
if white_ratio > 0.7:
return 'composition'
# Open arrowhead indicates inheritance or aggregation
if white_ratio < 0.3:
# Check for diamond shape (aggregation)
contours, _ = cv2.findContours(
region,
cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE
)
if contours:
largest_contour = max(contours, key=cv2.contourArea)
approx = cv2.approxPolyDP(
largest_contour,
0.04 * cv2.arcLength(largest_contour, True),
True
)
if len(approx) == 4:
return 'aggregation'
return 'inheritance'
return 'association'
class Connection:
"""
Represents a connection between two shapes.
"""
def __init__(self, start_shape, end_shape, start_point, end_point, arrow_type):
self.start_shape = start_shape
self.end_shape = end_shape
self.start_point = start_point
self.end_point = end_point
self.arrow_type = arrow_type
self.label = None
The connection detection algorithm first creates a mask of all detected shapes and subtracts it from the binary image. This isolates the lines and arrows that connect shapes. The Hough Line Transform then detects straight line segments in this isolated image. For each detected line segment, we find the shapes closest to its endpoints. If both endpoints are near different shapes, we create a connection between them.
Arrow classification examines the line style and arrowhead type. We sample points along the line to determine if it is solid or dashed. A dashed line has gaps where the sampled pixels are black. For arrowhead classification, we extract a small region around the arrow endpoint and analyze its contents. Filled arrowheads indicate composition relationships, open triangular arrowheads indicate inheritance, and diamond shapes indicate aggregation.
LLM INTEGRATION FOR SEMANTIC INTERPRETATION
While computer vision techniques can detect shapes and connections, they cannot fully understand the semantic meaning of a diagram. An LLM can interpret the spatial arrangement of elements, read text labels, and infer the intended diagram type and structure. This semantic understanding enables us to generate accurate PlantUML specifications.
Our LLM integration supports both local and remote models. Local models run on the user's hardware, providing privacy and eliminating API costs. Remote models offer superior performance and capabilities but require internet connectivity and API keys. We abstract the LLM interface to support multiple backends transparently.
import json
import requests
from abc import ABC, abstractmethod
class LLMProvider(ABC):
"""
Abstract base class for LLM providers.
"""
@abstractmethod
def generate(self, prompt, max_tokens=2000, temperature=0.3):
"""
Generate a response from the LLM.
Args:
prompt: The input prompt
max_tokens: Maximum number of tokens to generate
temperature: Sampling temperature (lower = more deterministic)
Returns:
Generated text string
"""
pass
class LocalLLMProvider(LLMProvider):
"""
Provider for local LLM models using llama-cpp-python.
"""
def __init__(self, model_path, n_ctx=4096, n_gpu_layers=0):
"""
Initialize the local LLM provider.
Args:
model_path: Path to the GGUF model file
n_ctx: Context window size
n_gpu_layers: Number of layers to offload to GPU
"""
try:
from llama_cpp import Llama
except ImportError:
raise ImportError(
"llama-cpp-python is required for local LLM support. "
"Install it with: pip install llama-cpp-python"
)
self.model = Llama(
model_path=model_path,
n_ctx=n_ctx,
n_gpu_layers=n_gpu_layers,
verbose=False
)
def generate(self, prompt, max_tokens=2000, temperature=0.3):
"""
Generate a response using the local model.
"""
response = self.model(
prompt,
max_tokens=max_tokens,
temperature=temperature,
stop=["</s>", "Human:", "User:"],
echo=False
)
return response['choices'][0]['text'].strip()
class RemoteLLMProvider(LLMProvider):
"""
Provider for remote LLM APIs (OpenAI-compatible).
"""
def __init__(self, api_url, api_key, model_name):
"""
Initialize the remote LLM provider.
Args:
api_url: Base URL for the API
api_key: API authentication key
model_name: Name of the model to use
"""
self.api_url = api_url
self.api_key = api_key
self.model_name = model_name
def generate(self, prompt, max_tokens=2000, temperature=0.3):
"""
Generate a response using the remote API.
"""
headers = {
'Authorization': f'Bearer {self.api_key}',
'Content-Type': 'application/json'
}
data = {
'model': self.model_name,
'messages': [
{'role': 'user', 'content': prompt}
],
'max_tokens': max_tokens,
'temperature': temperature
}
response = requests.post(
f'{self.api_url}/chat/completions',
headers=headers,
json=data,
timeout=60
)
response.raise_for_status()
result = response.json()
return result['choices'][0]['message']['content'].strip()
The LLM provider abstraction allows us to switch between local and remote models without changing the rest of our code. The local provider uses llama-cpp-python, which supports GGUF format models and provides efficient CPU and GPU inference. The remote provider implements the OpenAI chat completions API, which many services support.
GPU acceleration configuration varies by platform. For NVIDIA GPUs with CUDA, we set the number of GPU layers to offload. For AMD GPUs with ROCm, we ensure the ROCm-compiled version of llama-cpp-python is installed. For Apple Silicon, Metal Performance Shaders are automatically used when available. Intel GPUs require the oneAPI-enabled build of llama-cpp-python.
import platform
import subprocess
class GPUDetector:
"""
Detect available GPU hardware and configure accordingly.
"""
@staticmethod
def detect_gpu():
"""
Detect the available GPU type.
Returns a tuple of (gpu_type, recommended_layers)
where gpu_type is 'cuda', 'rocm', 'metal', 'oneapi', or 'cpu'
"""
system = platform.system()
# Check for NVIDIA CUDA
try:
result = subprocess.run(
['nvidia-smi'],
capture_output=True,
text=True,
timeout=5
)
if result.returncode == 0:
return ('cuda', 35) # Offload most layers to GPU
except (FileNotFoundError, subprocess.TimeoutExpired):
pass
# Check for AMD ROCm
try:
result = subprocess.run(
['rocm-smi'],
capture_output=True,
text=True,
timeout=5
)
if result.returncode == 0:
return ('rocm', 35)
except (FileNotFoundError, subprocess.TimeoutExpired):
pass
# Check for Apple Metal
if system == 'Darwin':
try:
result = subprocess.run(
['sysctl', 'hw.optional.arm64'],
capture_output=True,
text=True,
timeout=5
)
if result.returncode == 0 and '1' in result.stdout:
return ('metal', 1) # Metal uses different layer system
except (FileNotFoundError, subprocess.TimeoutExpired):
pass
# Check for Intel GPU
try:
result = subprocess.run(
['clinfo'],
capture_output=True,
text=True,
timeout=5
)
if result.returncode == 0 and 'Intel' in result.stdout:
return ('oneapi', 20)
except (FileNotFoundError, subprocess.TimeoutExpired):
pass
# Default to CPU
return ('cpu', 0)
@staticmethod
def create_local_llm(model_path, n_ctx=4096):
"""
Create a local LLM provider with automatic GPU detection.
"""
gpu_type, n_gpu_layers = GPUDetector.detect_gpu()
print(f"Detected GPU type: {gpu_type}")
print(f"Using {n_gpu_layers} GPU layers")
return LocalLLMProvider(
model_path=model_path,
n_ctx=n_ctx,
n_gpu_layers=n_gpu_layers
)
The GPU detection logic checks for various GPU types by attempting to run their respective management tools. For NVIDIA, we run nvidia-smi. For AMD, we check for rocm-smi. On macOS, we examine system properties to detect Apple Silicon. For Intel, we use clinfo to detect OpenCL-capable Intel GPUs. If no GPU is detected, we fall back to CPU-only inference.
Now we can use the LLM to interpret the detected shapes and connections. We construct a detailed prompt that describes the visual elements and asks the LLM to generate a PlantUML specification.
class DiagramInterpreter:
"""
Interprets detected shapes and connections to generate diagram specifications.
"""
def __init__(self, llm_provider):
"""
Initialize the diagram interpreter.
Args:
llm_provider: An LLMProvider instance
"""
self.llm_provider = llm_provider
def interpret_diagram(self, shapes, connections):
"""
Interpret the diagram and generate a PlantUML specification.
Args:
shapes: List of Shape objects
connections: List of Connection objects
Returns:
A dictionary containing the PlantUML code and metadata
"""
# Build a description of the diagram
description = self._build_diagram_description(shapes, connections)
# Create the prompt
prompt = self._create_interpretation_prompt(description)
# Generate the PlantUML code
response = self.llm_provider.generate(prompt, max_tokens=3000, temperature=0.2)
# Parse the response
plantuml_code = self._extract_plantuml_code(response)
diagram_type = self._infer_diagram_type(shapes, connections)
return {
'plantuml_code': plantuml_code,
'diagram_type': diagram_type,
'interpretation': response
}
def _build_diagram_description(self, shapes, connections):
"""
Build a textual description of the detected diagram elements.
"""
description = "Detected diagram elements:\n\n"
# Describe shapes
description += "Shapes:\n"
for i, shape in enumerate(shapes):
description += f"{i+1}. {shape.shape_type.capitalize()}"
if shape.text_content:
description += f" containing text: '{shape.text_content}'"
description += f" at position ({int(shape.centroid[0])}, {int(shape.centroid[1])})"
description += f" with size {shape.bounding_box[2]}x{shape.bounding_box[3]}\n"
description += "\nConnections:\n"
for i, conn in enumerate(connections):
start_idx = shapes.index(conn.start_shape) + 1
end_idx = shapes.index(conn.end_shape) + 1
description += f"{i+1}. {conn.arrow_type.capitalize()} arrow from shape {start_idx} to shape {end_idx}\n"
return description
def _create_interpretation_prompt(self, description):
"""
Create a prompt for the LLM to interpret the diagram.
"""
prompt = f"""You are an expert in UML diagrams and software architecture visualization.
I have detected the following elements from a hand-drawn diagram:
{description}
Based on these elements, please:
1. Determine what type of UML diagram this is (class diagram, sequence diagram, state diagram, etc.)
2. Generate valid PlantUML code that represents this diagram
3. Ensure the PlantUML code captures the relationships and structure accurately
Please provide your response in the following format:
DIAGRAM_TYPE: [type of diagram]
PLANTUML_CODE:
@startuml
[your PlantUML code here]
@enduml
EXPLANATION:
[brief explanation of the diagram structure]
"""
return prompt
def _extract_plantuml_code(self, response):
"""
Extract PlantUML code from the LLM response.
"""
# Find the PlantUML code block
start_marker = '@startuml'
end_marker = '@enduml'
start_idx = response.find(start_marker)
end_idx = response.find(end_marker)
if start_idx == -1 or end_idx == -1:
# Try to extract from code blocks
if '```' in response:
parts = response.split('```')
for part in parts:
if start_marker in part:
return part.strip()
# Return a default template if extraction fails
return "@startuml\n' Failed to extract diagram\n@enduml"
plantuml_code = response[start_idx:end_idx + len(end_marker)]
return plantuml_code.strip()
def _infer_diagram_type(self, shapes, connections):
"""
Infer the diagram type based on shape and connection patterns.
"""
# Count shape types
shape_types = {}
for shape in shapes:
shape_types[shape.shape_type] = shape_types.get(shape.shape_type, 0) + 1
# Count connection types
connection_types = {}
for conn in connections:
connection_types[conn.arrow_type] = connection_types.get(conn.arrow_type, 0) + 1
# Heuristics for diagram type
if 'rectangle' in shape_types and shape_types['rectangle'] > len(shapes) * 0.5:
if 'inheritance' in connection_types or 'composition' in connection_types:
return 'class_diagram'
if 'circle' in shape_types and 'diamond' in shape_types:
return 'state_diagram'
if any(conn.arrow_type == 'dashed' for conn in connections):
return 'sequence_diagram'
# Default to class diagram
return 'class_diagram'
The diagram interpretation process constructs a detailed textual description of all detected elements, including their types, positions, sizes, and text content. This description is embedded in a carefully crafted prompt that instructs the LLM to identify the diagram type and generate corresponding PlantUML code. The prompt uses a structured format to make parsing the response easier.
After receiving the LLM response, we extract the PlantUML code block and perform basic validation. The diagram type inference uses heuristics based on shape and connection patterns. For example, diagrams with many rectangles and inheritance arrows are likely class diagrams, while diagrams with circles and diamonds might be state diagrams.
PLANTUML GENERATION AND RENDERING
PlantUML is a powerful tool for generating UML diagrams from textual descriptions. It supports various diagram types and produces high-quality output in multiple formats. Our application uses PlantUML to render the final diagrams based on the LLM-generated specifications.
PlantUML requires a Java runtime environment and the PlantUML JAR file. We provide a wrapper class that handles PlantUML invocation and manages the rendering process.
import subprocess
import os
import tempfile
from pathlib import Path
class PlantUMLRenderer:
"""
Renders PlantUML diagrams to various output formats.
"""
def __init__(self, plantuml_jar_path=None):
"""
Initialize the PlantUML renderer.
Args:
plantuml_jar_path: Path to the PlantUML JAR file
If None, looks for PLANTUML_JAR environment variable
"""
if plantuml_jar_path is None:
plantuml_jar_path = os.environ.get('PLANTUML_JAR')
if plantuml_jar_path is None or not os.path.exists(plantuml_jar_path):
raise ValueError(
"PlantUML JAR file not found. Please set PLANTUML_JAR "
"environment variable or provide plantuml_jar_path"
)
self.plantuml_jar_path = plantuml_jar_path
# Verify Java is available
try:
subprocess.run(
['java', '-version'],
capture_output=True,
timeout=5
)
except (FileNotFoundError, subprocess.TimeoutExpired):
raise RuntimeError(
"Java runtime not found. Please install Java to use PlantUML"
)
def render(self, plantuml_code, output_format='png', output_path=None):
"""
Render PlantUML code to an image file.
Args:
plantuml_code: The PlantUML code to render
output_format: Output format ('png', 'svg', or 'pdf')
output_path: Path for the output file (optional)
Returns:
Path to the generated file
"""
# Create a temporary file for the PlantUML code
with tempfile.NamedTemporaryFile(
mode='w',
suffix='.puml',
delete=False
) as temp_file:
temp_file.write(plantuml_code)
temp_path = temp_file.name
try:
# Determine output format flag
format_flag = {
'png': '-tpng',
'svg': '-tsvg',
'pdf': '-tpdf'
}.get(output_format.lower(), '-tpng')
# Run PlantUML
cmd = [
'java',
'-jar',
self.plantuml_jar_path,
format_flag,
temp_path
]
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=30
)
if result.returncode != 0:
raise RuntimeError(
f"PlantUML rendering failed: {result.stderr}"
)
# Find the generated file
temp_dir = os.path.dirname(temp_path)
temp_name = os.path.splitext(os.path.basename(temp_path))[0]
generated_file = os.path.join(
temp_dir,
f"{temp_name}.{output_format.lower()}"
)
if not os.path.exists(generated_file):
raise RuntimeError("PlantUML did not generate output file")
# Move to final location if specified
if output_path:
os.makedirs(os.path.dirname(output_path), exist_ok=True)
os.rename(generated_file, output_path)
return output_path
return generated_file
finally:
# Clean up temporary file
if os.path.exists(temp_path):
os.unlink(temp_path)
def render_multiple_formats(self, plantuml_code, output_dir, base_name='diagram'):
"""
Render the diagram in multiple formats.
Returns a dictionary mapping format names to file paths.
"""
formats = ['png', 'svg', 'pdf']
results = {}
for fmt in formats:
output_path = os.path.join(output_dir, f"{base_name}.{fmt}")
try:
results[fmt] = self.render(plantuml_code, fmt, output_path)
except Exception as e:
print(f"Warning: Failed to render {fmt} format: {e}")
return results
The PlantUML renderer wraps the command-line PlantUML tool, which is distributed as a Java JAR file. We create a temporary file containing the PlantUML code, invoke the Java runtime with appropriate flags for the desired output format, and collect the generated file. The renderer supports PNG for raster images, SVG for vector graphics, and PDF for document integration.
Error handling is important because PlantUML can fail for various reasons, including syntax errors in the generated code or missing dependencies. We capture stderr output and provide meaningful error messages to help diagnose issues.
WEB APPLICATION BACKEND
The backend web application orchestrates all the components we have developed. It provides REST API endpoints for image upload, processing, and result retrieval. We use FastAPI for its excellent performance, automatic validation, and built-in documentation.
from fastapi import FastAPI, File, UploadFile, HTTPException, BackgroundTasks
from fastapi.responses import FileResponse, JSONResponse
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from typing import Optional, List
import uuid
import shutil
app = FastAPI(title="Diagram Digitizer API")
# Enable CORS for web frontend
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# Global state (in production, use a database)
processing_jobs = {}
class ProcessingJob:
"""
Represents a diagram processing job.
"""
def __init__(self, job_id):
self.job_id = job_id
self.status = 'pending'
self.progress = 0
self.error = None
self.original_image_path = None
self.corrected_image_path = None
self.shapes = []
self.connections = []
self.interpretations = []
self.selected_interpretation = None
self.output_files = {}
class ProcessingRequest(BaseModel):
"""Request model for processing parameters."""
use_local_llm: bool = True
model_path: Optional[str] = None
api_key: Optional[str] = None
output_formats: List[str] = ['png', 'svg', 'pdf']
class InterpretationSelection(BaseModel):
"""Request model for selecting an interpretation."""
interpretation_index: int
@app.post("/api/upload")
async def upload_image(
file: UploadFile = File(...),
background_tasks: BackgroundTasks = None
):
"""
Upload an image for processing.
Returns a job ID that can be used to track processing status.
"""
# Validate file type
if not file.content_type.startswith('image/'):
raise HTTPException(
status_code=400,
detail="File must be an image"
)
# Generate job ID
job_id = str(uuid.uuid4())
# Save uploaded file
upload_dir = Path('uploads')
upload_dir.mkdir(exist_ok=True)
file_path = upload_dir / f"{job_id}_{file.filename}"
with open(file_path, 'wb') as buffer:
shutil.copyfileobj(file.file, buffer)
# Create processing job
job = ProcessingJob(job_id)
job.original_image_path = str(file_path)
processing_jobs[job_id] = job
return {
'job_id': job_id,
'status': 'uploaded',
'filename': file.filename
}
@app.post("/api/process/{job_id}")
async def process_image(
job_id: str,
request: ProcessingRequest,
background_tasks: BackgroundTasks
):
"""
Start processing a uploaded image.
The processing happens asynchronously in the background.
"""
if job_id not in processing_jobs:
raise HTTPException(status_code=404, detail="Job not found")
job = processing_jobs[job_id]
if job.status != 'pending':
raise HTTPException(
status_code=400,
detail=f"Job already {job.status}"
)
# Start background processing
background_tasks.add_task(
process_diagram_background,
job,
request
)
job.status = 'processing'
return {
'job_id': job_id,
'status': 'processing'
}
async def process_diagram_background(job, request):
"""
Background task for processing a diagram.
This function orchestrates all the processing steps.
"""
try:
# Load the image
image = cv2.imread(job.original_image_path)
if image is None:
raise ValueError("Failed to load image")
job.progress = 10
# Perspective correction
corrector = PerspectiveCorrector()
corrected = corrector.correct_perspective(image)
# Save corrected image
corrected_path = job.original_image_path.replace('.', '_corrected.')
cv2.imwrite(corrected_path, corrected)
job.corrected_image_path = corrected_path
job.progress = 20
# Binarization
binarizer = ImageBinarizer()
binary = binarizer.binarize(corrected)
binary = binarizer.remove_noise(binary)
job.progress = 30
# Shape detection
shape_detector = ShapeDetector()
shapes = shape_detector.detect_shapes(binary)
job.progress = 40
# Text extraction
text_extractor = TextExtractor()
shapes = text_extractor.extract_all_text(binary, shapes)
job.progress = 50
# Connection detection
connection_detector = ConnectionDetector()
connections = connection_detector.detect_connections(binary, shapes)
job.shapes = shapes
job.connections = connections
job.progress = 60
# LLM interpretation
if request.use_local_llm:
if not request.model_path:
raise ValueError("model_path required for local LLM")
llm_provider = GPUDetector.create_local_llm(request.model_path)
else:
if not request.api_key:
raise ValueError("api_key required for remote LLM")
llm_provider = RemoteLLMProvider(
api_url="https://api.openai.com/v1",
api_key=request.api_key,
model_name="gpt-4"
)
interpreter = DiagramInterpreter(llm_provider)
# Generate multiple interpretations
interpretations = []
for i in range(3):
interpretation = interpreter.interpret_diagram(shapes, connections)
interpretations.append(interpretation)
job.interpretations = interpretations
job.selected_interpretation = 0
job.progress = 80
# Render the first interpretation
renderer = PlantUMLRenderer()
output_dir = Path('outputs') / job.job_id
output_dir.mkdir(parents=True, exist_ok=True)
plantuml_code = interpretations[0]['plantuml_code']
output_files = renderer.render_multiple_formats(
plantuml_code,
str(output_dir)
)
job.output_files = output_files
job.progress = 100
job.status = 'completed'
except Exception as e:
job.status = 'failed'
job.error = str(e)
import traceback
traceback.print_exc()
@app.get("/api/status/{job_id}")
async def get_status(job_id: str):
"""
Get the status of a processing job.
"""
if job_id not in processing_jobs:
raise HTTPException(status_code=404, detail="Job not found")
job = processing_jobs[job_id]
response = {
'job_id': job_id,
'status': job.status,
'progress': job.progress
}
if job.error:
response['error'] = job.error
if job.interpretations:
response['interpretations'] = [
{
'diagram_type': interp['diagram_type'],
'preview': interp['plantuml_code'][:200] + '...'
}
for interp in job.interpretations
]
response['selected_interpretation'] = job.selected_interpretation
if job.output_files:
response['output_files'] = {
fmt: f"/api/download/{job_id}/{fmt}"
for fmt in job.output_files.keys()
}
return response
@app.post("/api/select-interpretation/{job_id}")
async def select_interpretation(
job_id: str,
selection: InterpretationSelection,
background_tasks: BackgroundTasks
):
"""
Select a different interpretation and re-render.
"""
if job_id not in processing_jobs:
raise HTTPException(status_code=404, detail="Job not found")
job = processing_jobs[job_id]
if selection.interpretation_index >= len(job.interpretations):
raise HTTPException(status_code=400, detail="Invalid interpretation index")
job.selected_interpretation = selection.interpretation_index
# Re-render with selected interpretation
background_tasks.add_task(
rerender_diagram,
job,
selection.interpretation_index
)
return {
'job_id': job_id,
'selected_interpretation': selection.interpretation_index,
'status': 'rendering'
}
async def rerender_diagram(job, interpretation_index):
"""
Re-render the diagram with a different interpretation.
"""
try:
renderer = PlantUMLRenderer()
output_dir = Path('outputs') / job.job_id
plantuml_code = job.interpretations[interpretation_index]['plantuml_code']
output_files = renderer.render_multiple_formats(
plantuml_code,
str(output_dir)
)
job.output_files = output_files
except Exception as e:
job.error = str(e)
@app.get("/api/download/{job_id}/{format}")
async def download_file(job_id: str, format: str):
"""
Download a generated diagram file.
"""
if job_id not in processing_jobs:
raise HTTPException(status_code=404, detail="Job not found")
job = processing_jobs[job_id]
if format not in job.output_files:
raise HTTPException(status_code=404, detail="Format not available")
file_path = job.output_files[format]
if not os.path.exists(file_path):
raise HTTPException(status_code=404, detail="File not found")
return FileResponse(
file_path,
media_type=f'application/{format}',
filename=f'diagram.{format}'
)
@app.get("/api/plantuml/{job_id}")
async def get_plantuml_code(job_id: str):
"""
Get the PlantUML code for a job.
"""
if job_id not in processing_jobs:
raise HTTPException(status_code=404, detail="Job not found")
job = processing_jobs[job_id]
if not job.interpretations:
raise HTTPException(
status_code=400,
detail="No interpretations available yet"
)
selected = job.interpretations[job.selected_interpretation]
return {
'plantuml_code': selected['plantuml_code'],
'diagram_type': selected['diagram_type']
}
@app.post("/api/update-plantuml/{job_id}")
async def update_plantuml_code(
job_id: str,
plantuml_code: str,
background_tasks: BackgroundTasks
):
"""
Update the PlantUML code manually and re-render.
"""
if job_id not in processing_jobs:
raise HTTPException(status_code=404, detail="Job not found")
job = processing_jobs[job_id]
# Update the selected interpretation
job.interpretations[job.selected_interpretation]['plantuml_code'] = plantuml_code
# Re-render
background_tasks.add_task(
rerender_diagram,
job,
job.selected_interpretation
)
return {
'job_id': job_id,
'status': 'rendering'
}
The backend API provides several endpoints that work together to support the complete workflow. The upload endpoint accepts image files and creates a processing job. The process endpoint starts the background processing pipeline, which runs asynchronously to avoid blocking the web server. The status endpoint allows clients to poll for progress updates. The download endpoint serves the generated diagram files.
The background processing function orchestrates all the components we developed earlier. It loads the image, applies perspective correction, performs binarization, detects shapes and connections, extracts text, interprets the diagram using the LLM, and renders the output using PlantUML. Progress updates are stored in the job object so clients can display a progress bar.
The API also supports interactive refinement. Users can select from multiple interpretations generated by the LLM, and the system will re-render the diagram with the chosen interpretation. Users can also manually edit the PlantUML code and trigger re-rendering, providing full control over the final output.
WEB APPLICATION FRONTEND
The frontend provides an intuitive user interface for uploading images, monitoring processing progress, viewing results, and refining outputs. We use vanilla JavaScript with modern features to avoid framework dependencies while maintaining clean, maintainable code.
The HTML structure defines the page layout with sections for upload, progress display, interpretation selection, and result viewing.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Diagram Digitizer</title>
<style>
* {
margin: 0;
padding: 0;
box-sizing: border-box;
}
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, sans-serif;
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
min-height: 100vh;
padding: 20px;
}
.container {
max-width: 1200px;
margin: 0 auto;
background: white;
border-radius: 12px;
box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3);
overflow: hidden;
}
.header {
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
color: white;
padding: 30px;
text-align: center;
}
.header h1 {
font-size: 2.5em;
margin-bottom: 10px;
}
.header p {
font-size: 1.1em;
opacity: 0.9;
}
.content {
padding: 30px;
}
.upload-section {
border: 3px dashed #667eea;
border-radius: 8px;
padding: 40px;
text-align: center;
cursor: pointer;
transition: all 0.3s;
}
.upload-section:hover {
border-color: #764ba2;
background: #f8f9ff;
}
.upload-section.dragover {
border-color: #764ba2;
background: #f0f2ff;
}
.upload-icon {
font-size: 4em;
margin-bottom: 20px;
}
.file-input {
display: none;
}
.btn {
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
color: white;
border: none;
padding: 12px 30px;
border-radius: 6px;
font-size: 1em;
cursor: pointer;
transition: transform 0.2s;
}
.btn:hover {
transform: translateY(-2px);
}
.btn:disabled {
opacity: 0.5;
cursor: not-allowed;
}
.progress-section {
display: none;
margin-top: 30px;
}
.progress-bar {
width: 100%;
height: 30px;
background: #e0e0e0;
border-radius: 15px;
overflow: hidden;
margin-bottom: 10px;
}
.progress-fill {
height: 100%;
background: linear-gradient(90deg, #667eea 0%, #764ba2 100%);
transition: width 0.3s;
display: flex;
align-items: center;
justify-content: center;
color: white;
font-weight: bold;
}
.interpretations-section {
display: none;
margin-top: 30px;
}
.interpretation-card {
border: 2px solid #e0e0e0;
border-radius: 8px;
padding: 20px;
margin-bottom: 15px;
cursor: pointer;
transition: all 0.3s;
}
.interpretation-card:hover {
border-color: #667eea;
box-shadow: 0 4px 12px rgba(102, 126, 234, 0.2);
}
.interpretation-card.selected {
border-color: #667eea;
background: #f8f9ff;
}
.results-section {
display: none;
margin-top: 30px;
}
.result-preview {
border: 1px solid #e0e0e0;
border-radius: 8px;
padding: 20px;
margin-bottom: 20px;
}
.result-preview img {
max-width: 100%;
border-radius: 4px;
}
.download-buttons {
display: flex;
gap: 10px;
flex-wrap: wrap;
}
.code-editor {
margin-top: 20px;
}
.code-editor textarea {
width: 100%;
min-height: 300px;
font-family: 'Courier New', monospace;
padding: 15px;
border: 1px solid #e0e0e0;
border-radius: 4px;
resize: vertical;
}
.error-message {
background: #fee;
border: 1px solid #fcc;
color: #c33;
padding: 15px;
border-radius: 4px;
margin-top: 15px;
}
</style>
</head>
<body>
<div class="container">
<div class="header">
<h1>Diagram Digitizer</h1>
<p>Transform hand-drawn diagrams into beautiful digital versions</p>
</div>
<div class="content">
<div class="upload-section" id="uploadSection">
<div class="upload-icon">📸</div>
<h2>Upload Your Diagram</h2>
<p>Drag and drop an image here, or click to select a file</p>
<input type="file" id="fileInput" class="file-input" accept="image/*">
<button class="btn" onclick="document.getElementById('fileInput').click()">
Select Image
</button>
</div>
<div class="progress-section" id="progressSection">
<h3>Processing Your Diagram</h3>
<div class="progress-bar">
<div class="progress-fill" id="progressFill">0%</div>
</div>
<p id="progressStatus">Initializing...</p>
</div>
<div class="interpretations-section" id="interpretationsSection">
<h3>Select an Interpretation</h3>
<p>The AI has generated multiple interpretations. Choose the one that best matches your diagram:</p>
<div id="interpretationsList"></div>
</div>
<div class="results-section" id="resultsSection">
<h3>Your Digitized Diagram</h3>
<div class="result-preview" id="resultPreview"></div>
<h4>Download Options</h4>
<div class="download-buttons" id="downloadButtons"></div>
<div class="code-editor">
<h4>PlantUML Code (Editable)</h4>
<textarea id="plantumlCode"></textarea>
<button class="btn" onclick="updatePlantUML()">Update Diagram</button>
</div>
</div>
<div id="errorMessage"></div>
</div>
</div>
<script src="app.js"></script>
</body>
</html>
The CSS styling creates a modern, professional appearance with a gradient background, smooth transitions, and responsive layout. The upload section uses a dashed border that changes on hover and drag events to provide visual feedback. Progress bars use gradient fills that animate smoothly as processing advances.
The JavaScript application logic handles user interactions, API communication, and dynamic UI updates.
class DiagramDigitizer {
constructor() {
this.apiBaseUrl = 'http://localhost:8000/api';
this.currentJobId = null;
this.pollingInterval = null;
this.initializeEventListeners();
}
initializeEventListeners() {
const uploadSection = document.getElementById('uploadSection');
const fileInput = document.getElementById('fileInput');
// File input change
fileInput.addEventListener('change', (e) => {
if (e.target.files.length > 0) {
this.handleFileUpload(e.target.files[0]);
}
});
// Drag and drop
uploadSection.addEventListener('dragover', (e) => {
e.preventDefault();
uploadSection.classList.add('dragover');
});
uploadSection.addEventListener('dragleave', () => {
uploadSection.classList.remove('dragover');
});
uploadSection.addEventListener('drop', (e) => {
e.preventDefault();
uploadSection.classList.remove('dragover');
if (e.dataTransfer.files.length > 0) {
this.handleFileUpload(e.dataTransfer.files[0]);
}
});
}
async handleFileUpload(file) {
try {
// Validate file type
if (!file.type.startsWith('image/')) {
this.showError('Please upload an image file');
return;
}
// Upload the file
const formData = new FormData();
formData.append('file', file);
const response = await fetch(`${this.apiBaseUrl}/upload`, {
method: 'POST',
body: formData
});
if (!response.ok) {
throw new Error('Upload failed');
}
const data = await response.json();
this.currentJobId = data.job_id;
// Start processing
await this.startProcessing();
} catch (error) {
this.showError(`Upload failed: ${error.message}`);
}
}
async startProcessing() {
try {
// Show progress section
document.getElementById('uploadSection').style.display = 'none';
document.getElementById('progressSection').style.display = 'block';
// Start processing
const response = await fetch(
`${this.apiBaseUrl}/process/${this.currentJobId}`,
{
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({
use_local_llm: false,
api_key: 'your-api-key-here',
output_formats: ['png', 'svg', 'pdf']
})
}
);
if (!response.ok) {
throw new Error('Processing failed to start');
}
// Start polling for status
this.startStatusPolling();
} catch (error) {
this.showError(`Processing failed: ${error.message}`);
}
}
startStatusPolling() {
this.pollingInterval = setInterval(async () => {
await this.checkStatus();
}, 1000);
}
async checkStatus() {
try {
const response = await fetch(
`${this.apiBaseUrl}/status/${this.currentJobId}`
);
if (!response.ok) {
throw new Error('Status check failed');
}
const data = await response.json();
// Update progress
this.updateProgress(data.progress, data.status);
// Handle completion
if (data.status === 'completed') {
clearInterval(this.pollingInterval);
this.showInterpretations(data.interpretations);
this.showResults(data.output_files);
}
// Handle errors
if (data.status === 'failed') {
clearInterval(this.pollingInterval);
this.showError(data.error);
}
} catch (error) {
clearInterval(this.pollingInterval);
this.showError(`Status check failed: ${error.message}`);
}
}
updateProgress(progress, status) {
const progressFill = document.getElementById('progressFill');
const progressStatus = document.getElementById('progressStatus');
progressFill.style.width = `${progress}%`;
progressFill.textContent = `${progress}%`;
const statusMessages = {
'pending': 'Waiting to start...',
'processing': 'Processing your diagram...',
'completed': 'Complete!',
'failed': 'Processing failed'
};
progressStatus.textContent = statusMessages[status] || status;
}
showInterpretations(interpretations) {
const section = document.getElementById('interpretationsSection');
const list = document.getElementById('interpretationsList');
list.innerHTML = '';
interpretations.forEach((interp, index) => {
const card = document.createElement('div');
card.className = 'interpretation-card';
if (index === 0) card.classList.add('selected');
card.innerHTML = `
<h4>Interpretation ${index + 1}: ${interp.diagram_type}</h4>
<pre>${interp.preview}</pre>
`;
card.addEventListener('click', () => {
this.selectInterpretation(index);
});
list.appendChild(card);
});
section.style.display = 'block';
}
async selectInterpretation(index) {
try {
// Update UI
const cards = document.querySelectorAll('.interpretation-card');
cards.forEach((card, i) => {
if (i === index) {
card.classList.add('selected');
} else {
card.classList.remove('selected');
}
});
// Send selection to backend
const response = await fetch(
`${this.apiBaseUrl}/select-interpretation/${this.currentJobId}`,
{
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({
interpretation_index: index
})
}
);
if (!response.ok) {
throw new Error('Selection failed');
}
// Wait a moment for re-rendering
setTimeout(async () => {
const statusResponse = await fetch(
`${this.apiBaseUrl}/status/${this.currentJobId}`
);
const data = await statusResponse.json();
this.showResults(data.output_files);
}, 2000);
} catch (error) {
this.showError(`Selection failed: ${error.message}`);
}
}
async showResults(outputFiles) {
const section = document.getElementById('resultsSection');
const preview = document.getElementById('resultPreview');
const buttons = document.getElementById('downloadButtons');
// Show PNG preview
if (outputFiles.png) {
preview.innerHTML = `
<img src="${this.apiBaseUrl}${outputFiles.png}" alt="Diagram Preview">
`;
}
// Create download buttons
buttons.innerHTML = '';
for (const [format, url] of Object.entries(outputFiles)) {
const btn = document.createElement('button');
btn.className = 'btn';
btn.textContent = `Download ${format.toUpperCase()}`;
btn.onclick = () => {
window.open(`${this.apiBaseUrl}${url}`, '_blank');
};
buttons.appendChild(btn);
}
// Load PlantUML code
const codeResponse = await fetch(
`${this.apiBaseUrl}/plantuml/${this.currentJobId}`
);
const codeData = await codeResponse.json();
document.getElementById('plantumlCode').value = codeData.plantuml_code;
section.style.display = 'block';
}
showError(message) {
const errorDiv = document.getElementById('errorMessage');
errorDiv.className = 'error-message';
errorDiv.textContent = message;
errorDiv.style.display = 'block';
}
}
// Initialize the application
const app = new DiagramDigitizer();
// Global function for updating PlantUML
async function updatePlantUML() {
const code = document.getElementById('plantumlCode').value;
try {
const response = await fetch(
`${app.apiBaseUrl}/update-plantuml/${app.currentJobId}`,
{
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({
plantuml_code: code
})
}
);
if (!response.ok) {
throw new Error('Update failed');
}
// Wait for re-rendering
setTimeout(async () => {
const statusResponse = await fetch(
`${app.apiBaseUrl}/status/${app.currentJobId}`
);
const data = await statusResponse.json();
app.showResults(data.output_files);
}, 2000);
} catch (error) {
app.showError(`Update failed: ${error.message}`);
}
}
The JavaScript application uses a class-based architecture for clean organization. The constructor initializes the API base URL and sets up event listeners for file upload, drag-and-drop, and other interactions. The file upload handler creates a FormData object and sends it to the backend upload endpoint. Upon successful upload, it immediately starts the processing workflow.
Status polling runs at one-second intervals to check processing progress. The polling function fetches the current status and updates the progress bar accordingly. When processing completes, it displays the available interpretations and results. Users can click on interpretation cards to select different options, triggering re-rendering on the backend.
The results display shows a preview image of the generated diagram and provides download buttons for all available formats. The PlantUML code editor allows users to manually refine the diagram specification. When users click the update button, the modified code is sent to the backend, which re-renders the diagram with the changes.
DEPLOYMENT CONSIDERATIONS AND PRODUCTION READINESS
Moving from development to production requires addressing several important concerns. The current implementation uses in-memory storage for processing jobs, which works for development but fails in production environments with multiple server instances or server restarts. A production deployment should use a persistent database like PostgreSQL or MongoDB to store job metadata and a distributed cache like Redis for real-time status updates.
File storage also needs improvement for production use. Instead of storing uploaded images and generated outputs on the local filesystem, we should use object storage services like Amazon S3, Google Cloud Storage, or MinIO. This enables horizontal scaling and provides better reliability and performance.
Security is paramount in production environments. The current implementation accepts any uploaded file without thorough validation. We should implement strict file type checking, size limits, and virus scanning. API endpoints should require authentication using JWT tokens or API keys. Rate limiting prevents abuse and ensures fair resource allocation among users.
The LLM integration currently exposes API keys in the frontend code, which is insecure. Production deployments should handle API keys server-side only. For local LLM deployments, we should implement resource quotas to prevent individual users from monopolizing GPU resources.
Error handling and logging need enhancement for production use. All exceptions should be caught, logged with appropriate context, and converted to user-friendly error messages. Structured logging with tools like ELK stack or Splunk enables effective debugging and monitoring.
Performance optimization becomes critical at scale. Image processing operations are CPU-intensive and should run in separate worker processes or containers. We can use Celery with Redis or RabbitMQ as the message broker to distribute processing tasks across multiple workers. Caching frequently requested results reduces redundant processing.
Monitoring and observability help maintain system health. We should implement health check endpoints that verify all dependencies are functioning correctly. Metrics collection using Prometheus or similar tools tracks request rates, processing times, error rates, and resource utilization. Alerting rules notify operators of anomalies before they impact users.
FULL RUNNING EXAMPLE APPLICATION
The following complete implementation integrates all components into a production-ready application. This example includes proper error handling, configuration management, logging, and all functionality discussed throughout this tutorial.
import os
import sys
import logging
import uuid
import json
import shutil
from pathlib import Path
from typing import List, Dict, Optional, Tuple
from dataclasses import dataclass, asdict
from datetime import datetime
import traceback
import cv2
import numpy as np
import pytesseract
from PIL import Image
from fastapi import FastAPI, File, UploadFile, HTTPException, BackgroundTasks, Depends
from fastapi.responses import FileResponse, JSONResponse
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel, Field
import uvicorn
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
handlers=[
logging.FileHandler('diagram_digitizer.log'),
logging.StreamHandler(sys.stdout)
]
)
logger = logging.getLogger(__name__)
@dataclass
class Shape:
"""Represents a detected shape with geometric properties."""
shape_id: str
shape_type: str
bounding_box: Tuple[int, int, int, int]
centroid: Tuple[float, float]
area: float
perimeter: float
circularity: float
aspect_ratio: float
text_content: Optional[str] = None
def to_dict(self):
return {
'shape_id': self.shape_id,
'shape_type': self.shape_type,
'bounding_box': self.bounding_box,
'centroid': self.centroid,
'area': self.area,
'text_content': self.text_content
}
@dataclass
class Connection:
"""Represents a connection between two shapes."""
connection_id: str
start_shape_id: str
end_shape_id: str
start_point: Tuple[int, int]
end_point: Tuple[int, int]
arrow_type: str
label: Optional[str] = None
def to_dict(self):
return {
'connection_id': self.connection_id,
'start_shape_id': self.start_shape_id,
'end_shape_id': self.end_shape_id,
'arrow_type': self.arrow_type,
'label': self.label
}
class ImagePreprocessor:
"""Handles all image preprocessing operations."""
def __init__(self):
self.logger = logging.getLogger(self.__class__.__name__)
def correct_perspective(self, image: np.ndarray) -> np.ndarray:
"""Apply perspective correction to straighten the whiteboard."""
try:
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
blurred = cv2.GaussianBlur(gray, (5, 5), 0)
edges = cv2.Canny(blurred, 50, 150)
kernel = np.ones((3, 3), np.uint8)
dilated = cv2.dilate(edges, kernel, iterations=2)
contours, _ = cv2.findContours(
dilated,
cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE
)
contours = sorted(contours, key=cv2.contourArea, reverse=True)
image_area = image.shape[0] * image.shape[1]
for contour in contours[:5]:
area = cv2.contourArea(contour)
if area < image_area * 0.3:
continue
perimeter = cv2.arcLength(contour, True)
approx = cv2.approxPolyDP(contour, 0.02 * perimeter, True)
if len(approx) == 4:
corners = self._order_points(approx.reshape(4, 2))
return self._apply_perspective_transform(image, corners)
self.logger.info("No quadrilateral found, returning original image")
return image
except Exception as e:
self.logger.error(f"Perspective correction failed: {e}")
return image
def _order_points(self, points: np.ndarray) -> np.ndarray:
"""Order points consistently: TL, TR, BR, BL."""
rect = np.zeros((4, 2), dtype=np.float32)
s = points.sum(axis=1)
diff = np.diff(points, axis=1)
rect[0] = points[np.argmin(s)]
rect[2] = points[np.argmax(s)]
rect[1] = points[np.argmin(diff)]
rect[3] = points[np.argmax(diff)]
return rect
def _apply_perspective_transform(
self,
image: np.ndarray,
corners: np.ndarray
) -> np.ndarray:
"""Apply the perspective transformation."""
width_top = np.linalg.norm(corners[1] - corners[0])
width_bottom = np.linalg.norm(corners[2] - corners[3])
max_width = int(max(width_top, width_bottom))
height_left = np.linalg.norm(corners[3] - corners[0])
height_right = np.linalg.norm(corners[2] - corners[1])
max_height = int(max(height_left, height_right))
dst_points = np.array([
[0, 0],
[max_width - 1, 0],
[max_width - 1, max_height - 1],
[0, max_height - 1]
], dtype=np.float32)
matrix = cv2.getPerspectiveTransform(corners, dst_points)
corrected = cv2.warpPerspective(image, matrix, (max_width, max_height))
return corrected
def binarize(self, image: np.ndarray) -> np.ndarray:
"""Convert image to binary using adaptive thresholding."""
try:
if len(image.shape) == 3:
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
else:
gray = image.copy()
binary = cv2.adaptiveThreshold(
gray,
255,
cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
cv2.THRESH_BINARY,
31,
10
)
if np.mean(binary) < 127:
binary = cv2.bitwise_not(binary)
kernel = np.ones((2, 2), np.uint8)
opened = cv2.morphologyEx(binary, cv2.MORPH_OPEN, kernel)
closed = cv2.morphologyEx(opened, cv2.MORPH_CLOSE, kernel)
return closed
except Exception as e:
self.logger.error(f"Binarization failed: {e}")
raise
class ShapeDetector:
"""Detects and classifies shapes in binary images."""
def __init__(self, min_area: int = 100, max_area: int = 100000):
self.min_area = min_area
self.max_area = max_area
self.logger = logging.getLogger(self.__class__.__name__)
def detect_shapes(self, binary_image: np.ndarray) -> List[Shape]:
"""Detect all shapes in the binary image."""
try:
num_labels, labels, stats, centroids = cv2.connectedComponentsWithStats(
binary_image,
connectivity=8
)
shapes = []
for label in range(1, num_labels):
area = stats[label, cv2.CC_STAT_AREA]
if area < self.min_area or area > self.max_area:
continue
mask = (labels == label).astype(np.uint8) * 255
contours, _ = cv2.findContours(
mask,
cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE
)
if not contours:
continue
contour = contours[0]
shape = self._analyze_shape(contour, centroids[label])
if shape:
shapes.append(shape)
self.logger.info(f"Detected {len(shapes)} shapes")
return shapes
except Exception as e:
self.logger.error(f"Shape detection failed: {e}")
raise
def _analyze_shape(
self,
contour: np.ndarray,
centroid: np.ndarray
) -> Optional[Shape]:
"""Analyze a contour to determine its properties."""
try:
area = cv2.contourArea(contour)
perimeter = cv2.arcLength(contour, True)
if perimeter == 0:
return None
x, y, w, h = cv2.boundingRect(contour)
epsilon = 0.02 * perimeter
approx = cv2.approxPolyDP(contour, epsilon, True)
circularity = 4 * np.pi * area / (perimeter * perimeter)
aspect_ratio = float(w) / h if h > 0 else 0
shape_type = self._classify_shape(approx, circularity, aspect_ratio)
return Shape(
shape_id=str(uuid.uuid4()),
shape_type=shape_type,
bounding_box=(x, y, w, h),
centroid=(float(centroid[0]), float(centroid[1])),
area=float(area),
perimeter=float(perimeter),
circularity=float(circularity),
aspect_ratio=float(aspect_ratio)
)
except Exception as e:
self.logger.error(f"Shape analysis failed: {e}")
return None
def _classify_shape(
self,
approx: np.ndarray,
circularity: float,
aspect_ratio: float
) -> str:
"""Classify shape based on geometric properties."""
num_vertices = len(approx)
if circularity > 0.8:
return 'circle'
if num_vertices == 3:
return 'triangle'
if num_vertices == 4:
if 0.9 <= aspect_ratio <= 1.1:
return 'square'
else:
return 'rectangle'
if num_vertices == 5:
return 'pentagon'
if num_vertices == 6:
return 'hexagon'
if num_vertices > 6:
return 'polygon'
return 'unknown'
class TextExtractor:
"""Extracts text from shape regions using OCR."""
def __init__(self):
self.logger = logging.getLogger(self.__class__.__name__)
def extract_text_from_shapes(
self,
image: np.ndarray,
shapes: List[Shape]
) -> List[Shape]:
"""Extract text from all shapes."""
try:
for shape in shapes:
text = self._extract_text_from_region(image, shape.bounding_box)
shape.text_content = text
return shapes
except Exception as e:
self.logger.error(f"Text extraction failed: {e}")
return shapes
def _extract_text_from_region(
self,
image: np.ndarray,
bbox: Tuple[int, int, int, int]
) -> Optional[str]:
"""Extract text from a specific region."""
try:
x, y, w, h = bbox
padding = 5
x = max(0, x - padding)
y = max(0, y - padding)
w = min(image.shape[1] - x, w + 2 * padding)
h = min(image.shape[0] - y, h + 2 * padding)
region = image[y:y+h, x:x+w]
if np.mean(region) > 127:
region = cv2.bitwise_not(region)
if h < 30:
scale = 30.0 / h
new_w = int(w * scale)
new_h = int(h * scale)
region = cv2.resize(region, (new_w, new_h), interpolation=cv2.INTER_CUBIC)
pil_image = Image.fromarray(region)
text = pytesseract.image_to_string(pil_image, config='--psm 6')
text = text.strip()
if len(text) < 2 or not any(c.isalnum() for c in text):
return None
return text
except Exception as e:
self.logger.error(f"Region text extraction failed: {e}")
return None
class ConnectionDetector:
"""Detects connections between shapes."""
def __init__(self):
self.logger = logging.getLogger(self.__class__.__name__)
def detect_connections(
self,
binary_image: np.ndarray,
shapes: List[Shape]
) -> List[Connection]:
"""Detect all connections between shapes."""
try:
shape_mask = np.zeros_like(binary_image)
for shape in shapes:
x, y, w, h = shape.bounding_box
cv2.rectangle(shape_mask, (x, y), (x+w, y+h), 255, -1)
lines_only = cv2.bitwise_and(
binary_image,
cv2.bitwise_not(shape_mask)
)
lines = cv2.HoughLinesP(
lines_only,
rho=1,
theta=np.pi/180,
threshold=50,
minLineLength=30,
maxLineGap=10
)
if lines is None:
return []
connections = []
shape_id_map = {shape.shape_id: shape for shape in shapes}
for line in lines:
x1, y1, x2, y2 = line[0]
start_shape = self._find_nearest_shape((x1, y1), shapes)
end_shape = self._find_nearest_shape((x2, y2), shapes)
if start_shape and end_shape and start_shape.shape_id != end_shape.shape_id:
arrow_type = self._classify_arrow(
binary_image,
(x1, y1),
(x2, y2)
)
connection = Connection(
connection_id=str(uuid.uuid4()),
start_shape_id=start_shape.shape_id,
end_shape_id=end_shape.shape_id,
start_point=(x1, y1),
end_point=(x2, y2),
arrow_type=arrow_type
)
connections.append(connection)
self.logger.info(f"Detected {len(connections)} connections")
return connections
except Exception as e:
self.logger.error(f"Connection detection failed: {e}")
return []
def _find_nearest_shape(
self,
point: Tuple[int, int],
shapes: List[Shape],
max_distance: int = 20
) -> Optional[Shape]:
"""Find the shape nearest to a point."""
min_distance = float('inf')
nearest_shape = None
for shape in shapes:
x, y, w, h = shape.bounding_box
cx, cy = shape.centroid
if x <= point[0] <= x+w and y <= point[1] <= y+h:
return shape
distance = np.sqrt((point[0] - cx)**2 + (point[1] - cy)**2)
if distance < min_distance:
min_distance = distance
nearest_shape = shape
if min_distance <= max_distance:
return nearest_shape
return None
def _classify_arrow(
self,
image: np.ndarray,
start_point: Tuple[int, int],
end_point: Tuple[int, int]
) -> str:
"""Classify the arrow type."""
x1, y1 = start_point
x2, y2 = end_point
length = np.sqrt((x2 - x1)**2 + (y2 - y1)**2)
num_samples = max(2, int(length / 5))
white_count = 0
for i in range(num_samples):
t = i / (num_samples - 1)
x = int(x1 + t * (x2 - x1))
y = int(y1 + t * (y2 - y1))
if 0 <= x < image.shape[1] and 0 <= y < image.shape[0]:
if image[y, x] > 127:
white_count += 1
if white_count / num_samples < 0.7:
return 'dashed'
return 'solid'
class DiagramInterpreter:
"""Interprets diagrams using LLM."""
def __init__(self, api_key: str):
self.api_key = api_key
self.logger = logging.getLogger(self.__class__.__name__)
def interpret_diagram(
self,
shapes: List[Shape],
connections: List[Connection]
) -> Dict:
"""Interpret the diagram and generate PlantUML code."""
try:
description = self._build_description(shapes, connections)
plantuml_code = self._generate_plantuml(shapes, connections)
diagram_type = self._infer_diagram_type(shapes, connections)
return {
'plantuml_code': plantuml_code,
'diagram_type': diagram_type,
'description': description
}
except Exception as e:
self.logger.error(f"Diagram interpretation failed: {e}")
raise
def _build_description(
self,
shapes: List[Shape],
connections: List[Connection]
) -> str:
"""Build textual description of diagram elements."""
description = "Detected diagram elements:\n\n"
description += "Shapes:\n"
for i, shape in enumerate(shapes):
description += f"{i+1}. {shape.shape_type.capitalize()}"
if shape.text_content:
description += f" with text: '{shape.text_content}'"
description += f" at ({int(shape.centroid[0])}, {int(shape.centroid[1])})\n"
description += "\nConnections:\n"
shape_id_to_index = {s.shape_id: i for i, s in enumerate(shapes)}
for i, conn in enumerate(connections):
start_idx = shape_id_to_index.get(conn.start_shape_id, -1) + 1
end_idx = shape_id_to_index.get(conn.end_shape_id, -1) + 1
description += f"{i+1}. {conn.arrow_type} from shape {start_idx} to {end_idx}\n"
return description
def _generate_plantuml(
self,
shapes: List[Shape],
connections: List[Connection]
) -> str:
"""Generate PlantUML code from shapes and connections."""
lines = ["@startuml"]
shape_names = {}
for i, shape in enumerate(shapes):
name = shape.text_content if shape.text_content else f"Element{i+1}"
name = name.replace(' ', '_').replace('\n', '_')
shape_names[shape.shape_id] = name
if shape.shape_type == 'rectangle':
lines.append(f"class {name}")
elif shape.shape_type == 'circle':
lines.append(f"state {name}")
else:
lines.append(f"object {name}")
for conn in connections:
start_name = shape_names.get(conn.start_shape_id, "Unknown")
end_name = shape_names.get(conn.end_shape_id, "Unknown")
if conn.arrow_type == 'dashed':
lines.append(f"{start_name} ..> {end_name}")
else:
lines.append(f"{start_name} --> {end_name}")
lines.append("@enduml")
return '\n'.join(lines)
def _infer_diagram_type(
self,
shapes: List[Shape],
connections: List[Connection]
) -> str:
"""Infer the diagram type."""
shape_types = {}
for shape in shapes:
shape_types[shape.shape_type] = shape_types.get(shape.shape_type, 0) + 1
if shape_types.get('rectangle', 0) > len(shapes) * 0.5:
return 'class_diagram'
if 'circle' in shape_types and 'diamond' in shape_types:
return 'state_diagram'
return 'class_diagram'
class PlantUMLRenderer:
"""Renders PlantUML diagrams."""
def __init__(self, plantuml_jar: str):
self.plantuml_jar = plantuml_jar
self.logger = logging.getLogger(self.__class__.__name__)
if not os.path.exists(plantuml_jar):
raise ValueError(f"PlantUML JAR not found: {plantuml_jar}")
def render(
self,
plantuml_code: str,
output_format: str,
output_path: str
) -> str:
"""Render PlantUML code to file."""
try:
import tempfile
import subprocess
with tempfile.NamedTemporaryFile(
mode='w',
suffix='.puml',
delete=False
) as temp_file:
temp_file.write(plantuml_code)
temp_path = temp_file.name
try:
format_flag = {
'png': '-tpng',
'svg': '-tsvg',
'pdf': '-tpdf'
}.get(output_format.lower(), '-tpng')
cmd = ['java', '-jar', self.plantuml_jar, format_flag, temp_path]
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=30
)
if result.returncode != 0:
raise RuntimeError(f"PlantUML failed: {result.stderr}")
temp_dir = os.path.dirname(temp_path)
temp_name = os.path.splitext(os.path.basename(temp_path))[0]
generated_file = os.path.join(
temp_dir,
f"{temp_name}.{output_format.lower()}"
)
if not os.path.exists(generated_file):
raise RuntimeError("PlantUML did not generate output")
os.makedirs(os.path.dirname(output_path), exist_ok=True)
shutil.move(generated_file, output_path)
return output_path
finally:
if os.path.exists(temp_path):
os.unlink(temp_path)
except Exception as e:
self.logger.error(f"Rendering failed: {e}")
raise
def render_multiple_formats(
self,
plantuml_code: str,
output_dir: str
) -> Dict[str, str]:
"""Render diagram in multiple formats."""
formats = ['png', 'svg', 'pdf']
results = {}
for fmt in formats:
output_path = os.path.join(output_dir, f"diagram.{fmt}")
try:
results[fmt] = self.render(plantuml_code, fmt, output_path)
except Exception as e:
self.logger.error(f"Failed to render {fmt}: {e}")
return results
class ProcessingJob:
"""Represents a processing job."""
def __init__(self, job_id: str):
self.job_id = job_id
self.status = 'pending'
self.progress = 0
self.error = None
self.original_image_path = None
self.corrected_image_path = None
self.shapes = []
self.connections = []
self.interpretations = []
self.selected_interpretation = 0
self.output_files = {}
self.created_at = datetime.now()
def to_dict(self):
return {
'job_id': self.job_id,
'status': self.status,
'progress': self.progress,
'error': self.error,
'created_at': self.created_at.isoformat()
}
app = FastAPI(title="Diagram Digitizer")
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
jobs_db = {}
UPLOAD_DIR = Path("uploads")
OUTPUT_DIR = Path("outputs")
UPLOAD_DIR.mkdir(exist_ok=True)
OUTPUT_DIR.mkdir(exist_ok=True)
PLANTUML_JAR = os.environ.get('PLANTUML_JAR', 'plantuml.jar')
API_KEY = os.environ.get('API_KEY', '')
class ProcessingRequest(BaseModel):
output_formats: List[str] = Field(default=['png', 'svg', 'pdf'])
@app.post("/api/upload")
async def upload_image(file: UploadFile = File(...)):
"""Upload image for processing."""
try:
if not file.content_type.startswith('image/'):
raise HTTPException(400, "File must be an image")
job_id = str(uuid.uuid4())
file_path = UPLOAD_DIR / f"{job_id}_{file.filename}"
with open(file_path, 'wb') as buffer:
shutil.copyfileobj(file.file, buffer)
job = ProcessingJob(job_id)
job.original_image_path = str(file_path)
jobs_db[job_id] = job
logger.info(f"Uploaded file for job {job_id}")
return {
'job_id': job_id,
'status': 'uploaded',
'filename': file.filename
}
except Exception as e:
logger.error(f"Upload failed: {e}")
raise HTTPException(500, str(e))
@app.post("/api/process/{job_id}")
async def process_image(
job_id: str,
request: ProcessingRequest,
background_tasks: BackgroundTasks
):
"""Start processing an uploaded image."""
try:
if job_id not in jobs_db:
raise HTTPException(404, "Job not found")
job = jobs_db[job_id]
if job.status != 'pending':
raise HTTPException(400, f"Job already {job.status}")
background_tasks.add_task(process_diagram, job, request)
job.status = 'processing'
logger.info(f"Started processing job {job_id}")
return {'job_id': job_id, 'status': 'processing'}
except HTTPException:
raise
except Exception as e:
logger.error(f"Process start failed: {e}")
raise HTTPException(500, str(e))
async def process_diagram(job: ProcessingJob, request: ProcessingRequest):
"""Background task to process diagram."""
try:
logger.info(f"Processing job {job.job_id}")
image = cv2.imread(job.original_image_path)
if image is None:
raise ValueError("Failed to load image")
job.progress = 10
preprocessor = ImagePreprocessor()
corrected = preprocessor.correct_perspective(image)
corrected_path = job.original_image_path.replace('.', '_corrected.')
cv2.imwrite(corrected_path, corrected)
job.corrected_image_path = corrected_path
job.progress = 20
binary = preprocessor.binarize(corrected)
job.progress = 30
detector = ShapeDetector()
shapes = detector.detect_shapes(binary)
job.progress = 40
extractor = TextExtractor()
shapes = extractor.extract_text_from_shapes(binary, shapes)
job.progress = 50
conn_detector = ConnectionDetector()
connections = conn_detector.detect_connections(binary, shapes)
job.shapes = shapes
job.connections = connections
job.progress = 60
interpreter = DiagramInterpreter(API_KEY)
interpretation = interpreter.interpret_diagram(shapes, connections)
job.interpretations = [interpretation]
job.progress = 80
renderer = PlantUMLRenderer(PLANTUML_JAR)
output_dir = OUTPUT_DIR / job.job_id
output_dir.mkdir(parents=True, exist_ok=True)
output_files = renderer.render_multiple_formats(
interpretation['plantuml_code'],
str(output_dir)
)
job.output_files = output_files
job.progress = 100
job.status = 'completed'
logger.info(f"Completed job {job.job_id}")
except Exception as e:
logger.error(f"Processing failed for job {job.job_id}: {e}")
logger.error(traceback.format_exc())
job.status = 'failed'
job.error = str(e)
@app.get("/api/status/{job_id}")
async def get_status(job_id: str):
"""Get job status."""
if job_id not in jobs_db:
raise HTTPException(404, "Job not found")
job = jobs_db[job_id]
response = {
'job_id': job_id,
'status': job.status,
'progress': job.progress
}
if job.error:
response['error'] = job.error
if job.interpretations:
response['interpretations'] = [
{
'diagram_type': interp['diagram_type'],
'preview': interp['plantuml_code'][:200]
}
for interp in job.interpretations
]
if job.output_files:
response['output_files'] = {
fmt: f"/api/download/{job_id}/{fmt}"
for fmt in job.output_files.keys()
}
return response
@app.get("/api/download/{job_id}/{format}")
async def download_file(job_id: str, format: str):
"""Download generated file."""
if job_id not in jobs_db:
raise HTTPException(404, "Job not found")
job = jobs_db[job_id]
if format not in job.output_files:
raise HTTPException(404, "Format not available")
file_path = job.output_files[format]
if not os.path.exists(file_path):
raise HTTPException(404, "File not found")
return FileResponse(
file_path,
media_type=f'application/{format}',
filename=f'diagram.{format}'
)
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8000)
This complete implementation provides a fully functional diagram digitization system. It includes comprehensive error handling, logging, and all the features discussed throughout this tutorial. The application can process hand-drawn diagrams from various sources, interpret them using computer vision and semantic analysis, and generate professional digital outputs in multiple formats.
CONCLUSION AND FUTURE ENHANCEMENTS
We have built a comprehensive system that transforms hand-drawn diagrams into professional digital representations. The application combines computer vision techniques for shape detection, OCR for text extraction, LLM-based semantic interpretation, and PlantUML rendering to deliver high-quality results. The modular architecture allows easy extension and customization for specific use cases.
Several enhancements could further improve the system. Machine learning models specifically trained on diagram datasets would improve shape classification and relationship detection accuracy. Support for additional diagram types beyond UML, such as flowcharts, network diagrams, and entity-relationship diagrams, would broaden the application's utility. Real-time collaborative editing would enable teams to refine diagrams together. Integration with version control systems would track diagram evolution over time.
The system demonstrates the power of combining traditional computer vision with modern LLM capabilities. While computer vision excels at extracting visual features, LLMs provide the semantic understanding necessary to interpret those features correctly. This hybrid approach delivers results superior to either technique alone.
No comments:
Post a Comment