Saturday, November 21, 2026

BUILDING AN INTELLIGENT WHITEBOARD-TO-DIGITAL DIAGRAM CONVERTER

 



INTRODUCTION AND PROBLEM STATEMENT

Software developers, architects, and designers frequently sketch diagrams on whiteboards, paper, or digital notebooks during brainstorming sessions and design meetings. These hand-drawn artifacts capture valuable insights about system architecture, class relationships, and component interactions. However, converting these sketches into professional, shareable digital diagrams traditionally requires manual redrawing using specialized tools, which is time-consuming and error-prone.

This tutorial presents a comprehensive solution: a web-based application that leverages Large Language Models and computer vision to automatically convert photographs of hand-drawn UML diagrams, system sketches, and CRC cards into beautified digital representations. The application analyzes uploaded images, interprets the diagrammatic elements, and generates clean, professional outputs in formats like PDF, PNG, and SVG using PlantUML and similar rendering engines.

The core challenge involves multiple technical domains. First, we must extract visual information from photographs that may contain varying lighting conditions, perspectives, and drawing styles. Second, we need to interpret the semantic meaning of shapes, arrows, text labels, and spatial relationships. Third, we must translate this understanding into formal diagram specifications that rendering tools can process. Finally, we provide an interactive refinement interface where users can guide the system toward their intended interpretation.

ARCHITECTURAL OVERVIEW AND TECHNOLOGY STACK

Our application follows a modular architecture with clear separation of concerns. The frontend provides an intuitive interface for image upload, preview, and interaction. The backend orchestrates the processing pipeline, coordinating between computer vision components, LLM-based interpretation, and diagram generation services. We prioritize open-source technologies to ensure accessibility and customizability.

The technology stack includes FastAPI for the backend web framework, which offers excellent performance and automatic API documentation. For the frontend, we use vanilla JavaScript with modern ES6+ features, avoiding heavy framework dependencies while maintaining clean, maintainable code. The computer vision pipeline combines OpenCV for image preprocessing with specialized object detection models. The LLM integration supports both local models through libraries like llama-cpp-python and remote APIs, ensuring flexibility across deployment scenarios.

GPU acceleration presents unique challenges across different hardware platforms. NVIDIA CUDA remains the most mature ecosystem, but we must also support AMD ROCm for AMD GPUs, Apple Metal Performance Shaders for M-series chips, and Intel oneAPI for Intel discrete graphics. Our implementation uses abstraction layers that detect available hardware and configure the appropriate backends automatically.

IMAGE PREPROCESSING AND FEATURE EXTRACTION

Before we can interpret diagram semantics, we must prepare the input image through a series of preprocessing steps. Photographs of whiteboards often suffer from perspective distortion, uneven lighting, shadows, and background noise. Our preprocessing pipeline addresses these issues systematically.

The first stage performs perspective correction. When users photograph a whiteboard at an angle, rectangular shapes appear as trapezoids. We detect the whiteboard boundaries using edge detection and Hough transforms, then apply a homography transformation to correct the perspective. This ensures that spatial relationships in the diagram are preserved accurately.

Here is a Python implementation of the perspective correction module:

import cv2
import numpy as np

class PerspectiveCorrector:
    def __init__(self, min_area_ratio=0.3):
        """
        Initialize the perspective corrector.
        
        Args:
            min_area_ratio: Minimum ratio of detected quadrilateral area 
                           to image area to consider it a valid whiteboard
        """
        self.min_area_ratio = min_area_ratio
    
    def detect_whiteboard_corners(self, image):
        """
        Detect the four corners of the whiteboard in the image.
        
        This method uses edge detection and contour analysis to find
        the largest quadrilateral in the image, which we assume is
        the whiteboard boundary.
        """
        gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
        blurred = cv2.GaussianBlur(gray, (5, 5), 0)
        edges = cv2.Canny(blurred, 50, 150)
        
        # Dilate edges to close gaps
        kernel = np.ones((3, 3), np.uint8)
        dilated = cv2.dilate(edges, kernel, iterations=2)
        
        # Find contours
        contours, _ = cv2.findContours(
            dilated, 
            cv2.RETR_EXTERNAL, 
            cv2.CHAIN_APPROX_SIMPLE
        )
        
        # Sort contours by area and find the largest quadrilateral
        contours = sorted(contours, key=cv2.contourArea, reverse=True)
        
        image_area = image.shape[0] * image.shape[1]
        
        for contour in contours:
            area = cv2.contourArea(contour)
            if area < image_area * self.min_area_ratio:
                continue
            
            # Approximate the contour to a polygon
            perimeter = cv2.arcLength(contour, True)
            approx = cv2.approxPolyDP(contour, 0.02 * perimeter, True)
            
            # Check if we have a quadrilateral
            if len(approx) == 4:
                return self._order_points(approx.reshape(4, 2))
        
        # If no quadrilateral found, return image corners
        h, w = image.shape[:2]
        return np.array([
            [0, 0],
            [w - 1, 0],
            [w - 1, h - 1],
            [0, h - 1]
        ], dtype=np.float32)
    
    def _order_points(self, points):
        """
        Order points in consistent order: top-left, top-right, 
        bottom-right, bottom-left.
        """
        rect = np.zeros((4, 2), dtype=np.float32)
        
        # Sum and difference to find corners
        s = points.sum(axis=1)
        diff = np.diff(points, axis=1)
        
        rect[0] = points[np.argmin(s)]      # Top-left has smallest sum
        rect[2] = points[np.argmax(s)]      # Bottom-right has largest sum
        rect[1] = points[np.argmin(diff)]   # Top-right has smallest difference
        rect[3] = points[np.argmax(diff)]   # Bottom-left has largest difference
        
        return rect
    
    def correct_perspective(self, image):
        """
        Apply perspective correction to the image.
        
        Returns the corrected image with the whiteboard content
        transformed to a rectangular view.
        """
        corners = self.detect_whiteboard_corners(image)
        
        # Calculate the width and height of the corrected image
        width_top = np.linalg.norm(corners[1] - corners[0])
        width_bottom = np.linalg.norm(corners[2] - corners[3])
        max_width = int(max(width_top, width_bottom))
        
        height_left = np.linalg.norm(corners[3] - corners[0])
        height_right = np.linalg.norm(corners[2] - corners[1])
        max_height = int(max(height_left, height_right))
        
        # Define destination points for the perspective transform
        dst_points = np.array([
            [0, 0],
            [max_width - 1, 0],
            [max_width - 1, max_height - 1],
            [0, max_height - 1]
        ], dtype=np.float32)
        
        # Compute the perspective transform matrix
        matrix = cv2.getPerspectiveTransform(corners, dst_points)
        
        # Apply the transformation
        corrected = cv2.warpPerspective(
            image, 
            matrix, 
            (max_width, max_height)
        )
        
        return corrected

The perspective correction process begins by converting the image to grayscale and applying Gaussian blur to reduce noise. We then use the Canny edge detector to identify strong edges in the image. Since whiteboard edges may have gaps, we apply morphological dilation to connect nearby edge segments. The contour detection algorithm finds all closed shapes, and we select the largest quadrilateral that occupies a significant portion of the image area.

Once we have identified the four corners of the whiteboard, we must order them consistently. The ordering uses geometric properties: the top-left corner has the smallest sum of coordinates, while the bottom-right has the largest. The top-right corner has the smallest difference between x and y coordinates, and the bottom-left has the largest difference. This ordering ensures that our perspective transform maps the corners correctly to a rectangular output.

The perspective transform itself uses a homography matrix computed from the source and destination point correspondences. OpenCV provides efficient implementations of this transformation that handle interpolation and boundary conditions automatically.

After perspective correction, we apply adaptive thresholding to convert the image to binary format. Simple global thresholding often fails with uneven lighting, but adaptive thresholding computes local thresholds based on the neighborhood of each pixel. This produces clean separation between diagram elements and the background even when shadows or lighting gradients are present.

class ImageBinarizer:
    def __init__(self, block_size=31, c_value=10):
        """
        Initialize the image binarizer.
        
        Args:
            block_size: Size of the local neighborhood for adaptive thresholding
            c_value: Constant subtracted from the weighted mean
        """
        self.block_size = block_size
        self.c_value = c_value
    
    def binarize(self, image):
        """
        Convert the image to binary using adaptive thresholding.
        
        This method handles uneven lighting by computing local thresholds
        rather than using a single global threshold value.
        """
        # Convert to grayscale if needed
        if len(image.shape) == 3:
            gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
        else:
            gray = image.copy()
        
        # Apply adaptive thresholding
        binary = cv2.adaptiveThreshold(
            gray,
            255,
            cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
            cv2.THRESH_BINARY,
            self.block_size,
            self.c_value
        )
        
        # Invert if background is darker than foreground
        if np.mean(binary) < 127:
            binary = cv2.bitwise_not(binary)
        
        return binary
    
    def remove_noise(self, binary_image):
        """
        Remove small noise artifacts from the binary image.
        
        Uses morphological operations to clean up the image while
        preserving the main diagram elements.
        """
        # Remove small white noise on black background
        kernel = np.ones((2, 2), np.uint8)
        opened = cv2.morphologyEx(binary_image, cv2.MORPH_OPEN, kernel)
        
        # Remove small black noise on white background
        closed = cv2.morphologyEx(opened, cv2.MORPH_CLOSE, kernel)
        
        return closed

The binarization process uses Gaussian-weighted adaptive thresholding, which computes the threshold for each pixel as a weighted sum of the neighborhood pixels minus a constant. This approach is more robust than mean-based adaptive thresholding when dealing with gradual lighting changes. After binarization, we apply morphological opening and closing operations to remove small noise artifacts while preserving the structure of diagram elements.

SHAPE DETECTION AND CLASSIFICATION

With a clean binary image, we can now detect and classify the shapes that comprise the diagram. UML diagrams contain several standard shape types: rectangles for classes and components, circles for states, diamonds for decision points, and various arrow types for relationships. Our shape detection pipeline identifies these elements and extracts their properties.

Connected component analysis forms the foundation of shape detection. This algorithm groups adjacent white pixels into distinct regions, each representing a potential diagram element. We filter components by size to eliminate noise while retaining meaningful shapes.

class ShapeDetector:
    def __init__(self, min_area=100, max_area=100000):
        """
        Initialize the shape detector.
        
        Args:
            min_area: Minimum area in pixels for a valid shape
            max_area: Maximum area in pixels for a valid shape
        """
        self.min_area = min_area
        self.max_area = max_area
    
    def detect_shapes(self, binary_image):
        """
        Detect all shapes in the binary image.
        
        Returns a list of Shape objects containing geometric properties
        and classification information.
        """
        # Find connected components
        num_labels, labels, stats, centroids = cv2.connectedComponentsWithStats(
            binary_image,
            connectivity=8
        )
        
        shapes = []
        
        # Skip label 0 (background)
        for label in range(1, num_labels):
            area = stats[label, cv2.CC_STAT_AREA]
            
            # Filter by area
            if area < self.min_area or area > self.max_area:
                continue
            
            # Extract the component mask
            mask = (labels == label).astype(np.uint8) * 255
            
            # Find contour
            contours, _ = cv2.findContours(
                mask,
                cv2.RETR_EXTERNAL,
                cv2.CHAIN_APPROX_SIMPLE
            )
            
            if not contours:
                continue
            
            contour = contours[0]
            
            # Create shape object
            shape = self._analyze_shape(contour, mask, centroids[label])
            if shape:
                shapes.append(shape)
        
        return shapes
    
    def _analyze_shape(self, contour, mask, centroid):
        """
        Analyze a contour to determine its geometric properties.
        
        Returns a Shape object with classification and properties.
        """
        area = cv2.contourArea(contour)
        perimeter = cv2.arcLength(contour, True)
        
        # Compute bounding rectangle
        x, y, w, h = cv2.boundingRect(contour)
        
        # Compute minimum area rectangle
        min_rect = cv2.minAreaRect(contour)
        
        # Approximate the contour to a polygon
        epsilon = 0.02 * perimeter
        approx = cv2.approxPolyDP(contour, epsilon, True)
        
        # Compute circularity
        circularity = 4 * np.pi * area / (perimeter * perimeter) if perimeter > 0 else 0
        
        # Compute aspect ratio
        aspect_ratio = float(w) / h if h > 0 else 0
        
        # Classify the shape
        shape_type = self._classify_shape(
            approx,
            circularity,
            aspect_ratio,
            area
        )
        
        return Shape(
            shape_type=shape_type,
            contour=contour,
            approx_polygon=approx,
            bounding_box=(x, y, w, h),
            centroid=centroid,
            area=area,
            perimeter=perimeter,
            circularity=circularity,
            aspect_ratio=aspect_ratio
        )
    
    def _classify_shape(self, approx, circularity, aspect_ratio, area):
        """
        Classify a shape based on its geometric properties.
        
        Returns a string indicating the shape type.
        """
        num_vertices = len(approx)
        
        # Circle detection
        if circularity > 0.8:
            return 'circle'
        
        # Triangle detection
        if num_vertices == 3:
            return 'triangle'
        
        # Rectangle/Square detection
        if num_vertices == 4:
            if 0.9 <= aspect_ratio <= 1.1:
                return 'square'
            else:
                return 'rectangle'
        
        # Diamond detection (rotated square)
        if num_vertices == 4 and 0.8 <= aspect_ratio <= 1.2:
            # Check if it's rotated approximately 45 degrees
            angles = []
            for i in range(4):
                p1 = approx[i][0]
                p2 = approx[(i + 1) % 4][0]
                angle = np.arctan2(p2[1] - p1[1], p2[0] - p1[0])
                angles.append(abs(angle))
            
            avg_angle = np.mean(angles)
            if 0.6 < avg_angle < 1.0:  # Approximately 45 degrees
                return 'diamond'
        
        # Pentagon and higher polygons
        if num_vertices == 5:
            return 'pentagon'
        
        if num_vertices == 6:
            return 'hexagon'
        
        # Default to polygon for complex shapes
        if num_vertices > 6:
            return 'polygon'
        
        return 'unknown'


class Shape:
    """
    Represents a detected shape with its geometric properties.
    """
    def __init__(self, shape_type, contour, approx_polygon, bounding_box,
                 centroid, area, perimeter, circularity, aspect_ratio):
        self.shape_type = shape_type
        self.contour = contour
        self.approx_polygon = approx_polygon
        self.bounding_box = bounding_box
        self.centroid = centroid
        self.area = area
        self.perimeter = perimeter
        self.circularity = circularity
        self.aspect_ratio = aspect_ratio
        self.text_content = None
        self.connections = []
    
    def get_center(self):
        """Return the center point of the shape."""
        return tuple(self.centroid.astype(int))
    
    def contains_point(self, point):
        """Check if a point is inside the shape."""
        return cv2.pointPolygonTest(self.contour, point, False) >= 0

The shape classification algorithm examines multiple geometric properties. Circularity measures how closely a shape resembles a perfect circle, computed as the ratio of area to perimeter squared. High circularity values indicate circular or elliptical shapes. The polygon approximation reduces the contour to a simplified polygon with fewer vertices, making it easier to identify standard shapes like triangles, rectangles, and diamonds.

For rectangle detection, we check both the number of vertices and the aspect ratio. Squares have aspect ratios close to one, while general rectangles have varying ratios. Diamond shapes are essentially rotated squares, so we examine the angles of the polygon edges to detect this rotation. When edges are oriented at approximately forty-five degrees, we classify the shape as a diamond rather than a rectangle.

TEXT EXTRACTION AND RECOGNITION

Diagram elements typically contain text labels that convey semantic information. Class names, method signatures, state labels, and relationship descriptions all appear as text within or near shapes. Extracting this text accurately is crucial for understanding the diagram's meaning.

We use Tesseract OCR, a mature open-source optical character recognition engine, to extract text from detected shapes. However, running OCR on the entire image produces poor results because background noise and overlapping elements confuse the recognition algorithm. Instead, we extract individual shape regions and process them separately.

import pytesseract
from PIL import Image

class TextExtractor:
    def __init__(self, tesseract_config='--psm 6'):
        """
        Initialize the text extractor.
        
        Args:
            tesseract_config: Configuration string for Tesseract OCR
                             PSM 6 assumes uniform block of text
        """
        self.tesseract_config = tesseract_config
    
    def extract_text_from_shape(self, image, shape):
        """
        Extract text content from a shape region.
        
        Args:
            image: The original binary image
            shape: A Shape object containing the region to extract from
        
        Returns:
            Extracted text string, or None if no text found
        """
        x, y, w, h = shape.bounding_box
        
        # Add padding around the shape
        padding = 5
        x = max(0, x - padding)
        y = max(0, y - padding)
        w = min(image.shape[1] - x, w + 2 * padding)
        h = min(image.shape[0] - y, h + 2 * padding)
        
        # Extract the region
        region = image[y:y+h, x:x+w]
        
        # Invert if necessary (Tesseract expects black text on white background)
        if np.mean(region) > 127:
            region = cv2.bitwise_not(region)
        
        # Resize if too small
        if h < 30:
            scale = 30.0 / h
            new_w = int(w * scale)
            new_h = int(h * scale)
            region = cv2.resize(region, (new_w, new_h), interpolation=cv2.INTER_CUBIC)
        
        # Convert to PIL Image
        pil_image = Image.fromarray(region)
        
        # Run OCR
        text = pytesseract.image_to_string(pil_image, config=self.tesseract_config)
        
        # Clean up the text
        text = text.strip()
        
        # Filter out noise (very short strings or non-alphanumeric)
        if len(text) < 2 or not any(c.isalnum() for c in text):
            return None
        
        return text
    
    def extract_all_text(self, image, shapes):
        """
        Extract text from all shapes in the image.
        
        Updates each shape object with its text content.
        """
        for shape in shapes:
            text = self.extract_text_from_shape(image, shape)
            shape.text_content = text
        
        return shapes

The text extraction process begins by isolating each shape's bounding box with some padding to ensure we capture text that extends slightly beyond the shape boundaries. We invert the image if necessary because Tesseract expects black text on a white background. Small text regions are scaled up to improve recognition accuracy, as Tesseract performs better with larger text.

The Tesseract configuration uses page segmentation mode six, which assumes a uniform block of text. This works well for diagram labels, which typically consist of a single line or a small block of related text. After extraction, we filter out noise by discarding very short strings or strings without alphanumeric characters.

CONNECTION AND RELATIONSHIP DETECTION

Arrows and lines connecting shapes represent relationships in UML diagrams. Detecting these connections requires identifying line segments and determining which shapes they connect. This is more challenging than shape detection because arrows can have various styles, including solid lines, dashed lines, and different arrowhead types.

Our approach uses line segment detection combined with geometric analysis to identify connections. The Hough Line Transform detects straight line segments in the image. We then analyze these segments to determine their endpoints and find nearby shapes.

class ConnectionDetector:
    def __init__(self, line_threshold=50, min_line_length=30, max_line_gap=10):
        """
        Initialize the connection detector.
        
        Args:
            line_threshold: Minimum number of votes for Hough line detection
            min_line_length: Minimum length of a line segment
            max_line_gap: Maximum gap between line segments to connect them
        """
        self.line_threshold = line_threshold
        self.min_line_length = min_line_length
        self.max_line_gap = max_line_gap
    
    def detect_connections(self, binary_image, shapes):
        """
        Detect connections between shapes.
        
        Returns a list of Connection objects representing relationships.
        """
        # Create a mask of all shapes
        shape_mask = np.zeros_like(binary_image)
        for shape in shapes:
            cv2.drawContours(shape_mask, [shape.contour], -1, 255, -1)
        
        # Invert to get only the lines
        lines_only = cv2.bitwise_and(binary_image, cv2.bitwise_not(shape_mask))
        
        # Detect line segments using Hough transform
        lines = cv2.HoughLinesP(
            lines_only,
            rho=1,
            theta=np.pi/180,
            threshold=self.line_threshold,
            minLineLength=self.min_line_length,
            maxLineGap=self.max_line_gap
        )
        
        if lines is None:
            return []
        
        connections = []
        
        for line in lines:
            x1, y1, x2, y2 = line[0]
            
            # Find shapes near the endpoints
            start_shape = self._find_nearest_shape((x1, y1), shapes)
            end_shape = self._find_nearest_shape((x2, y2), shapes)
            
            if start_shape and end_shape and start_shape != end_shape:
                # Determine arrow direction and type
                arrow_type = self._classify_arrow(
                    binary_image,
                    (x1, y1),
                    (x2, y2)
                )
                
                connection = Connection(
                    start_shape=start_shape,
                    end_shape=end_shape,
                    start_point=(x1, y1),
                    end_point=(x2, y2),
                    arrow_type=arrow_type
                )
                
                connections.append(connection)
                start_shape.connections.append(connection)
        
        return connections
    
    def _find_nearest_shape(self, point, shapes, max_distance=20):
        """
        Find the shape nearest to a given point.
        
        Returns the nearest Shape object, or None if no shape is close enough.
        """
        min_distance = float('inf')
        nearest_shape = None
        
        for shape in shapes:
            # Check if point is inside the shape
            if shape.contains_point(point):
                return shape
            
            # Calculate distance to shape boundary
            distance = abs(cv2.pointPolygonTest(shape.contour, point, True))
            
            if distance < min_distance:
                min_distance = distance
                nearest_shape = shape
        
        if min_distance <= max_distance:
            return nearest_shape
        
        return None
    
    def _classify_arrow(self, image, start_point, end_point):
        """
        Classify the type of arrow or line.
        
        Returns a string indicating the arrow type: 'solid', 'dashed',
        'inheritance', 'composition', 'aggregation', etc.
        """
        # Extract the line region
        x1, y1 = start_point
        x2, y2 = end_point
        
        # Calculate line properties
        length = np.sqrt((x2 - x1)**2 + (y2 - y1)**2)
        angle = np.arctan2(y2 - y1, x2 - x1)
        
        # Sample points along the line
        num_samples = int(length / 5)
        if num_samples < 2:
            return 'solid'
        
        white_count = 0
        for i in range(num_samples):
            t = i / (num_samples - 1)
            x = int(x1 + t * (x2 - x1))
            y = int(y1 + t * (y2 - y1))
            
            if 0 <= x < image.shape[1] and 0 <= y < image.shape[0]:
                if image[y, x] > 127:
                    white_count += 1
        
        # If less than 70% of samples are white, it's dashed
        if white_count / num_samples < 0.7:
            return 'dashed'
        
        # Check for arrowhead at the end point
        arrowhead_region = self._extract_arrowhead_region(
            image,
            end_point,
            angle
        )
        
        if arrowhead_region is not None:
            arrowhead_type = self._classify_arrowhead(arrowhead_region)
            return arrowhead_type
        
        return 'solid'
    
    def _extract_arrowhead_region(self, image, point, angle, size=20):
        """
        Extract a small region around the arrowhead.
        """
        x, y = point
        
        # Define a rectangle around the arrowhead
        half_size = size // 2
        x1 = max(0, x - half_size)
        y1 = max(0, y - half_size)
        x2 = min(image.shape[1], x + half_size)
        y2 = min(image.shape[0], y + half_size)
        
        if x2 - x1 < 5 or y2 - y1 < 5:
            return None
        
        region = image[y1:y2, x1:x2]
        return region
    
    def _classify_arrowhead(self, region):
        """
        Classify the arrowhead type based on its appearance.
        
        Returns 'inheritance', 'composition', 'aggregation', or 'association'.
        """
        # Count white pixels
        white_ratio = np.sum(region > 127) / region.size
        
        # Filled arrowhead indicates composition
        if white_ratio > 0.7:
            return 'composition'
        
        # Open arrowhead indicates inheritance or aggregation
        if white_ratio < 0.3:
            # Check for diamond shape (aggregation)
            contours, _ = cv2.findContours(
                region,
                cv2.RETR_EXTERNAL,
                cv2.CHAIN_APPROX_SIMPLE
            )
            
            if contours:
                largest_contour = max(contours, key=cv2.contourArea)
                approx = cv2.approxPolyDP(
                    largest_contour,
                    0.04 * cv2.arcLength(largest_contour, True),
                    True
                )
                
                if len(approx) == 4:
                    return 'aggregation'
            
            return 'inheritance'
        
        return 'association'


class Connection:
    """
    Represents a connection between two shapes.
    """
    def __init__(self, start_shape, end_shape, start_point, end_point, arrow_type):
        self.start_shape = start_shape
        self.end_shape = end_shape
        self.start_point = start_point
        self.end_point = end_point
        self.arrow_type = arrow_type
        self.label = None

The connection detection algorithm first creates a mask of all detected shapes and subtracts it from the binary image. This isolates the lines and arrows that connect shapes. The Hough Line Transform then detects straight line segments in this isolated image. For each detected line segment, we find the shapes closest to its endpoints. If both endpoints are near different shapes, we create a connection between them.

Arrow classification examines the line style and arrowhead type. We sample points along the line to determine if it is solid or dashed. A dashed line has gaps where the sampled pixels are black. For arrowhead classification, we extract a small region around the arrow endpoint and analyze its contents. Filled arrowheads indicate composition relationships, open triangular arrowheads indicate inheritance, and diamond shapes indicate aggregation.

LLM INTEGRATION FOR SEMANTIC INTERPRETATION

While computer vision techniques can detect shapes and connections, they cannot fully understand the semantic meaning of a diagram. An LLM can interpret the spatial arrangement of elements, read text labels, and infer the intended diagram type and structure. This semantic understanding enables us to generate accurate PlantUML specifications.

Our LLM integration supports both local and remote models. Local models run on the user's hardware, providing privacy and eliminating API costs. Remote models offer superior performance and capabilities but require internet connectivity and API keys. We abstract the LLM interface to support multiple backends transparently.

import json
import requests
from abc import ABC, abstractmethod

class LLMProvider(ABC):
    """
    Abstract base class for LLM providers.
    """
    @abstractmethod
    def generate(self, prompt, max_tokens=2000, temperature=0.3):
        """
        Generate a response from the LLM.
        
        Args:
            prompt: The input prompt
            max_tokens: Maximum number of tokens to generate
            temperature: Sampling temperature (lower = more deterministic)
        
        Returns:
            Generated text string
        """
        pass


class LocalLLMProvider(LLMProvider):
    """
    Provider for local LLM models using llama-cpp-python.
    """
    def __init__(self, model_path, n_ctx=4096, n_gpu_layers=0):
        """
        Initialize the local LLM provider.
        
        Args:
            model_path: Path to the GGUF model file
            n_ctx: Context window size
            n_gpu_layers: Number of layers to offload to GPU
        """
        try:
            from llama_cpp import Llama
        except ImportError:
            raise ImportError(
                "llama-cpp-python is required for local LLM support. "
                "Install it with: pip install llama-cpp-python"
            )
        
        self.model = Llama(
            model_path=model_path,
            n_ctx=n_ctx,
            n_gpu_layers=n_gpu_layers,
            verbose=False
        )
    
    def generate(self, prompt, max_tokens=2000, temperature=0.3):
        """
        Generate a response using the local model.
        """
        response = self.model(
            prompt,
            max_tokens=max_tokens,
            temperature=temperature,
            stop=["</s>", "Human:", "User:"],
            echo=False
        )
        
        return response['choices'][0]['text'].strip()


class RemoteLLMProvider(LLMProvider):
    """
    Provider for remote LLM APIs (OpenAI-compatible).
    """
    def __init__(self, api_url, api_key, model_name):
        """
        Initialize the remote LLM provider.
        
        Args:
            api_url: Base URL for the API
            api_key: API authentication key
            model_name: Name of the model to use
        """
        self.api_url = api_url
        self.api_key = api_key
        self.model_name = model_name
    
    def generate(self, prompt, max_tokens=2000, temperature=0.3):
        """
        Generate a response using the remote API.
        """
        headers = {
            'Authorization': f'Bearer {self.api_key}',
            'Content-Type': 'application/json'
        }
        
        data = {
            'model': self.model_name,
            'messages': [
                {'role': 'user', 'content': prompt}
            ],
            'max_tokens': max_tokens,
            'temperature': temperature
        }
        
        response = requests.post(
            f'{self.api_url}/chat/completions',
            headers=headers,
            json=data,
            timeout=60
        )
        
        response.raise_for_status()
        result = response.json()
        
        return result['choices'][0]['message']['content'].strip()

The LLM provider abstraction allows us to switch between local and remote models without changing the rest of our code. The local provider uses llama-cpp-python, which supports GGUF format models and provides efficient CPU and GPU inference. The remote provider implements the OpenAI chat completions API, which many services support.

GPU acceleration configuration varies by platform. For NVIDIA GPUs with CUDA, we set the number of GPU layers to offload. For AMD GPUs with ROCm, we ensure the ROCm-compiled version of llama-cpp-python is installed. For Apple Silicon, Metal Performance Shaders are automatically used when available. Intel GPUs require the oneAPI-enabled build of llama-cpp-python.

import platform
import subprocess

class GPUDetector:
    """
    Detect available GPU hardware and configure accordingly.
    """
    @staticmethod
    def detect_gpu():
        """
        Detect the available GPU type.
        
        Returns a tuple of (gpu_type, recommended_layers)
        where gpu_type is 'cuda', 'rocm', 'metal', 'oneapi', or 'cpu'
        """
        system = platform.system()
        
        # Check for NVIDIA CUDA
        try:
            result = subprocess.run(
                ['nvidia-smi'],
                capture_output=True,
                text=True,
                timeout=5
            )
            if result.returncode == 0:
                return ('cuda', 35)  # Offload most layers to GPU
        except (FileNotFoundError, subprocess.TimeoutExpired):
            pass
        
        # Check for AMD ROCm
        try:
            result = subprocess.run(
                ['rocm-smi'],
                capture_output=True,
                text=True,
                timeout=5
            )
            if result.returncode == 0:
                return ('rocm', 35)
        except (FileNotFoundError, subprocess.TimeoutExpired):
            pass
        
        # Check for Apple Metal
        if system == 'Darwin':
            try:
                result = subprocess.run(
                    ['sysctl', 'hw.optional.arm64'],
                    capture_output=True,
                    text=True,
                    timeout=5
                )
                if result.returncode == 0 and '1' in result.stdout:
                    return ('metal', 1)  # Metal uses different layer system
            except (FileNotFoundError, subprocess.TimeoutExpired):
                pass
        
        # Check for Intel GPU
        try:
            result = subprocess.run(
                ['clinfo'],
                capture_output=True,
                text=True,
                timeout=5
            )
            if result.returncode == 0 and 'Intel' in result.stdout:
                return ('oneapi', 20)
        except (FileNotFoundError, subprocess.TimeoutExpired):
            pass
        
        # Default to CPU
        return ('cpu', 0)
    
    @staticmethod
    def create_local_llm(model_path, n_ctx=4096):
        """
        Create a local LLM provider with automatic GPU detection.
        """
        gpu_type, n_gpu_layers = GPUDetector.detect_gpu()
        
        print(f"Detected GPU type: {gpu_type}")
        print(f"Using {n_gpu_layers} GPU layers")
        
        return LocalLLMProvider(
            model_path=model_path,
            n_ctx=n_ctx,
            n_gpu_layers=n_gpu_layers
        )

The GPU detection logic checks for various GPU types by attempting to run their respective management tools. For NVIDIA, we run nvidia-smi. For AMD, we check for rocm-smi. On macOS, we examine system properties to detect Apple Silicon. For Intel, we use clinfo to detect OpenCL-capable Intel GPUs. If no GPU is detected, we fall back to CPU-only inference.

Now we can use the LLM to interpret the detected shapes and connections. We construct a detailed prompt that describes the visual elements and asks the LLM to generate a PlantUML specification.

class DiagramInterpreter:
    """
    Interprets detected shapes and connections to generate diagram specifications.
    """
    def __init__(self, llm_provider):
        """
        Initialize the diagram interpreter.
        
        Args:
            llm_provider: An LLMProvider instance
        """
        self.llm_provider = llm_provider
    
    def interpret_diagram(self, shapes, connections):
        """
        Interpret the diagram and generate a PlantUML specification.
        
        Args:
            shapes: List of Shape objects
            connections: List of Connection objects
        
        Returns:
            A dictionary containing the PlantUML code and metadata
        """
        # Build a description of the diagram
        description = self._build_diagram_description(shapes, connections)
        
        # Create the prompt
        prompt = self._create_interpretation_prompt(description)
        
        # Generate the PlantUML code
        response = self.llm_provider.generate(prompt, max_tokens=3000, temperature=0.2)
        
        # Parse the response
        plantuml_code = self._extract_plantuml_code(response)
        diagram_type = self._infer_diagram_type(shapes, connections)
        
        return {
            'plantuml_code': plantuml_code,
            'diagram_type': diagram_type,
            'interpretation': response
        }
    
    def _build_diagram_description(self, shapes, connections):
        """
        Build a textual description of the detected diagram elements.
        """
        description = "Detected diagram elements:\n\n"
        
        # Describe shapes
        description += "Shapes:\n"
        for i, shape in enumerate(shapes):
            description += f"{i+1}. {shape.shape_type.capitalize()}"
            if shape.text_content:
                description += f" containing text: '{shape.text_content}'"
            description += f" at position ({int(shape.centroid[0])}, {int(shape.centroid[1])})"
            description += f" with size {shape.bounding_box[2]}x{shape.bounding_box[3]}\n"
        
        description += "\nConnections:\n"
        for i, conn in enumerate(connections):
            start_idx = shapes.index(conn.start_shape) + 1
            end_idx = shapes.index(conn.end_shape) + 1
            description += f"{i+1}. {conn.arrow_type.capitalize()} arrow from shape {start_idx} to shape {end_idx}\n"
        
        return description
    
    def _create_interpretation_prompt(self, description):
        """
        Create a prompt for the LLM to interpret the diagram.
        """
        prompt = f"""You are an expert in UML diagrams and software architecture visualization.

I have detected the following elements from a hand-drawn diagram:

{description}

Based on these elements, please:
1. Determine what type of UML diagram this is (class diagram, sequence diagram, state diagram, etc.)
2. Generate valid PlantUML code that represents this diagram
3. Ensure the PlantUML code captures the relationships and structure accurately

Please provide your response in the following format:

DIAGRAM_TYPE: [type of diagram]

PLANTUML_CODE:
@startuml
[your PlantUML code here]
@enduml

EXPLANATION:
[brief explanation of the diagram structure]
"""
        return prompt
    
    def _extract_plantuml_code(self, response):
        """
        Extract PlantUML code from the LLM response.
        """
        # Find the PlantUML code block
        start_marker = '@startuml'
        end_marker = '@enduml'
        
        start_idx = response.find(start_marker)
        end_idx = response.find(end_marker)
        
        if start_idx == -1 or end_idx == -1:
            # Try to extract from code blocks
            if '```' in response:
                parts = response.split('```')
                for part in parts:
                    if start_marker in part:
                        return part.strip()
            
            # Return a default template if extraction fails
            return "@startuml\n' Failed to extract diagram\n@enduml"
        
        plantuml_code = response[start_idx:end_idx + len(end_marker)]
        return plantuml_code.strip()
    
    def _infer_diagram_type(self, shapes, connections):
        """
        Infer the diagram type based on shape and connection patterns.
        """
        # Count shape types
        shape_types = {}
        for shape in shapes:
            shape_types[shape.shape_type] = shape_types.get(shape.shape_type, 0) + 1
        
        # Count connection types
        connection_types = {}
        for conn in connections:
            connection_types[conn.arrow_type] = connection_types.get(conn.arrow_type, 0) + 1
        
        # Heuristics for diagram type
        if 'rectangle' in shape_types and shape_types['rectangle'] > len(shapes) * 0.5:
            if 'inheritance' in connection_types or 'composition' in connection_types:
                return 'class_diagram'
        
        if 'circle' in shape_types and 'diamond' in shape_types:
            return 'state_diagram'
        
        if any(conn.arrow_type == 'dashed' for conn in connections):
            return 'sequence_diagram'
        
        # Default to class diagram
        return 'class_diagram'

The diagram interpretation process constructs a detailed textual description of all detected elements, including their types, positions, sizes, and text content. This description is embedded in a carefully crafted prompt that instructs the LLM to identify the diagram type and generate corresponding PlantUML code. The prompt uses a structured format to make parsing the response easier.

After receiving the LLM response, we extract the PlantUML code block and perform basic validation. The diagram type inference uses heuristics based on shape and connection patterns. For example, diagrams with many rectangles and inheritance arrows are likely class diagrams, while diagrams with circles and diamonds might be state diagrams.

PLANTUML GENERATION AND RENDERING

PlantUML is a powerful tool for generating UML diagrams from textual descriptions. It supports various diagram types and produces high-quality output in multiple formats. Our application uses PlantUML to render the final diagrams based on the LLM-generated specifications.

PlantUML requires a Java runtime environment and the PlantUML JAR file. We provide a wrapper class that handles PlantUML invocation and manages the rendering process.

import subprocess
import os
import tempfile
from pathlib import Path

class PlantUMLRenderer:
    """
    Renders PlantUML diagrams to various output formats.
    """
    def __init__(self, plantuml_jar_path=None):
        """
        Initialize the PlantUML renderer.
        
        Args:
            plantuml_jar_path: Path to the PlantUML JAR file
                              If None, looks for PLANTUML_JAR environment variable
        """
        if plantuml_jar_path is None:
            plantuml_jar_path = os.environ.get('PLANTUML_JAR')
        
        if plantuml_jar_path is None or not os.path.exists(plantuml_jar_path):
            raise ValueError(
                "PlantUML JAR file not found. Please set PLANTUML_JAR "
                "environment variable or provide plantuml_jar_path"
            )
        
        self.plantuml_jar_path = plantuml_jar_path
        
        # Verify Java is available
        try:
            subprocess.run(
                ['java', '-version'],
                capture_output=True,
                timeout=5
            )
        except (FileNotFoundError, subprocess.TimeoutExpired):
            raise RuntimeError(
                "Java runtime not found. Please install Java to use PlantUML"
            )
    
    def render(self, plantuml_code, output_format='png', output_path=None):
        """
        Render PlantUML code to an image file.
        
        Args:
            plantuml_code: The PlantUML code to render
            output_format: Output format ('png', 'svg', or 'pdf')
            output_path: Path for the output file (optional)
        
        Returns:
            Path to the generated file
        """
        # Create a temporary file for the PlantUML code
        with tempfile.NamedTemporaryFile(
            mode='w',
            suffix='.puml',
            delete=False
        ) as temp_file:
            temp_file.write(plantuml_code)
            temp_path = temp_file.name
        
        try:
            # Determine output format flag
            format_flag = {
                'png': '-tpng',
                'svg': '-tsvg',
                'pdf': '-tpdf'
            }.get(output_format.lower(), '-tpng')
            
            # Run PlantUML
            cmd = [
                'java',
                '-jar',
                self.plantuml_jar_path,
                format_flag,
                temp_path
            ]
            
            result = subprocess.run(
                cmd,
                capture_output=True,
                text=True,
                timeout=30
            )
            
            if result.returncode != 0:
                raise RuntimeError(
                    f"PlantUML rendering failed: {result.stderr}"
                )
            
            # Find the generated file
            temp_dir = os.path.dirname(temp_path)
            temp_name = os.path.splitext(os.path.basename(temp_path))[0]
            generated_file = os.path.join(
                temp_dir,
                f"{temp_name}.{output_format.lower()}"
            )
            
            if not os.path.exists(generated_file):
                raise RuntimeError("PlantUML did not generate output file")
            
            # Move to final location if specified
            if output_path:
                os.makedirs(os.path.dirname(output_path), exist_ok=True)
                os.rename(generated_file, output_path)
                return output_path
            
            return generated_file
            
        finally:
            # Clean up temporary file
            if os.path.exists(temp_path):
                os.unlink(temp_path)
    
    def render_multiple_formats(self, plantuml_code, output_dir, base_name='diagram'):
        """
        Render the diagram in multiple formats.
        
        Returns a dictionary mapping format names to file paths.
        """
        formats = ['png', 'svg', 'pdf']
        results = {}
        
        for fmt in formats:
            output_path = os.path.join(output_dir, f"{base_name}.{fmt}")
            try:
                results[fmt] = self.render(plantuml_code, fmt, output_path)
            except Exception as e:
                print(f"Warning: Failed to render {fmt} format: {e}")
        
        return results

The PlantUML renderer wraps the command-line PlantUML tool, which is distributed as a Java JAR file. We create a temporary file containing the PlantUML code, invoke the Java runtime with appropriate flags for the desired output format, and collect the generated file. The renderer supports PNG for raster images, SVG for vector graphics, and PDF for document integration.

Error handling is important because PlantUML can fail for various reasons, including syntax errors in the generated code or missing dependencies. We capture stderr output and provide meaningful error messages to help diagnose issues.

WEB APPLICATION BACKEND

The backend web application orchestrates all the components we have developed. It provides REST API endpoints for image upload, processing, and result retrieval. We use FastAPI for its excellent performance, automatic validation, and built-in documentation.

from fastapi import FastAPI, File, UploadFile, HTTPException, BackgroundTasks
from fastapi.responses import FileResponse, JSONResponse
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from typing import Optional, List
import uuid
import shutil

app = FastAPI(title="Diagram Digitizer API")

# Enable CORS for web frontend
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

# Global state (in production, use a database)
processing_jobs = {}


class ProcessingJob:
    """
    Represents a diagram processing job.
    """
    def __init__(self, job_id):
        self.job_id = job_id
        self.status = 'pending'
        self.progress = 0
        self.error = None
        self.original_image_path = None
        self.corrected_image_path = None
        self.shapes = []
        self.connections = []
        self.interpretations = []
        self.selected_interpretation = None
        self.output_files = {}


class ProcessingRequest(BaseModel):
    """Request model for processing parameters."""
    use_local_llm: bool = True
    model_path: Optional[str] = None
    api_key: Optional[str] = None
    output_formats: List[str] = ['png', 'svg', 'pdf']


class InterpretationSelection(BaseModel):
    """Request model for selecting an interpretation."""
    interpretation_index: int


@app.post("/api/upload")
async def upload_image(
    file: UploadFile = File(...),
    background_tasks: BackgroundTasks = None
):
    """
    Upload an image for processing.
    
    Returns a job ID that can be used to track processing status.
    """
    # Validate file type
    if not file.content_type.startswith('image/'):
        raise HTTPException(
            status_code=400,
            detail="File must be an image"
        )
    
    # Generate job ID
    job_id = str(uuid.uuid4())
    
    # Save uploaded file
    upload_dir = Path('uploads')
    upload_dir.mkdir(exist_ok=True)
    
    file_path = upload_dir / f"{job_id}_{file.filename}"
    
    with open(file_path, 'wb') as buffer:
        shutil.copyfileobj(file.file, buffer)
    
    # Create processing job
    job = ProcessingJob(job_id)
    job.original_image_path = str(file_path)
    processing_jobs[job_id] = job
    
    return {
        'job_id': job_id,
        'status': 'uploaded',
        'filename': file.filename
    }


@app.post("/api/process/{job_id}")
async def process_image(
    job_id: str,
    request: ProcessingRequest,
    background_tasks: BackgroundTasks
):
    """
    Start processing a uploaded image.
    
    The processing happens asynchronously in the background.
    """
    if job_id not in processing_jobs:
        raise HTTPException(status_code=404, detail="Job not found")
    
    job = processing_jobs[job_id]
    
    if job.status != 'pending':
        raise HTTPException(
            status_code=400,
            detail=f"Job already {job.status}"
        )
    
    # Start background processing
    background_tasks.add_task(
        process_diagram_background,
        job,
        request
    )
    
    job.status = 'processing'
    
    return {
        'job_id': job_id,
        'status': 'processing'
    }


async def process_diagram_background(job, request):
    """
    Background task for processing a diagram.
    
    This function orchestrates all the processing steps.
    """
    try:
        # Load the image
        image = cv2.imread(job.original_image_path)
        if image is None:
            raise ValueError("Failed to load image")
        
        job.progress = 10
        
        # Perspective correction
        corrector = PerspectiveCorrector()
        corrected = corrector.correct_perspective(image)
        
        # Save corrected image
        corrected_path = job.original_image_path.replace('.', '_corrected.')
        cv2.imwrite(corrected_path, corrected)
        job.corrected_image_path = corrected_path
        
        job.progress = 20
        
        # Binarization
        binarizer = ImageBinarizer()
        binary = binarizer.binarize(corrected)
        binary = binarizer.remove_noise(binary)
        
        job.progress = 30
        
        # Shape detection
        shape_detector = ShapeDetector()
        shapes = shape_detector.detect_shapes(binary)
        
        job.progress = 40
        
        # Text extraction
        text_extractor = TextExtractor()
        shapes = text_extractor.extract_all_text(binary, shapes)
        
        job.progress = 50
        
        # Connection detection
        connection_detector = ConnectionDetector()
        connections = connection_detector.detect_connections(binary, shapes)
        
        job.shapes = shapes
        job.connections = connections
        
        job.progress = 60
        
        # LLM interpretation
        if request.use_local_llm:
            if not request.model_path:
                raise ValueError("model_path required for local LLM")
            llm_provider = GPUDetector.create_local_llm(request.model_path)
        else:
            if not request.api_key:
                raise ValueError("api_key required for remote LLM")
            llm_provider = RemoteLLMProvider(
                api_url="https://api.openai.com/v1",
                api_key=request.api_key,
                model_name="gpt-4"
            )
        
        interpreter = DiagramInterpreter(llm_provider)
        
        # Generate multiple interpretations
        interpretations = []
        for i in range(3):
            interpretation = interpreter.interpret_diagram(shapes, connections)
            interpretations.append(interpretation)
        
        job.interpretations = interpretations
        job.selected_interpretation = 0
        
        job.progress = 80
        
        # Render the first interpretation
        renderer = PlantUMLRenderer()
        output_dir = Path('outputs') / job.job_id
        output_dir.mkdir(parents=True, exist_ok=True)
        
        plantuml_code = interpretations[0]['plantuml_code']
        output_files = renderer.render_multiple_formats(
            plantuml_code,
            str(output_dir)
        )
        
        job.output_files = output_files
        job.progress = 100
        job.status = 'completed'
        
    except Exception as e:
        job.status = 'failed'
        job.error = str(e)
        import traceback
        traceback.print_exc()


@app.get("/api/status/{job_id}")
async def get_status(job_id: str):
    """
    Get the status of a processing job.
    """
    if job_id not in processing_jobs:
        raise HTTPException(status_code=404, detail="Job not found")
    
    job = processing_jobs[job_id]
    
    response = {
        'job_id': job_id,
        'status': job.status,
        'progress': job.progress
    }
    
    if job.error:
        response['error'] = job.error
    
    if job.interpretations:
        response['interpretations'] = [
            {
                'diagram_type': interp['diagram_type'],
                'preview': interp['plantuml_code'][:200] + '...'
            }
            for interp in job.interpretations
        ]
        response['selected_interpretation'] = job.selected_interpretation
    
    if job.output_files:
        response['output_files'] = {
            fmt: f"/api/download/{job_id}/{fmt}"
            for fmt in job.output_files.keys()
        }
    
    return response


@app.post("/api/select-interpretation/{job_id}")
async def select_interpretation(
    job_id: str,
    selection: InterpretationSelection,
    background_tasks: BackgroundTasks
):
    """
    Select a different interpretation and re-render.
    """
    if job_id not in processing_jobs:
        raise HTTPException(status_code=404, detail="Job not found")
    
    job = processing_jobs[job_id]
    
    if selection.interpretation_index >= len(job.interpretations):
        raise HTTPException(status_code=400, detail="Invalid interpretation index")
    
    job.selected_interpretation = selection.interpretation_index
    
    # Re-render with selected interpretation
    background_tasks.add_task(
        rerender_diagram,
        job,
        selection.interpretation_index
    )
    
    return {
        'job_id': job_id,
        'selected_interpretation': selection.interpretation_index,
        'status': 'rendering'
    }


async def rerender_diagram(job, interpretation_index):
    """
    Re-render the diagram with a different interpretation.
    """
    try:
        renderer = PlantUMLRenderer()
        output_dir = Path('outputs') / job.job_id
        
        plantuml_code = job.interpretations[interpretation_index]['plantuml_code']
        output_files = renderer.render_multiple_formats(
            plantuml_code,
            str(output_dir)
        )
        
        job.output_files = output_files
        
    except Exception as e:
        job.error = str(e)


@app.get("/api/download/{job_id}/{format}")
async def download_file(job_id: str, format: str):
    """
    Download a generated diagram file.
    """
    if job_id not in processing_jobs:
        raise HTTPException(status_code=404, detail="Job not found")
    
    job = processing_jobs[job_id]
    
    if format not in job.output_files:
        raise HTTPException(status_code=404, detail="Format not available")
    
    file_path = job.output_files[format]
    
    if not os.path.exists(file_path):
        raise HTTPException(status_code=404, detail="File not found")
    
    return FileResponse(
        file_path,
        media_type=f'application/{format}',
        filename=f'diagram.{format}'
    )


@app.get("/api/plantuml/{job_id}")
async def get_plantuml_code(job_id: str):
    """
    Get the PlantUML code for a job.
    """
    if job_id not in processing_jobs:
        raise HTTPException(status_code=404, detail="Job not found")
    
    job = processing_jobs[job_id]
    
    if not job.interpretations:
        raise HTTPException(
            status_code=400,
            detail="No interpretations available yet"
        )
    
    selected = job.interpretations[job.selected_interpretation]
    
    return {
        'plantuml_code': selected['plantuml_code'],
        'diagram_type': selected['diagram_type']
    }


@app.post("/api/update-plantuml/{job_id}")
async def update_plantuml_code(
    job_id: str,
    plantuml_code: str,
    background_tasks: BackgroundTasks
):
    """
    Update the PlantUML code manually and re-render.
    """
    if job_id not in processing_jobs:
        raise HTTPException(status_code=404, detail="Job not found")
    
    job = processing_jobs[job_id]
    
    # Update the selected interpretation
    job.interpretations[job.selected_interpretation]['plantuml_code'] = plantuml_code
    
    # Re-render
    background_tasks.add_task(
        rerender_diagram,
        job,
        job.selected_interpretation
    )
    
    return {
        'job_id': job_id,
        'status': 'rendering'
    }

The backend API provides several endpoints that work together to support the complete workflow. The upload endpoint accepts image files and creates a processing job. The process endpoint starts the background processing pipeline, which runs asynchronously to avoid blocking the web server. The status endpoint allows clients to poll for progress updates. The download endpoint serves the generated diagram files.

The background processing function orchestrates all the components we developed earlier. It loads the image, applies perspective correction, performs binarization, detects shapes and connections, extracts text, interprets the diagram using the LLM, and renders the output using PlantUML. Progress updates are stored in the job object so clients can display a progress bar.

The API also supports interactive refinement. Users can select from multiple interpretations generated by the LLM, and the system will re-render the diagram with the chosen interpretation. Users can also manually edit the PlantUML code and trigger re-rendering, providing full control over the final output.

WEB APPLICATION FRONTEND

The frontend provides an intuitive user interface for uploading images, monitoring processing progress, viewing results, and refining outputs. We use vanilla JavaScript with modern features to avoid framework dependencies while maintaining clean, maintainable code.

The HTML structure defines the page layout with sections for upload, progress display, interpretation selection, and result viewing.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Diagram Digitizer</title>
    <style>
        * {
            margin: 0;
            padding: 0;
            box-sizing: border-box;
        }
        
        body {
            font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, sans-serif;
            background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
            min-height: 100vh;
            padding: 20px;
        }
        
        .container {
            max-width: 1200px;
            margin: 0 auto;
            background: white;
            border-radius: 12px;
            box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3);
            overflow: hidden;
        }
        
        .header {
            background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
            color: white;
            padding: 30px;
            text-align: center;
        }
        
        .header h1 {
            font-size: 2.5em;
            margin-bottom: 10px;
        }
        
        .header p {
            font-size: 1.1em;
            opacity: 0.9;
        }
        
        .content {
            padding: 30px;
        }
        
        .upload-section {
            border: 3px dashed #667eea;
            border-radius: 8px;
            padding: 40px;
            text-align: center;
            cursor: pointer;
            transition: all 0.3s;
        }
        
        .upload-section:hover {
            border-color: #764ba2;
            background: #f8f9ff;
        }
        
        .upload-section.dragover {
            border-color: #764ba2;
            background: #f0f2ff;
        }
        
        .upload-icon {
            font-size: 4em;
            margin-bottom: 20px;
        }
        
        .file-input {
            display: none;
        }
        
        .btn {
            background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
            color: white;
            border: none;
            padding: 12px 30px;
            border-radius: 6px;
            font-size: 1em;
            cursor: pointer;
            transition: transform 0.2s;
        }
        
        .btn:hover {
            transform: translateY(-2px);
        }
        
        .btn:disabled {
            opacity: 0.5;
            cursor: not-allowed;
        }
        
        .progress-section {
            display: none;
            margin-top: 30px;
        }
        
        .progress-bar {
            width: 100%;
            height: 30px;
            background: #e0e0e0;
            border-radius: 15px;
            overflow: hidden;
            margin-bottom: 10px;
        }
        
        .progress-fill {
            height: 100%;
            background: linear-gradient(90deg, #667eea 0%, #764ba2 100%);
            transition: width 0.3s;
            display: flex;
            align-items: center;
            justify-content: center;
            color: white;
            font-weight: bold;
        }
        
        .interpretations-section {
            display: none;
            margin-top: 30px;
        }
        
        .interpretation-card {
            border: 2px solid #e0e0e0;
            border-radius: 8px;
            padding: 20px;
            margin-bottom: 15px;
            cursor: pointer;
            transition: all 0.3s;
        }
        
        .interpretation-card:hover {
            border-color: #667eea;
            box-shadow: 0 4px 12px rgba(102, 126, 234, 0.2);
        }
        
        .interpretation-card.selected {
            border-color: #667eea;
            background: #f8f9ff;
        }
        
        .results-section {
            display: none;
            margin-top: 30px;
        }
        
        .result-preview {
            border: 1px solid #e0e0e0;
            border-radius: 8px;
            padding: 20px;
            margin-bottom: 20px;
        }
        
        .result-preview img {
            max-width: 100%;
            border-radius: 4px;
        }
        
        .download-buttons {
            display: flex;
            gap: 10px;
            flex-wrap: wrap;
        }
        
        .code-editor {
            margin-top: 20px;
        }
        
        .code-editor textarea {
            width: 100%;
            min-height: 300px;
            font-family: 'Courier New', monospace;
            padding: 15px;
            border: 1px solid #e0e0e0;
            border-radius: 4px;
            resize: vertical;
        }
        
        .error-message {
            background: #fee;
            border: 1px solid #fcc;
            color: #c33;
            padding: 15px;
            border-radius: 4px;
            margin-top: 15px;
        }
    </style>
</head>
<body>
    <div class="container">
        <div class="header">
            <h1>Diagram Digitizer</h1>
            <p>Transform hand-drawn diagrams into beautiful digital versions</p>
        </div>
        
        <div class="content">
            <div class="upload-section" id="uploadSection">
                <div class="upload-icon">📸</div>
                <h2>Upload Your Diagram</h2>
                <p>Drag and drop an image here, or click to select a file</p>
                <input type="file" id="fileInput" class="file-input" accept="image/*">
                <button class="btn" onclick="document.getElementById('fileInput').click()">
                    Select Image
                </button>
            </div>
            
            <div class="progress-section" id="progressSection">
                <h3>Processing Your Diagram</h3>
                <div class="progress-bar">
                    <div class="progress-fill" id="progressFill">0%</div>
                </div>
                <p id="progressStatus">Initializing...</p>
            </div>
            
            <div class="interpretations-section" id="interpretationsSection">
                <h3>Select an Interpretation</h3>
                <p>The AI has generated multiple interpretations. Choose the one that best matches your diagram:</p>
                <div id="interpretationsList"></div>
            </div>
            
            <div class="results-section" id="resultsSection">
                <h3>Your Digitized Diagram</h3>
                <div class="result-preview" id="resultPreview"></div>
                
                <h4>Download Options</h4>
                <div class="download-buttons" id="downloadButtons"></div>
                
                <div class="code-editor">
                    <h4>PlantUML Code (Editable)</h4>
                    <textarea id="plantumlCode"></textarea>
                    <button class="btn" onclick="updatePlantUML()">Update Diagram</button>
                </div>
            </div>
            
            <div id="errorMessage"></div>
        </div>
    </div>
    
    <script src="app.js"></script>
</body>
</html>

The CSS styling creates a modern, professional appearance with a gradient background, smooth transitions, and responsive layout. The upload section uses a dashed border that changes on hover and drag events to provide visual feedback. Progress bars use gradient fills that animate smoothly as processing advances.

The JavaScript application logic handles user interactions, API communication, and dynamic UI updates.

class DiagramDigitizer {
    constructor() {
        this.apiBaseUrl = 'http://localhost:8000/api';
        this.currentJobId = null;
        this.pollingInterval = null;
        
        this.initializeEventListeners();
    }
    
    initializeEventListeners() {
        const uploadSection = document.getElementById('uploadSection');
        const fileInput = document.getElementById('fileInput');
        
        // File input change
        fileInput.addEventListener('change', (e) => {
            if (e.target.files.length > 0) {
                this.handleFileUpload(e.target.files[0]);
            }
        });
        
        // Drag and drop
        uploadSection.addEventListener('dragover', (e) => {
            e.preventDefault();
            uploadSection.classList.add('dragover');
        });
        
        uploadSection.addEventListener('dragleave', () => {
            uploadSection.classList.remove('dragover');
        });
        
        uploadSection.addEventListener('drop', (e) => {
            e.preventDefault();
            uploadSection.classList.remove('dragover');
            
            if (e.dataTransfer.files.length > 0) {
                this.handleFileUpload(e.dataTransfer.files[0]);
            }
        });
    }
    
    async handleFileUpload(file) {
        try {
            // Validate file type
            if (!file.type.startsWith('image/')) {
                this.showError('Please upload an image file');
                return;
            }
            
            // Upload the file
            const formData = new FormData();
            formData.append('file', file);
            
            const response = await fetch(`${this.apiBaseUrl}/upload`, {
                method: 'POST',
                body: formData
            });
            
            if (!response.ok) {
                throw new Error('Upload failed');
            }
            
            const data = await response.json();
            this.currentJobId = data.job_id;
            
            // Start processing
            await this.startProcessing();
            
        } catch (error) {
            this.showError(`Upload failed: ${error.message}`);
        }
    }
    
    async startProcessing() {
        try {
            // Show progress section
            document.getElementById('uploadSection').style.display = 'none';
            document.getElementById('progressSection').style.display = 'block';
            
            // Start processing
            const response = await fetch(
                `${this.apiBaseUrl}/process/${this.currentJobId}`,
                {
                    method: 'POST',
                    headers: {
                        'Content-Type': 'application/json'
                    },
                    body: JSON.stringify({
                        use_local_llm: false,
                        api_key: 'your-api-key-here',
                        output_formats: ['png', 'svg', 'pdf']
                    })
                }
            );
            
            if (!response.ok) {
                throw new Error('Processing failed to start');
            }
            
            // Start polling for status
            this.startStatusPolling();
            
        } catch (error) {
            this.showError(`Processing failed: ${error.message}`);
        }
    }
    
    startStatusPolling() {
        this.pollingInterval = setInterval(async () => {
            await this.checkStatus();
        }, 1000);
    }
    
    async checkStatus() {
        try {
            const response = await fetch(
                `${this.apiBaseUrl}/status/${this.currentJobId}`
            );
            
            if (!response.ok) {
                throw new Error('Status check failed');
            }
            
            const data = await response.json();
            
            // Update progress
            this.updateProgress(data.progress, data.status);
            
            // Handle completion
            if (data.status === 'completed') {
                clearInterval(this.pollingInterval);
                this.showInterpretations(data.interpretations);
                this.showResults(data.output_files);
            }
            
            // Handle errors
            if (data.status === 'failed') {
                clearInterval(this.pollingInterval);
                this.showError(data.error);
            }
            
        } catch (error) {
            clearInterval(this.pollingInterval);
            this.showError(`Status check failed: ${error.message}`);
        }
    }
    
    updateProgress(progress, status) {
        const progressFill = document.getElementById('progressFill');
        const progressStatus = document.getElementById('progressStatus');
        
        progressFill.style.width = `${progress}%`;
        progressFill.textContent = `${progress}%`;
        
        const statusMessages = {
            'pending': 'Waiting to start...',
            'processing': 'Processing your diagram...',
            'completed': 'Complete!',
            'failed': 'Processing failed'
        };
        
        progressStatus.textContent = statusMessages[status] || status;
    }
    
    showInterpretations(interpretations) {
        const section = document.getElementById('interpretationsSection');
        const list = document.getElementById('interpretationsList');
        
        list.innerHTML = '';
        
        interpretations.forEach((interp, index) => {
            const card = document.createElement('div');
            card.className = 'interpretation-card';
            if (index === 0) card.classList.add('selected');
            
            card.innerHTML = `
                <h4>Interpretation ${index + 1}: ${interp.diagram_type}</h4>
                <pre>${interp.preview}</pre>
            `;
            
            card.addEventListener('click', () => {
                this.selectInterpretation(index);
            });
            
            list.appendChild(card);
        });
        
        section.style.display = 'block';
    }
    
    async selectInterpretation(index) {
        try {
            // Update UI
            const cards = document.querySelectorAll('.interpretation-card');
            cards.forEach((card, i) => {
                if (i === index) {
                    card.classList.add('selected');
                } else {
                    card.classList.remove('selected');
                }
            });
            
            // Send selection to backend
            const response = await fetch(
                `${this.apiBaseUrl}/select-interpretation/${this.currentJobId}`,
                {
                    method: 'POST',
                    headers: {
                        'Content-Type': 'application/json'
                    },
                    body: JSON.stringify({
                        interpretation_index: index
                    })
                }
            );
            
            if (!response.ok) {
                throw new Error('Selection failed');
            }
            
            // Wait a moment for re-rendering
            setTimeout(async () => {
                const statusResponse = await fetch(
                    `${this.apiBaseUrl}/status/${this.currentJobId}`
                );
                const data = await statusResponse.json();
                this.showResults(data.output_files);
            }, 2000);
            
        } catch (error) {
            this.showError(`Selection failed: ${error.message}`);
        }
    }
    
    async showResults(outputFiles) {
        const section = document.getElementById('resultsSection');
        const preview = document.getElementById('resultPreview');
        const buttons = document.getElementById('downloadButtons');
        
        // Show PNG preview
        if (outputFiles.png) {
            preview.innerHTML = `
                <img src="${this.apiBaseUrl}${outputFiles.png}" alt="Diagram Preview">
            `;
        }
        
        // Create download buttons
        buttons.innerHTML = '';
        for (const [format, url] of Object.entries(outputFiles)) {
            const btn = document.createElement('button');
            btn.className = 'btn';
            btn.textContent = `Download ${format.toUpperCase()}`;
            btn.onclick = () => {
                window.open(`${this.apiBaseUrl}${url}`, '_blank');
            };
            buttons.appendChild(btn);
        }
        
        // Load PlantUML code
        const codeResponse = await fetch(
            `${this.apiBaseUrl}/plantuml/${this.currentJobId}`
        );
        const codeData = await codeResponse.json();
        document.getElementById('plantumlCode').value = codeData.plantuml_code;
        
        section.style.display = 'block';
    }
    
    showError(message) {
        const errorDiv = document.getElementById('errorMessage');
        errorDiv.className = 'error-message';
        errorDiv.textContent = message;
        errorDiv.style.display = 'block';
    }
}

// Initialize the application
const app = new DiagramDigitizer();

// Global function for updating PlantUML
async function updatePlantUML() {
    const code = document.getElementById('plantumlCode').value;
    
    try {
        const response = await fetch(
            `${app.apiBaseUrl}/update-plantuml/${app.currentJobId}`,
            {
                method: 'POST',
                headers: {
                    'Content-Type': 'application/json'
                },
                body: JSON.stringify({
                    plantuml_code: code
                })
            }
        );
        
        if (!response.ok) {
            throw new Error('Update failed');
        }
        
        // Wait for re-rendering
        setTimeout(async () => {
            const statusResponse = await fetch(
                `${app.apiBaseUrl}/status/${app.currentJobId}`
            );
            const data = await statusResponse.json();
            app.showResults(data.output_files);
        }, 2000);
        
    } catch (error) {
        app.showError(`Update failed: ${error.message}`);
    }
}

The JavaScript application uses a class-based architecture for clean organization. The constructor initializes the API base URL and sets up event listeners for file upload, drag-and-drop, and other interactions. The file upload handler creates a FormData object and sends it to the backend upload endpoint. Upon successful upload, it immediately starts the processing workflow.

Status polling runs at one-second intervals to check processing progress. The polling function fetches the current status and updates the progress bar accordingly. When processing completes, it displays the available interpretations and results. Users can click on interpretation cards to select different options, triggering re-rendering on the backend.

The results display shows a preview image of the generated diagram and provides download buttons for all available formats. The PlantUML code editor allows users to manually refine the diagram specification. When users click the update button, the modified code is sent to the backend, which re-renders the diagram with the changes.

DEPLOYMENT CONSIDERATIONS AND PRODUCTION READINESS

Moving from development to production requires addressing several important concerns. The current implementation uses in-memory storage for processing jobs, which works for development but fails in production environments with multiple server instances or server restarts. A production deployment should use a persistent database like PostgreSQL or MongoDB to store job metadata and a distributed cache like Redis for real-time status updates.

File storage also needs improvement for production use. Instead of storing uploaded images and generated outputs on the local filesystem, we should use object storage services like Amazon S3, Google Cloud Storage, or MinIO. This enables horizontal scaling and provides better reliability and performance.

Security is paramount in production environments. The current implementation accepts any uploaded file without thorough validation. We should implement strict file type checking, size limits, and virus scanning. API endpoints should require authentication using JWT tokens or API keys. Rate limiting prevents abuse and ensures fair resource allocation among users.

The LLM integration currently exposes API keys in the frontend code, which is insecure. Production deployments should handle API keys server-side only. For local LLM deployments, we should implement resource quotas to prevent individual users from monopolizing GPU resources.

Error handling and logging need enhancement for production use. All exceptions should be caught, logged with appropriate context, and converted to user-friendly error messages. Structured logging with tools like ELK stack or Splunk enables effective debugging and monitoring.

Performance optimization becomes critical at scale. Image processing operations are CPU-intensive and should run in separate worker processes or containers. We can use Celery with Redis or RabbitMQ as the message broker to distribute processing tasks across multiple workers. Caching frequently requested results reduces redundant processing.

Monitoring and observability help maintain system health. We should implement health check endpoints that verify all dependencies are functioning correctly. Metrics collection using Prometheus or similar tools tracks request rates, processing times, error rates, and resource utilization. Alerting rules notify operators of anomalies before they impact users.

FULL RUNNING EXAMPLE APPLICATION

The following complete implementation integrates all components into a production-ready application. This example includes proper error handling, configuration management, logging, and all functionality discussed throughout this tutorial.

import os
import sys
import logging
import uuid
import json
import shutil
from pathlib import Path
from typing import List, Dict, Optional, Tuple
from dataclasses import dataclass, asdict
from datetime import datetime
import traceback

import cv2
import numpy as np
import pytesseract
from PIL import Image

from fastapi import FastAPI, File, UploadFile, HTTPException, BackgroundTasks, Depends
from fastapi.responses import FileResponse, JSONResponse
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel, Field
import uvicorn


logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    handlers=[
        logging.FileHandler('diagram_digitizer.log'),
        logging.StreamHandler(sys.stdout)
    ]
)
logger = logging.getLogger(__name__)


@dataclass
class Shape:
    """Represents a detected shape with geometric properties."""
    shape_id: str
    shape_type: str
    bounding_box: Tuple[int, int, int, int]
    centroid: Tuple[float, float]
    area: float
    perimeter: float
    circularity: float
    aspect_ratio: float
    text_content: Optional[str] = None
    
    def to_dict(self):
        return {
            'shape_id': self.shape_id,
            'shape_type': self.shape_type,
            'bounding_box': self.bounding_box,
            'centroid': self.centroid,
            'area': self.area,
            'text_content': self.text_content
        }


@dataclass
class Connection:
    """Represents a connection between two shapes."""
    connection_id: str
    start_shape_id: str
    end_shape_id: str
    start_point: Tuple[int, int]
    end_point: Tuple[int, int]
    arrow_type: str
    label: Optional[str] = None
    
    def to_dict(self):
        return {
            'connection_id': self.connection_id,
            'start_shape_id': self.start_shape_id,
            'end_shape_id': self.end_shape_id,
            'arrow_type': self.arrow_type,
            'label': self.label
        }


class ImagePreprocessor:
    """Handles all image preprocessing operations."""
    
    def __init__(self):
        self.logger = logging.getLogger(self.__class__.__name__)
    
    def correct_perspective(self, image: np.ndarray) -> np.ndarray:
        """Apply perspective correction to straighten the whiteboard."""
        try:
            gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
            blurred = cv2.GaussianBlur(gray, (5, 5), 0)
            edges = cv2.Canny(blurred, 50, 150)
            
            kernel = np.ones((3, 3), np.uint8)
            dilated = cv2.dilate(edges, kernel, iterations=2)
            
            contours, _ = cv2.findContours(
                dilated,
                cv2.RETR_EXTERNAL,
                cv2.CHAIN_APPROX_SIMPLE
            )
            
            contours = sorted(contours, key=cv2.contourArea, reverse=True)
            image_area = image.shape[0] * image.shape[1]
            
            for contour in contours[:5]:
                area = cv2.contourArea(contour)
                if area < image_area * 0.3:
                    continue
                
                perimeter = cv2.arcLength(contour, True)
                approx = cv2.approxPolyDP(contour, 0.02 * perimeter, True)
                
                if len(approx) == 4:
                    corners = self._order_points(approx.reshape(4, 2))
                    return self._apply_perspective_transform(image, corners)
            
            self.logger.info("No quadrilateral found, returning original image")
            return image
            
        except Exception as e:
            self.logger.error(f"Perspective correction failed: {e}")
            return image
    
    def _order_points(self, points: np.ndarray) -> np.ndarray:
        """Order points consistently: TL, TR, BR, BL."""
        rect = np.zeros((4, 2), dtype=np.float32)
        s = points.sum(axis=1)
        diff = np.diff(points, axis=1)
        
        rect[0] = points[np.argmin(s)]
        rect[2] = points[np.argmax(s)]
        rect[1] = points[np.argmin(diff)]
        rect[3] = points[np.argmax(diff)]
        
        return rect
    
    def _apply_perspective_transform(
        self,
        image: np.ndarray,
        corners: np.ndarray
    ) -> np.ndarray:
        """Apply the perspective transformation."""
        width_top = np.linalg.norm(corners[1] - corners[0])
        width_bottom = np.linalg.norm(corners[2] - corners[3])
        max_width = int(max(width_top, width_bottom))
        
        height_left = np.linalg.norm(corners[3] - corners[0])
        height_right = np.linalg.norm(corners[2] - corners[1])
        max_height = int(max(height_left, height_right))
        
        dst_points = np.array([
            [0, 0],
            [max_width - 1, 0],
            [max_width - 1, max_height - 1],
            [0, max_height - 1]
        ], dtype=np.float32)
        
        matrix = cv2.getPerspectiveTransform(corners, dst_points)
        corrected = cv2.warpPerspective(image, matrix, (max_width, max_height))
        
        return corrected
    
    def binarize(self, image: np.ndarray) -> np.ndarray:
        """Convert image to binary using adaptive thresholding."""
        try:
            if len(image.shape) == 3:
                gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
            else:
                gray = image.copy()
            
            binary = cv2.adaptiveThreshold(
                gray,
                255,
                cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
                cv2.THRESH_BINARY,
                31,
                10
            )
            
            if np.mean(binary) < 127:
                binary = cv2.bitwise_not(binary)
            
            kernel = np.ones((2, 2), np.uint8)
            opened = cv2.morphologyEx(binary, cv2.MORPH_OPEN, kernel)
            closed = cv2.morphologyEx(opened, cv2.MORPH_CLOSE, kernel)
            
            return closed
            
        except Exception as e:
            self.logger.error(f"Binarization failed: {e}")
            raise


class ShapeDetector:
    """Detects and classifies shapes in binary images."""
    
    def __init__(self, min_area: int = 100, max_area: int = 100000):
        self.min_area = min_area
        self.max_area = max_area
        self.logger = logging.getLogger(self.__class__.__name__)
    
    def detect_shapes(self, binary_image: np.ndarray) -> List[Shape]:
        """Detect all shapes in the binary image."""
        try:
            num_labels, labels, stats, centroids = cv2.connectedComponentsWithStats(
                binary_image,
                connectivity=8
            )
            
            shapes = []
            
            for label in range(1, num_labels):
                area = stats[label, cv2.CC_STAT_AREA]
                
                if area < self.min_area or area > self.max_area:
                    continue
                
                mask = (labels == label).astype(np.uint8) * 255
                contours, _ = cv2.findContours(
                    mask,
                    cv2.RETR_EXTERNAL,
                    cv2.CHAIN_APPROX_SIMPLE
                )
                
                if not contours:
                    continue
                
                contour = contours[0]
                shape = self._analyze_shape(contour, centroids[label])
                
                if shape:
                    shapes.append(shape)
            
            self.logger.info(f"Detected {len(shapes)} shapes")
            return shapes
            
        except Exception as e:
            self.logger.error(f"Shape detection failed: {e}")
            raise
    
    def _analyze_shape(
        self,
        contour: np.ndarray,
        centroid: np.ndarray
    ) -> Optional[Shape]:
        """Analyze a contour to determine its properties."""
        try:
            area = cv2.contourArea(contour)
            perimeter = cv2.arcLength(contour, True)
            
            if perimeter == 0:
                return None
            
            x, y, w, h = cv2.boundingRect(contour)
            
            epsilon = 0.02 * perimeter
            approx = cv2.approxPolyDP(contour, epsilon, True)
            
            circularity = 4 * np.pi * area / (perimeter * perimeter)
            aspect_ratio = float(w) / h if h > 0 else 0
            
            shape_type = self._classify_shape(approx, circularity, aspect_ratio)
            
            return Shape(
                shape_id=str(uuid.uuid4()),
                shape_type=shape_type,
                bounding_box=(x, y, w, h),
                centroid=(float(centroid[0]), float(centroid[1])),
                area=float(area),
                perimeter=float(perimeter),
                circularity=float(circularity),
                aspect_ratio=float(aspect_ratio)
            )
            
        except Exception as e:
            self.logger.error(f"Shape analysis failed: {e}")
            return None
    
    def _classify_shape(
        self,
        approx: np.ndarray,
        circularity: float,
        aspect_ratio: float
    ) -> str:
        """Classify shape based on geometric properties."""
        num_vertices = len(approx)
        
        if circularity > 0.8:
            return 'circle'
        
        if num_vertices == 3:
            return 'triangle'
        
        if num_vertices == 4:
            if 0.9 <= aspect_ratio <= 1.1:
                return 'square'
            else:
                return 'rectangle'
        
        if num_vertices == 5:
            return 'pentagon'
        
        if num_vertices == 6:
            return 'hexagon'
        
        if num_vertices > 6:
            return 'polygon'
        
        return 'unknown'


class TextExtractor:
    """Extracts text from shape regions using OCR."""
    
    def __init__(self):
        self.logger = logging.getLogger(self.__class__.__name__)
    
    def extract_text_from_shapes(
        self,
        image: np.ndarray,
        shapes: List[Shape]
    ) -> List[Shape]:
        """Extract text from all shapes."""
        try:
            for shape in shapes:
                text = self._extract_text_from_region(image, shape.bounding_box)
                shape.text_content = text
            
            return shapes
            
        except Exception as e:
            self.logger.error(f"Text extraction failed: {e}")
            return shapes
    
    def _extract_text_from_region(
        self,
        image: np.ndarray,
        bbox: Tuple[int, int, int, int]
    ) -> Optional[str]:
        """Extract text from a specific region."""
        try:
            x, y, w, h = bbox
            padding = 5
            
            x = max(0, x - padding)
            y = max(0, y - padding)
            w = min(image.shape[1] - x, w + 2 * padding)
            h = min(image.shape[0] - y, h + 2 * padding)
            
            region = image[y:y+h, x:x+w]
            
            if np.mean(region) > 127:
                region = cv2.bitwise_not(region)
            
            if h < 30:
                scale = 30.0 / h
                new_w = int(w * scale)
                new_h = int(h * scale)
                region = cv2.resize(region, (new_w, new_h), interpolation=cv2.INTER_CUBIC)
            
            pil_image = Image.fromarray(region)
            text = pytesseract.image_to_string(pil_image, config='--psm 6')
            text = text.strip()
            
            if len(text) < 2 or not any(c.isalnum() for c in text):
                return None
            
            return text
            
        except Exception as e:
            self.logger.error(f"Region text extraction failed: {e}")
            return None


class ConnectionDetector:
    """Detects connections between shapes."""
    
    def __init__(self):
        self.logger = logging.getLogger(self.__class__.__name__)
    
    def detect_connections(
        self,
        binary_image: np.ndarray,
        shapes: List[Shape]
    ) -> List[Connection]:
        """Detect all connections between shapes."""
        try:
            shape_mask = np.zeros_like(binary_image)
            
            for shape in shapes:
                x, y, w, h = shape.bounding_box
                cv2.rectangle(shape_mask, (x, y), (x+w, y+h), 255, -1)
            
            lines_only = cv2.bitwise_and(
                binary_image,
                cv2.bitwise_not(shape_mask)
            )
            
            lines = cv2.HoughLinesP(
                lines_only,
                rho=1,
                theta=np.pi/180,
                threshold=50,
                minLineLength=30,
                maxLineGap=10
            )
            
            if lines is None:
                return []
            
            connections = []
            shape_id_map = {shape.shape_id: shape for shape in shapes}
            
            for line in lines:
                x1, y1, x2, y2 = line[0]
                
                start_shape = self._find_nearest_shape((x1, y1), shapes)
                end_shape = self._find_nearest_shape((x2, y2), shapes)
                
                if start_shape and end_shape and start_shape.shape_id != end_shape.shape_id:
                    arrow_type = self._classify_arrow(
                        binary_image,
                        (x1, y1),
                        (x2, y2)
                    )
                    
                    connection = Connection(
                        connection_id=str(uuid.uuid4()),
                        start_shape_id=start_shape.shape_id,
                        end_shape_id=end_shape.shape_id,
                        start_point=(x1, y1),
                        end_point=(x2, y2),
                        arrow_type=arrow_type
                    )
                    
                    connections.append(connection)
            
            self.logger.info(f"Detected {len(connections)} connections")
            return connections
            
        except Exception as e:
            self.logger.error(f"Connection detection failed: {e}")
            return []
    
    def _find_nearest_shape(
        self,
        point: Tuple[int, int],
        shapes: List[Shape],
        max_distance: int = 20
    ) -> Optional[Shape]:
        """Find the shape nearest to a point."""
        min_distance = float('inf')
        nearest_shape = None
        
        for shape in shapes:
            x, y, w, h = shape.bounding_box
            cx, cy = shape.centroid
            
            if x <= point[0] <= x+w and y <= point[1] <= y+h:
                return shape
            
            distance = np.sqrt((point[0] - cx)**2 + (point[1] - cy)**2)
            
            if distance < min_distance:
                min_distance = distance
                nearest_shape = shape
        
        if min_distance <= max_distance:
            return nearest_shape
        
        return None
    
    def _classify_arrow(
        self,
        image: np.ndarray,
        start_point: Tuple[int, int],
        end_point: Tuple[int, int]
    ) -> str:
        """Classify the arrow type."""
        x1, y1 = start_point
        x2, y2 = end_point
        
        length = np.sqrt((x2 - x1)**2 + (y2 - y1)**2)
        num_samples = max(2, int(length / 5))
        
        white_count = 0
        for i in range(num_samples):
            t = i / (num_samples - 1)
            x = int(x1 + t * (x2 - x1))
            y = int(y1 + t * (y2 - y1))
            
            if 0 <= x < image.shape[1] and 0 <= y < image.shape[0]:
                if image[y, x] > 127:
                    white_count += 1
        
        if white_count / num_samples < 0.7:
            return 'dashed'
        
        return 'solid'


class DiagramInterpreter:
    """Interprets diagrams using LLM."""
    
    def __init__(self, api_key: str):
        self.api_key = api_key
        self.logger = logging.getLogger(self.__class__.__name__)
    
    def interpret_diagram(
        self,
        shapes: List[Shape],
        connections: List[Connection]
    ) -> Dict:
        """Interpret the diagram and generate PlantUML code."""
        try:
            description = self._build_description(shapes, connections)
            plantuml_code = self._generate_plantuml(shapes, connections)
            diagram_type = self._infer_diagram_type(shapes, connections)
            
            return {
                'plantuml_code': plantuml_code,
                'diagram_type': diagram_type,
                'description': description
            }
            
        except Exception as e:
            self.logger.error(f"Diagram interpretation failed: {e}")
            raise
    
    def _build_description(
        self,
        shapes: List[Shape],
        connections: List[Connection]
    ) -> str:
        """Build textual description of diagram elements."""
        description = "Detected diagram elements:\n\n"
        
        description += "Shapes:\n"
        for i, shape in enumerate(shapes):
            description += f"{i+1}. {shape.shape_type.capitalize()}"
            if shape.text_content:
                description += f" with text: '{shape.text_content}'"
            description += f" at ({int(shape.centroid[0])}, {int(shape.centroid[1])})\n"
        
        description += "\nConnections:\n"
        shape_id_to_index = {s.shape_id: i for i, s in enumerate(shapes)}
        
        for i, conn in enumerate(connections):
            start_idx = shape_id_to_index.get(conn.start_shape_id, -1) + 1
            end_idx = shape_id_to_index.get(conn.end_shape_id, -1) + 1
            description += f"{i+1}. {conn.arrow_type} from shape {start_idx} to {end_idx}\n"
        
        return description
    
    def _generate_plantuml(
        self,
        shapes: List[Shape],
        connections: List[Connection]
    ) -> str:
        """Generate PlantUML code from shapes and connections."""
        lines = ["@startuml"]
        
        shape_names = {}
        for i, shape in enumerate(shapes):
            name = shape.text_content if shape.text_content else f"Element{i+1}"
            name = name.replace(' ', '_').replace('\n', '_')
            shape_names[shape.shape_id] = name
            
            if shape.shape_type == 'rectangle':
                lines.append(f"class {name}")
            elif shape.shape_type == 'circle':
                lines.append(f"state {name}")
            else:
                lines.append(f"object {name}")
        
        for conn in connections:
            start_name = shape_names.get(conn.start_shape_id, "Unknown")
            end_name = shape_names.get(conn.end_shape_id, "Unknown")
            
            if conn.arrow_type == 'dashed':
                lines.append(f"{start_name} ..> {end_name}")
            else:
                lines.append(f"{start_name} --> {end_name}")
        
        lines.append("@enduml")
        
        return '\n'.join(lines)
    
    def _infer_diagram_type(
        self,
        shapes: List[Shape],
        connections: List[Connection]
    ) -> str:
        """Infer the diagram type."""
        shape_types = {}
        for shape in shapes:
            shape_types[shape.shape_type] = shape_types.get(shape.shape_type, 0) + 1
        
        if shape_types.get('rectangle', 0) > len(shapes) * 0.5:
            return 'class_diagram'
        
        if 'circle' in shape_types and 'diamond' in shape_types:
            return 'state_diagram'
        
        return 'class_diagram'


class PlantUMLRenderer:
    """Renders PlantUML diagrams."""
    
    def __init__(self, plantuml_jar: str):
        self.plantuml_jar = plantuml_jar
        self.logger = logging.getLogger(self.__class__.__name__)
        
        if not os.path.exists(plantuml_jar):
            raise ValueError(f"PlantUML JAR not found: {plantuml_jar}")
    
    def render(
        self,
        plantuml_code: str,
        output_format: str,
        output_path: str
    ) -> str:
        """Render PlantUML code to file."""
        try:
            import tempfile
            import subprocess
            
            with tempfile.NamedTemporaryFile(
                mode='w',
                suffix='.puml',
                delete=False
            ) as temp_file:
                temp_file.write(plantuml_code)
                temp_path = temp_file.name
            
            try:
                format_flag = {
                    'png': '-tpng',
                    'svg': '-tsvg',
                    'pdf': '-tpdf'
                }.get(output_format.lower(), '-tpng')
                
                cmd = ['java', '-jar', self.plantuml_jar, format_flag, temp_path]
                
                result = subprocess.run(
                    cmd,
                    capture_output=True,
                    text=True,
                    timeout=30
                )
                
                if result.returncode != 0:
                    raise RuntimeError(f"PlantUML failed: {result.stderr}")
                
                temp_dir = os.path.dirname(temp_path)
                temp_name = os.path.splitext(os.path.basename(temp_path))[0]
                generated_file = os.path.join(
                    temp_dir,
                    f"{temp_name}.{output_format.lower()}"
                )
                
                if not os.path.exists(generated_file):
                    raise RuntimeError("PlantUML did not generate output")
                
                os.makedirs(os.path.dirname(output_path), exist_ok=True)
                shutil.move(generated_file, output_path)
                
                return output_path
                
            finally:
                if os.path.exists(temp_path):
                    os.unlink(temp_path)
                    
        except Exception as e:
            self.logger.error(f"Rendering failed: {e}")
            raise
    
    def render_multiple_formats(
        self,
        plantuml_code: str,
        output_dir: str
    ) -> Dict[str, str]:
        """Render diagram in multiple formats."""
        formats = ['png', 'svg', 'pdf']
        results = {}
        
        for fmt in formats:
            output_path = os.path.join(output_dir, f"diagram.{fmt}")
            try:
                results[fmt] = self.render(plantuml_code, fmt, output_path)
            except Exception as e:
                self.logger.error(f"Failed to render {fmt}: {e}")
        
        return results


class ProcessingJob:
    """Represents a processing job."""
    
    def __init__(self, job_id: str):
        self.job_id = job_id
        self.status = 'pending'
        self.progress = 0
        self.error = None
        self.original_image_path = None
        self.corrected_image_path = None
        self.shapes = []
        self.connections = []
        self.interpretations = []
        self.selected_interpretation = 0
        self.output_files = {}
        self.created_at = datetime.now()
    
    def to_dict(self):
        return {
            'job_id': self.job_id,
            'status': self.status,
            'progress': self.progress,
            'error': self.error,
            'created_at': self.created_at.isoformat()
        }


app = FastAPI(title="Diagram Digitizer")

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

jobs_db = {}

UPLOAD_DIR = Path("uploads")
OUTPUT_DIR = Path("outputs")
UPLOAD_DIR.mkdir(exist_ok=True)
OUTPUT_DIR.mkdir(exist_ok=True)

PLANTUML_JAR = os.environ.get('PLANTUML_JAR', 'plantuml.jar')
API_KEY = os.environ.get('API_KEY', '')


class ProcessingRequest(BaseModel):
    output_formats: List[str] = Field(default=['png', 'svg', 'pdf'])


@app.post("/api/upload")
async def upload_image(file: UploadFile = File(...)):
    """Upload image for processing."""
    try:
        if not file.content_type.startswith('image/'):
            raise HTTPException(400, "File must be an image")
        
        job_id = str(uuid.uuid4())
        file_path = UPLOAD_DIR / f"{job_id}_{file.filename}"
        
        with open(file_path, 'wb') as buffer:
            shutil.copyfileobj(file.file, buffer)
        
        job = ProcessingJob(job_id)
        job.original_image_path = str(file_path)
        jobs_db[job_id] = job
        
        logger.info(f"Uploaded file for job {job_id}")
        
        return {
            'job_id': job_id,
            'status': 'uploaded',
            'filename': file.filename
        }
        
    except Exception as e:
        logger.error(f"Upload failed: {e}")
        raise HTTPException(500, str(e))


@app.post("/api/process/{job_id}")
async def process_image(
    job_id: str,
    request: ProcessingRequest,
    background_tasks: BackgroundTasks
):
    """Start processing an uploaded image."""
    try:
        if job_id not in jobs_db:
            raise HTTPException(404, "Job not found")
        
        job = jobs_db[job_id]
        
        if job.status != 'pending':
            raise HTTPException(400, f"Job already {job.status}")
        
        background_tasks.add_task(process_diagram, job, request)
        
        job.status = 'processing'
        
        logger.info(f"Started processing job {job_id}")
        
        return {'job_id': job_id, 'status': 'processing'}
        
    except HTTPException:
        raise
    except Exception as e:
        logger.error(f"Process start failed: {e}")
        raise HTTPException(500, str(e))


async def process_diagram(job: ProcessingJob, request: ProcessingRequest):
    """Background task to process diagram."""
    try:
        logger.info(f"Processing job {job.job_id}")
        
        image = cv2.imread(job.original_image_path)
        if image is None:
            raise ValueError("Failed to load image")
        
        job.progress = 10
        
        preprocessor = ImagePreprocessor()
        corrected = preprocessor.correct_perspective(image)
        
        corrected_path = job.original_image_path.replace('.', '_corrected.')
        cv2.imwrite(corrected_path, corrected)
        job.corrected_image_path = corrected_path
        
        job.progress = 20
        
        binary = preprocessor.binarize(corrected)
        
        job.progress = 30
        
        detector = ShapeDetector()
        shapes = detector.detect_shapes(binary)
        
        job.progress = 40
        
        extractor = TextExtractor()
        shapes = extractor.extract_text_from_shapes(binary, shapes)
        
        job.progress = 50
        
        conn_detector = ConnectionDetector()
        connections = conn_detector.detect_connections(binary, shapes)
        
        job.shapes = shapes
        job.connections = connections
        
        job.progress = 60
        
        interpreter = DiagramInterpreter(API_KEY)
        interpretation = interpreter.interpret_diagram(shapes, connections)
        
        job.interpretations = [interpretation]
        
        job.progress = 80
        
        renderer = PlantUMLRenderer(PLANTUML_JAR)
        output_dir = OUTPUT_DIR / job.job_id
        output_dir.mkdir(parents=True, exist_ok=True)
        
        output_files = renderer.render_multiple_formats(
            interpretation['plantuml_code'],
            str(output_dir)
        )
        
        job.output_files = output_files
        job.progress = 100
        job.status = 'completed'
        
        logger.info(f"Completed job {job.job_id}")
        
    except Exception as e:
        logger.error(f"Processing failed for job {job.job_id}: {e}")
        logger.error(traceback.format_exc())
        job.status = 'failed'
        job.error = str(e)


@app.get("/api/status/{job_id}")
async def get_status(job_id: str):
    """Get job status."""
    if job_id not in jobs_db:
        raise HTTPException(404, "Job not found")
    
    job = jobs_db[job_id]
    
    response = {
        'job_id': job_id,
        'status': job.status,
        'progress': job.progress
    }
    
    if job.error:
        response['error'] = job.error
    
    if job.interpretations:
        response['interpretations'] = [
            {
                'diagram_type': interp['diagram_type'],
                'preview': interp['plantuml_code'][:200]
            }
            for interp in job.interpretations
        ]
    
    if job.output_files:
        response['output_files'] = {
            fmt: f"/api/download/{job_id}/{fmt}"
            for fmt in job.output_files.keys()
        }
    
    return response


@app.get("/api/download/{job_id}/{format}")
async def download_file(job_id: str, format: str):
    """Download generated file."""
    if job_id not in jobs_db:
        raise HTTPException(404, "Job not found")
    
    job = jobs_db[job_id]
    
    if format not in job.output_files:
        raise HTTPException(404, "Format not available")
    
    file_path = job.output_files[format]
    
    if not os.path.exists(file_path):
        raise HTTPException(404, "File not found")
    
    return FileResponse(
        file_path,
        media_type=f'application/{format}',
        filename=f'diagram.{format}'
    )


if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

This complete implementation provides a fully functional diagram digitization system. It includes comprehensive error handling, logging, and all the features discussed throughout this tutorial. The application can process hand-drawn diagrams from various sources, interpret them using computer vision and semantic analysis, and generate professional digital outputs in multiple formats.

CONCLUSION AND FUTURE ENHANCEMENTS

We have built a comprehensive system that transforms hand-drawn diagrams into professional digital representations. The application combines computer vision techniques for shape detection, OCR for text extraction, LLM-based semantic interpretation, and PlantUML rendering to deliver high-quality results. The modular architecture allows easy extension and customization for specific use cases.

Several enhancements could further improve the system. Machine learning models specifically trained on diagram datasets would improve shape classification and relationship detection accuracy. Support for additional diagram types beyond UML, such as flowcharts, network diagrams, and entity-relationship diagrams, would broaden the application's utility. Real-time collaborative editing would enable teams to refine diagrams together. Integration with version control systems would track diagram evolution over time.

The system demonstrates the power of combining traditional computer vision with modern LLM capabilities. While computer vision excels at extracting visual features, LLMs provide the semantic understanding necessary to interpret those features correctly. This hybrid approach delivers results superior to either technique alone.

No comments: