👁️ Visual Layout Parsing

SLM Vision Parser

Extract text nodes, bounding boxes, chart labels, and table grids locally on CPU using Microsoft's MIT-licensed Florence-2-large model.

🚀 Overview & Capabilities

The SLM Vision Parser uses the sequence-to-sequence Florence-2-large model to translate image structures directly. It is designed to run offline on CPU under 1.5 GB memory footprint, making it ideal for visual PDF OCR, diagram relationship mappings, and whiteboard extractions.

💻 Installation

Install the local CPU-optimized package using pip:

Terminal

              # Install from PyPI
pip install slm-vision-parser
            

🤖 Truly Agentic Workflow

The Vision Parser maps visual coordinate dimensions into clean layout logs:

OCR Node Extraction: Translates raster text strings from diagrams.
Region Mapping: Generates bounding box parameters mapping visual coordinates for web rendering.
Semantic Image Captioning: Extracts visual flow structures (e.g., arrows, shapes, labels) into a raw description.
Agentic LLM Refinement: Pipe Florence-2's raw description or OCR nodes directly into the local Phi-3.5 ONNX model. By guiding the text model with a template prompt, you can synthesize exact custom descriptions (e.g., describing step hierarchies and the exact count of connecting arrows).

⚡ CPU Performance Tuning Guidelines

Florence-2 utilizes complex visual transformers. To run model prediction tasks efficiently on CPU under 2.0 GB memory:

Sequential execution: Process page scans sequentially rather than in batches to avoid CPU memory thrashing.
Garbage Collection: Clear cache blocks after processing high-resolution images:
```
import gc
gc.collect()
```

🎯 Accuracy Improvement Tips

Tip for Scanned Diagrams: Contrast is key for visual OCR models. Preprocess low-contrast diagram images by resizing them to a standard width of 1024px and scaling contrast values using Pillow before parsing.

Task Tag Wrapper: Always encapsulate Florence-2 model instructions in explicit brackets (e.g., "<OCR>" or "<DETAILED_CAPTION>") to trigger correct task headers.

API Reference

`SLMVisionParser` Initialization

from slm_vision_parser.vision_parser import SLMVisionParser

parser = SLMVisionParser()

Parameter	Type	Description
model_path	str	Local directory containing pre-downloaded Florence-2 checkpoints. Defaults to "../../models/florence-2-large".
system_prompt	str \| None	Optional custom system prompt instructions overriding the default template.
user_input	str \| None	Optional additional user-supplied target parameters or variables.

`parse_image` Method

Run coordinate mappings or caption generation by supplying task tags. Below is the input flowchart diagram processed by the vision model:

from slm_vision_parser.vision_parser import SLMVisionParser

parser = SLMVisionParser()

# Describe the flowchart diagram in text using the local vision LLM
result = parser.parse_image("flowchart.png", task="<DETAILED_CAPTION>")
print(result)

Image Description Response Output:

"A flowchart showing a start step ('Start Process') and a next step ('Next Step Link') connected with two arrows from the start step to the next step."

🐙 Checkout from GitHub

Clone only this agent's folder from the monorepo using Git sparse-checkout — no need to download the full repository:

Option 1 — Sparse Checkout (Recommended)

Terminal — Git Sparse Checkout

              # 1. Create and enter a new directory
$ mkdir slm_vision_parser && cd slm_vision_parser

# 2. Initialise empty git repo and add remote
$ git init
$ git remote add origin https://github.com/t00114218-stack/SLMAgents.git

# 3. Enable sparse-checkout and set target folder
$ git sparse-checkout init --cone
$ git sparse-checkout set slm_vision_parser

# 4. Pull only that agent's source
$ git pull origin main
            

Option 2 — Full Repository Clone

Terminal — Full Clone

              $ git clone https://github.com/t00114218-stack/SLMAgents.git
$ cd SLMAgents/slm_vision_parser
            

💡 Tip: After checkout, install the package locally with pip install -e ./slm_vision_parser to run in editable mode without publishing to PyPI.