🔍 Layout Sanitization

SLM Web Scraper

Extract structured JSON directly from dirty, obfuscated HTML layouts offline using BeautifulSoup and Phi-3.5-mini.

🚀 Overview & Capabilities

The SLM Web Scraper parses raw HTML strings, stripping scripts, styles, duplicate nav tags, and ads. The sanitized flat content is compiled locally into structured formats using target JSON schemas.

💻 Installation

Install the local CPU-optimized package using pip:

Terminal

              # Install from PyPI
pip install slm-web-scraper
            

🤖 Truly Agentic Workflow

The Web Scraper filters raw DOM elements to produce clean extractions:

DOM Pruning: Automatically strips scripts, styles, metadata tags, and ad headers using BeautifulSoup.
Visual Image OCR: Identifies image tags (<img>) on the page, downloads them, and runs them through the local SLMVisionParser to generate natural language text descriptions, replacing the tags with the descriptions.
Whitespaces Compacting: Collapses contiguous whitespace blocks into single space delimiters.
Schema Alignment: Instructs the local Phi-3.5 model to match and structure the sanitized text into clean JSON.

⚡ CPU Performance Tuning Guidelines

Follow these configuration rules to maximize scraping speed on standard hardware:

Sanitize First: Always clean webpage source code before parsing. Running raw HTML through the model increases token count and latency.
Memory Management: Clear page memory scopes after heavy parsing cycles to keep footprint under 1.5 GB RAM.

🎯 Accuracy Improvement Tips

Tip for Layout Cleaning: Condensing tables or pricing matrices into columnar string lists (e.g. "Item 1 | Price 1") preserves formatting structure for the model, improving data extraction accuracy.

Keep Schema Flat: Keep the target schema flat. Deep nesting can confuse smaller models, causing schema validation errors.

API Reference

`SLMWebScraper` Initialization

from slm_web_scraper.web_scraper import SLMWebScraper

scraper = SLMWebScraper()

Parameter	Type	Description
model_path	str	Local path to Phi-3.5 weights. Defaults to "../../models/phi-3.5-mini-instruct-onnx".
system_prompt	str \| None	Optional custom system prompt instructions overriding the default template.
user_input	str \| None	Optional additional user-supplied target parameters or variables.

`scrape_url` Method

Fetches a URL, filters out header/footer menus, dropdown lists, sidebars, and ads, and automatically parses tables and image content:

Hybrid Visual Scraping (Tables & Images)

This example scrapes the vision parser documentation page https://www.slmagents.ai/vision_parser.html, describing its parameters table and flowchart image automatically:

from slm_web_scraper.web_scraper import SLMWebScraper

scraper = SLMWebScraper()

# Scrapes page and converts tables/images to descriptions automatically
clean_text = scraper.scrape_url("https://www.slmagents.ai/vision_parser.html")
print(clean_text)

Cleaned Webpage Text Output with Visual Descriptions:

SLM Vision Parser | Documentation
Overview
The SLM Vision Parser uses the sequence-to-sequence Florence-2-large model to translate image structures directly.

API Reference
`SLMVisionParser` Initialization
from slm_vision_parser.vision_parser import SLMVisionParser
parser = SLMVisionParser()

[Table Description: The HTML table represents the constructor parameter configuration for SLMVisionParser, detailing the "model_path" parameter of type string (str), which references the local directory containing pre-downloaded checkpoints and defaults to "../../models/florence-2-large".]

`parse_image` Method
Run coordinate mappings or caption generation by supplying task tags. Below is the input flowchart diagram processed by the vision model:
[Image Description: A flowchart showing a start step ('Start Process') and a next step ('Next Step Link') connected with two arrows from the start step to the next step.]

🐙 Checkout from GitHub

Clone only this agent's folder from the monorepo using Git sparse-checkout — no need to download the full repository:

Option 1 — Sparse Checkout (Recommended)

Terminal — Git Sparse Checkout

              # 1. Create and enter a new directory
$ mkdir slm_web_scraper && cd slm_web_scraper

# 2. Initialise empty git repo and add remote
$ git init
$ git remote add origin https://github.com/t00114218-stack/SLMAgents.git

# 3. Enable sparse-checkout and set target folder
$ git sparse-checkout init --cone
$ git sparse-checkout set slm_web_scraper

# 4. Pull only that agent's source
$ git pull origin main
            

Option 2 — Full Repository Clone

Terminal — Full Clone

              $ git clone https://github.com/t00114218-stack/SLMAgents.git
$ cd SLMAgents/slm_web_scraper
            

💡 Tip: After checkout, install the package locally with pip install -e ./slm_web_scraper to run in editable mode without publishing to PyPI.