What It Does

This tool crawls a website and downloads:

• All HTML pages
• CSS files (including referenced assets)
• JavaScript files
• Images (PNG, JPG, SVG, etc.)
• Fonts
• Media files
• Internal linked pages (based on depth setting)

It then rewrites links so everything works locally without internet access.

Key Features

Complete Website Backup

Downloads full site structure including all static assets.

Offline Viewing

Built-in local HTTP server lets you view the website exactly as if it were live.

Link Rewriting Engine

• Converts absolute URLs to relative paths
• Ensures internal navigation works offline
• Handles CSS asset references

JavaScript Rendering Support

Optional Selenium integration to capture dynamically loaded content.

Graphical User Interface

• Clean Windows GUI built with tkinter
• No command-line knowledge required
• Simple input fields and controls

Progress Tracking

• Real-time progress bar
• Detailed log window
• Error reporting with retry logic

System Requirements

• Windows OS
• Python 3.7 or higher
• Required Python packages:
• requests
• beautifulsoup4
• urllib3
• selenium (optional for JS-heavy sites)

How It Works

1. Enter the target website URL.
2. Choose an output folder.
3. Set crawl depth (default: 2).
4. Enable JavaScript rendering if needed.
5. Click “Start Scraping”.

The tool will:

• Crawl pages
• Download assets
• Rewrite links
• Save everything in a structured directory

When complete, click “View Offline” to launch the local server and browse the site offline.

JavaScript-Heavy Website Support

For websites that load content dynamically:

• Enable “Wait for JavaScript”
• Set custom wait time
• Requires Chrome + ChromeDriver
• Uses Selenium for rendering

This captures dynamic content that normal scrapers would miss.

Output Structure

The saved backup mirrors the website’s structure:

scraped_websites/

└── domain.com/

├── index.html

├── css/

├── js/

├── images/

└── ...

Use Cases

• Website backup before redesign
• Archiving client projects
• Offline demos
• Research & analysis
• Development sandbox copies
• Security auditing preparation
• Compliance archiving

Advantages Over Basic Downloaders

• GUI-based (no technical expertise required)
• Built-in local server
• Handles CSS asset extraction
• Optional JavaScript rendering
• Structured and organized output
• Smart link rewriting

Important Notes

• Always respect robots.txt and website terms of service.
• Some external APIs cannot function offline.
• Very large websites may require significant disk space.
• Dynamic third-party integrations may not fully work offline.

System Requirements – Website Scraper

Before running the Website Scraper application, make sure your system meets the following requirements.

1. Python Installation

• Python 3.7 or newer (recommended: latest stable version)
• tkinter (required for the graphical interface)

Notes:

• On Windows/macOS, tkinter is usually bundled with Python.
• On Linux, install tkinter manually:

sudo apt install python3-tk

2. Required Python Packages

From the project folder, run:

-> pip install -r requirements.txt

Or install manually:

pip install requests beautifulsoup4 urllib3 selenium

Required Versions

• requests ≥ 2.31.0
• beautifulsoup4 ≥ 4.12.0
• urllib3 ≥ 2.0.0
• selenium ≥ 4.15.0 (optional – only for JavaScript rendering)

3. Optional – JavaScript Rendering Support

If you do not enable “Wait for JavaScript” in the application, no additional setup is required.

If you do enable JavaScript rendering:

• Google Chrome must be installed
• ChromeDriver must be installed and:
• Available in your system PATH
• Or compatible with your Chrome version

Note: Selenium 4 can automatically manage ChromeDriver in many cases.

4. How to Run the Application

From the project directory:

--> python website_scraper.py

The graphical interface will open.

You can then:

• Enter a website URL
• Choose output folder
• Set crawl depth
• Enable optional JavaScript rendering
• Start scraping

Quick Summary

Minimum required:

• Python 3
• tkinter
• Required Python packages

For JavaScript-heavy websites:

• Google Chrome
• ChromeDriver (if Selenium does not auto-detect)