[Bug]: background-images seems to be skipped

[dpalic]

crawl4ai version

0.7.8

Expected Behavior

I am having multiple pages which images used as background images. The thing is they get skipped.
Any decisions how to handle all the possible background images at all?

Current Behavior

background images are not reliably scraped and reported

Is this reproducible?

Yes

Inputs Causing the Bug

https://bader-kulturreisen.de/reisen/winter-glacier-bernina-express-2026/

Steps to Reproduce

Code snippets

OS

linux

Python version

3.10.12

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response

[SohamKukreti]

Root Cause Analysis: Incomplete Media Extraction (Images and Videos)

Issue Summary

The crawler is missing background images and some videos when extracting media from pages that use modern CSS-based background images and data attributes (e.g., Elementor page builders, modern React/Vue sites).

Affected URL

• Test URL: https://bader-kulturreisen.de/reisen/winter-glacier-bernina-express-2026/
• Expected: Extract all background images from data-dce-background-image-url attributes and all video elements
• Actual: Only extracts <img> tags and standard <video> elements with direct src attributes

Root Cause

Primary Issue: Background Images Not Extracted

The current implementation in content_scraping_strategy.py only extracts images from <img> tags:

# Line 314: Only searches for <img> elements
images = element.xpath(".//img")

Missing patterns:

1. CSS Background Images: Images set via style="background-image: url(...)"
2. Data Attributes: Images stored in custom data attributes like:
   • data-dce-background-image-url (Elementor/Dynamic Content)
   • data-background-image
   • data-bg
   • Other framework-specific attributes

Example from affected site:

<div data-dce-background-image-url="https://bader-kulturreisen.de/wp-content/uploads/reisen/glacierExpress/winterreise/Glacier-im-Schnee.jpg" 
     class="elementor-element ...">

These images are not extracted because they're not in <img> tags.

Secondary Issue: Video Extraction Limitations

The video extraction logic (lines 343-359) has gaps:

1. Limited Attribute Checking: Only checks elem.get("src") - doesn't check data-src, data-video-src, or other lazy-loading attributes
2. Source Tag Handling: While it processes <source> tags, the logic doesn't handle cases where:
   • The main video element has no src but only <source> children
   • Multiple source tags need proper deduplication
   • Videos are in data attributes

Example from affected site:

<video class="elementor-background-video-hosted" 
       src="https://bader-kulturreisen.de/wp-content/uploads/reisen/glacierExpress/winterreise/Winter-Glacier-Express-Schweiz03.mp4">

This should be captured, but videos with alternative loading patterns may be missed.

Currently Extracted:

• Standard <img> tags (logo images) ✅
• Videos with direct src attributes ✅
• Videos with <source> tags ✅

Missing:

• Background images in data attributes ❌
• Background images in CSS style attributes ❌
• Videos with data-src or lazy-loading patterns (if any) ❌

Code Location

File: crawl4ai/content_scraping_strategy.py

Lines:

• Image extraction: Lines 313-341
• Video/audio extraction: Lines 343-359

Impact

• Severity: Medium
• Affected Sites: Modern CMS/page builders (WordPress/Elementor, Webflow, Framer, etc.)
• User Experience: Incomplete media extraction leads to:
  • Missing visual content in scraped data
  • Incomplete media galleries
  • Loss of important background imagery used for design/content

Recommended Solution Approach

1. Add Background Image Extraction:

   • Search all elements (not just <img>) for:
     • Common data attributes: data-dce-background-image-url, data-background-image, data-bg, etc.
     • CSS background-image in style attributes (regex: background-image\s*:\s*url\(["']?([^"']+)["']?\))
   • Normalize and validate URLs
   • Apply same filtering/exclusion logic as regular images

2. Enhance Video Extraction:

   • Check multiple attributes: src, data-src, data-video-src
   • Better handling of <source> tags when main element has no src
   • Proper deduplication