[Bug]: scroll_delay config ignored in take_screenshot_scroller() — hardcoded 0.01s delay between scroll segments

[mkaesler44]

crawl4ai version

0.7.8

Expected Behavior

When performing full page screenshots with a scroll_delay, the scrolls between screenshots should respect the scroll_delay parameter and wait that amount of time before screenshotting

Current Behavior

Currently there is a hardcoded asyncio.sleep(0.01) between each of the screenshots in the take_screenshot_scroller() function, thus each screenshot only waits 0.01 seconds before scrolling for the next screenshot

Is this reproducible?

Yes

Inputs Causing the Bug

Steps to Reproduce

Code snippets

OS

macOS

Python version

3.12.0

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response

[Ahmed-Tawfik94]

Root Cause Analysis

• Primary cause:
  The take_screenshot_scroller() method at line 1725 in crawl4ai/async_crawler_strategy.py contains a hardcoded await asyncio.sleep(0.01) instead of using the configurable scroll_delay parameter.

• Contributing factors:

  1. Missing parameter propagation: The take_screenshot() function is called at lines 1016-1018 and 1584-1586 without passing scroll_delay from the config
  2. Incomplete implementation: The scroll_delay parameter exists in CrawlerRunConfig (default: 0.2) and is properly used in _handle_full_page_scan() for regular crawling, but was not implemented for the screenshot scrolling logic
  3. Inconsistent behavior: Users expect scroll_delay to work consistently across all scrolling operations, but screenshots bypass this configuration

• Evidence:

  • Line 1725: await asyncio.sleep(0.01) # wait for render (hardcoded)
  • Lines 1016-1018: take_screenshot() called with only screenshot_height_threshold, missing scroll_delay
  • Lines 1584-1586: Second call site also missing scroll_delay parameter
  • Line 1279: scroll_delay: float = 0.2 defined in CrawlerRunConfig but not utilized for screenshots

Suggested Fixes (Backward-Compatible)

Fix 1: Pass scroll_delay to take_screenshot() (Lines 1016-1018)

# Before:
screenshot_data = await self.take_screenshot(
    page, screenshot_height_threshold=config.screenshot_height_threshold
)

# After:
screenshot_data = await self.take_screenshot(
    page,
    screenshot_height_threshold=config.screenshot_height_threshold,
    scroll_delay=config.scroll_delay
)

Fix 2: Pass scroll_delay to take_screenshot() (Lines 1584-1586)

# Before:
screenshot_data = await self.take_screenshot(
    page, screenshot_height_threshold=screenshot_height_threshold
)

# After:
screenshot_data = await self.take_screenshot(
    page,
    screenshot_height_threshold=screenshot_height_threshold,
    scroll_delay=getattr(config, 'scroll_delay', 0.2)
)

Fix 3: Use scroll_delay in take_screenshot_scroller() (Line 1725)

# Add at the beginning of take_screenshot_scroller():
scroll_delay = kwargs.get('scroll_delay', 0.2)

# Replace at line 1725:
# Before:
await asyncio.sleep(0.01)  # wait for render

# After:
await asyncio.sleep(scroll_delay)  # wait for render

[ntohidi]

Root Cause Analysis

• Primary cause:
  The take_screenshot_scroller() method at line 1725 in crawl4ai/async_crawler_strategy.py contains a hardcoded await asyncio.sleep(0.01) instead of using the configurable scroll_delay parameter.

• Contributing factors:

  1. Missing parameter propagation: The take_screenshot() function is called at lines 1016-1018 and 1584-1586 without passing scroll_delay from the config
  2. Incomplete implementation: The scroll_delay parameter exists in CrawlerRunConfig (default: 0.2) and is properly used in _handle_full_page_scan() for regular crawling, but was not implemented for the screenshot scrolling logic
  3. Inconsistent behavior: Users expect scroll_delay to work consistently across all scrolling operations, but screenshots bypass this configuration

• Evidence:

  • Line 1725: await asyncio.sleep(0.01) # wait for render (hardcoded)
  • Lines 1016-1018: take_screenshot() called with only screenshot_height_threshold, missing scroll_delay
  • Lines 1584-1586: Second call site also missing scroll_delay parameter
  • Line 1279: scroll_delay: float = 0.2 defined in CrawlerRunConfig but not utilized for screenshots

Suggested Fixes (Backward-Compatible)

Fix 1: Pass scroll_delay to take_screenshot() (Lines 1016-1018)

Before:

screenshot_data = await self.take_screenshot(
page, screenshot_height_threshold=config.screenshot_height_threshold
)

After:

screenshot_data = await self.take_screenshot(
page,
screenshot_height_threshold=config.screenshot_height_threshold,
scroll_delay=config.scroll_delay
)

Fix 2: Pass scroll_delay to take_screenshot() (Lines 1584-1586)

Before:

screenshot_data = await self.take_screenshot(
page, screenshot_height_threshold=screenshot_height_threshold
)

After:

screenshot_data = await self.take_screenshot(
page,
screenshot_height_threshold=screenshot_height_threshold,
scroll_delay=getattr(config, 'scroll_delay', 0.2)
)

Fix 3: Use scroll_delay in take_screenshot_scroller() (Line 1725)

Add at the beginning of take_screenshot_scroller():

scroll_delay = kwargs.get('scroll_delay', 0.2)

Replace at line 1725:

Before:

await asyncio.sleep(0.01) # wait for render

After:

await asyncio.sleep(scroll_delay) # wait for render

Heads up — the first call site already passes force_viewport_screenshot (line 1080), so make sure to keep that when adding scroll_delay. The fix should look like:

  screenshot_data = await self.take_screenshot(
      page,
      screenshot_height_threshold=config.screenshot_height_threshold,
      force_viewport_screenshot=config.force_viewport_screenshot,
      scroll_delay=config.scroll_delay
  )

The rest of the approach (Fix 2 and Fix 3) looks good.

In crawl4ai/async_crawler_strategy.py:1848: await asyncio.sleep(0.01) # wait for render

take_screenshot_scroller() hardcodes a 0.01s delay between scroll-and-capture segments. The scroll_delay config parameter (default 0.2s) exists in CrawlerRunConfig and is used by _handle_full_page_scan(), but it's never passed through to take_screenshot_scroller().

The call chain is:

1. take_screenshot(page, **kwargs) (line 1077) — only receives screenshot_height_threshold and force_viewport_screenshot
2. → take_screenshot_scroller(page, **kwargs) (line 1756) — same kwargs, no scroll_delay
3. → hardcoded asyncio.sleep(0.01) (line 1848)

Fix: Pass scroll_delay from the config through to take_screenshot_scroller and use it instead of the hardcoded 0.01s. The simplest approach is to add scroll_delay to the kwargs at the call site (line 1077) and use kwargs.get("scroll_delay", 0.2) at line 1848.