Skip to content
Autowright

Handling Dynamic Content

Handling Dynamic Content

Modern web applications load content dynamically. This guide covers techniques for reliably automating dynamic content.

The Challenge

Dynamic content issues include:

  • Elements appearing after AJAX calls
  • Content loading based on scroll position
  • SPAs updating the DOM without page loads
  • Animations and transitions
  • Lazy-loaded images and components

Waiting Strategies

Wait for Selector

Wait for an element to appear:

.step('Wait for results', async (ctx) => {
  // Wait up to 30 seconds for element
  await ctx.page.waitForSelector('.search-results', {
    timeout: 30000,
  });
})

Wait for Load State

Wait for page to reach a specific load state:

.step('Wait for page load', async (ctx) => {
  // Wait for network to be idle
  await ctx.page.waitForLoadState('networkidle');


  // Or wait for DOM content
  await ctx.page.waitForLoadState('domcontentloaded');


  // Or wait for full load
  await ctx.page.waitForLoadState('load');
})

Wait for URL

Wait for navigation to complete:

.step('Submit and wait', async (ctx) => {
  await ctx.page.click('#submit');


  // Wait for specific URL pattern
  await ctx.page.waitForURL('**/dashboard/**');


  // Or with regex
  await ctx.page.waitForURL(/\/user\/\d+/);
})

Wait for Response

Wait for API calls to complete:

.step('Wait for API', async (ctx) => {
  // Wait for specific API response
  const response = await ctx.page.waitForResponse(
    (res) => res.url().includes('/api/data') && res.status() === 200
  );


  const data = await response.json();
  ctx.data.apiResponse = data;
})

Wait for Function

Wait for a custom condition:

.step('Wait for condition', async (ctx) => {
  // Wait for a condition to be true in the page
  await ctx.page.waitForFunction(() => {
    const items = document.querySelectorAll('.item');
    return items.length >= 10;
  });
})

Handling AJAX Content

Intercepting Requests

Monitor and wait for specific requests:

.step('Load more items', async (ctx) => {
  // Start waiting before triggering
  const responsePromise = ctx.page.waitForResponse(
    (res) => res.url().includes('/api/items')
  );


  // Trigger the request
  await ctx.page.click('#load-more');


  // Wait for response
  await responsePromise;


  ctx.log('Items loaded');
})

Handling Pagination

.step('Scrape all pages', async (ctx) => {
  let hasNextPage = true;


  while (hasNextPage) {
    // Extract current page data
    const items = await ctx.page.locator('.item').allTextContents();
    ctx.data.items.push(...items);


    // Check for next page
    const nextButton = ctx.page.locator('.next-page:not([disabled])');
    hasNextPage = await nextButton.count() > 0;


    if (hasNextPage) {
      // Wait for next page load
      const navPromise = ctx.page.waitForResponse(
        (res) => res.url().includes('/page/')
      );
      await nextButton.click();
      await navPromise;
    }
  }
})

Handling SPAs

Route Changes

SPAs change routes without full page loads:

.step('Navigate in SPA', async (ctx) => {
  // Click navigation link
  await ctx.page.click('a[href="/dashboard"]');


  // Wait for component to mount
  await ctx.page.waitForSelector('[data-component="dashboard"]', {
    state: 'visible',
  });


  // Or wait for URL change
  await ctx.page.waitForURL('**/dashboard');
})

React/Vue/Angular Apps

.step('Wait for React render', async (ctx) => {
  // Wait for React to finish rendering
  await ctx.page.waitForFunction(() => {
    // Check for loading indicators
    const loader = document.querySelector('.loading-spinner');
    return !loader || loader.style.display === 'none';
  });


  // Verify content is rendered
  await ctx.page.waitForSelector('[data-testid="content"]');
})

Handling Animations

Wait for Animation End

.step('Wait for modal', async (ctx) => {
  await ctx.page.click('#open-modal');


  // Wait for animation to complete
  await ctx.page.waitForSelector('.modal', {
    state: 'visible',
  });


  // Additional wait for CSS animation
  await ctx.page.waitForTimeout(300);  // Animation duration


  // Or better: wait for stable state
  await ctx.page.waitForFunction(() => {
    const modal = document.querySelector('.modal');
    if (!modal) return false;
    const style = getComputedStyle(modal);
    return style.opacity === '1' && style.transform === 'none';
  });
})

Handling Lazy Loading

Scroll to Load

.step('Load all lazy content', async (ctx) => {
  let previousHeight = 0;
  let currentHeight = await ctx.page.evaluate(() => document.body.scrollHeight);


  while (previousHeight < currentHeight) {
    // Scroll to bottom
    await ctx.page.evaluate(() => {
      window.scrollTo(0, document.body.scrollHeight);
    });


    // Wait for new content
    await ctx.page.waitForTimeout(1000);  // Or waitForResponse


    previousHeight = currentHeight;
    currentHeight = await ctx.page.evaluate(() => document.body.scrollHeight);
  }


  ctx.log('All content loaded');
})

Lazy Images

.step('Wait for images', async (ctx) => {
  // Scroll to element to trigger lazy load
  const element = ctx.page.locator('#product-gallery');
  await element.scrollIntoViewIfNeeded();


  // Wait for images to load
  await ctx.page.waitForFunction(() => {
    const images = document.querySelectorAll('#product-gallery img');
    return Array.from(images).every(img => img.complete);
  });
})

Using Retries for Flaky Elements

Step-Level Retry

.step('Click flaky button', async (ctx) => {
  const button = ctx.page.locator('#sometimes-slow-button');
  await button.click();
}, {
  retry: {
    times: 3,
    delay: 1000,
    backoff: 'exponential',
  },
})

Checkpoint-Level Retry

.checkpoint('Submit Form', { maxRetries: 2 })
  .step('Fill form', async (ctx) => {
    await ctx.page.fill('#email', ctx.data.email);
    await ctx.page.fill('#message', ctx.data.message);
  })
  .step('Submit', async (ctx) => {
    await ctx.page.click('#submit');
    await ctx.page.waitForSelector('.success-message');
  })
.endCheckpoint()

Debugging Dynamic Content Issues

Take Screenshots at Key Points

.step('Debug step', async (ctx) => {
  await ctx.screenshot('before-click');


  await ctx.page.click('#trigger');


  await ctx.screenshot('after-click');


  await ctx.page.waitForSelector('.result');


  await ctx.screenshot('after-result');
})

Log Page State

.step('Debug state', async (ctx) => {
  // Log current URL
  ctx.log('URL:', ctx.page.url());


  // Log element count
  const count = await ctx.page.locator('.item').count();
  ctx.log('Item count:', count);


  // Log element visibility
  const visible = await ctx.page.locator('#modal').isVisible();
  ctx.log('Modal visible:', visible);


  // Capture DOM for analysis
  const html = await ctx.getDOM('#container');
  ctx.log('Container HTML length:', html.length);
})

Best Practices

1. Prefer Explicit Waits

await ctx.page.waitForSelector('.results');
await ctx.page.waitForResponse(r => r.url().includes('/api'));

2. Use Auto-Waiting Locators

Playwright locators auto-wait by default:

// Good: Uses auto-waiting
const button = ctx.page.locator('#submit');
await button.click();  // Auto-waits for element


// Also good for assertions
await expect(ctx.page.locator('.result')).toBeVisible();

3. Handle Loading States

.step('Handle loading', async (ctx) => {
  // Wait for loader to disappear
  await ctx.page.waitForSelector('.loading', { state: 'hidden' });


  // Then interact with content
  await ctx.page.click('.content-item');
})

4. Set Reasonable Timeouts

.config({
  defaultTimeout: 30000,  // 30 seconds default
})


.step('Quick operation', async (ctx) => {
  // ...
}, {
  timeout: 5000,  // Override for fast operations
})


.step('Slow operation', async (ctx) => {
  // ...
}, {
  timeout: 60000,  // Override for slow operations
})

Next Steps