2.27Crawling Web Page Content (Crawler)

Updated on Aug 22, 2026~8 min read

This section shows three ways to crawl web pages: direct fetching plus an HTML parsing library, and headless browser crawling for JavaScript-rendered pages.

With the HTTP remote request module, you can easily crawl any web page content on the internet (commonly known as a "crawler"). Combined with an HTML parsing library (such as AngleSharp or HtmlAgilityPack), you can further extract the required data; for pages that rely on JavaScript rendering, you can also use a headless browser (such as Playwright) to crawl the fully rendered content.

The following uses crawling the blog titles on the cnblogs homepage as an example to demonstrate how to use the two parsing libraries and the headless browser respectively.

Using AngleSharp

1. Install the NuGet package

bash
Install-Package AngleSharp

2. Write the crawling code

cs
// Get the cnblogs homepage HTML stringvar cnblogs = await httpRemoteService.SendAsStringAsync(HttpRequestBuilder.Get("https://www.cnblogs.com/")     .SetUserAgent(UserAgents.GetRandom())); // Random browser User-Agent// Create a context for parsing web pages using the default configurationvar context = BrowsingContext.New(Configuration.Default);// Parse the fetched HTML string as the content of a virtual request into an operable document objectvar document = await context.OpenAsync(req => req.Content(cnblogs));// Use CSS selectors to query all matching elements in the document: post-item-title class elements under the post_list idvar elements = document.QuerySelectorAll("#post_list .post-item-title");// Extract the text content of each element (i.e., the blog title)var titles = elements.Select(u => u.TextContent).ToList();

Using HtmlAgilityPack

1. Install the NuGet package

bash
Install-Package HtmlAgilityPack

2. Write the crawling code

cs
// Get the cnblogs homepage HTML stringvar cnblogs = await httpRemoteService.SendAsStringAsync(HttpRequestBuilder.Get("https://www.cnblogs.com/")     .SetUserAgent(UserAgents.GetRandom())); // Random browser User-Agent// Create an HtmlDocument instance for parsing HTMLvar document = new HtmlDocument();// Load the fetched HTML stringdocument.LoadHtml(cnblogs);// Use XPath to get a node collection: select all child elements whose class contains post-item-title within the element whose id is post_listvar nodes = document.DocumentNode.SelectNodes("//*[@id='post_list']//*[contains(@class,'post-item-title')]");// Extract the inner text of each node (i.e., the blog title)var titles =  nodes.Select(node => node.InnerText).ToList();

Using a Headless Browser (Playwright)

Some pages rely on JavaScript dynamic rendering (such as single-page applications SPA or asynchronously loaded lists), so fetching the raw HTML directly does not yield the final content. In this case, you can use a headless browser to execute the page scripts in a real browser engine and then extract the rendered data. Playwright for .NET (the Microsoft.Playwright package) is currently the most mainstream headless browser library in the C# ecosystem (with over 60 million cumulative NuGet downloads), officially maintained by Microsoft, runs headless by default, and supports the Chromium, Firefox, and WebKit engines. If you only need Chromium, PuppeteerSharp is also a good choice.

1. Install the NuGet package

bash
Install-Package Microsoft.Playwright

2. Download the browser engine

Download the browser engine before the first use (taking Chromium as an example):

bash
dotnet tool install --global Microsoft.Playwright.CLIplaywright install chromium

3. Write the crawling code

cs
// Launch Playwright (Chromium runs in headless mode by default)using var playwright = await Playwright.CreateAsync();await using var browser = await playwright.Chromium.LaunchAsync();// Create a page and set the viewport size (to simulate a real browser environment)var page = await browser.NewPageAsync();await page.SetViewportSizeAsync(1920, 1080);// Visit the page and wait for the network to become idle to ensure JavaScript has finished executingawait page.GotoAsync("https://www.cnblogs.com/", new() { WaitUntil = WaitUntilState.NetworkIdle });// Wait for the target element to appear (SPA pages may render asynchronously)await page.WaitForSelectorAsync("#post_list .post-item-title");// Extract the text of all matching elements (i.e., the blog titles); page.ContentAsync() returns the fully rendered HTMLvar titles = await page.Locator("#post_list .post-item-title").AllTextContentsAsync();

A headless browser actually executes page scripts and can crawl the final JavaScript-rendered content — the strongest of the three approaches; see the notes below for its overhead and usage guidance.

Notes

  • Legal compliance: be sure to comply with the target website's robots.txt protocol and relevant laws and regulations, control the request frequency reasonably, and avoid putting pressure on the server.
  • Anti-crawler strategies: you can appropriately configure strategies such as User-Agent, delayed waiting, and proxy IP to improve crawling stability.
  • Parsing methods: AngleSharp supports CSS selectors, and its syntax is closer to frontend development; HtmlAgilityPack is based on XPath. Both have their own strengths and can be chosen as needed.
  • Headless browser: the browser engine must be downloaded before the first use (hundreds of MB), and it consumes more resources than direct fetching, so use it only when the page relies on JavaScript rendering; for regular static pages, prefer the direct fetching approaches in the previous two sections, which are lighter and more efficient.

By combining HTTP remote requests, parsing libraries, and a headless browser, you can quickly meet various web page data collection needs.