5.20Simulating a Browser Environment (Crawler Detection)

Updated on Aug 20, 2026~3 min read

When developing a crawler program, the target website may provide different page versions based on the user agent (User-Agent) or other factors, such as the PC version and the mobile version. In addition, some websites have anti-crawler mechanisms that can identify and block crawler program access. To address these problems, we can configure the request headers to simulate a real browser environment for making requests.

HTTP declarative requests simulate a browser environment via the SimulateBrowserAttribute attribute. The corresponding HTTP declarative extractor is implemented as the SimulateBrowserDeclarativeExtractor type, which is responsible for parsing the SimulateBrowserAttribute attribute and building the HttpRequestBuilder instance configuration required to simulate a browser environment.

cs
// Apply on the interface definition, affecting all methods[SimulateBrowser]public interface IHttpService : IHttpDeclarative{    [Get("https://furion.net/")]    Task<string> GetStringAsync();    // Apply on a method    [SimulateBrowser(IsMobile = true)]   // Simulate a mobile browser environment    [Get("https://furion.net/")]    Task<string> GetStringAsync();}

After adding this attribute, the HTTP request will automatically attach the following request headers before sending to ensure that the server can accurately identify and process the request:

bash
# PC browser user agentMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36 Edg/142.0.0.0# Mobile browser user agentMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Mobile Safari/537.36 Edg/142.0.0.0

SimulateBrowserAttribute includes the following constructors and properties:

  • Constructors:

    • new(): applies to a method or interface, enabling browser environment simulation.

  • Properties:

    • IsMobile: whether it is mobile (bool type); the default value is false (that is, the desktop version).