3.50Simulating a Browser Environment (Crawler Detection)

Created on Aug 17, 2026~1 min read

When developing a crawler, the target website may serve different page versions — such as PC and mobile — based on the user agent (User-Agent) or other factors. In addition, some websites implement anti-crawler mechanisms that can identify and block crawler access. To address these issues, we can configure request headers to simulate a real browser environment when making requests.

cs
HttpRequestBuilder.Get("https://www.baidu.com/")    .SimulateBrowser(); // Simulate a PC browser environmentHttpRequestBuilder.Get("https://www.baidu.com/")    .SimulateBrowser(isMobile: true); // Simulate a mobile browser environment

After adding this operation, the HTTP request will automatically include the following request headers before being sent, ensuring that the server can accurately identify and process the request:

bash
# PC browser user agentMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36 Edg/142.0.0.0# Mobile browser user agentMozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Mobile Safari/537.36 Edg/142.0.0.0