'System.Net.WebException' in HtmlAgilityPack using Parallel.ForEach loop
See the question and my original answer on StackOverflowI think it's because you have a limit on the maximum number of simultaneous HttpWebRequest connections to a site. Check this .NET setting: ConnectionManagement Element (Network Settings).
You can also do it programmatically: How can I programmatically remove the 2 connection limit in WebClient
It works with a browser in parallel because it uses another process.