HtmlAgilityPack & Selenium Webdriver returns random results
See the question and my original answer on StackOverflowAs others said, the page from this site loads itself dynamically using some javascript, so the Html Agility Pack just gets the first items.
Web Scraping can be tough, especially with modern sites which use more and more javascript, and it's in general very specific to the target site (I'm not even talking about the legal issues..). You can use various techniques to determine how to get the information you require.
In this case, if you use any network analyzer, you'll quickly see the site uses an 'sz'
(for Size I guess) query string parameter that allows you to specify the number of items you want.
So, just modify your url for this:
var url = @"http://www.roots.com/ca/en/men/tops/shirts-and-polos/?sz=9999";
and get any numbers of items you want.