Scanning web site subfolder

Question

Can I search any website subfolder with C#?
For example
search ->  www.domain.com/

find ->
www.domain.com/numbers/
www.domain.com/magazine/

if I can, I wanna try to find page
www.domain.com/index.php
www.domain.com/page.htm etc.

Some program such as Internet Download Manager can find all pages under folder
I wanna learn How can they do this?
Can u help me any idea , way or code?