Link Checker (Site Crawler)
This tool will crawl any site given and get all the links on the same domain checking they all exist. The result of this will be a report containing any URL's that 404 or any URL's that redirect and the type of redirect they do along with the page they're on. Data is cached for 5 days so any subsequent requests for a URL within that period will not be "live" results.
The engine will show up as xcitebot all the time and ignores robots.txt files. So it is possible to prevent access by watching the useragent specifically. It will never be spoofed if used via this website.
N.B.: All access to this tool is logged, if it's abused, it will be removed. Please make sure you have javascript enabled and that you're not using any masking tools to hide your identity in any way. There is also a limit on the number of page this tool will download, trying to create a complete map of huge sites in a single PHP script isn't something this server can do without some serious code re-writes!