February 25, 2007
Xenu is your Friend
There are several tools that I reach for when testing sites. The first is almost always Tilman Hausherr's Xenu's Link Sleuth™. Xenu's Link Sleuth™ is a free Windows app that quickly and flexibly crawls your site for missing objects and generates easy to digest reports you can use to identify bad or stale links.
I often run Xenu on a new or potential client's site to get an idea of how large the site may be and what shape it's in; this helps us better understand the level of effort required to meet the client's needs, and gives us a head start when an RFP is short on detail.
Speaking of reports, you get three: available in the application itself, exported as tab-delimited format, or in html format.
When viewed in the application, you can sort columns; sorting by error type allows you to quickly identify your problem areas. Tab-delimited exports are useful when you want to isolate sections in a spreadsheet. The html report lets you view your broken links ordered by link or page, separately lists redirected and ftp url, orphan files and presents useful stats on scanned site content.
Xenu usage is fairly self-explanatory, but should you need a little help, a nice third party manual by Frank Visser is available, as is a Yahoo users group for Xenu devotees.
Xenu's Link Sleuth™ is eminently useful, straightforward, wickedly fast, and free; a combination that can't be beat and I could hardly recommend it more.
Posted by Lewis Francis at February 25, 2007 2:37 PM
While I agree it is a great little tool, one 'gotcha' I had once was when I ran it over a LARGE clients site, it bought the server down, due to the fact every hit was a new session, and the sessions on this particular site where very "heavy"
And what about using google webmaster for broken links?
Has XENU anything better?
You can add a line to Xenu's .ini file under [Options] that enables cookies:
Xenu's doc warns, though, that if you have links that commit or delete data then you should be very careful when using this option or you might break things. If your 'dangerous' links are unique, you could add them to the 'Do not check...' field and exclude them from the scan.
Hope this helps!
Thanks for reminding me about Google's Webmaster tools. I added my site last year and planned on a review but frankly forgot about it till now.
I need to reacquaint myself with the Google tools, but I'd say the main advantage of a tool like Xenu (I plan to write about other link checkers in future posts) is that you can run them on-demand and on your development environment, which presumably isn't public and accessible to online tools. Google reports will only be up to date as of the last crawl.
Right now my Google Web Crawl stats are not reporting bad links, though Xenu tells me I have many in older posts that have rotten over time (I confess I've been lazy of late); this may be because I haven't yet uploaded a site map or may point to some other configuration I haven't sussed out yet.
The funniest thing (for me) is that this software is named after a mythical character in the Scientology cult. So mythical that we aren't allowed to even know its existance.
Tilman Hausherr is a long time and influetial critic of the Scienos and by naming his application this way he advertizes his opposition.
See his home page for more info on Scientology.
...and thus my multi-level post heading. ;)
Regarding Xenu's ability to identify orphan files via ftp; Xenu can also spider and identify orphans with local access.