February 25, 2007
Xenu is your Friend
There are several tools that I reach for when testing sites. The first is almost always Tilman Hausherr's Xenu's Link Sleuth™. Xenu's Link Sleuth™ is a free Windows app that quickly and flexibly crawls your site for missing objects and generates easy to digest reports you can use to identify bad or stale links.
I often run Xenu on a new or potential client's site to get an idea of how large the site may be and what shape it's in; this helps us better understand the level of effort required to meet the client's needs, and gives us a head start when an RFP is short on detail.
Xenu can check for orphan files on ftp-accessible site roots. This feature can be a great help in cleaning up old, crufty sites, but care should be taken to avoid deleting files that are actually in use but invisible to Xenu's scan; such files might be Javascript controlled roll-over states, plug-in/degradation content, etc. Some sites also host standalone rich mail, newsletter, or podcast content that may generate false positive in Orphan reports.
Speaking of reports, you get three: available in the application itself, exported as tab-delimited format, or in html format.
When viewed in the application, you can sort columns; sorting by error type allows you to quickly identify your problem areas. Tab-delimited exports are useful when you want to isolate sections in a spreadsheet. The html report lets you view your broken links ordered by link or page, separately lists redirected and ftp url, orphan files and presents useful stats on scanned site content.
Plentiful options include limiting scans to certain areas of a site, scan depth, external links checking and even combining multiple sites into one report. Additions to Xenu's .ini file can track Javascript links, often used in dhtml content and for opening sized pop-up windows.
Xenu usage is fairly self-explanatory, but should you need a little help, a nice third party manual by Frank Visser is available, as is a Yahoo users group for Xenu devotees.
Xenu's Link Sleuth™ is eminently useful, straightforward, wickedly fast, and free; a combination that can't be beat and I could hardly recommend it more.
Posted by Lewis Francis at February 25, 2007 2:37 PMWhile I agree it is a great little tool, one 'gotcha' I had once was when I ran it over a LARGE clients site, it bought the server down, due to the fact every hit was a new session, and the sessions on this particular site where very "heavy"
Do you know if there is a way to get Xenu to use cookies so the server can persist a single session for the process?
Posted by: David Harris at March 4, 2007 3:27 PMAnd what about using google webmaster for broken links?
Has XENU anything better?
Thanks.
Posted by: Ahmet at March 4, 2007 3:38 PMHey David,
You can add a line to Xenu's .ini file under [Options] that enables cookies:
[Options]
AllowCookies=1
Xenu's doc warns, though, that if you have links that commit or delete data then you should be very careful when using this option or you might break things. If your 'dangerous' links are unique, you could add them to the 'Do not check...' field and exclude them from the scan.
Hope this helps!
Posted by: Lewis Francis at March 4, 2007 4:01 PMHey Ahmet,
Thanks for reminding me about Google's Webmaster tools. I added my site last year and planned on a review but frankly forgot about it till now.
I need to reacquaint myself with the Google tools, but I'd say the main advantage of a tool like Xenu (I plan to write about other link checkers in future posts) is that you can run them on-demand and on your development environment, which presumably isn't public and accessible to online tools. Google reports will only be up to date as of the last crawl.
Right now my Google Web Crawl stats are not reporting bad links, though Xenu tells me I have many in older posts that have rotten over time (I confess I've been lazy of late); this may be because I haven't yet uploaded a site map or may point to some other configuration I haven't sussed out yet.
Posted by: Lewis Francis at March 4, 2007 4:23 PMThe funniest thing (for me) is that this software is named after a mythical character in the Scientology cult. So mythical that we aren't allowed to even know its existance.
Tilman Hausherr is a long time and influetial critic of the Scienos and by naming his application this way he advertizes his opposition.
See his home page for more info on Scientology.
...and thus my multi-level post heading. ;)
Posted by: Lewis Francis at March 4, 2007 8:45 PMRegarding Xenu's ability to identify orphan files via ftp; Xenu can also spider and identify orphans with local access.
Posted by: Lewis Francis at March 15, 2007 10:44 AMhttp://informationgift.com/mt/mt-tb.cgi/117
Listed below are links to weblogs that reference 'Xenu is your Friend' from Information Gift.
