"The crawler could not communicate with the server. Check that the server is available and that the firewall access is configured correctly.."
This error sometimes shows up in the Crawl Log in the SSP.
The issue in my case was a custom database that the crawler account did not have access to. Of course an easy way to figure this out is to login to the website as the crawler account -- to see what it sees... It worked. A quick adjustment of SQL permissions and voila, we're crawling again.
Pesky little thing. MS introduced a loopback security check for websites accessed locally on the server.
To fix:http://support.microsoft.com/kb/896861
Click Start, click Run, type regedit, and then click OK.
In Registry Editor, locate and then click the following registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
Right-click Lsa, point to New, and then click DWORD Value.
Type DisableLoopbackCheck, and then press ENTER.
Right-click DisableLoopbackCheck, and then click Modify.
In the Value data box, type 1, and then click OK.
Quit Registry Editor, and then restart your computer.