The EasyAssist call with the Microsoft Engineer is finished. It seems that a proxy setting caused all the trouble. We went through a lot of settings like BackConnectionHostnames, DisableLoopbackCheck, crawl rules, security settings on the databases etc.
But using the commandline tool proxycfg triggered it. See the screenshot below. The proxycfg command shows if there are proxies configured. In the client environment there is a proxy configured at the management network:
As you can see, a bypass list is not configured. Therefore the crawler wants to use this proxy setting to crawl the site. That is not possible and this results in an error. With Microsoft we reconfigured these proxy settings and included a bypass list. See below:
Now the crawler will bypass the proxy settings and is able to crawl the complete site.
By the way: the proxy is needed by WSUS to deploy updates to the servers over the management lan.