Lync 2013 Frontend service not starting after CU

I have experienced a situation where a Lync Server 2013 front end service would not start after updating from RTM to CU10 (January 2015). RTM version servers were running fine. The deployment is a migration from Lync Server 2010 to Lync Server 2013.

Following errors were seen in the eventlog:
Event ID:14562
Event ID: 14651
Event ID: 12303 (twice)

Details:
Log Name:      Lync Server
Source:        LS Protocol Stack
Date:          21-01-2015 11:46:26
Event ID:      14562
Task Category: (1001)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE01.domain.local
Description:
Two servers cannot be configured at the same FQDN with different 'Outbound Only' options.

Cannot configure a server at FQDN [0.0.0.0] because another server is already configured there with a different 'Outbound Only' option.
Cause: This is a configuration problem.
Resolution:
Review the server roles that are configured at this FQDN and ensure that they have identical trust options.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Protocol Stack" />
    <EventID Qualifiers="50153">14562</EventID>
    <Level>2</Level>
    <Task>1001</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-01-21T10:46:26.000000000Z" />
    <EventRecordID>8402</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE01.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>0.0.0.0</Data>
  </EventData>

</Event>

Log Name:      Lync Server
Source:        LS Protocol Stack
Date:          21-01-2015 11:46:26
Event ID:      14651
Task Category: (1001)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE01.domain.local
Description:
A serious configuration problem is preventing Lync Server from functioning.

An unexpected error occurred while configuring the internal servers table. Contact Product Support Services.

Cause: The Lync Server failed to initialize due to invalid settings from CMS.
Resolution:
Review and correct the CMS configuration, then start the service again.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Protocol Stack" />
    <EventID Qualifiers="50153">14651</EventID>
    <Level>2</Level>
    <Task>1001</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-01-21T10:46:26.000000000Z" />
    <EventRecordID>8403</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE01.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>An unexpected error occurred while configuring the internal servers table. Contact Product Support Services.</Data>
  </EventData>
</Event>

Log Name:      Lync Server
Source:        LS Server
Date:          21-01-2015 11:46:26
Event ID:      12303
Task Category: (1000)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE01.domain.local
Description:
The protocol stack reported a critical error: code C3E93C66 (SIPPROXY_E_MULTIPLE_INCOMPATIBLE_TRUST_OPTIONS). The service has to stop.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Server" />
    <EventID Qualifiers="50152">12303</EventID>
    <Level>2</Level>
    <Task>1000</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-01-21T10:46:26.000000000Z" />
    <EventRecordID>8404</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE01.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>C3E93C66</Data>
    <Data>SIPPROXY_E_MULTIPLE_INCOMPATIBLE_TRUST_OPTIONS</Data>
  </EventData>

</Event>

Log Name:      Lync Server
Source:        LS Server
Date:          21-01-2015 11:46:26
Event ID:      12303
Task Category: (1000)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE01.domain.local
Description:
The protocol stack reported a critical error: code C3E93C66 (Configuration failure prevented the server from starting up). The service has to stop.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Server" />
    <EventID Qualifiers="50152">12303</EventID>
    <Level>2</Level>
    <Task>1000</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-01-21T10:46:26.000000000Z" />
    <EventRecordID>8405</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE01.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>C3E93C66</Data>
    <Data>Configuration failure prevented the server from starting up</Data>
  </EventData>

</Event>

Working with Microsoft Premier Support on this, we found that 2 application servers were giving problems, both indicating they had FQDN [0.0.0.0] (which is also noticed in the event ID 14562).

In the logging we also saw the registred name of the servers were IP addresses and also on which ports communication should be attempted. These IP hosts were not responding to ping or telnet to the specified TCP port. These were no PTR records registered to these IP addresses.

The servers were Trusted Application servers that are not used anymore (old obsolete gateway from OCS2007R2 and an RCC test server), so deleting the trusted application and the trusted application pool was safe. 

After a management store replication, the front end service (RTCSRV) could start on FE01.domain.local and the rest of the servers were updated and the CU10 was successfully deployed.

Lesson learned: Always check the status of ALL the servers before a migration is started.