Wednesday 21 January 2015

Lync 2013 Frontend service not starting after CU

I have experienced a situation where a Lync Server 2013 front end service would not start after updating from RTM to CU10 (January 2015). RTM version servers were running fine. The deployment is a migration from Lync Server 2010 to Lync Server 2013.

Following errors were seen in the eventlog:
Event ID:14562
Event ID: 14651
Event ID: 12303 (twice)

Details:
Log Name:      Lync Server
Source:        LS Protocol Stack
Date:          21-01-2015 11:46:26
Event ID:      14562
Task Category: (1001)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE01.domain.local
Description:
Two servers cannot be configured at the same FQDN with different 'Outbound Only' options.

Cannot configure a server at FQDN [0.0.0.0] because another server is already configured there with a different 'Outbound Only' option.
Cause: This is a configuration problem.
Resolution:
Review the server roles that are configured at this FQDN and ensure that they have identical trust options.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Protocol Stack" />
    <EventID Qualifiers="50153">14562</EventID>
    <Level>2</Level>
    <Task>1001</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-01-21T10:46:26.000000000Z" />
    <EventRecordID>8402</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE01.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>0.0.0.0</Data>
  </EventData>

</Event>

Log Name:      Lync Server
Source:        LS Protocol Stack
Date:          21-01-2015 11:46:26
Event ID:      14651
Task Category: (1001)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE01.domain.local
Description:
A serious configuration problem is preventing Lync Server from functioning.

An unexpected error occurred while configuring the internal servers table. Contact Product Support Services.

Cause: The Lync Server failed to initialize due to invalid settings from CMS.
Resolution:
Review and correct the CMS configuration, then start the service again.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Protocol Stack" />
    <EventID Qualifiers="50153">14651</EventID>
    <Level>2</Level>
    <Task>1001</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-01-21T10:46:26.000000000Z" />
    <EventRecordID>8403</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE01.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>An unexpected error occurred while configuring the internal servers table. Contact Product Support Services.</Data>
  </EventData>
</Event>

Log Name:      Lync Server
Source:        LS Server
Date:          21-01-2015 11:46:26
Event ID:      12303
Task Category: (1000)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE01.domain.local
Description:
The protocol stack reported a critical error: code C3E93C66 (SIPPROXY_E_MULTIPLE_INCOMPATIBLE_TRUST_OPTIONS). The service has to stop.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Server" />
    <EventID Qualifiers="50152">12303</EventID>
    <Level>2</Level>
    <Task>1000</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-01-21T10:46:26.000000000Z" />
    <EventRecordID>8404</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE01.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>C3E93C66</Data>
    <Data>SIPPROXY_E_MULTIPLE_INCOMPATIBLE_TRUST_OPTIONS</Data>
  </EventData>

</Event>

Log Name:      Lync Server
Source:        LS Server
Date:          21-01-2015 11:46:26
Event ID:      12303
Task Category: (1000)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE01.domain.local
Description:
The protocol stack reported a critical error: code C3E93C66 (Configuration failure prevented the server from starting up). The service has to stop.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Server" />
    <EventID Qualifiers="50152">12303</EventID>
    <Level>2</Level>
    <Task>1000</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-01-21T10:46:26.000000000Z" />
    <EventRecordID>8405</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE01.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>C3E93C66</Data>
    <Data>Configuration failure prevented the server from starting up</Data>
  </EventData>

</Event>

Working with Microsoft Premier Support on this, we found that 2 application servers were giving problems, both indicating they had FQDN [0.0.0.0] (which is also noticed in the event ID 14562).

In the logging we also saw the registred name of the servers were IP addresses and also on which ports communication should be attempted. These IP hosts were not responding to ping or telnet to the specified TCP port. These were no PTR records registered to these IP addresses.

The servers were Trusted Application servers that are not used anymore (old obsolete gateway from OCS2007R2 and an RCC test server), so deleting the trusted application and the trusted application pool was safe. 

After a management store replication, the front end service (RTCSRV) could start on FE01.domain.local and the rest of the servers were updated and the CU10 was successfully deployed.

Lesson learned: Always check the status of ALL the servers before a migration is started.

Tuesday 20 January 2015

PortqryUI

PortQryUI is a wonderful little tool, that I use quite often for troubleshooting network connectivity and checking e.g. loadbalancers etc.

The tool is used for checking if the ports are listening to traffic or not. Results can be the following:
Listening (TCP or UDP) indicates a service are listening on the port.
Filtered (TCP or UDP) indicates a service might be listening on the port.
Not listening (TCP or UDP) indicates a service is not listening on the port.

The tool can also be used from command line interface, syntax for commands can be found here: http://technet.microsoft.com/en-us/library/cc779626.aspx

However checking all the Lync services by hand, using this tool, can be a little cumbersome, so I have added the following lines to the config.xml file:

  <Service Name="Lync hardware loadbalancer only">
    <Port Name="SIP TLS" Value="5061" Protocol="TCP"/>
    <Port Name="HTTPS Focus - FE" Value="444" Protocol="TCP"/>
    <Port Name="DCOM / RPC" Value="135" Protocol="TCP"/>
    <Port Name="HTTP internal" Value="80" Protocol="TCP"/>
    <Port Name="HTTP external" Value="8080" Protocol="TCP"/>
    <Port Name="HTTPS internal" Value="443" Protocol="TCP"/>
    <Port Name="HTTPS external" Value="4443" Protocol="TCP"/>
    <Port Name="SIP mediation" Value="5070" Protocol="TCP"/>
    <Port Name="SIP response group" Value="5071" Protocol="TCP"/>
    <Port Name="SIP attendant dialin" Value="5072" Protocol="TCP"/>
    <Port Name="SIP conf announcement" Value="5073" Protocol="TCP"/>
    <Port Name="SIP call park" Value="5075" Protocol="TCP"/>
    <Port Name="SIP audio test" Value="5076" Protocol="TCP"/>
    <Port Name="CAC Edge TURN" Value="5080" Protocol="TCP"/>
    <Port Name="CAC FE" Value="448" Protocol="TCP"/>
  </Service>
  <Service Name="Lync hardware loadbalancer DNS">
    <Port Name="HTTP internal" Value="80" Protocol="TCP"/>
    <Port Name="HTTP external" Value="8080" Protocol="TCP"/>
    <Port Name="HTTPS internal" Value="443" Protocol="TCP"/>
    <Port Name="HTTPS external" Value="4443" Protocol="TCP"/>
  </Service>


Then I can easily select the service I want to check and just enter the DNS name for the pool.
 


The PortQry tool can be downloaded from here: http://www.microsoft.com/en-us/download/details.aspx?id=24009