Wednesday 23 September 2015

Skype4B / IIS 8 not presenting new certificate in browser.

After replacing a couple of old, soon to expire, default certificates on the frontend pool, I discovered that the old certificate was still being presented to the client.

Checking the IIS manager binding for the website, I saw the correctly assigned new certificate.

Running this command:
netsh http show sslcert
Showed that some of the ippools still having the thumbprint of the old certificate, so going back into bindings for the website. I changed the IP address assignment from Any unused to the IP address of the server, resetting the IIS (iisreset.exe) and changing the IP address assignment back to Any unused.

Ran the command again and this time the correct thumbprint was shown for all the ippools.

Test showed the correct certificate now being presented to clients.

Thursday 10 September 2015

Skype4B web plugin cannot establish media to meeting

Over some time I have been troubleshooting a problem where a customer cannot connect to Skype4B Online meetings using the Skype Web plugin. Connecting media failed.


Research showed that connecting to a Lync 2013 online meeting using the Lync Web App plugin worked fine, but the connecting to the same meeting using the Skype4B Web App plugin didn't work.


The customer has a Websense proxy server farm set up using SSL inspection, so we tried to remove SSL inspection as this is unsupported (according to Microsoft PSS - not documented).


Tracing the proxy server, we saw the Skype4B not authenticating towards the Websense proxy. Websense requesting Kerberos, NTLM and Basic authentication. Removing authentication for the meeting.domain did not change anything.


Still no resolution, but it does look like a bug in the Skype4B Web App plugin.

Thursday 9 July 2015

Migrate a migration scenario to Skype for Business Server 2015

At a large customer, we are stuck in a migration scenario between Lync Server 2010 and Lync Server 2013, because of dependant non-Lync projects, like migration from a "legacy" e-mail/calendering system to Exchange 2013. We are unable to move users to Lync Server 2013 for now.

Last week, the same customer requested an upgrade to Skype for Business Server 2015 and asked if it was possible to in-place upgrade their Lync Server 2013 pool to Skype for Business Server 2015. My first reaction was off course: No. You cannot. It is unsupported to have more than 1 legacy version installed before Skype4B.

Customer responded: I didn't ask if it is supported, I asked if it is possible.

So taking this request to the lab servers, I installed a new AD with a few user, a Lync Server 2010 SE enabled some users. Then a Lync Server 2013 SE and a Skype4B management server.

I started the Skype4B topology builder and downloaded the topology, and got to this result:
Nope, not possible... Clicked OK.

Tried anyway to upgrade my Lync Server 2013 pool: 
Right click the pool and press Upgrade to Skype for Business Server 2015

Published the topology

 Opened the to-do text file at the end

This still gives me the creeps after 5 Skype4B upgrades


On the Lync Server 2013, I could mount the ISO file and start the installation (at least after adding another 20Gb of disk space to the VM). 

In the end, all services are running and users can be moved to, and connect to, the upgraded server.

So, it is possible to migrate a migration scenario to Skype for Business Server 2015

Thursday 7 May 2015

Bug in inplace upgrade to Skype for Business Server 2015: Replica Replicator Agent won't start

As many in the community, we (@torstenegebirk from egebirk-public.sharepoint.com/blog and me) was excited to see Skype for Business Server 2015 being released on May 1st, 2015.
First installation in lab went apparently fine, after going through the prerequisites we added an extra 35Gb harddisk to the VM, formatted and assigned drive letter E. Rest of the installation process is out of scope for this post.
After installation all services very running with their lovely new "short" service names. Databases were where we expected them to be. Lovely.
As the new E drive didn't contain any data, we removed the harddisk, and rebooted the server. Then problems started.
After the reboot the Skype For Business Server Replica Replicator Agent service would'nt start (thanks to the Redundant Department of Redundancy). In the logging we saw event ID 3007 on the LS Replica Replicator Agent Service, see screenshot:


The problem is obviously a missing share on the E drive.
During the inplace upgrade the xds-replica shared folder was without notification moved from the original place, could be C or D (C in our case), to the drive with most available space. This is a problem, if you, for some reason, have placed your xds-replica folder on a certain drive for data storage.


The quick fix is to do following:
  • Recreate an E drive
  • Create a folder structure E:\RtcReplicaRoot\xds-replica\
  • Restart the Server service (this will "reshare" the folder after detecting that the physical location now exists)
  • Make sure NETWORK SERVICE and RTC Local Config Replicator (local security group on the front end servers) has Full Control in the share permissions
  • Make sure RTC Local Config Replicator local security group has full control in NTFS permissions
  • Start the Skype for Business Server 2015 Local Replica Replicator Agent service again
  • When service is started run the Invoke-CsManagementStoreReplication cmdlet
  • Wait a minute or so (depending on size of deployment) and check the status with the Get-CsManagementStoreReplicationStatus cmdlet
  • All servers should have Up-To-Date set to True
Then the folder share is available again with the correct share permissions



You could also follow Ken Lasko's post here: http://ucken.blogspot.com/2012/04/resetting-lync-cms-replication.html
The problem is still not solved, the xds-replica fileshare is not in the same place as before the upgrade.

Imagine this scenario. When first deploying Lync 2013 there was only a C drive with let's say 100 GB of space. Then later for some reason you create another disk on the server let's say to have a seperate drive for CLS logging. If the new drive has more space free than the C drive you will probably run into this problem.

As we know, during installation of Skype For Business Server 2015 the xds-replica share is always created on drive with most available space. During an upgrade scenario this should not happen. 

Bug has been reported to the Skype For Business server 2015 product team.

Wednesday 21 January 2015

Lync 2013 Frontend service not starting after CU

I have experienced a situation where a Lync Server 2013 front end service would not start after updating from RTM to CU10 (January 2015). RTM version servers were running fine. The deployment is a migration from Lync Server 2010 to Lync Server 2013.

Following errors were seen in the eventlog:
Event ID:14562
Event ID: 14651
Event ID: 12303 (twice)

Details:
Log Name:      Lync Server
Source:        LS Protocol Stack
Date:          21-01-2015 11:46:26
Event ID:      14562
Task Category: (1001)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE01.domain.local
Description:
Two servers cannot be configured at the same FQDN with different 'Outbound Only' options.

Cannot configure a server at FQDN [0.0.0.0] because another server is already configured there with a different 'Outbound Only' option.
Cause: This is a configuration problem.
Resolution:
Review the server roles that are configured at this FQDN and ensure that they have identical trust options.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Protocol Stack" />
    <EventID Qualifiers="50153">14562</EventID>
    <Level>2</Level>
    <Task>1001</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-01-21T10:46:26.000000000Z" />
    <EventRecordID>8402</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE01.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>0.0.0.0</Data>
  </EventData>

</Event>

Log Name:      Lync Server
Source:        LS Protocol Stack
Date:          21-01-2015 11:46:26
Event ID:      14651
Task Category: (1001)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE01.domain.local
Description:
A serious configuration problem is preventing Lync Server from functioning.

An unexpected error occurred while configuring the internal servers table. Contact Product Support Services.

Cause: The Lync Server failed to initialize due to invalid settings from CMS.
Resolution:
Review and correct the CMS configuration, then start the service again.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Protocol Stack" />
    <EventID Qualifiers="50153">14651</EventID>
    <Level>2</Level>
    <Task>1001</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-01-21T10:46:26.000000000Z" />
    <EventRecordID>8403</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE01.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>An unexpected error occurred while configuring the internal servers table. Contact Product Support Services.</Data>
  </EventData>
</Event>

Log Name:      Lync Server
Source:        LS Server
Date:          21-01-2015 11:46:26
Event ID:      12303
Task Category: (1000)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE01.domain.local
Description:
The protocol stack reported a critical error: code C3E93C66 (SIPPROXY_E_MULTIPLE_INCOMPATIBLE_TRUST_OPTIONS). The service has to stop.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Server" />
    <EventID Qualifiers="50152">12303</EventID>
    <Level>2</Level>
    <Task>1000</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-01-21T10:46:26.000000000Z" />
    <EventRecordID>8404</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE01.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>C3E93C66</Data>
    <Data>SIPPROXY_E_MULTIPLE_INCOMPATIBLE_TRUST_OPTIONS</Data>
  </EventData>

</Event>

Log Name:      Lync Server
Source:        LS Server
Date:          21-01-2015 11:46:26
Event ID:      12303
Task Category: (1000)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE01.domain.local
Description:
The protocol stack reported a critical error: code C3E93C66 (Configuration failure prevented the server from starting up). The service has to stop.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Server" />
    <EventID Qualifiers="50152">12303</EventID>
    <Level>2</Level>
    <Task>1000</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-01-21T10:46:26.000000000Z" />
    <EventRecordID>8405</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE01.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>C3E93C66</Data>
    <Data>Configuration failure prevented the server from starting up</Data>
  </EventData>

</Event>

Working with Microsoft Premier Support on this, we found that 2 application servers were giving problems, both indicating they had FQDN [0.0.0.0] (which is also noticed in the event ID 14562).

In the logging we also saw the registred name of the servers were IP addresses and also on which ports communication should be attempted. These IP hosts were not responding to ping or telnet to the specified TCP port. These were no PTR records registered to these IP addresses.

The servers were Trusted Application servers that are not used anymore (old obsolete gateway from OCS2007R2 and an RCC test server), so deleting the trusted application and the trusted application pool was safe. 

After a management store replication, the front end service (RTCSRV) could start on FE01.domain.local and the rest of the servers were updated and the CU10 was successfully deployed.

Lesson learned: Always check the status of ALL the servers before a migration is started.

Tuesday 20 January 2015

PortqryUI

PortQryUI is a wonderful little tool, that I use quite often for troubleshooting network connectivity and checking e.g. loadbalancers etc.

The tool is used for checking if the ports are listening to traffic or not. Results can be the following:
Listening (TCP or UDP) indicates a service are listening on the port.
Filtered (TCP or UDP) indicates a service might be listening on the port.
Not listening (TCP or UDP) indicates a service is not listening on the port.

The tool can also be used from command line interface, syntax for commands can be found here: http://technet.microsoft.com/en-us/library/cc779626.aspx

However checking all the Lync services by hand, using this tool, can be a little cumbersome, so I have added the following lines to the config.xml file:

  <Service Name="Lync hardware loadbalancer only">
    <Port Name="SIP TLS" Value="5061" Protocol="TCP"/>
    <Port Name="HTTPS Focus - FE" Value="444" Protocol="TCP"/>
    <Port Name="DCOM / RPC" Value="135" Protocol="TCP"/>
    <Port Name="HTTP internal" Value="80" Protocol="TCP"/>
    <Port Name="HTTP external" Value="8080" Protocol="TCP"/>
    <Port Name="HTTPS internal" Value="443" Protocol="TCP"/>
    <Port Name="HTTPS external" Value="4443" Protocol="TCP"/>
    <Port Name="SIP mediation" Value="5070" Protocol="TCP"/>
    <Port Name="SIP response group" Value="5071" Protocol="TCP"/>
    <Port Name="SIP attendant dialin" Value="5072" Protocol="TCP"/>
    <Port Name="SIP conf announcement" Value="5073" Protocol="TCP"/>
    <Port Name="SIP call park" Value="5075" Protocol="TCP"/>
    <Port Name="SIP audio test" Value="5076" Protocol="TCP"/>
    <Port Name="CAC Edge TURN" Value="5080" Protocol="TCP"/>
    <Port Name="CAC FE" Value="448" Protocol="TCP"/>
  </Service>
  <Service Name="Lync hardware loadbalancer DNS">
    <Port Name="HTTP internal" Value="80" Protocol="TCP"/>
    <Port Name="HTTP external" Value="8080" Protocol="TCP"/>
    <Port Name="HTTPS internal" Value="443" Protocol="TCP"/>
    <Port Name="HTTPS external" Value="4443" Protocol="TCP"/>
  </Service>


Then I can easily select the service I want to check and just enter the DNS name for the pool.
 


The PortQry tool can be downloaded from here: http://www.microsoft.com/en-us/download/details.aspx?id=24009