Migrate a migration scenario to Skype for Business Server 2015

At a large customer, we are stuck in a migration scenario between Lync Server 2010 and Lync Server 2013, because of dependant non-Lync projects, like migration from a "legacy" e-mail/calendering system to Exchange 2013. We are unable to move users to Lync Server 2013 for now.

Last week, the same customer requested an upgrade to Skype for Business Server 2015 and asked if it was possible to in-place upgrade their Lync Server 2013 pool to Skype for Business Server 2015. My first reaction was off course: No. You cannot. It is unsupported to have more than 1 legacy version installed before Skype4B.

Customer responded: I didn't ask if it is supported, I asked if it is possible.

So taking this request to the lab servers, I installed a new AD with a few user, a Lync Server 2010 SE enabled some users. Then a Lync Server 2013 SE and a Skype4B management server.

I started the Skype4B topology builder and downloaded the topology, and got to this result:

Nope, not possible... Clicked OK.

Tried anyway to upgrade my Lync Server 2013 pool: 

Right click the pool and press Upgrade to Skype for Business Server 2015

Published the topology

 Opened the to-do text file at the end

This still gives me the creeps after 5 Skype4B upgrades

On the Lync Server 2013, I could mount the ISO file and start the installation (at least after adding another 20Gb of disk space to the VM). 

In the end, all services are running and users can be moved to, and connect to, the upgraded server.

So, it is possible to migrate a migration scenario to Skype for Business Server 2015

Bug in inplace upgrade to Skype for Business Server 2015: Replica Replicator Agent won't start

As many in the community, we (@torstenegebirk from egebirk-public.sharepoint.com/blog and me) was excited to see Skype for Business Server 2015 being released on May 1st, 2015.

First installation in lab went apparently fine, after going through the prerequisites we added an extra 35Gb harddisk to the VM, formatted and assigned drive letter E. Rest of the installation process is out of scope for this post.

After installation all services very running with their lovely new "short" service names. Databases were where we expected them to be. Lovely.

As the new E drive didn't contain any data, we removed the harddisk, and rebooted the server. Then problems started.

After the reboot the Skype For Business Server Replica Replicator Agent service would'nt start (thanks to the Redundant Department of Redundancy). In the logging we saw event ID 3007 on the LS Replica Replicator Agent Service, see screenshot:

The problem is obviously a missing share on the E drive.

During the inplace upgrade the xds-replica shared folder was without notification moved from the original place, could be C or D (C in our case), to the drive with most available space. This is a problem, if you, for some reason, have placed your xds-replica folder on a certain drive for data storage.

The quick fix is to do following:

• Recreate an E drive
• Create a folder structure E:\RtcReplicaRoot\xds-replica\
• Restart the Server service (this will "reshare" the folder after detecting that the physical location now exists)
• Make sure NETWORK SERVICE and RTC Local Config Replicator (local security group on the front end servers) has Full Control in the share permissions
• Make sure RTC Local Config Replicator local security group has full control in NTFS permissions
• Start the Skype for Business Server 2015 Local Replica Replicator Agent service again
• When service is started run the Invoke-CsManagementStoreReplication cmdlet
• Wait a minute or so (depending on size of deployment) and check the status with the Get-CsManagementStoreReplicationStatus cmdlet
• All servers should have Up-To-Date set to True

Then the folder share is available again with the correct share permissions

You could also follow Ken Lasko's post here: http://ucken.blogspot.com/2012/04/resetting-lync-cms-replication.html

The problem is still not solved, the xds-replica fileshare is not in the same place as before the upgrade.

Imagine this scenario. When first deploying Lync 2013 there was only a C drive with let's say 100 GB of space. Then later for some reason you create another disk on the server let's say to have a seperate drive for CLS logging. If the new drive has more space free than the C drive you will probably run into this problem.

As we know, during installation of Skype For Business Server 2015 the xds-replica share is always created on drive with most available space. During an upgrade scenario this should not happen. 

Bug has been reported to the Skype For Business server 2015 product team.

Lync 2013 Frontend service not starting after CU

I have experienced a situation where a Lync Server 2013 front end service would not start after updating from RTM to CU10 (January 2015). RTM version servers were running fine. The deployment is a migration from Lync Server 2010 to Lync Server 2013.

Following errors were seen in the eventlog:
Event ID:14562
Event ID: 14651
Event ID: 12303 (twice)

Details:
Log Name:      Lync Server
Source:        LS Protocol Stack
Date:          21-01-2015 11:46:26
Event ID:      14562
Task Category: (1001)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE01.domain.local
Description:
Two servers cannot be configured at the same FQDN with different 'Outbound Only' options.

Cannot configure a server at FQDN [0.0.0.0] because another server is already configured there with a different 'Outbound Only' option.
Cause: This is a configuration problem.
Resolution:
Review the server roles that are configured at this FQDN and ensure that they have identical trust options.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Protocol Stack" />
    <EventID Qualifiers="50153">14562</EventID>
    <Level>2</Level>
    <Task>1001</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-01-21T10:46:26.000000000Z" />
    <EventRecordID>8402</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE01.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>0.0.0.0</Data>
  </EventData>

</Event>

Log Name:      Lync Server
Source:        LS Protocol Stack
Date:          21-01-2015 11:46:26
Event ID:      14651
Task Category: (1001)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE01.domain.local
Description:
A serious configuration problem is preventing Lync Server from functioning.

An unexpected error occurred while configuring the internal servers table. Contact Product Support Services.

Cause: The Lync Server failed to initialize due to invalid settings from CMS.
Resolution:
Review and correct the CMS configuration, then start the service again.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Protocol Stack" />
    <EventID Qualifiers="50153">14651</EventID>
    <Level>2</Level>
    <Task>1001</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-01-21T10:46:26.000000000Z" />
    <EventRecordID>8403</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE01.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>An unexpected error occurred while configuring the internal servers table. Contact Product Support Services.</Data>
  </EventData>
</Event>

Log Name:      Lync Server
Source:        LS Server
Date:          21-01-2015 11:46:26
Event ID:      12303
Task Category: (1000)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE01.domain.local
Description:
The protocol stack reported a critical error: code C3E93C66 (SIPPROXY_E_MULTIPLE_INCOMPATIBLE_TRUST_OPTIONS). The service has to stop.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Server" />
    <EventID Qualifiers="50152">12303</EventID>
    <Level>2</Level>
    <Task>1000</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-01-21T10:46:26.000000000Z" />
    <EventRecordID>8404</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE01.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>C3E93C66</Data>
    <Data>SIPPROXY_E_MULTIPLE_INCOMPATIBLE_TRUST_OPTIONS</Data>
  </EventData>

</Event>

Log Name:      Lync Server
Source:        LS Server
Date:          21-01-2015 11:46:26
Event ID:      12303
Task Category: (1000)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      FE01.domain.local
Description:
The protocol stack reported a critical error: code C3E93C66 (Configuration failure prevented the server from starting up). The service has to stop.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="LS Server" />
    <EventID Qualifiers="50152">12303</EventID>
    <Level>2</Level>
    <Task>1000</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-01-21T10:46:26.000000000Z" />
    <EventRecordID>8405</EventRecordID>
    <Channel>Lync Server</Channel>
    <Computer>FE01.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>C3E93C66</Data>
    <Data>Configuration failure prevented the server from starting up</Data>
  </EventData>

</Event>

Working with Microsoft Premier Support on this, we found that 2 application servers were giving problems, both indicating they had FQDN [0.0.0.0] (which is also noticed in the event ID 14562).

In the logging we also saw the registred name of the servers were IP addresses and also on which ports communication should be attempted. These IP hosts were not responding to ping or telnet to the specified TCP port. These were no PTR records registered to these IP addresses.

The servers were Trusted Application servers that are not used anymore (old obsolete gateway from OCS2007R2 and an RCC test server), so deleting the trusted application and the trusted application pool was safe. 

After a management store replication, the front end service (RTCSRV) could start on FE01.domain.local and the rest of the servers were updated and the CU10 was successfully deployed.

Lesson learned: Always check the status of ALL the servers before a migration is started.

PortqryUI

PortQryUI is a wonderful little tool, that I use quite often for troubleshooting network connectivity and checking e.g. loadbalancers etc.

The tool is used for checking if the ports are listening to traffic or not. Results can be the following:
Listening (TCP or UDP) indicates a service are listening on the port.
Filtered (TCP or UDP) indicates a service might be listening on the port.
Not listening (TCP or UDP) indicates a service is not listening on the port.

The tool can also be used from command line interface, syntax for commands can be found here: http://technet.microsoft.com/en-us/library/cc779626.aspx

However checking all the Lync services by hand, using this tool, can be a little cumbersome, so I have added the following lines to the config.xml file:

  <Service Name="Lync hardware loadbalancer only">
    <Port Name="SIP TLS" Value="5061" Protocol="TCP"/>
    <Port Name="HTTPS Focus - FE" Value="444" Protocol="TCP"/>
    <Port Name="DCOM / RPC" Value="135" Protocol="TCP"/>
    <Port Name="HTTP internal" Value="80" Protocol="TCP"/>
    <Port Name="HTTP external" Value="8080" Protocol="TCP"/>
    <Port Name="HTTPS internal" Value="443" Protocol="TCP"/>
    <Port Name="HTTPS external" Value="4443" Protocol="TCP"/>
    <Port Name="SIP mediation" Value="5070" Protocol="TCP"/>
    <Port Name="SIP response group" Value="5071" Protocol="TCP"/>
    <Port Name="SIP attendant dialin" Value="5072" Protocol="TCP"/>
    <Port Name="SIP conf announcement" Value="5073" Protocol="TCP"/>
    <Port Name="SIP call park" Value="5075" Protocol="TCP"/>
    <Port Name="SIP audio test" Value="5076" Protocol="TCP"/>
    <Port Name="CAC Edge TURN" Value="5080" Protocol="TCP"/>
    <Port Name="CAC FE" Value="448" Protocol="TCP"/>
  </Service>
  <Service Name="Lync hardware loadbalancer DNS">
    <Port Name="HTTP internal" Value="80" Protocol="TCP"/>
    <Port Name="HTTP external" Value="8080" Protocol="TCP"/>
    <Port Name="HTTPS internal" Value="443" Protocol="TCP"/>
    <Port Name="HTTPS external" Value="4443" Protocol="TCP"/>
  </Service>

Then I can easily select the service I want to check and just enter the DNS name for the pool.

 



The PortQry tool can be downloaded from here: http://www.microsoft.com/en-us/download/details.aspx?id=24009

Response group queue overflow and timeout

A customer wanted to give an announcement to callers and disconnect the call, when the queue length was more than 2 calls. 

This must be done using Lync Management Shell.

$w = New-CsRgsPrompt -TextToSpeechPrompt "Announcement to caller"
$y = New-CsRgsCallAction -Prompt $w -Action Terminate
$z = Get-CsRgsQueue -Identity service:ApplicationServer:<pool-FQDN> -Name "<Queue-name>"
$z.OverflowAction = $y
$z.OverflowCandidate = "NewestCall"
$z.OverflowThreshold = "2"
Set-CsRgsQueue -Instance $z

Speaking of this, giving an announcement to callers before timeout disconnect can be done this way.

$w = New-CsRgsPrompt -TextToSpeechPrompt "Announcement to caller"
$y = New-CsRgsCallAction -Prompt $w -Action Terminate
$z = Get-CsRgsQueue -Identity service:ApplicationServer:<pool-FQDN> -Name "<Queue-name>"
$z.TimeoutAction = $y
$z.TimeoutThreshold = "60"
Set-CsRgsQueue -Instance $z

Happy shell'ing 

User is not validly homed on target

At a new deployment I saw an issue when some users were enabled for Lync, they could not sign in.

When we investigated the client certificates, I saw this error:

Disable and re-enable the user did not change anything.

Luckily we also had a Survivable Branch Appliance (SBA) in the deployment, so forcely moving the user to the SBA, did change something, so the user was able to login. Hey now things work, but having the user at the SBA was not what we wanted, so we moved the user back to main pool and we were back at the beginning... The user could not sign in. 

Luckily we had also a test user with this problem, so production users would work and we could do more investigation for this problem.

I escalated the problem to Microsoft PSS, who is working on it now... Hopefully this post will be updated soon.

Update: The problem was caused by some incorrect routing group IDs, the problem was identified by event id 30057 in the Lync Server eventlog, text is:
msRTCSIP-UserRoutingGroupId attribute for a user is incorrect in Active Directory domain:<AD-domain>
The user with these attributes has DN:<USER-DN>
These incorrect attribute cannot be updated in the database.
Source of replication is: LocalRegistrarReplication.
Resolution:
This is usually caused due to home pool information change in Active Directory using unsupported tools or direct edits to AD. To resolve this condition, move all the affected user back to the original Pool using Move-CsUser and then move the user again to this Pool gracefully using Move-CsUser.

Neither moving the user (who btw originated on this very pool), removing the content of MSRTCSIP-UserRoutingGroupID field of the object nor a full reset of the pool did resolve this issue. So we decided to renew the local databases.

First backup userdata (just in case):
Export-CsUserData -PoolFQDN <fqdn> -FileName <filename>

Then on all FEs:
Stop-CsWindowsServices
Install-CsDatabase -LocalDatabases -Clean
Restart-Server

After all the servers are back online the error was gone.

The problem was also causing some response groups to stay in offline mode with event id 30058 in the Lync Server eventlog, this was also solved with this action.

Forwarding hunt group calls to a group voicemail

During a deployment, we were met with the request of calls in a hunt group (or queue) should be able to go to an Exchange UM enabled mailbox for the department/function.

HG name is department@sipdomain e.g. test@kml.local
Exchange mailbox is department@sipdomain e.g. test@kml.local
UPN for the Exchange mailbox is the same as the e-mailaddress

The mailbox account is (should be) disabled in AD, so first move is to enable the account. Then I enable the account for Lync using Powershell:
Get-CsAdUser -Identity "test@kml.local"|Enable-CsUser -RegistrarPool Lyncpool.kml.local -SipDomain kml.local -SipAddress sip:vm.test@kml.local

The Lync user should be Enterprise Voice enabled to be able to utilize the UM mailbox
Set-CsUser -Identity test@kml.local -EnterpriseVoiceEnabled $True

Next step is UM enabling the mailbox (implies that a UM Mailbox Policy has already been defined) this could be done using the Exchange Control Panel or Powershell:
Enable-UMMailbox -Identity "test@kml.local" -UMMailboxPolicy UmPolicy1 -Extensions 2000 -PIN 13579 -SIPResourceIdentifier "test@kml.local"

Now we'll use SEFAUtil to set the Lync forwarding destination for test@kml.local.
.\SEFAUtil.exe /server:lyncpool.kml.local test@kml.local /enablefwdimmediate /setfwddestination:"sip:test@kml.local;app=voicemail"

Now we can assign the Forward to Voicemail destination in the HG setup and don't forget to record a greeting in the Exchange mailbox.

After recording a greeting, you should consider disabling the AD account again. If you decide to disable the account, the users of the mailbox cannot record a new greeting unless an administrator (you) reenable the account again.