Missing indexes in LcsCDR database

I investigated some users complaining over Skype for Business (Lync) Monitoring Reports for Response Groups that timed out and didn't return data.


The query in the report Response Group Usage Data uses the stored procedure CdrRGSUsageTrend in the monitoring database LcsCDR.


I went through the estimated execution plan in SQL for this stored proc

The execution plan showed a number of missing indexes, which I created (listed below)


CREATE NONCLUSTERED INDEX [IX_SessionDetails_Missing1] ON [dbo].[SessionDetails]
([ReplacesDialogIdTime] ASC,[SessionIdTime] ASC,[ReplacesDialogIdSeq] ASC,[CallFlag] ASC,[MediaTypes] ASC,[User1ClientVerId] ASC,[User2ClientVerId] ASC,[SessionIdSeq] ASC,[SessionStartedById] ASC,[User1Id] ASC,[User2Id] ASC,[CorrelationId] ASC,[ReferredById] ASC)
INCLUDE ([TargetUserId],[ResponseTime],[ResponseCode],[SessionEndTime]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 80) ON [PRIMARY]
GO


CREATE NONCLUSTERED INDEX [IX_SessionDetails_Missing2] ON [dbo].[SessionDetails]
([CorrelationId] ASC,[SessionIdTime] ASC,[ReplacesDialogIdTime] ASC,[ReplacesDialogIdSeq] ASC,[CallFlag] ASC,[MediaTypes] ASC,[User1ClientVerId] ASC,[User2ClientVerId] ASC,[SessionIdSeq] ASC,[SessionStartedById] ASC,[User1Id] ASC,[User2Id] ASC,[ReferredById] ASC)
INCLUDE ([TargetUserId],[ResponseTime],[ResponseCode],[SessionEndTime]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 80) ON [PRIMARY]
GO


CREATE NONCLUSTERED INDEX [IX_SessionDetails_Missing3] ON [dbo].[SessionDetails]
([ReplacesDialogIdTime] ASC,[ReplacesDialogIdSeq] ASC,[MediaTypes] ASC) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
GO


CREATE NONCLUSTERED INDEX [IX_SessionDetails_Missing4] ON [dbo].[SessionDetails]
([CorrelationId] ASC,[MediaTypes] ASC) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
GO


CREATE NONCLUSTERED INDEX [IX_SessionDetails_Missing5] ON [dbo].[SessionDetails]
([ReplacesDialogIdTime] ASC,[ReplacesDialogIdSeq] ASC,[SessionIdTime] ASC,[MediaTypes] ASC)
INCLUDE ([SessionIdSeq],[CorrelationId],[User1Id],[User2Id],[SessionStartedById],[User1ClientVerId],[User2ClientVerId],[CallFlag]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
GO


CREATE NONCLUSTERED INDEX [IX_SessionDetails_Missing6] ON [dbo].[SessionDetails]
([ReplacesDialogIdTime] ASC,[ReplacesDialogIdSeq] ASC,[SessionIdTime] ASC,[MediaTypes] ASC)
INCLUDE ([SessionIdSeq],[User1Id],[User2Id],[SessionStartedById],[ReferredById],[User1ClientVerId],[User2ClientVerId],[ResponseCode]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
GO


CREATE NONCLUSTERED INDEX [IX_SessionDetails_Missing7] ON [dbo].[SessionDetails]
([ReplacesDialogIdTime] ASC,[ReplacesDialogIdSeq] ASC,[SessionIdTime] ASC,[MediaTypes] ASC)
INCLUDE ([SessionIdSeq],[CorrelationId],[User1Id],[User2Id],[SessionStartedById],[User1ClientVerId],[User2ClientVerId],[ResponseCode],[CallFlag]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
GO


After creating these indexes, the report was succesfully generated.

Access denied when creating local databases

When I was installing a new Skype for Business Server 2015 frontend server pool for a customer, installation of the Skype for Business components failed with this useless error messages: 0x80004005 (Unspecified error) and failure code 1603 (which is a generic error code).

Digging into the LCSSetup_Commands.log showed this entry:

Creating database rtcdyn from scratch. Data File Path = D:\CsData\RtcDatabaseStore\rtclocal\DynDbPath, Log File Path= E:\CsData\RtcDatabaseStore\rtclocal\DynLogPath.
Exception Stack:
Type: Microsoft.SqlServer.Management.Smo.FailedOperationException, Message: Create failed for Database 'rtcdyn'.
Type: Microsoft.SqlServer.Management.Common.ExecutionFailureException, Message: An exception occurred while executing a Transact-SQL statement or batch.
Type: System.Data.SqlClient.SqlException, Message: CREATE FILE encountered operating system error 5(Access is denied.) while attempting to open or create the physical file 'D:\CsData\RtcDatabaseStore\rtclocal\DynDbPath\rtcdyn.mdf'.
CREATE DATABASE failed. Some file names listed could not be created. Check related errors.

Access denied from the operating system?? Investigating the Security eventlog I found som of these entries: 

Issue is related to insufficient filesystem priviliges.

Checking the filesystem priviliges, I found that the special permissions not granted in the picture above, was missing from the root of the drive. 

How it should look like:

Adding the default user permissions the root of the drive fixed the problem and the installation of the server could continue.

Problem was caused by additional security settings made by the customer.

Change SIP URI of a response group

A customer wanted me to change the SIP uri of a response group workflow. This is not an option using the Response Group Configuration Tool, so what you must do is delete the existing workflow and create a new with the same parameters, the workflow Name is a unique identifier.

This is not what I wanted, so I turned to Powershell.

Tried:
$wf = Get-CsRgsWorkflow | where-object {$_.Name -eq "RGS test"}
$wf.primaryuri = 'sip:testrgs@sip.dom'
Set-CsRgsWorkflow $wf
But this was, kind of expected, not possible. You cannot change the primary uri of a Response Group Workflow...

Off course I can, I just need to figure out how.

In the backend database of the frontend pool hosting the RGS application, there is a database RGSDYN, I have blogged about that previously here: https://uctales.blogspot.dk/2016/02/response-group-agent-state.html, this database contains all dynamic data on e.g. agents logged in state. There is also a RGSCONFIG database, this database contains all RGS configuration.

Going to SQL Server Management Studio and opening a query window and entered this query:
use rgsconfig
go
Select Name,PrimaryUri from Workflows where PrimaryUri like 'sip:rgstest%'
Shows the Name and SIP uri of the RGS test workflow.

Changing the SIP uri of the workflow using SSMS, from sip:rgstest@sip.dom to sip:testrgs@sip.dom.
use rgsconfig
go
update Workflows
set PrimaryUri = 'sip:testrgs@sip.dom'
where PrimaryUri = 'sip:rgstest@sip.dom'

Now we have changed the SIP uri of the workflow, but this is not enough, we must also change SIP uri and proxy address of the application contact object i Active Directory.

To find the application contact, we go to Powershell again and searches for the endpoints.
Get-CsApplicationEndpoint | Where-Object {$_.sipaddress -like "sip:testrgs*"}
This cmdlet returns the application endpoint identity, which we need to locate the AD object itself.



Go to ADSIedit.msc and open the configuration store -> Services -> RTC service -> Application Contacts and edit the object attributes msRTCSIP-PrimaryUserAddress to sip:testrgs@sip.dom and proxyAddresses to sip:testrgs@sip.dom

Tested workflow and everything works nicely. Job done.

Disclaimer: Editing the database entries and AD objects for Application Contacts, might lead to an undesirable state, unrecoverable failures, unsupported solution or smelly feet. Please only do this at your own risk and if you know what you are doing and are able to recover from these risks.

What is what when it comes to the contact card

I was wondering which attributes from an AD user object, is shown on the Lync/Skype4B contact card.

Let's check it out.

I took an old friend of mine from my lab, Mr. Baggins, you might know him, not so tall chap rather polite and always dresses perfectly for the occassion, gave him a few different values for his AD attributes.

Then I ran the Update-CsAddressbook cmdlet and waited for the event 21056 to appear. All good. Started the client and got the newest address book, ready to go.

Searching for Bilbo, got this result

Obviously we see the Display Name (red square), Job Title (green square) and Department (blue square).

Showing the contact card, rendered this result.

Here we see a clear mapping between the AD attributes and the contact card fields.

Using the ABSConfig.exe tool (part of the Lync/Skype4B reskit), we can clearly see these mappings.

We can also adapt the mapping between these fields, and leave some of them out, if necessary. There is an extra column to the right, with the label "Enabled" removing the tick from the box, removes the attribute from the contact card.

Response group agent state or how to check RGS agent state

A client recently had an issue with a response group (RGS) queue, where the call came nicely into the queue, but the call was not presented to the agents.

A small query into the Skype4B backend database gave us the answer: The agent were not logged into the RGS.

select T1.Name, T3.DisplayName, T4.[State] 
from [rgsconfig].dbo.AgentGroups as T1
join rgsconfig.dbo.AgentGroupsToAgentsMap as T2
on T1.ID = T2.AgentGroupId 
join rgsconfig.dbo.Agents as T3
on T2.AgentId = T3.ID
left join rgsdyn.dbo.AgentGroupSignInStates as T4
on (T2.AgentGroupId = T4.GroupId and T2.AgentId = T4.AgentId)
where T4.[State] in (0,1) and T1.Name like 'servicedesk 1%'
group by T1.Name,T1.ID,T3.DisplayName,T3.ID,T4.[State]

Gave us a nice little list of the users:

State 0 = not logged in
State 1 = logged in
If the user is not in the list, the user has never signed into the response group.

We can see that there are no users active in Servicedesk 1. line group, so this explains why no calls are presented to the agents

Database mirror monitoring

When a Skype for Business Server 2015 backend SQL server is using database mirroring for high availability, some information on issues with the mirroring relation is required.


Set up Database Mirroring Monitor:
Launch the Database Mirroring Monitor tool



Go to Menu -> Action and select Register Mirrored Database



Select the primary server in the drop down menu and tick all the databases you want to monitor



Select a database and go to Warnings pane, click Set Thresholds



Set thresholds like this



Repeat for every database mirror relation you want to monitor


Now you will get an informational event 32042 in the application event log when the size of unsent transaction exceeds 100KB



And you will recieve an informational event 32040 in the application event log when the age of the oldest unsent transaction surpasses 5 minutes (4 minutes on mirror server)



Now you will add the event log entries to your favorite monitoring tool and you should be covered from this event, if you react properly to these above event

Which occurs when the transaction log is full due to database mirroring stopped.

Skype4B / IIS 8 not presenting new certificate in browser.

After replacing a couple of old, soon to expire, default certificates on the frontend pool, I discovered that the old certificate was still being presented to the client.

Checking the IIS manager binding for the website, I saw the correctly assigned new certificate.

Running this command:
netsh http show sslcert
Showed that some of the ippools still having the thumbprint of the old certificate, so going back into bindings for the website. I changed the IP address assignment from Any unused to the IP address of the server, resetting the IIS (iisreset.exe) and changing the IP address assignment back to Any unused.

Ran the command again and this time the correct thumbprint was shown for all the ippools.

Test showed the correct certificate now being presented to clients.