Following on from previous article Citrix Cloud – Cloud Connector in-depth review(including Local Host Cache). which touched on LHC i have been keen to test this functionality as it has been a big concern for enterprises who raised the question around “What happens if the internet is not available?” which LHC for Citrix Cloud clearly addresses.
At the time of writing that article the LHC functionality has not yet reached my citrix cloud setup.It appears mid-march my citrix connector updated and the SQL binaries and database appeared which had not been there originally which indicated to me that LHC has been enabled.
Each resource location must have an on-premises StoreFront installed and configured. Local Host Cache works only in resource locations containing an on-premises StoreFront as the hosted citrix cloud storefront is dependant on access to Citrix Cloud.
In order to enter outage mode or return to normal operation after outage the storefront server must be online and available. LHC will not enter/exit Outage if this is online and the below message will be logged on the XaXdProxy Trace:-
OutageModeSupport.IsConnectorOutageModeEnabled – Disable LHC because last Nfuse activity expired after: 3.00:00:00
EntryExit Enter: OutageModeSupport.EnterOutageMode – Transitioning component OutageModeForced has reported a failure with reason = Forced Outage Mode Entry
EntryExit Exit: OutageModeSupport.EnterOutageMode – Cannot enter outage mode due to NoNFuseActivity (undefined)
Local Host Cache is supported for server-hosted applications and desktops, and static (assigned) desktops; it is not supported for pooled VDI desktops created by MCS or PVS.
What is unavailable during an outage, and other differences
- You cannot use the Manage functions (Studio) in Citrix Cloud for items in the resource location experiencing the outage, or run PowerShell cmdlets.
- Monitoring data is not sent to Citrix Cloud during an outage. So, the Monitor functions (Director) do not show activity from an outage interval.
- Hypervisor credentials cannot be obtained from the Host Service. All machines are in the unknown power state, and power operations cannot be issued. However, VMs on the host that are powered-on can be used for connection requests.
- Power-managed desktop VDAs in pooled Delivery Groups that have the “ShutDownDesktopsAfterUse” property enabled are placed into maintenance mode when an outage occurs.
- An assigned machine can be used only if the assignment occurred before the outage. New assignments cannot be made during an outage.
- Automatic enrollment and configuration of Remote PC Access machines is not possible. However, machines that were enrolled and configured before the outage can accept connections.
- Server-hosted applications and desktop users can use more sessions than their configured session limits, if the resources are in different resource locations.
Validating LHC Is Enabled
To validate this i ran the following powershell command using the XenApp and XenDesktop Remote SDK you can confirm the status of your Citrix Cloud
This confirmed the LocalHostCacheEnabled was set as True.
Local Host Cache Database
SQL will not appear to be installed via the normal add/remove programs but the binaries will be located in %ProgramFiles%\Microsoft SQL Server\120\LocalDB\
The local host database is stored in “C:\Windows\ServiceProfiles\NetworkService” HaDatabaseName.mdf & HaDatabaseName_log.ldf as shown below.
Force an Outage
To test the local host cache functionality add the following registry key which forces an outage ensuring
HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\DesktopServer\LHC DWORD : OutageModeForced VALUE : 1
0 – Takes the Cloud Connector out of outage mode
1 – Put the cloud connector in Outage mode
Running the following in command prompt will add the key and kick off the process of forcing an outage/simulating connectivity issue with Citrix Cloud.
REG ADD HKLM\SOFTWARE\Citrix\DesktopServer\LHC /f /v OutageModeForced /t REG_DWORD /d 1
Reviewing the Event logs the below is the sequence of events:-
The Citrix remote broker service will log Event ID 3001 indicating that an outage has been forced:-
The Citrix High Availability service will log Event ID 3502 indicating service has now become active and all user requests will be brokered until through the high availability service until the issue has been resolved.
The Citrix CofigSync service will log Event ID 507 indicating any updated config received will be abandoned until system is returned to normal mode.
This event is only likely created when outage mode is forced using the key and when citrix cloud is accessible via an internet connection as configuration changes would not be normally occur during a real outage
The Citrix remote broker service will continuously log Event ID 3003 indicating that an outage is still operating (extended outage)
The following video cover adding the registry key, validating outage in Application Log and testing resource enumeration using a locally installed storefront within the same resource location:-
During an outage, if a Cloud Connector is restarted:
- If that Cloud Connector is not the elected primary broker, the restart has no impact.
- If that Cloud Connector is the elected primary broker, a different Cloud Connector is elected, causing VDAs to register. After the restarted Cloud Connector powers on, it automatically takes over brokering, which causes VDAs to register again. In this scenario, performance can be affected during the registrations.
Revert back to Normal Mode
To go back to normal mode add the following registry key which forces an outage ensuring
HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\DesktopServer\LHC DWORD : OutageModeForced VALUE : 0
Running the following in command prompt will add the key and kick off the process of returning to normal mode/simulating connectivity restored with Citrix Cloud.
REG ADD HKLM\SOFTWARE\Citrix\DesktopServer\LHC /f /v OutageModeForced /t REG_DWORD /d 0
Once returning to normal mode, this can take up to 10-15 minutes, Event ID 3003 will indicate the registry has been detected and it can take a while for the brokering to be switched back to citrix cloud.
The Citrix High Availability service event ID 3503 is recorded indicating the issue with normal brokering has been resolved and that Citrix High Availability services will stop participating in broker requests.
As per previous event, event ID 2007/2008 for the Citrix High Availability service confirms the XML Services is stopped
And finally the Citrix Remote Broker Provider Service indicate Working Normally which indicates the Outage mode has returned to normal mode.
The following video cover adding the registry key, validating outage is returned to normal mode in Application Log:-
Storefront Bypass Duration
An Issue was identified with the StoreFront server marking the cloud connectors as offline when there was a network interruption. Although the 60 minute default applies if there is multiple cloud connectors added, a single cloud connector will be by default excluded for 5 minutes which means requests will be rejected until this has completed.
As per https://support.citrix.com/article/CTX233424 it is recommend to reduce the bypassduration setting in the storefront servers to 0.
1) Select the store on the StoreFront server
2) Manage delivery controllers
3) Edit Delivery controller
4) Select Settings under the Advanced settings
5)Change the bypass duration to 0
The LocalDB service can use approximately 1.2 GB of RAM (up to 1 GB for the database cache, plus 200 MB for running SQL Server Express LocalDB). The High Availability Service can use up to 1 GB of RAM if an outage lasts for an extended interval with many logons occurring (for example, 12 hours with 10K users). This is additional to the minimum Cloud Connector RAM requirements so 2.2GB should be added to your Cloud Connector VM/Allocated a higher spec Azure VM.
The number of cores available to the SQL Server Express LocalDB, directly affects Local Host Cache performance. This CPU overhead is observed only during the outage period when the database is unreachable and the High Availability service is active.
LocalDB is limited to the use of a Single Socket and the advise is allocate more cores with single socket as additional sockets/cores will not improve the performance.
The broker must have sufficient space on the drive where the LocalDB is installed to allow for the database growth during an outage. Citrix has provided an example, during a logon/logoff test running at 10 logons per second, the database grew by one MB every 2-3 minutes.
During an outage, one broker handles all the connections. In resource locations that load balance among multiple Cloud Connectors during normal operations, the elected broker might need to handle many more requests than normal during an outage. Therefore, CPU demands are higher. Every broker in the resource location must be able to handle the additional load imposed by LocalDB and all of the affected VDAs, because the broker elected during an outage can change.
As the LHC runs on SQL Server 2014 Express, the normal anti-virus exclusions are recommended. The following sql server data files should be excluded
This will ensure optimal performance is achieved on the LHC cache database.
Internet connectivity should never have single point of failure, high availability should be adopted to ensure a route is always available from the data centre/alternative data centre. Alternatively it should be the highest priority to for your IT department to restore internet connectivity via whatever means are possible.
Software Defined WAN (Such as Netscaler SD-WAN) could be utilised to have multiple paths to the internet over a single logical connection ensuring internet is still available even if a network outage occurs on a single path.
I hope you found this article useful and any feedback is greatly appreciated.