Last week, I got an email from my monitoring system that a certificate on a Server 2012 R2 Essentials machine was about to expire. I tracked it down to the computer’s local certificate, issued by the local Certification Authority (CA) almost 5 years ago. Since it hadn’t expired yet, it was a fairly simple matter to go into IIS, open Server Certificates, choose the expiring certificate, and click Renew.
Essentials Services Not Running
This week, I noticed that none of the Windows Server Essentials services were running. I didn’t even try to open the dashboard; I knew that wouldn’t work. Most of the services depend on the Windows Server Essentials Provider Registry Service. This article pointed me to the log patch, where I examined
C:\ProgramData\Microsoft\Windows Server\Logs\ProviderRegistryService.log
From the most recent error, the key lines are:
[1688] 201124.103151.9343: WssgCertMgmt: Found 0 matching certs without verification:
[1688] 201124.103151.9499: WssgCertMgmt: Collection Empty
[1688] 201124.103151.9499: IDENTITY: Local machine cert not found, trying to import the root cert backup to fix
I eventually found this 2012 thread, where Robert Pearman mentioned the registry key HKLM>SOFTWARE>Microsoft>Wwindows Server>IDENTITY. Sure enough, the LocalMachineCert value contained the thumbnail of the old certificate:
After manually replacing the LocalMachineCert value with the thumbprint of the new certificate (copied from the Personal store of Local Computer Certificates), I was once again able to start the Windows Server Essentials services and open the dashboard.
Remote Site Reporting .NET Warnings
While investigating the above issue, I also noticed multiple .NET warnings in the Application event log that started after the certificate renewal:
Log Name: Application
Source: ASP.NET 4.0.30319.0
Date: 11/24/2020 11:08:48 AM
Event ID: 1310
Task Category: Web Event
Level: Warning
Keywords: Classic
User: N/A
Computer: MYSERVER.MYDOMAIN.local
Description:
Event code: 3008
Event message: A configuration error has occurred.
Event time: 11/24/2020 11:08:48 AM
Event time (UTC): 11/24/2020 7:08:48 PM
Event ID: 748e9776d144445884b50993eba47579
Event sequence: 2
Event occurrence: 1
Event detail code: 0
Application information:
Application domain: /LM/W3SVC/1/ROOT/Remote-4-132507185287384759
Trust level: Full
Application Virtual Path: /Remote
Application Path: C:\Program Files\Windows Server\Bin\WebApps\RemoteAccess\
Machine name: MYSERVER
Process information:
Process ID: 4036
Process name: w3wp.exe
Account name: NT AUTHORITY\NETWORK SERVICE
Exception information:
Exception type: HttpException
Exception message: Exception has been thrown by the target of an invocation. (C:\Program Files\Windows Server\Bin\WebApps\RemoteAccess\web.config line 76)
at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)
at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)
Exception has been thrown by the target of an invocation. (C:\Program Files\Windows Server\Bin\WebApps\RemoteAccess\web.config line 76)
at System.Web.Configuration.ProvidersHelper.InstantiateProvider(ProviderSettings providerSettings, Type providerType)
at System.Web.Configuration.ProvidersHelper.InstantiateProviders(ProviderSettingsCollection configProviders, ProviderCollection providers, Type providerType)
at System.Web.Security.Membership.InitializeSettings(Boolean initializeGeneralSettings, RuntimeConfig appConfig, MembershipSection settings)
at System.Web.Security.Membership.Initialize()
at System.Web.Security.Membership.get_Provider()
at Microsoft.WindowsServerSolutions.Web.Security.AuthenticationHelper.add_PreLogOn(EventHandler value)
at Microsoft.WindowsServerSolutions.Web.RemoteAccessSite.Global.Application_Start(Object sender, EventArgs e)
Exception has been thrown by the target of an invocation.
at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
at System.Activator.CreateInstance(Type type, Boolean nonPublic)
at System.Activator.CreateInstance(Type type)
at System.Web.HttpRuntime.CreatePublicInstanceByWebObjectActivator(Type type)
at System.Web.Configuration.ProvidersHelper.InstantiateProvider(ProviderSettings providerSettings, Type providerType)
Machine certificate is not found.
Server stack trace:
at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ProductConfiguratorBase._CreateProxyEndpoint(Uri address, Type contractType, ProviderEndpointBehaviorAttribute endpointBehavior)
at Microsoft.WindowsServerSolutions.Common.ProviderFramework.Internal.ProviderFrameworkConfigurator.GetDuplexChannelFactory[T](ICollection`1 behaviors, ProviderInfo info, Object callback, NetworkCredential credential)
at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ProviderRegistryConnectionMgmt`1._CreateChannel(String targetComputer, IRegistryCallback callback, ProviderEndpointBehaviorAttribute endpointBehavior)
at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ProviderRegistryConnectionMgmt`1._GetFactory(String targetComputer, IRegistryCallback callback, ProviderEndpointBehaviorAttribute endpointBehavior)
at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ProviderRegistryConnectionMgmt`1._CreateRealProxy()
at Microsoft.WindowsServerSolutions.Common.ProviderFramework.Internal.AutoReconnecter`1._BeginConnect()
at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ProviderRegistryConnectionMgmt`1.ConnectWhenAvailable(Int32 spinWaitMillis, Action operation, IRegistryCallback informOfProviderUpdateCallback)
at Microsoft.WindowsServerSolutions.Common.ProviderFramework.Internal.ProviderRegistryFacade.InitProviderRegistryProxy()
at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ConnectorFactory._InitProxy()
at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ConnectorFactory.get__Proxy()
at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ConnectorFactory.SetupConnector[T](ProviderConnector`1 providerConnector)
at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ConnectorFactory.GetConnector[T](String identifier, Object callback)
at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ConnectorFactory.GetServerConnector[T](String identifier, Object callback, NetworkCredential credential)
at Microsoft.WindowsServerSolutions.Users.Internal.UserBackEnd.<>c__DisplayClass6.<CreateConnector>b__2()
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)
at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData)
at System.Action.EndInvoke(IAsyncResult result)
at Microsoft.WindowsServerSolutions.Users.Internal.UserBackEnd.CreateConnector(TimeSpan timeSpan)
at Microsoft.WindowsServerSolutions.Users.Internal.UserBackEnd.Microsoft.WindowsServerSolutions.Users.Internal.IUserBackEnd.ConnectAsync(TimeSpan timeSpan)
at Microsoft.WindowsServerSolutions.Users.Internal.UserBackEnd.Microsoft.WindowsServerSolutions.Users.Internal.IUserBackEnd.Connect(TimeSpan timeSpan)
at Microsoft.WindowsServerSolutions.Users.UserMgmtManager.Connect()
at Microsoft.WindowsServerSolutions.Web.Security.HSBSMembershipProvider..ctor()
Request information:
Request URL: https://mydomain.remotewebaccess.com:443/remote
Request path: /remote
User host address: 192.168.1.2
User:
Is authenticated: False
Authentication Type:
Thread account name: NT AUTHORITY\NETWORK SERVICE
Thread information:
Thread ID: 179
Thread account name: NT AUTHORITY\NETWORK SERVICE
Is impersonating: False
Stack trace: at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)
at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)
In the middle of that huge event is this telling line: “Machine certificate is not found.”
Hmm. The Remote Web Access site is working fine at the moment, and that error hasn’t recurred since I updated the registry key (first part of this article). Maybe this was a symptom of the same issue. Will have to wait to see if it continues.
This all does leave me wondering what the “right” way is to renew an expiring certificate on Essentials. Maybe using the Anywhere Access wizard, even though we’re talking about the certificate issued by the local CA and not the Remote Web Access public certificate?