IIS Application Pool Will Not Start – App Log Error 2307

After applying patches to a 2012 R2 server, I found one of our IIS websites was not working. While IIS was itself running, the DefaultAppPool was stopped. Attempts to start the app pool were unsuccessful; the pool would stop almost immediately, and error 2307 from “IIS-W3SVC-WP” was logged in the Application Log.

The error message itself contained the following information:

The worker process for application pool 'DefaultAppPool' encountered an error 'Cannot read configuration file' trying to read configuration data from file '\\?\<EMPTY>', line number '0'. The data field contains the error code.

Not the most helpful error message since it doesn’t specific the file that cannot be read. Some research online turned up references to specific IIS config files (web.config, etc.) having incorrect user permissions, but I looked into that and everything seemed correct. I used Sysinternals Process Monitor to see if I could determine the file in question, with no luck.

Eventually I stumbled on a post (which I unfortunately lost track of), which referenced a Microsoft KB article. The article pertains to Windows 10 and Server 2016, the 1709 Fall Creators Update, and a totally different event ID from a totally different service. That said, the resolution shown fixed the error I was having.

The actual fix is quick:

net stop WAS /y
rmdir /s /q C:\inetpub\temp\appPools
net start W3SVC

Evidently the patches I applied caused the same issue (broken symbolic links) as the Win 10 1709 upgrade.

Advertisements

Inaccessible Network Shares on Windows 10 1709

After upgrading a number of systems to the Fall Creators edition of Windows 10, version 1709, we began receiving reports of certain users being unable to access a specific set of shared folders. These were folders hosted on a 2012R2 file server, and the error received indicated the share was missing, rather than there being a permissions issue.

Searching for this type of error on the Fall Creators edition turns up two common results:

• SMBv1 has been disabled and/or removed from Win 10 1709
• Guest access in SMB2 disabled by default in Win 10 1709

We were pretty certain SMBv1 was not in use but ran Wireshark to sniff the SMB traffic and verify SMB2 was being used.

Likewise, we knew guest account access was already disabled, and the Microsoft KB article references log entries that would be present if this were causing a problem. There were no relevant log entries, so we crossed this off the list of possible causes.

Interestingly, not all shares were an issue; even other ones on the same server worked fine. That said, this share had been used for years without problems, with multiple versions of Windows, including three previous Win 10 builds.

I started reviewing the permissions on the share, just to see what might be different from other shares that were working, and quickly found a discrepancy.

The share in question was the parent folder for many sub-folders. The sub-folders have different permissions for different groups. We provide a shortcut directly to the parent folder and have access-based enumeration enabled so that users only see the folders they have access to. Nothing all that unusual here.

The parent folder had “Authenticated Users” being granted “Traverse Folder / Execute File” on the folder itself. (“This Folder Only”)

Certain user groups however had “Read” permissions on the parent folder. Some quick testing showed that the users with “Read” permissions could access the share, while those with “Traverse Folder” could not.

The entire purpose of “Traverse Folder” is to “[a]llow..moving through a restricted folder to reach files and folders beneath the restricted folder in the folder hierarchy.” Having this permission should be perfectly sufficient for what we were doing, and it has been for years, up until the Fall Creators version of Win 10.

Even stranger though is that “bypass traverse checking,” which is granted to everyone by default (and enabled in our domain) is supposed to entirely negate the need for the “Traverse Folder” right in the first place.

Something obviously changed in Fall Creators that has broken traverse checking; my guess is with all the file system work that Microsoft acknowledges they put into Fall Creators, they either inadvertently broke this or deliberately changed the behavior without communicating it to us. Although if this were a deliberate change it completely eliminates the value of traverse bypass in the first place.

The fix here was easy; I gave “Read” permissions to “Authenticated Users” for the parent folder only. This had no effect on any of the sub folders or users access to them, but it resolved the path not found error.

Oh well…back to beta testing the latest GA build of Win 10…

WSUS Error 0x80092026 – File Failed Post-Processing

As is typically the case with my sporadic blog posts, we recently ran into a situation where, as a colleague likes to put it, we ended up far down the rabbit hole. There are a few lessons to be learned here so I’m going to explain a bit about the troubleshooting process, not just how we eventually fixed the issue.

The initial complaint as reported was that one set of laptops was not working properly, but only during the first class period, and only in one classroom. Now, it’s not uncommon for mobile devices to take a little while to wake up, but in this case the machines were turning on an hour before class started, and again, we were told the problem was very limited in scope.

We were able to duplicate the problem, but again, only during the one period, and even stranger, only when the class was in the room. We use Meraki access points, and based on this particular AP not seeing its mesh neighbors all that well, we initially thought we were dealing with some localized RF interference. After extensive troubleshooting of the RF environment we could not find any explanation for the apparent interference. The AP was seemingly overloaded during this period of time, but there was no obvious cause.

One thing that did stand out was certain wireless clients were transferring an abnormal amount of data. We noted this with the intent to revisit it later, but because we had no reports of issues from other locations, we concluded this was unrelated to our current situation.

That was a mistake. We assumed that because there were no other problems reported, that there were no other problems. As it turns out, nobody else was using laptops at this time of day. The lesson here is to check your assumptions.

Once we realized that we could not draw any conclusions from the location of the problem, we expanded our investigation and revisited the previously mentioned excessively large amounts of data being downloaded. The Meraki APs do an excellent job of reporting certain info, and it quickly became obvious the data was coming from our WSUS patch server.

We had recently rebuilt WSUS, and with 2012 R2 there are (were) some important out-of-band patches that can result in failed downloads. We reviewed these and confirmed that on a fully-patched 2012 R2 install, these needed updates are now included in the normal security and critical updates that Microsoft pushes out. (Evidently you still must add the .esd mime type to IIS, but that’s unrelated to this post.)

We next turned to one of the laptops that was experiencing a problem. On Windows 10, getting a human-readable update log requires running the PowerShell command Get-WindowsUpdateLog; this creates the old-style WindowsUpdate.log you may be used to from earlier Windows versions.

Searching for errors, we came across the following:

Misc Validating signature for <file guid> with dwProvFlags 0x00000080:
Misc Error: 0x80092026 when verifying trust for <file guid>
DownloadManager File failed postprocessing, error = 80092026
DownloadManager Failed file: URL = <WSUSServer><File>, Local path = <local path>
DownloadManager Error 0x80092026 occurred while downloading update; notifying dependent calls. 

I should add that many of the affected laptops were not consistently reporting to the new WSUS server, we were not certain if this was related but it certainly seemed suspicious.

A number of online searches confirmed that this error had something to do with a certificate check failing for a particular update. Since our content filter does SSL decryption, we double-checked that we were not decrypting any traffic to Microsoft’s CRL or patch servers: we were not. Of course, if something were being blocked by the content filter we would expect every client to be impacted, but most of our PCs were working fine.

We found different files failing post-processing on different computers, so this did not seem to be related to a specific update. Was what more interesting is that despite only one downloaded update failing, WSUS (or BITS) seemingly discarded the entire batch of downloads, so the next time Windows Update started to run, it re-downloaded everything.

After further troubleshooting, I stumbled on this thread on Microsoft TechNet. User “lforbes” posted the following:

I found the offending key for those that are interested.
HKEY_USERS\S-1-5-18\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing
State SHOULD be 146432 or 0x00023c00
If it switches to 408576 or 0x00063c00 [t]hen it will cause these Crypto Errors 

Checking the registry on affected computers confirmed that the key in question was set to 0x00063c00. After changing it back to 0x0023c00, the trust provider errors stopped. I have no idea how he found that key, and what’s amazing is that this TechNet post seems to be the only reference to this error and this registry key out there.

I didn’t want to just make the change without understanding what it meant, so researching further lead to this post which clarifies that the “State” key is a bitwise flag for various settings of “Microsoft Trust verification services, which provide a common API for determining whether a specific subject can be trusted.” After a bit of hex to binary conversion and a comparison between the two sets of flags, I determined that the difference was the incorrect setting had enabled the flag to only “allow…items in personal trust database.” Seemingly this flag was preventing a system-level certificate from being used to validate a Microsoft signature. No clue why.

But of course things didn’t stop there.

Whether it was related to the original certificate error or not, many of the same PCs that had this error were also not reporting to WSUS. This at least I already had a partial fix for. Years ago I had put together a basic batch file that reset the SusClientId, deleted the SoftwareDistribution folder, and basically forced a full reset of the WSUS client. (I’m not taking credit for figuring this all out, but I have no idea where I first saw the info.)

The process involves running the following as a batch file on the affected PC:

net stop wuauserv

reg delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v AccountDomainSid /f

reg delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v PingID /f

reg delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v SusClientId /f

reg delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v SusClientIdValidation /f

del c:\windows\softwaredistribution /S /F /Q

net start wuauserv

wuauclt /resetauthorization /detectnow

This stops the Automatic Update service, clears out the registry keys that identity the system to WSUS, clears out the patch download cache, restarts the Automatic Update service, and then re-registers the system to WSUS. There’s one little problem. The “/detectnow” switch was apparently removed in the Win 10 version of wuauclt.exe, leaving no simple way to force a machine to scan WSUS for updates.

There are a couple of ways to get around this via VBScript or languages that directly call the Windows Update API. In our case however we already were licensed for WUInstall, which has this functionality built-in. We added an additional command at the end of the batch file to force WUInstall to scan for updates, and at this point, everything seems to be working properly.

Thanks for reading, and “lforbes,” if you’re out there, thank you for posting your fix.

VSS Snapshots Fail on Server 2008 or 2008 R2

Recently our Exchange 2013 server – running on Windows Server 2008 (not R2) – started having backup failures. We use FalconStor DiskSafe for backup, which, like most (all?) Exchange-aware backup application uses VSS to snap the server.

During the investigation with FalconStor support, we discovered VSS was timing out during the verification of the snapshot, and running “vssadmin list writers” showed a number of the VSS writers in a failed state. No updates had recently been applied to the server, nor had any configuration changes been made recently, so there was no obvious potential cause of the problem.

FalconStor support was able to point us to this link at IBM. While not an exact description of our problem, the article linked to a Microsoft utility, DevNodeClean. The description for that utility is:

“On a computer that is running Windows Server 2003 or a later version, a storage device that is connected by using a fiber channel or by using the iSCSI protocol may be connected for only a short time. When a storage device is connected, Windows creates registry information for the device. Over time, the registry may contain many entries for devices that will never be used again. This utility can be used to remove this information from the registry.”

Interesting. Searching a bit more I came across a 3^rd party complied version of DevNodeClean, which has more information about the problem it resolves.

I can’t find the link now, but I also came across a page that suggested reviewing the size of the registry itself. The actual registry files are located at c:\windows\system32\config. The SYSTEM file, which represents HKLM\SYSTEM, was in this case over 1.5GB! Checking a few other servers of similar age showed SYSTEM files of 50MB or less in most cases.

By now, I was pretty confident this was our issue. I ran DevNodeClean (the Microsoft provided one), which took – no joke – six days to complete. As it continued to delete unused devices from the registry, our backups began running again intermittently. After about five days of running the utility, all backups began working properly.

It seems what happens is that any backup product that calls VSS – FalconStor, Veeam, Microsoft DPM, etc. – causes a registry entry to be created for every single VSS snapshop. I believe the same happens when using Hyper-V snapshots. These registry entries are not cleaned-up, and over time, they bloat the registry to the point where it can essentially break VSS. In our case, we were snapping two LUNs every hour, so over the course of a few years of the server’s life, there were thousands of orphaned registry entries.

There is supposedly a hotfix for 2008 R2, but it’s not clear if that fixes the problem, and in any event it’s not available for 2008.

It seems the best way to deal with this is to run DevNodeClean as a scheduled task on a regular basis.

Unfortunately, the registry cannot be compacted online, so we will need to schedule a maintenance window to actually shrink the bloat.

Google Chrome Managed Bookmarks

The Enterprise (MSI) version of Google Chrome includes a comprehensive Group Policy template, allowing many settings to be centrally controlled. One of these settings is “Managed Bookmarks,” which allows the administrator to push out a fixed set of bookmarks to all users.

As shown here, there are three methods to create managed bookmarks:

• JSON strings in the GPO editor
• JSON strings in the registry editor
• Expanded JSON in the registry

Reviewing those options, I initially looked to method #3, as dealing directly with the registry seemed the easiest of the methods. I tested the registry entries that needed to be made, confirmed everything was OK, and proceeded to configure Group Policy to import the registry keys. And it didn’t work. To be specific, the registry keys would not import.

After spending way too much time troubleshooting, I found the problem: Chrome’s Managed Bookmarks are stored at “HKCU\Software\Policies\Google\Chrome\ManagedBookmarks” – this is in the current user’s registry hive so it should be writable. Except it isn’t. If you check permissions on “HCKU\Software\Policies”, you’ll find the current user only has READ permissions to this particular branch of the registry. You need administrator privileges to write to the “Policies” key.

The obvious question is why? The “Policies” key contains group policy settings, as set by administrative templates. If a standard user could change settings in this key, it would give them the ability to override group policy.

The difficulty this creates for the way Chrome handles Managed Bookmarks is that neither of the registry options are viable for non-admin users. I suppose they could be useful if you wanted to preload bookmarks in your default profile, but otherwise there’s no viable way to use the registry to control the bookmarks. You’re left with the first option, directly entering JSON into the Group Policy editor.

I wrote a utility that will take a folder of url shortcuts (like how IE handles favorites) and create the appropriate JSON code. It’s too rough to share here but if anyone has interest in it, please post below and perhaps I’ll clean it up.

Windows Update Logging on Windows 10

For many years and through multiple Windows versions, Windows Update logged to c:\windows\windowsupdate.log – simple and functional. But Microsoft’s vision with Windows 10 seems to be fixing things that aren’t broken, and this is no exception.

Opening c:\windows\windowsupdate.log on Win 10 gives you this:

Windows Update logs are now generated using ETW (Event Tracing for Windows).

Please run the Get-WindowsUpdateLog PowerShell command to convert ETW traces into a readable WindowsUpdate.log.

For more information, please visit http://go.microsoft.com/fwlink/?LinkId=518345

Basically, in order to have a human-readable update log, you need to open PowerShell and run Get-WindowsUpdateLog, which as noted above will convert ETW logs to an old-style WindowsUpdate.log which it places on the desktop. It’s an extra step but not too bad. When it works.

Unfortunately, in some (many?) cases you will end up with a log file filled with lines like these:

1601.01.01 01:00:00.0000000 692 5220 Unknown( 31): GUID=c232d8d6-c0ba-3f2d-4ee0-3c6f4b234595 (No Format Information found).

So what’s the problem? Believe it or not, in order for Windows 10 to convert the ETW files to a human-readable log, the system needs Internet access in order to reach Microsoft’s symbol servers. This is yet another functional change that seems to ignore the reality that many larger environments and virtually every school has content filtering in place, which blocks this very access.

In order to fix this, you will need to whitelist all traffic to http://msdl.microsoft.com/download/symbols. If that still does not fix things, you will need to delete the contents of %temp%\WindowsUpdateLog and re-run the PowerShell command.

After doing that, you should once again have a usable log file. Easy, right?

BGInfo for Windows 10

Years ago, Mark Russinovich of Sysinternals fame (since bought by Microsoft) released the BGInfo utility, which served two purposes: to put various system information such as the computer name and free hard drive space on the desktop, and likewise to put the same info on the lock screen. In a large environment, that latter feature was extremely useful as it enables users and technicians to quickly see the name of a PC without having to log on and find the info.

Unfortunately, Microsoft changed the way the lock screen was handled in Windows Vista and 7, which broke BGInfo’s lock screen functionality. Windows 10 changed things yet again and while BGInfo has been updated to support Win 10, we could not get it working for the lock screen. I don’t want to say it’s not possible, because there may be some combination of using the program along with group policy to display the info on the lock screen, but in our environment it did not work.

Anyway…we really wanted this functionality so I began to look into ways of hacking something together…and boy is this a hack.

Before I go any further, I want to clarify the terms I’m using:

The lock screen is the screen (wallpaper) displayed before a user has logged on, or when the logged-on user locks the computer. {WIN}+{L} for example. In Win 10 RTM and 1511, this disappears before the user is prompted for credentials.

The login screen is the screen where a user actually enters their credentials. On Win 10 RTM and 1511, this background will either be a solid color or the neon-blue Windows wallpaper. On 1607, the lock screen image will remain even once the login fields are shown.

As is seemingly always the case with Windows 10, half the battle is figuring out how aspects of the OS actually work; only then can you determine how to best control things. The below represents my best efforts at understanding what’s going on behind the scenes; if you know otherwise or can add to the discussion, please leave a comment. Also, while some of the settings below can be controlled through group policy (and one must be), I elected to deal with the registry directly as doing so kept all the changes in one place.

Additionally, the way the lock screen works is different in build 1511 (Threshold 2), 1607 (Anniversary) and 1607 with the September cumulative update. These scripts will work with either 1511 or a fully-patched 1607, at least as of this writing. They may however work differently between the two builds, as well as on an existing user profile versus a new one.

There are two sets of lock screen settings: the system-wide machine setting, and a user setting which is (predictably) specific to each user.

For the system-wide settings, we have:

HKLM\Software\Policies\Microsoft\Windows\Personalization\NoChangingLockScreen – Set to 0 to allow the user to change the lock screen, 1 to prevent changes.

HKLM\Software\Policies\Microsoft\Windows\Personalization\LockScreenImage – This is the full path to the lock screen file.

The in-box lock screens are located at c:\windows\web\screen and there does not appear to be a way to select anything other than these default images through the UI.

For the user-specific settings, we have:

HKCU\Software\Microsoft\Windows\CurrentVersin\ContentDeliveryManager\RotatingLockScreen – Set to 0 for a fixed image, or 1 for Windows Spotlight.

Windows Spotlight is a feature by which Win 10 downloads new lock screen images on its own, rotating them from time to time. If Spotlight is enabled, the below registry keys are relevant:

HKCU\Software\Microsoft\Windows\CurrentVersion\Lock Screen\Creative\HotspotImageFolderPath – Path to the user’s downloaded images

HKCU\Software\Microsoft\Windows\CurrentVersion\Lock Screen\Creative\PortraitAssetPath –> Full path to the lock screen file for portrait (vertical) layout

HKCU\Software\Microsoft\Windows\CurrentVersion\Lock Screen\Creative\LandscapeAssetPath –> Full path to the lock screen file for landscape (horizontal) layout

Now in theory, if a computer (HKLM) policy is set to prevent the user from changing the lock screen image, the user should receive the same lock screen, but what we found instead was in some cases on build 1511 the user’s setting would override the computer’s setting. In other words, we would have a fixed lock screen for the computer when nobody was logged in, but once a user logged in, their lock screen settings (which default to having Spotlight enabled) would take precedence. This meant we’d have one lock screen for the computer, and one lock screen for each user. 1607 seems to work properly in that setting a lock screen at the HKLM level prevents any user changes from being made.

On 1511, I recommend enabling the group policy Computer Config à Policies à Administrative Templates à Control Panel à Personalization à Prevent changing lock screen and login image. If you do not do this, the user retains the ability to change their lock screen even though they technically should not.

There are a lot of moving pieces here. The download includes both a startup and login script; on 1511 you need both. On 1607 you can get away with just the startup script however you will lose some functionality in doing so. Here’s what each piece does:

Startup Script

• Check if we’re running on Win 10; if not, exit.
• Choose a random picture from c:\windows\web\screen, and save it out to c:\programdata\bginfoscreens
• Determine the size of the display; this unfortunately running this at startup or shutdown is problematic, so we’re doing a few different things:
  • Attempt to determine the size; the default return value is 1024×768. If we get that back, assume we have bad info.
  • If we have bad info, attempt to read c:\program files\bginfo\resolution.out, which is a text file created by the login script, containing the correct resolution.
  • If we have resolution.out, use the resolution there; otherwise stick to the (likely incorrect) data we collected earlier.
• Resize the picture chosen earlier to our screen resolution.
• Write out our desired info to the resized picture. We attempt to adjust the text positioning to fit the image.
• There’s some code here for debugging purposes, where you can choose to display additional info on the screen if the system name matches a certain pattern.
• Save the image.
• Set the new lock screen by making changes to the registry.

Login Script

• Check if we’re running on Win 10; if not, exit.
• If you set $enablenetworkcollection to $true and provide a valid, user-writable path as $networksspath, the login script will copy the user’s Windows Spotlight backgrounds to a network share. This is useful if you want to have a larger pool of pictures for the computer lock screen. If you choose to do this, you will eventually want to disable $enablenetworkcollection and use whatever method you like to copy this collection of pictures to c:\windows\web\screen
• If you collect these images, you will need to rename them to .jpg and also filter out any that are not the proper resolution. I’ve included a quick and dirty script, “CheckResolution.ps1” which will do this.
• Determine the screen resolution; unlike the startup script, this should always be correct. We save it to a file so the startup script can access it on subsequent reboots.
• Read the registry to determine where the user’s lock screens are located; if we don’t find the keys the rest of the script is meaningless, so we exit.
• Find a random image to work with, checking if it’s 1920px wide.
• Code to save images to a network location if $enablenetworkcollection is true.
• Resize the picture chosen earlier to our screen resolution.
• Write out our desired info to the resized picture. We attempt to adjust the text positioning to fit the image.
• Save the image.
• Set the new lock screen by making changes to the registry.

The combination of these two scripts provides a lock screen that rotates on startup and on login (1511) only, as well as displays the info you choose to show. It’s a hack as I said earlier, but it works for us at least, and if nothing else it should provide a starting point for your own environment.

Download the scripts here: http://media.islipufsd.org/Scripts/BGInfoScripts.zip

Note the scripts themselves credits certain other sites and posts I’ve taken code from.

EDIT: These are tested and working on both 1709 and 1803.