NTE_BAD_FLAGS 0x80090009: Invalid Flags Fix

0x80090009 means you passed an invalid flag to a CryptoAPI function. It's almost always a wrong dwFlags value in CryptAcquireContext, CryptGenKey, or CryptImportKey.

Quick answer: 0x80090009 is NTE_BAD_FLAGS — the dwFlags value you handed to a CryptoAPI function (usually CryptAcquireContext, CryptGenKey, CryptImportKey, or CryptSignHash) doesn't match what that CSP/provider accepts. Read the docs for the specific call, then fix the flag.

This one shows up more than people expect. A dev ships an app that calls CryptAcquireContext with both CRYPT_VERIFYCONTEXT and CRYPT_NEWKEYSET set at the same time, tests pass on their machine because their provider tolerates it, and then the app explodes on a customer's box running a different CSP. Sometimes it's a legacy app hitting a modern Windows 10 22H2 or Windows 11 23H2 build where the default provider has changed behavior. The error itself is honest: you sent flags the provider doesn't recognize or that conflict with each other.

Here's the thing — this error is almost never Microsoft's fault. It's caller error 99% of the time. The provider's job is to validate flags against its supported set. If you pass garbage, you get NTE_BAD_FLAGS. The fix is on your side.

Fix steps

Find the exact failing call. Turn on first-chance exception debugging in Visual Studio (Debug > Windows > Exception Settings, check Win32 exceptions). Reproduce the error. The debugger will break at the CryptoAPI call that returned 0x80090009. Write down the function name and every parameter you passed.
Look at the dwFlags argument. That's the culprit 90% of the time. Common combos that trigger this:
- CRYPT_VERIFYCONTEXT | CRYPT_NEWKEYSET — these are mutually exclusive. Verify context means "don't touch my keyset," new keyset means "make me one." Pick one.
- CRYPT_MACHINE_KEYSET passed to a user-scoped provider that doesn't support machine keys.
- A custom bitmask value like 0x00000040 that isn't defined in wincrypt.h.
- CRYPT_SILENT on a provider that doesn't implement silent mode (some smart card CSPs).
Check the provider you're targeting. If you passed a specific pszProvider like "Microsoft Enhanced RSA and AES Cryptographic Provider", confirm which flags that provider actually supports. Run this to list what's installed:
```
certutil -csplist
```
You'll see entries like Microsoft Software Key Storage Provider and Microsoft RSA SChannel Cryptographic Provider. Each has its own flag support matrix. A flag that works on the legacy CSP won't necessarily work on the KSP (Key Storage Provider).
Fix the flag and rebuild. Nine times out of ten the correct call looks like this for a signature verification use case where you don't want a persistent keyset:
```
if (!CryptAcquireContext(&hProv, NULL, NULL,
        PROV_RSA_FULL,
        CRYPT_VERIFYCONTEXT | CRYPT_SILENT)) {
    DWORD err = GetLastError();
    // err == 0x80090009 means your flags are wrong
}
```
Notice: no CRYPT_NEWKEYSET, no CRYPT_MACHINE_KEYSET. Just the two flags that coexist cleanly.
Test on a clean VM. Don't trust your dev machine. Spin up a fresh Windows 10 VM or Windows Server 2022 instance, install only your app, and run it. Different providers are present depending on whether the OS is client or server, and whether certain roles are installed. What works on your laptop may not work on a stripped-down server core image.

If the main fix doesn't work

Try the modern NCrypt API instead. If you're on Windows Vista or later and doing key storage, NCryptOpenStorageProvider plus NCryptCreatePersistedKey replaces the old CryptoAPI calls. It has cleaner flag semantics and the CNG providers behave more predictably:

NCRYPT_PROV_HANDLE hProv;
SECURITY_STATUS status = NCryptOpenStorageProvider(
    &hProv, MS_KEY_STORAGE_PROVIDER, 0);
// flags parameter here is almost always 0

Check for a shim or AppCompat layer. If the app runs fine without compatibility mode but fails with it, a shim is rewriting your API calls in a way that mangles flags. Look at HKCU\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers. Remove any entry for your exe and retest.

Look for a third-party CSP hijacking the default. Some VPN clients, smart card middleware, and HSM vendors install a CSP that registers itself as the default provider for common algorithms. Run certutil -csplist and check whether the first provider listed is the one you expect. If a vendor CSP jumped to the top, that explains why your flags suddenly "stopped working" — they work fine on the Microsoft provider.

Validate the handle type. A subtle variant: passing an HCRYPTPROV handle to a CNG function (or vice versa) can surface as NTE_BAD_FLAGS because the function reads garbage where it expected a flags field. If you're mixing CryptoAPI and CNG in the same codebase, audit every crossover carefully.

Prevention

Never OR together flags from memory. Open wincrypt.h and read the actual definitions. Write unit tests that call your crypto init path with each provider you intend to support — Microsoft Software KSP, Microsoft Smart Card KSP, and any vendor CSP your customers use. When you upgrade a provider or move from a legacy CSP to a KSP, retest the flag matrix, because the supported set isn't identical. The 20 minutes you spend on that test saves you from a support ticket storm later.