Using the DirectXShaderCompiler C++ API

The old fxc.exe compiles to DXBC and only supports up to Shader Model 5.1. Microsoft has since introduced their new llvm-based compiler DirectXShaderCompiler (DXC) which compiles to DXIL (whereas FXC compiles to DXBC). It’s completely open-source on GitHub. It both provides command line tools and a C++ API for compiling, validating and using shaders with SM 6.0 and up.

FXC has existed for quite a long time now and there are guides for it everywhere however DXC is quite a bit younger and I didn’t find too much about it. The documentation for it is quite minimal and incomplete which is surprising considering D3D is generally really well documented. Using DXC’s commandline tools are quite straight forward and self-explanatory however I’ve always liked to compile my shaders at runtime because it makes it easier to recompile on the fly. Unfortunately, I didn’t find too much documentation on the C++ API.

I’ve found a few articles here and there about the very basic setup for this however most of the were very minimal and didn’t really cover things like compile-arguments, defines, include handling, validation, reflection, debug-data, … which are quite important to know about.

EDIT (20/03/2020): DirectX Developer day has happened and there’s been a super good talk by one of the DXC developers and they gave a great walkthrough of the interface. I’d definitely recommend watching that. The general usage is the same but with the latest few updates, it has become a bit easier to use and manipulate.

The main thing that got updated are the introduction of IDxcCompiler3 which is has the new interfaces that are streamlined with the CLI. Secondly, it’s now a lot clearer what “parts” the compiler outputs and it’s more easy to isolate certain parts like reflection and PDBs and possibly strip them to be processed later.

Getting Started

At the time of writing this, there is not yet an official release with the new DXC updates however the latest build binaries can be downloaded from the repo’s AppVeyor.

There are quite a lot of binaries in there but to use the C++ API you only need a few:

• dxcompiler.lib - The compiler dll link lib
• dxcompiler.dll - The compiler’s frontend
• dxcapi.h - The header with the interfaces. You only need this single header. (/include/dxc/dxcapi.h)

Compiling

The new update comes with a new interface IDxcUtils, this interface essentially replaces IDxcLibrary. The ‘Utils’ interface provides all the functionality to create data blobs. Besides that, just like before, creating a blob remains the same:

ComPtr<IDxcUtils> pUtils;
DxcCreateInstance(CLSID_DxcUtils, IID_PPV_ARGS(pUtils.GetAddressOf()));
ComPtr<IDxcBlobEncoding> pSource;
pUtils->CreateBlob(pShaderSource, shaderSourceSize, CP_UTF8, pSource.GetAddressOf());

Now introducing IDxcCompiler3, the Compile function no longer provides separate input arguments for defines. This now all needs to get passed through as compile arguments using the -D argument. Note the use of std::vector is just for simplicity’s sake.

std::vector<LPWSTR> arguments;
// -E for the entry point (eg. 'main')
arguments.push_back(L"-E");
arguments.push_back(entryPoint);

// -T for the target profile (eg. 'ps_6_6')
arguments.push_back(L"-T");
arguments.push_back(target);

// Strip reflection data and pdbs (see later)
arguments.push_back(L"-Qstrip_debug");
arguments.push_back(L"-Qstrip_reflect");

arguments.push_back(DXC_ARG_WARNINGS_ARE_ERRORS); //-WX
arguments.push_back(DXC_ARG_DEBUG); //-Zi

for (const std::wstring& define : defines)
{
    arguments.push_back(L"-D");
    arguments.push_back(define.c_str());
}

DxcBuffer sourceBuffer;
sourceBuffer.Ptr = pSource->GetBufferPointer();
sourceBuffer.Size = pSource->GetBufferSize();
sourceBuffer.Encoding = 0;

ComPtr<IDxcResult> pCompileResult;
HR(pCompiler->Compile(&sourceBuffer, arguments.data(), (uint32)arguments.size(), nullptr, IID_PPV_ARGS(pCompileResult.GetAddressOf())));

// Error Handling. Note that this will also include warnings unless disabled.
ComPtr<IDxcBlobUtf8> pErrors;
pCompileResult->GetOutput(DXC_OUT_ERRORS, IID_PPV_ARGS(pErrors.GetAddressOf()), nullptr);
if (pErrors && pErrors->GetStringLength() > 0)
{
    MyLogFunction(Error, (char*)pErrors->GetBufferPointer());
}

Now before you stop reading and being happy that you compile a shader, wait! You’re missing out on the main reason why this update is such a big deal! You might have noticed the two arguments -Qstrip_debug and -Qstrip_reflect.

Stripping parts

I’ve found that previously, stripping debug data or reflection data hasn’t always been doing what I would expect. With the new updates, this becomes a lot easier.

The output of the compiler now is a IDxcResult (as opposed to IDxcOperationResult). It has a method GetOutput which allows you to extract a part of the output. This can be one of the following: (defined in dxcapi.h)

• DXC_OUT_OBJECT
• DXC_OUT_ERRORS
• DXC_OUT_PDB
• DXC_OUT_SHADER_HASH
• DXC_OUT_DISASSEMBLY
• DXC_OUT_HLSL
• DXC_OUT_TEXT
• DXC_OUT_REFLECTION
• DXC_OUT_ROOT_SIGNATURE

In the code example above, I added the two arguments -Qstrip_debug and -Qstrip_reflect. The effect of this is that the compiler will strip both the shader PDBs and reflection data from the Object part. The important thing here is that it will still be in the compile result and can be extracted using DXC_OUT_PDB and DXC_OUT_REFLECTION respectively. The result of the flags are that it will no longer be embedded in the DXC_OUT_OBJECT part which keeps the actual shader object nice and slim. I would definitely advise to always use these flags. Pix also understands separate PDBs for shaders.

So now, it’s a lot more clear and easy to specifically get parts of the shader. Getting shader PDBs is done as follows:

ComPtr<IDxcBlob> pDebugData;
ComPtr<IDxcBlobUtf16> pDebugDataPath;
pCompileResult->GetOutput(DXC_OUT_PDB, IID_PPV_ARGS(pDebugData.GetAddressOf()), pDebugDataPath.GetAddressOf());

The function has one extra argument which can be retrieved as a IDxcBlobUtf16. This contains the path that is baked into the shader object to refer to the part in question. So if you want to save the PDBs to a separate file, use this name so that Pix will know where to find it.

For reflection data: (library reflection is similar but with ID3D12LibraryReflection)

ComPtr<IDxcBlob> pReflectionData;
pCompileResult->GetOutput(DXC_OUT_REFLECTION, IID_PPV_ARGS(pReflectionData.GetAddressOf()), nullptr);
DxcBuffer reflectionBuffer;
reflectionBuffer.Ptr = pReflectionData->GetBufferPointer();
reflectionBuffer.Size = pReflectionData->GetBufferSize();
reflectionBuffer.Encoding = 0;
ComPtr<ID3D12ShaderReflection> pShaderReflection;
pUtils->CreateReflection(&reflectionBuffer, IID_PPV_ARGS(pShaderReflection.GetAddressOf()));

The hash:

RefCountPtr<IDxcBlob> pHash;
if (SUCCEEDED(pCompileResult->GetOutput(DXC_OUT_SHADER_HASH, IID_PPV_ARGS(pHash.GetAddressOf()), nullptr)))
{
    DxcShaderHash* pHashBuf = (DxcShaderHash*)pHash->GetBufferPointer();
}

On a last note, as mentioned in the video, none of the DXC interfaces are thread-safe and it is advised to have an instance of each interface for each thread. DxcCreateInstance is an exception and is thread-safe.

Custom include handler

I’ve found no good examples of implementing a custom include handler and it’s really not straight forward at first. This is useful if you want to add some custom include logic or if you want to know which files were included to do some kind of shader hot-reloading. Here’s an example implementation that correctly loads include sources and makes sure not to include the same file twice. To add include directories, use the -I argument like for example -I /Resources/Shaders.

static ComPtr<IDxcUtils> pUtils;
if(!pUtils)
    VERIFY_HR(DxcCreateInstance(CLSID_DxcUtils, IID_PPV_ARGS(pUtils.GetAddressOf())));

class CustomIncludeHandler : public IDxcIncludeHandler
{
public:
    HRESULT STDMETHODCALLTYPE LoadSource(_In_ LPCWSTR pFilename, _COM_Outptr_result_maybenull_ IDxcBlob** ppIncludeSource) override
    {
        ComPtr<IDxcBlobEncoding> pEncoding;
        std::string path = Paths::Normalize(UNICODE_TO_MULTIBYTE(pFilename));
        if (IncludedFiles.find(path) != IncludedFiles.end())
        {
            // Return empty string blob if this file has been included before
            static const char nullStr[] = " ";
            pUtils->CreateBlobFromPinned(nullStr, ARRAYSIZE(nullStr), DXC_CP_ACP, pEncoding.GetAddressOf());
            *ppIncludeSource = pEncoding.Detach();
            return S_OK;
        }

        HRESULT hr = pUtils->LoadFile(pFilename, nullptr, pEncoding.GetAddressOf());
        if (SUCCEEDED(hr))
        {
            IncludedFiles.insert(path);
            *ppIncludeSource = pEncoding.Detach();
        }
        return hr;
    }

    HRESULT STDMETHODCALLTYPE QueryInterface(REFIID riid, _COM_Outptr_ void __RPC_FAR* __RPC_FAR* ppvObject) override { return E_NOINTERFACE; }
    ULONG STDMETHODCALLTYPE AddRef(void) override {	return 0; }
    ULONG STDMETHODCALLTYPE Release(void) override { return 0; }

    std::unordered_set<std::string> IncludedFiles;
};

All compiler arguments

I’ve also found it pretty hard to find the compiler options for DXC. When using FXC, most of the compiler options were defines and easy to find because they were all defined together in a unified way. That doesn’t seem to be the case for DXC.

I recently saw a conversation on the DirectX Discord server about when to use “-“ or “/” for compile argument but if you look at HLSLOptions.td in the DirectXShaderCompiler reposity, you’ll see “-“ works for all arguments while “/” only works for some. So it’s always best to use “-“.

When you execute dxc.exe -help, it shows you the most important options which I’ve formatted below, mostly for my own reference.

Version: dxcompiler.dll: 1.7 - 1.7.2212.12 (8c9d92be7); dxil.dll: 1.7(101.7.2212.14)

Updates

• 15/12/2021: Update DXC cheat sheet with options from DXC 1.6.2112.12
• 27/04/2021: Update DXC cheat sheet with options from DXC 1.7 (1.6.0.3119) with SM6.6 support
• 14/10/2021: Add custom include handler example
• 01/02/2023: Update DXC cheat sheet with options from DXC 1.7 (1.7.2212.12)

Some great articles that go more in-depth

• Signing DXIL post-compile - Graham Wihlidal
• Using a Custom Toolchain with MSBuild - Nathan Reed