Using the DirectXShaderCompiler C++ API

04 Mar 2020 - Simon Coenen - Reading time: 14 mins - Comments

Shader Model 6 has existed for quite a while now and after reading this great article from Francesco Cifariello Ciardi about scalarizing the light loop in tiled/clustered lighting, I wanted to try out Wave Intrisics. The old fxc.exe compiles to DXBC and only supports up to Shader Model 5.1. Microsoft has since introduced their new llvm-based compiler DirectXShaderCompiler (DXC) which compiles to DXIL (whereas FXC compiles to DXBC). It’s completely open-source on GitHub here: https://github.com/microsoft/DirectXShaderCompiler It both provides command line tools and a C++ API for compiling, validating and using shaders with SM 6.0 and up.

FXC has existed for quite a long time now and there are guides for it everywhere however DXC is quite a bit younger and I didn’t find too much about it. The documentation for it is quite minimal and incomplete which is surprising considering D3D is generally really well documented. Using DXC’s commandline tools are quite straight forward and self-explanatory however I’ve always liked to compile my shaders at runtime because it makes it easier to recompile on the fly. Unfortunately, I didn’t find too much documentation on the C++ API.

I’ve found a few articles here and there about the very basic setup for this however most of the were very minimal and didn’t really cover things like compile-arguments, defines, include handling, validation, reflection, debug-data, … which are quite important to know about.

EDIT (20/03/2020): DirectX Developer day has happened and there’s been a super good talk by one of the DXC developers and they gave a great walkthrough of the interface. I’d definitely recommend watching that. The general usage is the same but with the latest few updates, it has become a bit easier to use and manipulate.

The main thing that got updated are the introduction of IDxcCompiler3 which is has the new interfaces that are streamlined with the CLI. Secondly, it’s now a lot clearer what “parts” the compiler outputs and it’s more easy to isolate certain parts like reflection and PDBs and possibly strip them to be processed later.

Getting Started

At the time of writing this, there is not yet an official release with the new DXC updates however the latest build binaries can be downloaded from the repo’s AppVeyor.

There are quite a lot of binaries in there but to use the C++ API you only need a few:

  • dxcompiler.lib - The compiler dll link lib
  • dxcompiler.dll - The compiler’s frontend
  • dxcapi.h - The header with the interfaces. You only need this single header. (/include/dxc/dxcapi.h)

Compiling

The new update comes with a new interface IDxcUtils, this interface essentially replaces IDxcLibrary. The ‘Utils’ interface provides all the functionality to create data blobs. Besides that, just like before, creating a blob remains the same:

ComPtr<IDxcUtils> pUtils;
DxcCreateInstance(CLSID_DxcUtils, IID_PPV_ARGS(pUtils.GetAddressOf()));
ComPtr<IDxcBlobEncoding> pSource;
pUtils->CreateBlob(pShaderSource, shaderSourceSize, CP_UTF8, pSource.GetAddressOf());

Now introducing IDxcCompiler3, the Compile function no longer provides separate input arguments for defines. This now all needs to get passed through as compile arguments using the -D argument. Note the use of std::vector is just for simplicity’s sake.

std::vector<LPWSTR> arguments;
//-E for the entry point (eg. PSMain)
arguments.push_back(L"-E");
arguments.push_back(entryPoint);

//-T for the target profile (eg. ps_6_2)
arguments.push_back(L"-T");
arguments.push_back(target);

//Strip reflection data and pdbs (see later)
arguments.push_back(L"-Qstrip_debug");
arguments.push_back(L"-Qstrip_reflect");

arguments.push_back(DXC_ARG_WARNINGS_ARE_ERRORS); //-WX
arguments.push_back(DXC_ARG_DEBUG); //-Zi
arguments.push_back(DXC_ARG_PACK_MATRIX_ROW_MAJOR); //-Zp

for (const std::wstring& define : defines)
{
    arguments.push_back(L"-D");
    arguments.push_back(define.c_str());
}

DxcBuffer sourceBuffer;
sourceBuffer.Ptr = pSource->GetBufferPointer();
sourceBuffer.Size = pSource->GetBufferSize();
sourceBuffer.Encoding = 0;

ComPtr<IDxcResult> pCompileResult;
HR(pCompiler->Compile(&sourceBuffer, arguments.data(), (uint32)arguments.size(), nullptr, IID_PPV_ARGS(pCompileResult.GetAddressOf())));

//Error Handling
ComPtr<IDxcBlobUtf8> pErrors;
pCompileResult->GetOutput(DXC_OUT_ERRORS, IID_PPV_ARGS(pErrors.GetAddressOf()), nullptr);
if (pErrors && pErrors->GetStringLength() > 0)
{
    MyLogFunction(Error, (char*)pErrors->GetBufferPointer());
}

Now before you stop reading and being happy that you compile a shader, wait! You’re missing out on the main reason why this update is such a big deal! You might have noticed the two arguments -Qstrip_debug and -Qstrip_reflect.

Stripping parts

I’ve found that previously, stripping debug data or reflection data hasn’t always been doing what I would expect. With the new updates, this becomes a lot easier.

The output of the compiler now is a IDxcResult (as opposed to IDxcOperationResult). It has a method GetOutput which allows you to extract a part of the output. This can be one of the following: (defined in dxcapi.h)

  • DXC_OUT_OBJECT
  • DXC_OUT_ERRORS
  • DXC_OUT_PDB
  • DXC_OUT_SHADER_HASH
  • DXC_OUT_DISASSEMBLY
  • DXC_OUT_HLSL
  • DXC_OUT_TEXT
  • DXC_OUT_REFLECTION
  • DXC_OUT_ROOT_SIGNATURE

In the code example above, I added the two arguments -Qstrip_debug and -Qstrip_reflect. The effect of this is that the compiler will strip both the shader PDBs and reflection data from the Object part. The important thing here is that it will still be in the compile result and can be extracted using DXC_OUT_PDB and DXC_OUT_REFLECTION respectively. The result of the flags are that it will no longer be embedded in the DXC_OUT_OBJECT part which keeps the actual shader object nice and slim. I would definitely advise to always use these flags. Pix also understands separate PDBs for shaders.

So now, it’s a lot more clear and easy to specifically get parts of the shader. Getting shader PDBs is done as follows:

ComPtr<IDxcBlob> pDebugData;
ComPtr<IDxcBlobUtf16> pDebugDataPath;
pCompileResult->GetOutput(DXC_OUT_PDB, IID_PPV_ARGS(pDebugData.GetAddressOf()), pDebugDataPath.GetAddressOf());

The function has one extra argument which can be retrieved as a IDxcBlobUtf16. This contains the path that is baked into the shader object to refer to the part in question. So if you want to save the PDBs to a separate file, use this name so that Pix will know where to find it.

For reflection data: (library reflection is similar but with ID3D12LibraryReflection)

ComPtr<IDxcBlob> pReflectionData;
pCompileResult->GetOutput(DXC_OUT_REFLECTION, IID_PPV_ARGS(pReflectionData.GetAddressOf()), nullptr);
DxcBuffer reflectionBuffer;
reflectionBuffer.Ptr = pReflectionData->GetBufferPointer();
reflectionBuffer.Size = pReflectionData->GetBufferSize();
reflectionBuffer.Encoding = 0;
ComPtr<ID3D12ShaderReflection> pShaderReflection;
pUtils->CreateReflection(&reflectionBuffer, IID_PPV_ARGS(pShaderReflection.GetAddressOf()));

On a last note, as mentioned in the video, none of the DXC interfaces are thread-safe and it is advised to have an instance of each interface for each thread. DxcCreateInstance is an exception and is thread-safe.

All compiler arguments

I’ve also found it pretty hard to find the compiler options for DXC. When using FXC, most of the compiler options were defines and easy to find because they were all defined together in a unified way. That doesn’t seem to be the case for DXC.

I recently saw a conversation on the DirectX Discord server about when to use “-“ or “/” for compile argument but if you look at HLSLOptions.td in the DirectXShaderCompiler reposity, you’ll see “-“ works for all arguments while “/” only works for some. So it’s always best to use “-“.

When you execute dxc.exe -help, it shows you the most important options which I’ve formatted below, mostly for my own reference.

General Options:    
  -help Display available options
  -nologo Suppress copyright message
  -Qunused-arguments Don’t emit warning for unused driver arguments
Compilation Options:    
  -all_resources_bound Enables agressive flattening
  -auto-binding-space <value> Set auto binding space - enables auto resource binding in libraries
  -Cc Output color coded assembly listings
  -default-linkage <value> Set default linkage for non-shader functions when compiling or linking to a library target (internal, external)
  -denorm <value> select denormal value options (any, preserve, ftz). any is the default.
  -D <value> Define macro
  -enable-16bit-types Enable 16bit types and disable min precision types. Available in HLSL 2018 and shader model 6.2
  -export-shaders-only Only export shaders when compiling a library
  -exports <value> Specify exports when compiling a library: export1[[,export1_clone,…]=internal_name][;…]
  -E <value> Entry point name
  -Fc <file> Output assembly code listing file
  -Fd <file> Write debug information to the given file, or automatically named file in directory when ending in ‘'
  -Fe <file> Output warnings and errors to the given file
  -Fh <file> Output header file containing object code
  -flegacy-macro-expansion Expand the operands before performing token-pasting operation (fxc behavior)
  -flegacy-resource-reservation Reserve unused explicit register assignments for compatibility with shader model 5.0 and below
  -force_rootsig_ver <profile> force root signature version (rootsig_1_1 if omitted)
  -Fo <file> Output object file
  -Gec Enable backward compatibility mode
  -Ges Enable strict mode
  -Gfa Avoid flow control constructs
  -Gfp Prefer flow control constructs
  -Gis Force IEEE strictness
  -HV <value> HLSL version (2016, 2017, 2018). Default is 2018
  -H Show header includes and nesting depth
  -ignore-line-directives Ignore line directives
  -I <value> Add directory to include search path
  -Lx Output hexadecimal literals
  -Ni Output instruction numbers in assembly listings
  -no-warnings Suppress warnings
  -not_use_legacy_cbuf_load Do not use legacy cbuffer load
  -No Output instruction byte offsets in assembly listings
  -Odump Print the optimizer commands.
  -Od Disable optimizations
  -pack_optimized Optimize signature packing assuming identical signature provided for each connecting stage
  -pack_prefix_stable (default) Pack signatures preserving prefix-stable property - appended elements will not disturb placement of prior elements
  -recompile recompile from DXIL container with Debug Info or Debug Info bitcode file
  -res_may_alias Assume that UAVs/SRVs may alias
  -rootsig-define <value> Read root signature from a #define
  -T <profile> Set target profile.
  -Vd Disable validation
  -Vi Display details about the include process.
  -Vn <name> Use <name> as variable name in header file
  -WX Treat warnings as errors
  -Zi Enable debug information
  -Zpc Pack matrices in column-major order
  -Zpr Pack matrices in row-major order
  -Zsb Build debug name considering only output binary
  -Zss Build debug name considering source information
Optimization Options:    
  -O0 Optimization Level 0
  -O1 Optimization Level 1
  -O2 Optimization Level 2
  -O3 Optimization Level 3 (Default)
SPIR-V CodeGen Options:    
  -fspv-debug=<value> Specify whitelist of debug info category (file -> source -> line, tool)
  -fspv-extension=<value> Specify SPIR-V extension permitted to use
  -fspv-reflect Emit additional SPIR-V instructions to aid reflection
  -fspv-target-env=<value> Specify the target environment: vulkan1.0 (default) or vulkan1.1
  -fvk-b-shift <shift> <space> Specify Vulkan binding number shift for b-type register
  -fvk-bind-globals <binding> <set> Specify Vulkan binding number and set number for the $Globals cbuffer
  -fvk-bind-register <type-number> <space> <binding> <set> Specify Vulkan descriptor set and binding for a specific register
  -fvk-invert-y Negate SV_Position.y before writing to stage output in VS/DS/GS to accommodate Vulkan’s coordinate system
  -fvk-s-shift <shift> <space> Specify Vulkan binding number shift for s-type register
  -fvk-t-shift <shift> <space> Specify Vulkan binding number shift for t-type register
  -fvk-u-shift <shift> <space> Specify Vulkan binding number shift for u-type register
  -fvk-use-dx-layout Use DirectX memory layout for Vulkan resources
  -fvk-use-dx-position-w Reciprocate SV_Position.w after reading from stage input in PS to accommodate the difference between Vulkan and DirectX
  -fvk-use-gl-layout Use strict OpenGL std140/std430 memory layout for Vulkan resources
  -fvk-use-scalar-layout Use scalar memory layout for Vulkan resources
  -Oconfig=<value> Specify a comma-separated list of SPIRV-Tools passes to customize optimization configuration (see http://khr.io/hlsl2spirv#optimization)
  -spirv Generate SPIR-V code
Utility Options:    
-dumpbin Load a binary file rather than compiling  
-extractrootsignature Extract root signature from shader bytecode (must be used with /Fo <file>)  
-getprivate <file> Save private data from shader blob  
-P <value> Preprocess to file (must be used alone)  
-Qembed_debug Embed PDB in shader container (must be used with /Zi)  
-Qstrip_debug Strip debug information from 4_0+ shader bytecode (must be used with /Fo <file>)
-Qstrip_priv Strip private data from shader bytecode (must be used with /Fo <file>)
-Qstrip_reflect Strip reflection data from shader bytecode (must be used with /Fo <file>)
-Qstrip_rootsignature Strip root signature data from shader bytecode (must be used with /Fo <file>)  
-setprivate <file> Private data to add to compiled shader blob  
-setrootsignature <file> Attach root signature to shader bytecode  
-verifyrootsignature <file> Verify shader bytecode with root signature  

Some great articles that go more in-depth