Using the DirectXShaderCompiler C++ API

04 Mar 2020 - Simon Coenen - Reading time: 17 mins - Comments

Shader Model 6 has existed for quite a while now and after reading this great article from Francesco Cifariello Ciardi about scalarizing the light loop in tiled/clustered lighting, I wanted to try out Wave Intrisics. The old fxc.exe compiles to DXBC and only supports up to Shader Model 5.1. Microsoft has since introduced their new llvm-based compiler called DirectXShaderCompiler (DXC) which compiles to DXIL. It’s completely open-source on GitHub here: https://github.com/microsoft/DirectXShaderCompiler It both provides command line tools and a C++ API for compiling, validating and using shaders with SM 6.0 and up.

Fxc has existed for quite a long time now and there are guides for it everywhere however Dxc is quite a bit younger and I didn’t find too much about it. The documentation for it is quite minimal and incomplete which is surprising considering DirectX is generally really well documented. Using Dxc’s commandline tools are quite straight forward and self-explanatory however I’ve always like to compile my shaders at runtime because it eventually require no effort when modifying shaders and having to recompile them. Unfortunately, I didn’t find too much documentation on the C++ API.

I’ve found a few articles here and there about the very basic setup for this however most of the were very minimal and didn’t really cover things like compile-arguments, defines, include handling, validation, reflection, debug-data, … which are quite important to know about.

(20/03/2020): DirectX Developer day has happened and there’s been a super good talk by one of the DXC developers and they gave a great walkthrough of the interface. I’d definitely recommend watching that as things described in this post might not all be correct. Check it out here!

bool CompileShader(const wchar_t* pFilePath, const wchar_t* pTarget, const wchar_t* pEntryPoint, const std::vector<std::pair<std::wstring, std::string>>& defines, ComPtr<ID3DBlob>& pOutput)
{
	ComPtr<IDxcLibrary> pLibrary;
	HR(DxcCreateInstance(CLSID_DxcLibrary, IID_PPV_ARGS(pLibrary.GetAddressOf())));
	ComPtr<IDxcCompiler> pCompiler;
	HR(DxcCreateInstance(CLSID_DxcCompiler, IID_PPV_ARGS(pCompiler.GetAddressOf())));
	ComPtr<IDxcBlobEncoding> pSource;
	HR(pLibrary->CreateBlobWithEncodingFromPinned(source.c_str(), (uint32)source.size(), CP_UTF8, pSource.GetAddressOf()));

	wchar_t* pArgs[] =
	{
		L"-Zpr",			//Row-major matrices
		L"-WX",				//Warnings as errors
#ifdef _DEBUG
		L"-Zi",				//Debug info
		L"-Qembed_debug",	//Embed debug info into the shader
		L"-Od",				//Disable optimization
#else
		L"-O3",				//Optimization level 3
#endif
	};
	
	std::vector<DxcDefine> dxcDefines(defines.size());
	for (size_t i = 0; i < defines.size(); ++i)
	{
		DxcDefine& m = dxcDefines(i);
		m.Name = defines[i].first.c_str();
		m.Value = defines[i].second.c_str();
	}

	ComPtr<IDxcOperationResult> pCompileResult;
	HR(pCompiler->Compile(pSource.Get(), pFileName, pEntryPoint, pTarget, &pArgs[0], sizeof(pArgs) / sizeof(pArgs[0]), dxcDefines.data(), (uint32)dxcDefines.size(), nullptr, pCompileResult.GetAddressOf()));

	HRESULT hrCompilation;
	HR(pCompileResult->GetStatus(&hrCompilation));

	if (hrCompilation < 0)
	{
		ComPtr<IDxcBlobEncoding> pPrintBlob;
		HR(pCompileResult->GetErrorBuffer(pPrintBlob.GetAddressOf()));
		std::wcout << pPrintBlob->GetBufferPointer() << std::endl;
		return false;
	}
	IDxcBlob** pBlob = reinterpret_cast<IDxcBlob**>(pOutput.GetAddressOf());
	pCompileResult->GetResult(pBlob);
	return true;
}

Reflection

Reflection with Dxc output is only slightly different from Fxc, you’ll eventually end up with a ID3D12ShaderReflection reference and from then on you can just do the same as you would usually do.

void DoReflection(const ComPtr<IDxcBlob>& pShader)
{
	ComPtr<IDxcContainerReflection> pReflection;
	DxcCreateInstance(CLSID_DxcContainerReflection, IID_PPV_ARGS(pReflection.GetAddressOf()));
	HR(pReflection->Load(pShader.Get()));
	uint32 partIndex;

#ifndef MAKEFOURCC
#define MAKEFOURCC(a, b, c, d) (unsigned int)((unsigned char)(a) | (unsigned char)(b) << 8 | (unsigned char)(c) << 16 | (unsigned char)(d) << 24)
#endif
	HR(pReflection->FindFirstPartKind(MAKEFOURCC('D', 'X', 'I', 'L'), &partIndex));
#undef MAKEFOURCC

	ComPtr<ID3D12ShaderReflection> pShaderReflection;
	HR(pReflection->GetPartReflection(partIndex, IID_PPV_ARGS(pShaderReflection.GetAddressOf())));
	
	//...
	D3D12_SHADER_DESC shaderDesc;
	HR(pShaderReflection->GetDesc(&shaderDesc));
	//...
}

Stripping debug data

In some cases, you might want to compile the shader, process some of the debug data and then strip the debug data before serializing the shader. This can be done using the IDxcContainer. All the parts of a DXIL container are defined in DxilContainer.h and could be stripped however I found that stripping certain parts throw exceptions. It’s also important to know that if you’re not sure the container has a certain part, you have to check it first otherwise the function will fail.

Unfortunately, it doesn’t seem to be possible currently to strip out reflection data. A workaround for that could possibly to compile the shader twice, once with reflection data that can be thrown away and once without. In reality, this won’t really happen because usually shader compilation is done offline but in the case of having to support UGC, this could be an option.

The cool thing about this stripping is that you can separate shader parts into different blobs and serialize them in different ways. That way you could save shader PDBs separate (This can also easily be done with IDxcCompiler2::CompileWithDebug)

void StripDebugData(ComPtr<IDxcBlob>& pShader)
{
	ComPtr<IDxcContainerBuilder> pContainerBuilder;
	HR(DxcCreateInstance(CLSID_DxcContainerBuilder, IID_PPV_ARGS(pContainerBuilder.GetAddressOf())));
	HR(pContainerBuilder->Load(pShader.Get()));
	HR(pContainerBuilder->RemovePart(MAKEFOURCC('I', 'L', 'D', 'B'))); //DFCC_ShaderDebugInfoDXIL
	HR(pContainerBuilder->RemovePart(MAKEFOURCC('I', 'L', 'D', 'N'))); //DFCC_ShaderDebugName

	ComPtr<IDxcOperationResult> pResult;
	HR(pContainerBuilder->SerializeContainer(pResult.GetAddressOf()));
	//Do status checking
	//...
	HR(pResult->GetResult(pShader.GetAddressOf()));
}

Here is a list of all the parts:

enum DxilFourCC { 
   DFCC_Container                = DXIL_FOURCC('D', 'X', 'B', 'C'),
   DFCC_ResourceDef              = DXIL_FOURCC('R', 'D', 'E', 'F'), 
   DFCC_InputSignature           = DXIL_FOURCC('I', 'S', 'G', '1'), 
   DFCC_OutputSignature          = DXIL_FOURCC('O', 'S', 'G', '1'), 
   DFCC_PatchConstantSignature   = DXIL_FOURCC('P', 'S', 'G', '1'), 
   DFCC_ShaderStatistics         = DXIL_FOURCC('S', 'T', 'A', 'T'), 
   DFCC_ShaderDebugInfoDXIL      = DXIL_FOURCC('I', 'L', 'D', 'B'), 
   DFCC_ShaderDebugName          = DXIL_FOURCC('I', 'L', 'D', 'N'), 
   DFCC_FeatureInfo              = DXIL_FOURCC('S', 'F', 'I', '0'), 
   DFCC_PrivateData              = DXIL_FOURCC('P', 'R', 'I', 'V'), 
   DFCC_RootSignature            = DXIL_FOURCC('R', 'T', 'S', '0'), 
   DFCC_DXIL                     = DXIL_FOURCC('D', 'X', 'I', 'L'), 
   DFCC_PipelineStateValidation  = DXIL_FOURCC('P', 'S', 'V', '0'), 
   DFCC_RuntimeData              = DXIL_FOURCC('R', 'D', 'A', 'T'), 
   DFCC_ShaderHash               = DXIL_FOURCC('H', 'A', 'S', 'H'), 
 }; 

All compiler arguments

I’ve also found it pretty hard to find the compiler options for DXC. When using FXC, most of the compiler options were defines and easy to find because they were all defined together in a unified way. That doesn’t seem to be the case for DXC.

I recently saw a conversation on the DirectX Discord server about when to use “-“ or “/” for compile argument but if you look at HLSLOptions.td in the DirectXShaderCompiler reposity, you’ll see “-“ works for all arguments while “/” only works for some. So it’s always best to use “-“.

When you execute dxc.exe -help, it shows you the most important options which I’ve formatted below, mostly for my own reference.

General Options:    
  -help Display available options
  -nologo Suppress copyright message
  -Qunused-arguments Don’t emit warning for unused driver arguments
Compilation Options:    
  -all_resources_bound Enables agressive flattening
  -auto-binding-space <value> Set auto binding space - enables auto resource binding in libraries
  -Cc Output color coded assembly listings
  -default-linkage <value> Set default linkage for non-shader functions when compiling or linking to a library target (internal, external)
  -denorm <value> select denormal value options (any, preserve, ftz). any is the default.
  -D <value> Define macro
  -enable-16bit-types Enable 16bit types and disable min precision types. Available in HLSL 2018 and shader model 6.2
  -export-shaders-only Only export shaders when compiling a library
  -exports <value> Specify exports when compiling a library: export1[[,export1_clone,…]=internal_name][;…]
  -E <value> Entry point name
  -Fc <file> Output assembly code listing file
  -Fd <file> Write debug information to the given file, or automatically named file in directory when ending in ‘'
  -Fe <file> Output warnings and errors to the given file
  -Fh <file> Output header file containing object code
  -flegacy-macro-expansion Expand the operands before performing token-pasting operation (fxc behavior)
  -flegacy-resource-reservation Reserve unused explicit register assignments for compatibility with shader model 5.0 and below
  -force_rootsig_ver <profile> force root signature version (rootsig_1_1 if omitted)
  -Fo <file> Output object file
  -Gec Enable backward compatibility mode
  -Ges Enable strict mode
  -Gfa Avoid flow control constructs
  -Gfp Prefer flow control constructs
  -Gis Force IEEE strictness
  -HV <value> HLSL version (2016, 2017, 2018). Default is 2018
  -H Show header includes and nesting depth
  -ignore-line-directives Ignore line directives
  -I <value> Add directory to include search path
  -Lx Output hexadecimal literals
  -Ni Output instruction numbers in assembly listings
  -no-warnings Suppress warnings
  -not_use_legacy_cbuf_load Do not use legacy cbuffer load
  -No Output instruction byte offsets in assembly listings
  -Odump Print the optimizer commands.
  -Od Disable optimizations
  -pack_optimized Optimize signature packing assuming identical signature provided for each connecting stage
  -pack_prefix_stable (default) Pack signatures preserving prefix-stable property - appended elements will not disturb placement of prior elements
  -recompile recompile from DXIL container with Debug Info or Debug Info bitcode file
  -res_may_alias Assume that UAVs/SRVs may alias
  -rootsig-define <value> Read root signature from a #define
  -T <profile> Set target profile.
  -Vd Disable validation
  -Vi Display details about the include process.
  -Vn <name> Use <name> as variable name in header file
  -WX Treat warnings as errors
  -Zi Enable debug information
  -Zpc Pack matrices in column-major order
  -Zpr Pack matrices in row-major order
  -Zsb Build debug name considering only output binary
  -Zss Build debug name considering source information
Optimization Options:    
  -O0 Optimization Level 0
  -O1 Optimization Level 1
  -O2 Optimization Level 2
  -O3 Optimization Level 3 (Default)
SPIR-V CodeGen Options:    
  -fspv-debug=<value> Specify whitelist of debug info category (file -> source -> line, tool)
  -fspv-extension=<value> Specify SPIR-V extension permitted to use
  -fspv-reflect Emit additional SPIR-V instructions to aid reflection
  -fspv-target-env=<value> Specify the target environment: vulkan1.0 (default) or vulkan1.1
  -fvk-b-shift <shift> <space> Specify Vulkan binding number shift for b-type register
  -fvk-bind-globals <binding> <set> Specify Vulkan binding number and set number for the $Globals cbuffer
  -fvk-bind-register <type-number> <space> <binding> <set> Specify Vulkan descriptor set and binding for a specific register
  -fvk-invert-y Negate SV_Position.y before writing to stage output in VS/DS/GS to accommodate Vulkan’s coordinate system
  -fvk-s-shift <shift> <space> Specify Vulkan binding number shift for s-type register
  -fvk-t-shift <shift> <space> Specify Vulkan binding number shift for t-type register
  -fvk-u-shift <shift> <space> Specify Vulkan binding number shift for u-type register
  -fvk-use-dx-layout Use DirectX memory layout for Vulkan resources
  -fvk-use-dx-position-w Reciprocate SV_Position.w after reading from stage input in PS to accommodate the difference between Vulkan and DirectX
  -fvk-use-gl-layout Use strict OpenGL std140/std430 memory layout for Vulkan resources
  -fvk-use-scalar-layout Use scalar memory layout for Vulkan resources
  -Oconfig=<value> Specify a comma-separated list of SPIRV-Tools passes to customize optimization configuration (see http://khr.io/hlsl2spirv#optimization)
  -spirv Generate SPIR-V code
Utility Options:    
-dumpbin Load a binary file rather than compiling  
-extractrootsignature Extract root signature from shader bytecode (must be used with /Fo <file>)  
-getprivate <file> Save private data from shader blob  
-P <value> Preprocess to file (must be used alone)  
-Qembed_debug Embed PDB in shader container (must be used with /Zi)  
-Qstrip_debug Strip debug information from 4_0+ shader bytecode (must be used with /Fo <file>)
-Qstrip_priv Strip private data from shader bytecode (must be used with /Fo <file>)
-Qstrip_reflect Strip reflection data from shader bytecode (must be used with /Fo <file>)
-Qstrip_rootsignature Strip root signature data from shader bytecode (must be used with /Fo <file>)  
-setprivate <file> Private data to add to compiled shader blob  
-setrootsignature <file> Attach root signature to shader bytecode  
-verifyrootsignature <file> Verify shader bytecode with root signature  

Some great articles that go more in-depth