循迹研究室

UE中ASTC贴图压缩分析及效率优化

ASTC texture compression analysis and efficiency optimization in UE
发表于 分类于 ， 阅读次数：
本文字数： 4k 阅读时长 ≈ 10 分钟

ASTC是Adaptive Scalable Texture Compression的简称，是在移动端流行的贴图压缩方案。当平台使用ASTC打包时，UE默认使用Intel ISPC Texture Compressor压缩贴图，但它具有一些局限性，只支持8x8及以上的压缩规格，10x10和12x12则不支持，如果在项目中指定它们，也是会使用8x8的规格。除ISPC外，引擎中也提供了ARM的astc-encoder压缩方式，可以支持8x8以下的规格，但默认未启用，而且引擎中的集成压缩效率非常低，在大规模资源中，使用astc-encoder压缩贴图对Cook耗时是非常大的挑战。

本篇文章分析UE中贴图使用ASTC压缩的配置以及实现方式，以及引擎中astc-encoder的压缩效率和优化思路。

Texture的压缩配置

在Project Settings-Cooker-Texture可以设置默认的资源质量和大小的级别：CookerSettings.h

与ASTC块大小的对照：

1
2
3
4
5
0=12x12 
1=10x10 
2=8x8 
3=6x6 
4=4x4

引擎里的默认值是3：Engine/Config/BaseEngine.ini

在Texture的资源编辑中也可以针对某个Texture单独设置：

Lowest->Hightest对应着0-4的值，使用Default则使用项目设置中的配置。

并且，设置Compression Settings的类型也会对资源压缩的类型有差别，Default则是项目设置中的参数，如果设置成NormalMap的类型会是ASTC_4x4的。

Intel ISPC

Intel ISPC是引擎中默认的ASTC压缩器，但它只支持最大块大小8x8，像更高的10x10和12x12是不支持的。

它的Github README中有介绍：

ASTC (LDR, block sizes up to 8x8)

在Adaptive Scalable Texture Compression User Guide中也有具体的描述：

The Intel ISPC Texture Compressoris a compression library developed by Intel, supporting multiple texture formats include ASTC. It uses the Intel ISPC (Implicit SPMD Program Compiler) compilerto parallelize key parts of the compressor implementation for the target CPU SIMD instruction set.
The ASTC support in the tool only supports the 2D LDR profile and block sizes up to 8x8. It has image quality that is similar to astcenc -fast for photographic imagery, but can be up to three times faster at compressing. Its quality is measurably worse than astcenc -fast for non-photographic imagery, such as normal maps, mask maps, or cartoon-like color data.

这意味着在UE中，使用ASTC如果在项目配置中设置8x8以上的压缩设置会无效，因为这种ISPC压缩方式最大仅支持8x8的。

UE代码中的实现同样如此：Engine/Source/Developer/TextureFormatIntelISPCTexComp/Private/TextureFormatIntelISPCTexComp.cpp

ARM astcenc

UE还提供了了另一套ASTC的Texture压缩方式：ARM的astc-encoder。如果没有使用ISPC的话就会使用它。

1
2
3
4
5
6
// Developer\TextureFormatASTC\Private\TextureFormatASTC.cpp
#if PLATFORM_WINDOWS || PLATFORM_LINUX || PLATFORM_MAC
	#define SUPPORTS_ISPC_ASTC	1
#else
	#define SUPPORTS_ISPC_ASTC	0
#endif

引擎中的代码：TextureFormatASTC.cpp#L450

默认情况下，在UE4的版本中，只能通过修改SUPPORTS_ISPC_ASTC宏定义的值或修改引擎中上图CompressImage代码实现。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
static bool UseArmASTCCompressor()
{
	bool bUseArmASTC = false;
	GConfig->GetBool(TEXT("/Script/UnrealEd.CookerSettings"), TEXT("UsArmASTCCompressor"), bUseArmASTC, GEngineIni);
	return bUseArmASTC;
}
// for CompressImage
#if SUPPORTS_ISPC_ASTC
if (!UseArmASTCCompressor())
{
	return IntelISPCTexCompFormat.CompressImage(InImage, BuildSettings, bImageHasAlphaChannel, OutCompressedImage);
}
#endif
// arm astcenc...

编辑DefaultEngine.ini即可控制开关：

1
2
[/Script/UnrealEd.CookerSettings]
UsArmASTCCompressor=true

当关闭ISPC压缩时，替代的ARM astcenc会拉起引擎Binaries/ThirdParty下的astsenc程序来压缩贴图：

调用栈：

拉起单独的进程执行的，不是在UE进程内执行，在拉起astcenc执行压缩的过程中，UE的Cook进程默认是逐资源，虽然DDC的cache过程可以是异步的，但从Cook的角度看，当前资源仍然是阻塞执行的。并且因为是拉起单独的进程处理，可能会产生很多的进程启动和切换开销，这个过程会非常慢，可能一张贴图的耗时都要好几分钟，对于构建时间是比较大的挑战。

astcenc的执行，是根据贴图设置构造出astcenc的命令行参数：

1
2
3
4
# 4x4
../../../Engine/Binaries/ThirdParty/ARM/Win32/astcenc.exe -c "D:/UnrealProjects/AstcExample/Intermediate/Cache/60c572f7-471422f1-24309fb0-45c3d4f9-RGBToASTCIn.webp" "D:/UnrealProjects/AstcExample/Intermediate/Cache/60c572f7-471422f1-24309fb0-45c3d4f9-RGBToASTCOut.astc" 4x4 -thorough  -esw bgr1 -ch 1 1 1 0 "
# 12x12
../../../Engine/Binaries/ThirdParty/ARM/Win32/astcenc.exe -c "D:/UnrealProjects/AstcExample/Intermediate/Cache/ea8ed05a-4371fffd-333d47a1-cf7b6490-RGBToASTCIn.webp" "D:/UnrealProjects/AstcExample/Intermediate/Cache/ea8ed05a-4371fffd-333d47a1-cf7b6490-RGBToASTCOut.astc" 12x12 -thorough -normal_percep -esw bbbg -dsw rgba"

拉起来多个进程是要对贴图的多个Mip机别进行分别压缩：

每个Mip对应着一个astcenc的处理进程。调用栈：

注意：不同的Mip机别的压缩参数完全相同，但进行压缩的Image是不同的，这一点要注意，从上图的调用栈中也能看出来。

UE5

在UE5中，ASTC默认也是ISPC的压缩器，但可以通过cook.ASTCTextureCompressor来控制，0是ISPC，1是arm的。

并且在Project Settings-Cooker中可以配置使用的默认压缩器：

效率分析&优化

DDC Log

分析贴图的压缩过程，可以开启Log和LogTexture的Verbose的日志机别：

1
2
3
4
; DefaultEngine.ini
[Core.Log]
LogDerivedDataCache=Verbose
LogTexture=Verbose

在执行Cook过程会输出以下内容：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
LogDerivedDataCache: Verbose: GetSynchronous TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D'
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__EA6A229E23092C6517F13F71B2BF5C2E8C64457A
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) Cache miss on TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__EA6A229E23092C6517F13F71B2BF5C2E8C64457A
LogTexture: Display: Building textures: T_ForestGround_D (ASTC_RGBAuto, 2048X2048)
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogTextureFormatASTC: Display: Compressing to ASTC (options = '6x6 -fast  -esw bgra -ch 1 1 1 1')...
LogDerivedDataCache: Verbose: Put TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP0_2048x2048 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D'
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) queueing TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__47BCD9955FA7DFF7F5CFEFB8D7FBCF9D3D9B6312 for put
LogDerivedDataCache: Verbose: Put TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP1_1024x1024 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D'
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__47BCD9955FA7DFF7F5CFEFB8D7FBCF9D3D9B6312
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) queueing TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__F5074C647712694523D5FA76B15A7517E5FCEDF5 for put
LogDerivedDataCache: Verbose: Put TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP2_512x512 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D'
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) queueing TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__AFC2F7816F8391CA659275BFD85152D17743C125 for put
LogDerivedDataCache: Verbose: Put TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP3_256x256 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D'
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__F5074C647712694523D5FA76B15A7517E5FCEDF5
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) queueing TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__C1072EE744DD250899FAB02DB01DEEB85A98C532 for put
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__AFC2F7816F8391CA659275BFD85152D17743C125
LogDerivedDataCache: Verbose: Put TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP4_128x128 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D'
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__AFC2F7816F8391CA659275BFD85152D17743C125
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) queueing TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__4190A033AE05AC24DA30E7B5AD2E6827F260A716 for put
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__C1072EE744DD250899FAB02DB01DEEB85A98C532
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__C1072EE744DD250899FAB02DB01DEEB85A98C532
LogDerivedDataCache: Verbose: Put TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D'
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__4190A033AE05AC24DA30E7B5AD2E6827F260A716
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__F5074C647712694523D5FA76B15A7517E5FCEDF5
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) queueing TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__EA6A229E23092C6517F13F71B2BF5C2E8C64457A for put
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__4190A033AE05AC24DA30E7B5AD2E6827F260A716
LogTexture: Verbose: Storing texture in DDC:
  Key: TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B
  Format: ASTC_6x6
  Mip0 2048x2048 1871424 bytes TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP0_2048x2048
  Mip1 1024x1024 467856 bytes TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP1_1024x1024
  Mip2 512x512 118336 bytes TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP2_512x512
  Mip3 256x256 29584 bytes TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP3_256x256
  Mip4 128x128 7744 bytes TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP4_128x128
  Mip5 64x64 1936 bytes [inline] TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP5_64x64
  Mip6 32x32 576 bytes [inline] TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP6_32x32
  Mip7 16x16 144 bytes [inline] TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP7_16x16
  Mip8 8x8 64 bytes [inline] TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP8_8x8
  Mip9 4x4 16 bytes [inline] TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP9_4x4
  Mip10 2x2 16 bytes [inline] TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP10_2x2
  Mip11 1x1 16 bytes [inline] TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP11_1x1
  Derived Data: 4648 bytes
LogDerivedDataCache: Verbose: GetAsynchronous TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP0_2048x2048 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D', Handle 19506
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__AFC2F7816F8391CA659275BFD85152D17743C125
LogDerivedDataCache: Verbose: GetAsynchronous TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP1_1024x1024 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D', Handle 19507
LogDerivedDataCache: Verbose: GetAsynchronous TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP2_512x512 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D', Handle 19508
LogDerivedDataCache: Verbose: GetAsynchronous TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP3_256x256 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D', Handle 19509
LogDerivedDataCache: Verbose: GetAsynchronous TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CBBEBEB5538C060A_07_0000803F0000803F0000803F000000000000803F00000000000000000000803F0000000000000000020000000100000100000000000101000000040000803FFFFFFFFF00000000FF00FF00FFFF8180803B_MIP4_128x128 from '/Game/ExampleContent/Landscapes/Textures/T_ForestGround_D.T_ForestGround_D', Handle 19510
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) CacheHit from InFlightCache on TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__47BCD9955FA7DFF7F5CFEFB8D7FBCF9D3D9B6312
LogDerivedDataCache: Verbose: WaitAsynchronousCompletion, Handle 19506
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) CacheHit from InFlightCache on TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__F5074C647712694523D5FA76B15A7517E5FCEDF5
LogDerivedDataCache: Verbose: GetAsynchronousResults, bDataWasBuilt: 0, Handle 19506, SUCCESS
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) CacheHit from InFlightCache on TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__AFC2F7816F8391CA659275BFD85152D17743C125
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) CacheHit from InFlightCache on TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__C1072EE744DD250899FAB02DB01DEEB85A98C532
LogDerivedDataCache: Verbose: AsyncPutWrapper (HierarchicalDerivedDataBackend) CacheHit from InFlightCache on TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__4190A033AE05AC24DA30E7B5AD2E6827F260A716
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__47BCD9955FA7DFF7F5CFEFB8D7FBCF9D3D9B6312
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__4190A033AE05AC24DA30E7B5AD2E6827F260A716
LogDerivedDataCache: Verbose: WaitAsynchronousCompletion, Handle 19507
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__F5074C647712694523D5FA76B15A7517E5FCEDF5
LogDerivedDataCache: Verbose: GetAsynchronousResults, bDataWasBuilt: 0, Handle 19507, SUCCESS
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__C1072EE744DD250899FAB02DB01DEEB85A98C532
LogDerivedDataCache: Verbose: WaitAsynchronousCompletion, Handle 19508
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache: Successful cache put of TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__AFC2F7816F8391CA659275BFD85152D17743C125 to ../../../Engine/DerivedDataCache/7/7/2/TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAUTO_25638_39DF77B84020C752CB__AFC2F7816F8391CA659275BFD85152D17743C125.udd
LogDerivedDataCache: Verbose: GetAsynchronousResults, bDataWasBuilt: 0, Handle 19508, SUCCESS
LogDerivedDataCache: Verbose: WaitAsynchronousCompletion, Handle 19509
LogDerivedDataCache: Verbose: GetAsynchronousResults, bDataWasBuilt: 0, Handle 19509, SUCCESS
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=1 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__AFC2F7816F8391CA659275BFD85152D17743C125
LogDerivedDataCache: Verbose: WaitAsynchronousCompletion, Handle 19510
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__EA6A229E23092C6517F13F71B2BF5C2E8C64457A
LogDerivedDataCache: Verbose: GetAsynchronousResults, bDataWasBuilt: 0, Handle 19510, SUCCESS
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=0 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__47BCD9955FA7DFF7F5CFEFB8D7FBCF9D3D9B6312
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache: Successful cache put of TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__4190A033AE05AC24DA30E7B5AD2E6827F260A716 to ../../../Engine/DerivedDataCache/1/1/3/TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAUTO_25638_39DF77B84020C752CB__4190A033AE05AC24DA30E7B5AD2E6827F260A716.udd
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache: Successful cache put of TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__F5074C647712694523D5FA76B15A7517E5FCEDF5 to ../../../Engine/DerivedDataCache/7/1/7/TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAUTO_25638_39DF77B84020C752CB__F5074C647712694523D5FA76B15A7517E5FCEDF5.udd
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=1 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__4190A033AE05AC24DA30E7B5AD2E6827F260A716
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=1 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__F5074C647712694523D5FA76B15A7517E5FCEDF5
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache: Successful cache put of TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__EA6A229E23092C6517F13F71B2BF5C2E8C64457A to ../../../Engine/DerivedDataCache/5/8/2/TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAUTO_25638_39DF77B84020C752CB__EA6A229E23092C6517F13F71B2BF5C2E8C64457A.udd
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache: Successful cache put of TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__C1072EE744DD250899FAB02DB01DEEB85A98C532 to ../../../Engine/DerivedDataCache/2/1/5/TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAUTO_25638_39DF77B84020C752CB__C1072EE744DD250899FAB02DB01DEEB85A98C532.udd
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=1 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__EA6A229E23092C6517F13F71B2BF5C2E8C64457A
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache: Successful cache put of TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__47BCD9955FA7DFF7F5CFEFB8D7FBCF9D3D9B6312 to ../../../Engine/DerivedDataCache/2/6/1/TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAUTO_25638_39DF77B84020C752CB__47BCD9955FA7DFF7F5CFEFB8D7FBCF9D3D9B6312.udd
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=1 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__C1072EE744DD250899FAB02DB01DEEB85A98C532
LogDerivedDataCache: Verbose: ../../../Engine/DerivedDataCache CachedDataProbablyExists=1 for TEXTURE_564290F8998644E39A2118D5C683187B_ASTC_RGBAuto_25638_39DF77B84020C752CB__47BCD9955FA7DFF7F5CFEFB8D7FBCF9D3D9B6312

便于查看DDC和贴图压缩过程中的详细信息。

主要耗时

通过前面的分析，在使用ARM astcenc进行Cook时的贴图压缩时，单张贴图的开销情况大概有以下几部分：

  1. 文件的序列化（看到压缩时需要存盘/读取）
  2. 压缩进程的冷启动
  3. 等待压缩完毕（压缩进程退出）
  4. 多个Mip级别会拉起多个进程

在UE原始的Cook流程中，逐资源处理，没有办法很好地利用起来硬件的多核优势，本质上就是Cook一张贴图逐Mip拉起来进程压缩，这样串行循环的过程。

使用HotPatcher进行单张贴图的ARM astcenc测试，从几秒到数分钟不等：

1
2
3
Display: Package /Game/ExampleContent/Landscapes/Textures/T_ForestGround_D Time : 29.767000s
Display: Package /Game/ExampleContent/Landscapes/Textures/T_ForestGround_N Time : 145.753000s
Display: Package /Game/ExampleContent/ScriptExamples/PivotPainter/Textures/T_PalmTree_N Time : 232.641000s

Cook 28张贴图耗时14分钟：

而贴图又是工程中最大量的资源类型，完全DDC miss的情况下，Cook耗时难以想象，达到数十小时。

替换astcenc

在UE4中，引擎默认使用的预编译astcenc是版本非常古老的1.3：

1
2
3
ASTC codec version 1.3
Copyright (C) 2011-2013 ARM Limited
All rights reserved. Use of this software is subject to terms of its license.

并且是32位的，没有SSE/SSE2指令集优化：

1
2
3
4
5
6
7
8
9
10
FILE HEADER VALUES
             14C machine (x86)
               5 number of sections
        515B0518 time date stamp Wed Apr  3 00:19:36 2013
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
             102 characteristics
                   Executable
                   32 bit word machine

升级astcenc版本

看Github上的release note中的记录以及一些文章，较新版本的astcenc对于压缩性能还是有不少提升的。

在UE5中，集成的astcenc是3.2.0，并且是64位的版本，具有SSE的指令集优化。但UE5中的astcenc不能直接放到UE4里使用，版本间参数的差异还是挺大的，需要把UE5的实现移植到UE4内。让旧版引擎中构造的astcenc参数和新版本匹配

编译x64 SSE版本

这种方式，可以保持引擎内astcenc的参数兼容，无需对引擎实现产生变动，只需编译出一个64位的SSE指令集优化版本。

注：编译64位时默认启用了SSE指令集优化，无需再单独指定/arch:sse。

优化前：

优化后：

从原来的7m12s到1m24s，节省了约80%的耗时，有非常大的提升。

Win32 Win64 SSE
耗时（s） 432 84

并行CacheDDC

通过替代astcenc可以提高单张贴图的压缩效率，但逐资源地处理过程，哪怕所有Mip都并行处理，最多也就拉起Mip数量的astcenc进程，在超大核心CPU下，也不能够跑满机器性能。

所以，可以把单个资源的Cook过程拆开，批量地去Cache资源的DDC，但不执行序列化操作。通过异步任务拉起astcenc执行实际的压缩过程，可以实现单个UE进程同时处理多个贴图压缩的过程。

这样能够充分利用多核心CPU的性能，而避免逐资源逐Mip执行时，跑不满性能，CPU空转的情况。

但在这种单Cook进程，并行压缩贴图的情况，对于机器CPU性能要求较高。简单来说，如果一个4核心CPU拉起远超于核心数量的任务数，不仅起不到加速的作用，反而会因为进程的频繁切换而变慢（增加系统进程调度开销、进程间的CPU竞争等）。

合理的做法是，根据核心数决定任务数量，充分利用CPU的同时避免超负载。但一台机器的物理性能总归是有限的，对于海量的资源和单机瓶颈难以兼顾。

所以我实现了一种，多机器并行Cook的方案，不同于FastBuild与IB这种联机编译的实现，它们局限在代码和Shader的编译，但无法用在其他的资源类型中，在本文的这种情况下，贴图的压缩耗时已经远超于Shader的编译耗时。

单机的运行模拟：

在多机模式下，每个Agent可以按照机器性能分配一定的资源量，可以避免单机的机能限制。并且机器间可以共享互相产出的DDC数据，实现真正地并行Cook。

相关资料

全文完，若有不足之处请评论指正。

微信扫描二维码，关注我的公众号。

本文标题:UE中ASTC贴图压缩分析及效率优化
文章作者:查利鹏
发布时间:2022年09月08日 19时01分
本文字数:本文一共有4k字
原始链接:https://imzlp.com/posts/7181/
许可协议: CC BY-NC-SA 4.0
文章禁止全文转载，摘要转发请保留原文链接及作者信息，谢谢！
您的捐赠将鼓励我继续创作！