From cb5a5e46454c39284109c7b79a84cb9d85484e0b Mon Sep 17 00:00:00 2001 From: Josh Freeman Date: Wed, 13 Dec 2017 11:22:49 +0000 Subject: [PATCH] Update sqpack_dat_files.txt --- research/sqpack_dat_files.txt | 90 ++++++++++++++++++++++------------- 1 file changed, 58 insertions(+), 32 deletions(-) diff --git a/research/sqpack_dat_files.txt b/research/sqpack_dat_files.txt index 7a5ef3c5f..186063a37 100644 --- a/research/sqpack_dat_files.txt +++ b/research/sqpack_dat_files.txt @@ -1,49 +1,75 @@ -# SqPack Index Files +#### DAT FILE RESEARCH #### By Ioncannon -Big thanks to: http://ffxivexplorer.fragmenterworks.com/research.php +Current research on SqPack Dat files ---- - -#### INDEX FILE RESEARCH #### By Ioncannon - -Current research on SqPack Index files +Last Updated: 12/14/2014 ===SQPACK HEADER=== (0x400 in size) 0x000: Signature Int32; "SqPack", followed by 0's (12 bytes) 0x00c: Header Length Int32; -0x010: ~~Unknown~~ Int32; Unknown but repeated in other header +0x010: ~~Unknown~~ Int32; Unknown but repeated in other header 0x014: SqPack Type Int32; Type 0: SQDB, Type 1: Data, Type 2: Index 0x018: ~~Unknown~~ A lot of 0s, but 0xFFFF at 0x20 -0x3c0: SHA-1 of Header 20B; SHA-1 of bytes 0x000-0x3BF, starts 64bytes before end of header +0x3c0: SHA-1 of Header 20B; SHA-1 of bytes 0x000-0x3BF, starts 64bytes before end of header ~~~~~~~~Padding~~~~~~~~~ Padding of 0's till [Header Length] -===SEGMENT HEADER=== (starts after SQPACK HEADER) +===DATA HEADER=== (0x400 in size) 0x000: Header Length Int32; +0x004: NULL +0x008: ~~Unknown~~ Int32; Static Value of 0x10 +0x00c: Data Size Int32; From end of this header (usually 0x800) to EOF. Divided by 0x08. +0x010: Spanned DAT Int32; 0x01 = .dat0, 0x02 = .dat1 or .dat2, etc +0x014: NULL Int32; +0x018: Max File Size Int32; Always 0x77359400 or 2GB. Interestingly if a file is > 2GB, a dat1 is formed. In 13 dat, is 20MB? +0x01c: NULL Int32; +0x020: SHA1 of Data 20B; From end of this header (usually 0x800) to EOF +0x3c0: SHA1 of Header 20B; Starts 64bytes before end of header +~~~~~~~~~~Padding~~~~~~~ Padding of 0's till [Header Length] -Each Segment follows this (starting right after header length): -0x000: Unknown/Num Dats Int32; For segment 2 it's the number of dats for this archive (dat0, dat1, etc), for others unknown. -0x004: Segment Offset Int32; Offset to the segment -0x008: Segment Size Int32; How large a segment is -0x00c: SHA-1 of Segment 20B; Hash of the segment... [Segment Offset] to [Segment Offset] + [Segment Size] +-----DATA IS BELOW THIS LINE------ !!!Use Index to find entries!!! -Notes: --Segment 1 is usually files, Segment 2/3 is unknown, Segment 4 is folders. --Segments may not exist, but their position is still treated as if they did (benchmark has no segment 3 but you still need to skip it's bytes). --Each segment is followed by a padding of 0x28 0's, except the first one, which has 4 extra 0s. +~~~~~~~~~~~~~~ -===FILE SEGMENTS=== (each one is at each folder's [Files Offset], in segment 1) (Each is 16 bytes padded) -0x000: File ID1 Hash Int32; Hash to the file name -0x004: File ID2 Hash Int32; Hash to the file path -0x008: File Data Offset Int32; Multiply by 0x08, points to compressed data in .dat file -0x00b: Padding Padded to a total segment entry size of 16 bytes. +===DATA ENTRY HEADER=== (Minimum size is 0x80) +0x000: Header Length Int32; +0x004: Content Type Int32; 0x01 - Empty Placeholder, 0x02 - Binary, 0x03 - Model, 0x04 - Texture +0x008: Uncompressed Size Int32; +0x00c: ~~Unknown~~ Int32; +0x010: Block Buffer Size Int32; Buffer size need to read largest block +0x014: Num Blocks Int32; -===FOLDER SEGMENTS=== (seen in segment 3, points to files in segment 1) (Each is 16 bytes padded) -0x000: FOLDER ID Hash Int32; Hash to the folder name -0x004: Files Offset Int32; Offset to file list in segment 1. -0x008: Total Files Size Int32; Total size of all file segments for this folder. To find # files, divide by 0x10 (16). -0x00b: Padding Padded to a total segment entry size of 16 bytes. +===TYPE-2 BLOCK TABLE=== (Size of Num blocks) +0x000: Offset Int32; From end of this header (Check [File Entry Offset] + [Header Length]) +0x004: Block Size Short; Total Block Size, from beginning of header to end of padding. +0x006: Decompressed Data Size Short; -NOTES: +===TYPE-3 BLOCK TABLE=== (Size of Num blocks) --Check the final byte of the offset. If Offset & 0x000F == 2, then the offset is pointing to dat1 rather dat0. Also remember -to subtract the 0x2 out. Theoretically, 0x4 would mean dat2 but unknown as of now. +0x000: Unknown Int32; Always 0x05000001 +0x004: Frame Uncompressed Chunk 44 Bytes; Each Block Follows Pattern: 5 Ints, 12b of NULL, 3 Ints. Size when uncompressed, but rounded +0x030: Frame Size Chunk 44 Bytes; Size of each frame (can be more than 1 data block). +0x05C: Frame Offset Chunk 44 Bytes; Offset to start of frame. +0x088: Block Size Indexes Short; Indexes into the next table below??? +0x0D0: Block Size Table Num Blocks*Short; Size of each data block below + +===TYPE-4 BLOCK TABLE=== (Size of Num blocks) +0x000: Frame Offset Int32; Starting block for this frame. +0x004: Frame Size Int32; This is the total size of the whole frame. +0x008: Unknown Int32; +0x00C: Frame Blocksize Offset Int32; Offset starting from the end of the block table, to the first size. +0x010: Frame Blocksize Count Int32; How many blocks this frame contains. +0x000: Frame Block Size Short; The size of the block in this frame. + +~~~~~~~~~~~~~~ + +~~~EXTRACTED FILE HEADER MAY BE HERE, PREPEND TO EXTRACT FILE~~~ + +-----Zlib compressed data starts here------- !!!Used Block table to find each block!!! + +===BLOCK HEADER=== +0x000: Header Size Int32; Seems to be always 0x10bytes +0x004: NULL Int32; +0x008: Compressed Length Int32; If this is 32000, IT'S NOT COMPRESSED. Use decompressed length to read the data in and just append +0x00c: Decompressed Length Int32; Will be max 16kb. + +0x010: Compressed Data ; Size will be [Compressed Length]