Update sqpack_dat_files.txt

2026-07-24 02:49:10 +00:00 · 2017-12-13 11:22:49 +00:00
parent e8bd100f59
commit cb5a5e4645
1 changed files with 58 additions and 32 deletions
@@ -1,49 +1,75 @@
-# SqPack Index Files
+#### DAT FILE RESEARCH #### By Ioncannon

-Big thanks to: http://ffxivexplorer.fragmenterworks.com/research.php
+Current research on SqPack Dat files

---
-
-#### INDEX FILE RESEARCH #### By Ioncannon
-
-Current research on SqPack Index files
+Last Updated: 12/14/2014

 ===SQPACK HEADER=== (0x400 in size)
 0x000: Signature        Int32; "SqPack", followed by 0's (12 bytes)
 0x00c: Header Length    Int32;  
-0x010: ~~Unknown~~             Int32; Unknown but repeated in other header
+0x010: ~~Unknown~~      Int32; Unknown but repeated in other header
 0x014: SqPack Type      Int32; Type 0: SQDB, Type 1: Data, Type 2: Index
 0x018: ~~Unknown~~             A lot of 0s, but 0xFFFF at 0x20
-0x3c0: SHA-1 of Header    20B; SHA-1 of bytes 0x000-0x3BF, starts 64bytes before end of header
+0x3c0: SHA-1 of Header  20B;   SHA-1 of bytes 0x000-0x3BF, starts 64bytes before end of header
 ~~~~~~~~Padding~~~~~~~~~       Padding of 0's till [Header Length]

-===SEGMENT HEADER=== (starts after SQPACK HEADER)
+===DATA HEADER=== (0x400 in size)
 0x000: Header Length    Int32;
+0x004: NULL
+0x008: ~~Unknown~~      Int32; Static Value of 0x10
+0x00c: Data Size        Int32; From end of this header (usually 0x800) to EOF. Divided by 0x08.
+0x010: Spanned DAT      Int32; 0x01 = .dat0, 0x02 = .dat1 or .dat2, etc
+0x014: NULL             Int32;
+0x018: Max File Size    Int32; Always 0x77359400 or 2GB. Interestingly if a file is > 2GB, a dat1 is formed. In 13 dat, is 20MB?
+0x01c: NULL             Int32;
+0x020: SHA1 of Data     20B;   From end of this header (usually 0x800) to EOF
+0x3c0: SHA1 of Header   20B;   Starts 64bytes before end of header
+~~~~~~~~~~Padding~~~~~~~       Padding of 0's till [Header Length]

-Each Segment follows this (starting right after header length):
-0x000: Unknown/Num Dats Int32; For segment 2 it's the number of dats for this archive (dat0, dat1, etc), for others unknown.
-0x004: Segment Offset   Int32; Offset to the segment
-0x008: Segment Size     Int32; How large a segment is
-0x00c: SHA-1 of Segment   20B; Hash of the segment... [Segment Offset] to [Segment Offset] + [Segment Size]
+-----DATA IS BELOW THIS LINE------ !!!Use Index to find entries!!!

-Notes: 
-Segment 1 is usually files, Segment 2/3 is unknown, Segment 4 is folders.
-Segments may not exist, but their position is still treated as if they did (benchmark has no segment 3 but you still need to skip it's bytes).
-Each segment is followed by a padding of 0x28 0's, except the first one, which has 4 extra 0s.
+~~~~~~~~~~~~~~

-===FILE SEGMENTS=== (each one is at each folder's [Files Offset], in segment 1) (Each is 16 bytes padded)
-0x000: File ID1 Hash    Int32; Hash to the file name
-0x004: File ID2 Hash    Int32; Hash to the file path
-0x008: File Data Offset Int32; Multiply by 0x08, points to compressed data in .dat file
-0x00b: Padding                 Padded to a total segment entry size of 16 bytes.
+===DATA ENTRY HEADER=== (Minimum size is 0x80) 
+0x000: Header Length     Int32; 
+0x004: Content Type      Int32; 0x01 - Empty Placeholder, 0x02 - Binary, 0x03 - Model, 0x04 - Texture
+0x008: Uncompressed Size Int32;
+0x00c: ~~Unknown~~       Int32;
+0x010: Block Buffer Size Int32; Buffer size need to read largest block
+0x014: Num Blocks        Int32;  

-===FOLDER SEGMENTS=== (seen in segment 3, points to files in segment 1) (Each is 16 bytes padded)
-0x000: FOLDER ID Hash   Int32; Hash to the folder name
-0x004: Files Offset     Int32; Offset to file list in segment 1.
-0x008: Total Files Size Int32; Total size of all file segments for this folder. To find # files, divide by 0x10 (16).
-0x00b: Padding                 Padded to a total segment entry size of 16 bytes.
+===TYPE-2 BLOCK TABLE=== (Size of Num blocks)
+0x000: Offset                  Int32; From end of this header (Check [File Entry Offset] + [Header Length])
+0x004: Block Size              Short; Total Block Size, from beginning of header to end of padding.
+0x006: Decompressed Data Size  Short;

-NOTES:
+===TYPE-3 BLOCK TABLE=== (Size of Num blocks)

-Check the final byte of the offset. If Offset & 0x000F == 2, then the offset is pointing to dat1 rather dat0. Also remember 
-to subtract the 0x2 out. Theoretically, 0x4 would mean dat2 but unknown as of now. 
+0x000: Unknown					           Int32; Always 0x05000001
+0x004: Frame Uncompressed Chunk	        44 Bytes; Each Block Follows Pattern: 5 Ints, 12b of NULL, 3 Ints. Size when uncompressed, but rounded
+0x030: Frame Size Chunk			        44 Bytes; Size of each frame (can be more than 1 data block).
+0x05C: Frame Offset Chunk		        44 Bytes; Offset to start of frame. 
+0x088: Block Size Indexes				   Short; Indexes into the next table below???
+0x0D0: Block Size Table			Num Blocks*Short; Size of each data block below
+			
+===TYPE-4 BLOCK TABLE=== (Size of Num blocks)
+0x000: Frame Offset				Int32; Starting block for this frame.
+0x004: Frame Size				Int32; This is the total size of the whole frame.
+0x008: Unknown				    Int32;
+0x00C: Frame Blocksize Offset  	Int32; Offset starting from the end of the block table, to the first size.
+0x010: Frame Blocksize Count	Int32; How many blocks this frame contains.
+0x000: Frame Block Size         Short; The size of the block in this frame. 
+
+~~~~~~~~~~~~~~
+
+~~~EXTRACTED FILE HEADER MAY BE HERE, PREPEND TO EXTRACT FILE~~~
+
+-----Zlib compressed data starts here------- !!!Used Block table to find each block!!!
+
+===BLOCK HEADER===
+0x000: Header Size         Int32; Seems to be always 0x10bytes
+0x004: NULL                Int32;
+0x008: Compressed Length   Int32; If this is 32000, IT'S NOT COMPRESSED. Use decompressed length to read the data in and just append
+0x00c: Decompressed Length Int32; Will be max 16kb.
+
+0x010: Compressed Data          ; Size will be [Compressed Length]