Big clean up of structure and contents

.gitignore

@@ -0,0 +1 @@
+.idea

README.md

@@ -2,13 +2,21 @@
 
 This repository is to serve as a place to share data mining information related to Final Fantasy XIV. All findings and knowledge that each member discovers will be provided here for all in the FFXIV Community to read, learn and use.
 
-Huge thanks to the community team members involved:
+# Documents of Knowledge
 
-- [Ioncannon](https://www.reddit.com/user/Ioncannon) from [FFXIV Explorer](http://ffxivexplorer.fragmenterworks.com)
+- [Click here to view Documents on various FFXIV systems](docs/README.md)
-- [Clorifex](https://www.reddit.com/user/Clorifex) from [GarlandTools](http://www.garlandtools.org/db/)
-- [Hezkezl](https://www.reddit.com/user/Hezkezl) for being awesome
 
-## Updating CSV files:
+# Research and misc files
+
+- [Click here to view various research and resource files](research/README.md)
+
+# CSVs
+
+This repository keeps a record of CSV's extracted via SaintCoinach using the command `rawexd`, this is so they can be easily linked and referenced when datamining, so we have a git history of changes and just to make life easier in some cases :)
+
+- [Click here to view CSV files](csv/)
+
+### Updating CSV files:
 
 - Download and open **SaintCoinach.cmd**
 - Run: `rawexd`
@@ -19,8 +27,29 @@ The idea is to maintain an easy diffing view of what has changed during a patch
 
 It would also be very useful to keep a history of `ex.json` files from SaintCoinach for each patch as the Korean and Chinese versions are on different patches than the Official live client.
 
+### whats the "ex" folder?
 
-## Helping out
+Ex contains an archive list of the [SaintCoinach ex.json](https://github.com/ufx/SaintCoinach/blob/master/SaintCoinach/ex.json) file, ths main point was to keep a `ex.json` file per Patch, then if you wanted to extract data from an older patch (such as Korean or Chinese clients) you can hot-swap the `ex.json` file and extract data in the correct format.
+
+This kind of hot swap does not always work however as new structures are detected that are implemented into SaintCoinach, or you may have systems that expect data in a specific way (eg the great "Tooltip Descriptions Changes" of Patch 4.4)** ...
+
+You can just grab an old version from git history, which I may switch to linking to instead of keeping copies.
+
+
+### whats the "uld" folder?
+
+It's an extract of FFXIV UI Elements (images) extracted from SaintCoinach.
+
+---
+
+### Huge thanks to:
+
+- [Ioncannon](https://www.reddit.com/user/Ioncannon) from [FFXIV Explorer](http://ffxivexplorer.fragmenterworks.com)
+- [Clorifex](https://www.reddit.com/user/Clorifex) from [GarlandTools](http://www.garlandtools.org/db/)
+- [Hezkezl](https://www.reddit.com/user/Hezkezl) for being awesome
+
+
+### Becoming a dataminer
 
 Getting started in datamining is quite easy. If you enjoy digging around CSV database files then you've come to the right place! The best place to start would be:
 
@@ -33,30 +62,3 @@ If you find a connection between files or an identification of data, throw up an
 - Connection between files could be an number in 1 column that matches the numbers in another column or even another file
 - Identification of data could be values in a column that have a meaning and a pattern can be observed (eg ClassJob ID, or Craft Level)
 
-# The tools of the trade
-
-### Saint Coinach
-
-- Written by: Clorifex and Rogueadyn
-- GitHub: https://github.com/ufx/SaintCoinach
-- Downloads: https://github.com/ufx/SaintCoinach/releases
-	- SaintCoinach.Cmd is a tool to rip data (csv, music, icons, maps, etc)
-	- Godbert is a *viewer* tool to browse data.
-
-A great tool showing really impressive results, it will allow you to view and export a lot of data. The source is all open for your luxery.
-
-
----
-
-
-**Other cool projects**
-
-- REST API: http://xivapi.com/
-- (PHP) Lodestone Parser: https://github.com/viion/lodestone-php
-- (PHP CLI) Libra exporter: https://github.com/viion/libra-php
-- (PHP CLI) Lodestone Icon Ripper: https://github.com/viion/lodestone-icons
-- (PHP CLI) SaintCoinach CSV Parser: https://github.com/viion/saint-csv-parser
-- (C#) Memory Reader: https://github.com/Icehunter/sharlayan
-- (NodeJS) Dead Lodestone Parser: https://github.com/viion/lodestone-nodejs
-
-

docs/AchievementsRequirementTypes.md

@@ -1,4 +1,8 @@
-Achievements link to a "pre-requisite type" that defines it as completed. The list of Types are:
+# Achievements Requirement Types
+
+Achievements link to a "pre-requisite type" that defines it as completed.
+
+**This is a known list of types as of: 3rd October, 2018**
 
 ```
 0: complete specific Legacy thing

docs/CollectibleRewardInfo.md

@@ -1,3 +1,5 @@
+# Collectible Reward Info
+
 MasterpieceSupplyDuty is where all of the Collectable information is kept. Unknown how scrips are calculated for base value,
 but Bonus / High Bonus is calculated very easily, for both EXP and Scrips.
 
@@ -30,7 +32,7 @@ rewards across all classes for the exact same level (assuming an item isn't in d
 you can divide it by 1.2 and round up to get the info!) Gathering collectables I'd presume share a similar EXP systme, but I
 haven't mapped out those values yet.
 
-Level    = EXP 
+Level    = EXP
 ------------------
 Level 51 = 52920
 Level 52 = 63360

docs/CompanionAppApi.md

@@ -1,4 +1,5 @@
-# FFXIV Companion App
+# Companion App API
+
 - https://eu.finalfantasyxiv.com/companion/
 
 The FINAL FANTASY XIV companion app has the ability to query the game servers for a specific resource to pull information (market info, retainer items, player items, chat messages, etc). This document will list research and provide logic for obtaining data out of the app as well as quering the companion API.

docs/EmoteLogMessage.md

@@ -1,3 +1,9 @@
+# Emote Log Messages
+
+Attempting to debug the emote LogMessages
+
+---
+
 100	Standard Emotes	1	0	False
 
 

docs/FateExpPoints.md

@@ -1,3 +1,5 @@
+# FATE Experience Points
+
 Fate EXP is still a mystery, research so far:
 
 Equation: ` fate offset 22 * paramgrow offset 8 + paramgrow offset 8 = exp`

docs/FateLevels.md

@@ -1,3 +1,5 @@
+# FATE Levels
+
 The sync level for fates is always +5 of the fate level.
 
 ## Test Data

docs/ItemStats.md

@@ -1,3 +1,5 @@
+# Item Stats
+
 Most of these are kind of obvious, but will add here for safe keeping!
 
 # Attack speed

docs/MapCoordinates.md

@@ -1,8 +1,24 @@
 # Conversions to 2D Map
 
-*Code samples in C#*
 
-Conversion of a Level coordinate into a 2D map coordinate:
+### Version of an in-game coordinate to a 2D map coordinate
+
+**By: Clorifex, slightly modified by: Vekien - In C#**
+https://github.com/xivapi/xivapi-mappy/blob/master/Mappy/Helpers/MapHelper.cs
+
+```csharp
+public static double ConvertCoordinatesIntoMapPosition(double scale, double offset, double val)
+{
+    val = Math.Round(val, 3);
+    val = (val + offset) * scale;
+    return ((41.0 / scale) * ((val + 1024.0) / 2048.0)) + 1;
+}
+```
+
+### Conversion of a Level coordinate into a 2D map coordinate:
+
+**By: Clorifex - In C#**
+
 ```csharp
 private double ToMapCoordinate(double val) {
 	var c = Map.SizeFactor / 100.0;
@@ -12,7 +28,10 @@ private double ToMapCoordinate(double val) {
 }
 ```
 
-Conversion of a FishingSpot coordinate into a 2D map coordinate:
+### Conversion of a FishingSpot coordinate into a 2D map coordinate:
+
+**By: Clorifex - In C#**
+
 ```csharp
 private double ToMapCoordinate(double val) {
 	var c = TerritoryType.Map.SizeFactor / 100.0;
@@ -21,51 +40,20 @@ private double ToMapCoordinate(double val) {
 }
 ```
 
-# Conversions to Long/Lat (eg Leaflets/Google Maps)
+### Conversion of MapPosition to Pixels
 
-*Code samples in PHP*
+**By: Vekien - In C#**
+https://github.com/xivapi/xivapi-mappy/blob/master/Mappy/Helpers/MapHelper.cs
 
-You can translate the `levels.exd` values to Long/Lat using the `x`, `y` and **scale** would be the maps [size_factor](https://github.com/viion/XIV-Datamining/blob/master/offsets/3.1_list.txt#L679)
+```csharp
-
+public static int ConvertMapPositionToPixels(double value, double scale)
-```php
-//
-// Used due to spherical maps, world is round irl :P
-//
-function mapsRadiansToDegrees($rad)
 {
-    return $rad / (pi() / 180);
+    return Convert.ToInt32((value - 1) * 50 * scale);
-}
-
-//
-// This code will take the X, Y and map scale from the levels CSV and convert it into a LAT/LONG 
-// which can be used on leaflet or Google maps or just about any map tool
-//
-public function mapsTranslateXYZtoLatLong($x, $y, $scale = 100)
-{
-    if ($x == 0 || $y == 0 || $scale == 0) {
-        return false;
-    }
-    
-    $scale = $scale / 100;
-    
-    $tilesize = 2048 / $scale;
-    
-    $pixelsPerLonDegree = $tilesize / 360;
-    $pixelsPerLonRadian = $tilesize / (2 * pi());
-    
-    $lng = $x / $pixelsPerLonDegree;
-    $latRadians = $y / -$pixelsPerLonRadian;
-    $lat = $this->mapsRadiansToDegrees(2 * atan(exp($latRadians)) - pi() / 2);
-    
-    $coords = [
-        'x' => $lng,
-        'y' => $lat,
-    ];
-    
-    return $coords;
 }
 ```
 
+----
+
 To translate a `levels.exd` position to an in-game X/Y:
 
 ```php

docs/MapScales.md

@@ -0,0 +1,10 @@
+# Maps and various scales
+
+> Do not remember what purpose this had...
+
+ID: 31 = 100 = Lower La Noscea = 41.9
+ID: 52 = 200 = New Gridania = 21.4
+ID: 58 = 300 = The Tam-Tara Deepcroft = 14.6
+ID: 359 = 400 = The Navel = 8.6
+ID: 733 = 800 = Mizzenmast Inn = 6.0
+ID: 2200 = 95 = Coerthas Western Highlands = 44

docs/QuestExpReward.md

@@ -1,4 +1,7 @@
-Quest experience points are generated using a small equation. 
+# Quest Exp Reward
+
+Quest experience points are generated using a small equation.
+
 Why this is done I am not sure as the values always seem static.
 
 ### Formula updated: 28th May, 2018

docs/README.md

@@ -0,0 +1,66 @@
+# FFXIV Data-Mining
+
+- Research
+  - [Achievements Requirement Types](docs/AchievementsRequirementTypes.md)
+  - [Collectable Reward Info](docs/CollectibleRewardInfo.md)
+  - [CompanionApp API](docs/CompanionAppApi.md)
+  - [Emote Log Messages](docs/EmoteLogMessage.md)
+  - [FATE Experience Points](docs/FateExpPoints.md)
+  - [FATE Levels](docs/FateLevels.md)
+  - [Icon Paths](docs/IconPaths.md)
+  - [Item Actions](docs/ItemActions.md)
+  - [Item Stats](docs/ItemStats.md)
+  - [LeveToNpc](docs/LeveToNpc.md)
+  - [MapCoordinates](docs/MapCoordinates.md)
+  - [MapScales](docs/MapScales.md)
+  - [QuestExpReward](docs/QuestExpReward.md)
+  - [UIColor](docs/UIColor.md)
+  - [Weather](docs/Weather.md)
+
+- Tools of the trade
+  - [SaintCoinach](https://github.com/ufx/SaintCoinach) - The #1 datamining tool.
+  - [TextTools2](https://github.com/liinko/FFXIV_TexTools2) - Extract data, import game mods.
+  - [Sapphire Server Emulation](https://github.com/SapphireMordred/Sapphire) - FFXIV Server Emulation
+  - [Machina](https://github.com/ravahn/machina) - Network Packet Reader
+  - [Sharlayan](https://github.com/FFXIVAPP/sharlayan) - In-Game memory reader.
+  - [FFXIV MemoryReader](https://github.com/TamanegiMage/FFXIV_MemoryReader) - In-Game memory reader.
+  - [XIVAPI.com Source Code](https://github.com/xivapi/xivapi.com)
+  - [LibPomPom](https://github.com/Minoost/libpompom-sharp) - C# code that authenticates with the FFXIV Companion App.
+  - [3D Web Model Viewer](https://github.com/viion/ffxiv-3d)
+  - [Sapphire Ubuntu Setup Script](https://github.com/viion-misc/ffxiv-sapphire-setup) - a script to install Sapphire on ubuntu.
+  - [Custom FFXIV Launcher](https://github.com/xivapi/ffxiv-launcher)
+  - [FFXIV Mapping Tool](https://github.com/xivapi/xivapi-mappy) - Map Eorzea by grabbing position data of NPCs, Enemies and EObjs.
+  - [Lodestone Parser](https://github.com/xivapi/lodestone-parser) - PHP library that parses lodestone.
+  - [ClassJob Icons](https://github.com/xivapi/xivapi-classjobs) - A collection of HQ job class icons.
+  - [FFXIV Icons](http://xivapi.com/docs/Icons) - Download all sorts of FFXIV related icons.
+  - [XIVAPI Quick Search](https://chrome.google.com/webstore/detail/xivapi-quick-search/lgefpgdbbmcahllbifniibndmoignmfg) - Search XIVAPI via a chrome extension!
+  - [REST API](http://xivapi.com/) - A little REST API that has a few bits and bobs.
+
+- Archives
+  - [Extract of every FFXIV Patch since 2.0](https://github.com/viion/ffxiv-datamining-patches)
+
+- Dead
+  - [Data Tool](https://github.com/viion-misc/xivdb-data-tool) - A python library that extracts game data.
+  - [Libra PHP](https://github.com/viion-misc/libra-php) - PHP library that extracts data from the FFXIV Libra app.
+
+- Discords:
+  - [XIVAPI](https://discord.gg/MFFVHWC)
+  - [GarlandTools](https://discord.gg/)
+  - [TeamCraft](https://discord.gg/493Z6pQ)
+  - [Sapphire](https://discord.gg/xxcdCER)
+  - [Apkallu Falls](https://discord.gg/RNcV4eY)
+
+Do you have a discord for your FFXIV project? Provide a PR with your invite link added!
+
+---
+
+# Community Members
+
+Notable members that have contributed information in some way, If you're part of the datamining and FFXIV discovery community be sure to add your credit!
+
+- Vekien (xivapi)
+- Clorifex (GarlandTools)
+- Hezkezl (GamerEscape)
+- Icarus Twine (GamerEscape & Blue Mage overlord)
+- Miu (TeamCraft)
+- Ioncannon (FFXIV Explorer)

docs/UIColor.md

@@ -1,3 +1,12 @@
+# UI Color
+
+### Update 3rd October, 2018
+
+It is believed that:
+- 72 = Foreground
+- 73 = Glow (not background)
+
+
 In Patch 4.4, SE changed how colours work in tooltips in preperation of new UI "Skins". The logic for displaying colours changed quite drastically but was figured out quickly, here is the process and how it works now:
 
 ## (ID: 174) Bane tooltip

docs/Weather.md

@@ -1,6 +1,11 @@
+# Weather
+
 The current weather of a zone is based on the current unix epoch, and a zone weather table (WeatherRate).
+
 A forecast target integer is first calculated.  This value to the frequency of the zone's weather.
+
 When the number is less than the cumulative rate at which a weather pattern occurs, that weather will appear in the zone.
+
 For example, in Gridania it's raining 20% of the time, and foggy 10%. It rains when the number is < 20, and foggy when it's >= 20 and < 30, and so on.
 
 ````js
@@ -29,4 +34,4 @@ calculateForecastTarget: function(lDate) {
     // 0x64 = 100
     return step2 % 100;
 }
-````
+````

research/README.md

@@ -1,18 +1,6 @@
 # Research
 
-This folder contains various research on the FFXIV database structure, file structure and various formulas and the craziness that comes with FFXIV :)
+Random stuff, research, useful files, etc
-
-### Achievements
-
-- Requirement Types: https://github.com/viion/ffxiv-datamining/blob/master/research/achievements_requirement_type.md
-
-### Collectables
-
-- https://github.com/viion/ffxiv-datamining/blob/master/research/collectable_reward_info.txt
-
-### Emotes
-
-- https://github.com/viion/ffxiv-datamining/blob/master/research/emotes.md
 
 ### DevNPCs
 
@@ -20,42 +8,7 @@ This really needs maintaining, there is a bunch of NPC's which are "dev" specifi
 
 - https://github.com/viion/ffxiv-datamining/blob/master/research/devnpcs.csv
 
-
+### File structure research
-### F.A.T.Es
-
-- EXP: https://github.com/viion/ffxiv-datamining/blob/master/research/fate_exp.md
-- Levels: https://github.com/viion/ffxiv-datamining/blob/master/research/fate_levels.md
-
-### Icons
-
-- https://github.com/viion/ffxiv-datamining/blob/master/research/icon_paths.md
-
-### Items
-
-- Stats: https://github.com/viion/ffxiv-datamining/blob/master/research/item_stats.md
-
-### Leves
-
-- Leve to NPC: https://github.com/viion/ffxiv-datamining/blob/master/research/leve_to_npc.md
-
-### Maps
-
-- Scales: https://github.com/viion/ffxiv-datamining/blob/master/research/map_scales.md
-- Coordinate conversion: https://github.com/viion/ffxiv-datamining/blob/master/research/map_coordinates.md
-
-### Quests
-
-- EXP Formula: https://github.com/viion/ffxiv-datamining/blob/master/research/quest_exp_reward.md
-
-### Weather
-
-- https://github.com/viion/ffxiv-datamining/blob/master/research/weather.md
-
-
----
-
-
-# File structure research
 
 **Huge thank you to FFXIV Explorer Ioncannon for providing these**
 

research/map_scales.md

@@ -1,11 +0,0 @@
-31 = 100 = Lower La Noscea = 41.9
-
-52 = 200 = New Gridania = 21.4
-
-58 = 300 = The Tam-Tara Deepcroft = 14.6
-
-359 = 400 = The Navel = 8.6
-
-733 = 800 = Mizzenmast Inn = 6.0
-
-2200 = 95 = Coerthas Western Highlands = 44