Final Fantasy Wiki
Register
Advertisement
FFWiki forum logo
Forums: Index > Rin's Travel Agency > Archive > Final Fantasy XIV database



Live for the module!

With the help of JBed, Techno, and others, I've been datamining FFXIV for items. With ~13k items, it would be utterly impractical to collect information by hand, but I've developed a much faster system. This process involves taking *.csv files (spreadsheets) from FFXIV Explorer and turning them into Lua tables on the wiki through a Python script.

But Catuse, you would be wise to ask, why not just generate wikitables from the spreadsheets? It would be easier for newcomers to edit. Well, there's a number of reasons not to. They're easier to update for new patches and the formatting of all of them can be changed in one fell swoop. But the best reason is that you can pull up these tables from anywhere. So while List of Final Fantasy XIV weapons/Arcanist will have a row for the T9 SCH arm, that same data set can be put in an article about T9 or Second Coil, or a general article about Allagan equipment, or even this forum page, like so:

NameTypeiLvDescription
EquipPatch
MaterialsCrafting
High Allagan Grimoire of HealingScholar's Arm115

Auto Attack: 56.53
Damage: 53
Delay: 3.2
Magic Damage: 77
Vitality: 47
Mind: 45
Piety: 24
Direct Hit Rate: 9
critical hit: 46

SCH 502.2
High Allagan Grimoire of CastingArcanist's Grimoire115

Auto Attack: 56.53
Damage: 53
Delay: 3.2
Magic Damage: 77
Vitality: 47
Intelligence: 45
critical hit: 46
Spell Speed: 32

ACN SMN 502.2

The item database has been uploaded to the wiki, but implementation is far from complete. Things to fix before this can be rolled out into mainspace, in order of urgency:

  1. While the data has been uploaded, it hasn't been implemented into the module itself, which currently only loads SCoB gear. This fix won't take long, I could set it up in under an hour, but I want to wait until... - Medicines and Meals has a few kinks to work out, but for the most part all items work as intended.
  2. ...we decide on appropriate table formatting. As you can see above, the current table is hideous, and there's something of a trade-off between "more information" and "more visual appeal", especially because FFXIV has so much useless information attached to its items. (SCoB gear prices don't matter!) The tables also need to be formatted to work on mobile. I would think the best way to do this is to have different formats for Arms, Tools, Materials, etc. to better show the information useful to that item type. Still, I'd like to see other people's table styles before deciding on a final format. - I'm more or less satisfied with the current format. If someone wants to change it up, you're welcome to go right ahead.
  3. Update for 3.21 (or, hell, if the other problems take more time to clear up, 3.3x). The most recent version of FFXIV I can datamine is 3.10, so someone with a current subscription to FFXIV and who knows how to use FFXIV Explorer to generate CSVs will have to help here. - thanks SCM!
  4. Assigning anchors and uploading icon images (see below). Can be done after implementation if we don't mind a lot of broken file links where we don't have icon images. - anchors can be added later if we so choose.
  5. Not urgent, but there are a number of indices, such as the "Desynthesis ID" (dsyn = 7666 for T9 SCH arm) that I cannot make sense of. I included these mystery values in the Lua table on the wiki, so that if a dataminer can figure out how to interpret them at a later date, we can easily implement them. As it is, they can be ignored.

I also hope to expand the database to include Quests (~2000), enemies (a couple thousand), Crafting (who knows?) and so on. But let's get Items working as a proof-of-concept before attempting anything crazy.

Wiki designers and gnomes should probably be aware of the technical restraints of this system. First, the user-unfriendly formatting and the fact that items are divided into multiple modules. <textareas> become difficult to load after a couple hundred KB, and MediaWiki does not allow submissions greater than 2 MB. By getting rid of whitespace and taking terse variable names, the file sizes of the database are greatly reduced, but since the raw CSV was still 6 MB long, that's still not enough and it has to be broken into multiple files. (Notice that the Armor table is still nearly 1 MB).

The other problem is that the datamining pulls the numbers (and strings and bools) from the game itself, but not the wiki. Therefore, any links for {{LA}} are not accounted for, and the script can only guess at the correct icon file (File:FFXIV [ItemName] Icon.png, which works with most of our current icon images, but not all). There is a way to override this, with the "page" and "img" attributes for each item. For example, if an item had the icon file "Gobbiedoom.png" and parent page "Gobtank", then in the database, one would add

img="Gobbiedoom.png",page="Gobtank",

to the end of its row, before the end " } ".

But with 13k items and overrides having to be added in manually, this process will take a while. We also don't have the icon files for the vast majority of items, afaik. I'll see if I can pull up the ones we don't have from FFXIV Explorer or Lodestone, but that's a long term project.

But even so, I think that this is the best way to handle the massive flood of data from FFXIV, at least until Wikia comes out with their own database that we keep hearing is coming Soon™. Cat (meowhunt) 06:08, March 12, 2016 (UTC)

I honestly feel the table looks fine, I realise I'm not contributing much by saying this but yeah, having a database for everything we can honestly cover in XIV is a great effort that we need to do. Hexed 06:23, March 12, 2016 (UTC)
If you've got a guide on how to create the CSVs somewhere I can handle that for you. -- Some Color Mage ~ (Talk) 06:26, March 12, 2016 (UTC)
Click on the file you want to export (in this case, sqpack/ffxiv/0a0000.win32.index/exd/item_0_en.exd) and press Ctrl + E. This also works for entire folders, like sqpack/ffxiv/0a0000.win32.index/exd/, but it takes a really long time. Cat (meowhunt) 06:32, March 12, 2016 (UTC)
Got it. -- Some Color Mage ~ (Talk) 06:42, March 12, 2016 (UTC)

Great job on the database for items so far. When you get to dungeons, just make sure to implement databases in the infobox and add item data to the loot tables. And when you get to enemy databases, I don't know what coverage of enemies is at the moment, but Adonzo seems to be taking care of that.--Magicite-ffvi-ios Technobliterator TC 12:32, March 12, 2016 (UTC)

Intangir Bot is currently uploading 9000 or so of the icons SCM extracted (in actuality, there's over 30k, but only some of them are relevant to this project). Then I'll work out some of the kinks and we can see about getting Items rolled out into mainspace later this week. Cat (meowhunt) 21:57, March 14, 2016 (UTC)

Catch me fixing redlinks on Weapons pages tomorrow. Then it's time to delve into the other item types. I'm picking up most bugs as I go along, but if you catch one that I didn't fix, please let me know here. Cat (meowhunt) 00:02, March 16, 2016 (UTC)

Advertisement