Staredit Network > Forums > Modding Discussion > Topic: Expanding DAT files
Expanding DAT files
May 22 2016, 6:00 am
By: O)FaRTy1billion[MM]  

May 22 2016, 6:00 am O)FaRTy1billion[MM] Post #1

👻 👾 👽 💪

GPTP Integration! 🥳

GPTP DatExtend Repo

PyMS: https://github.com/poiuyqwert/PyMS
Right click list in PyDAT and add entries.

More details in this post: http://www.staredit.net/377086/



Previous post:

Update:
Uploaded a modified DatCC so you can decompile dat files to ini and recompile them to have more entries with a -n command-line option. Also updated the plugin, at clokr's request, to have sprites.dat have every value for every entry. No real progress on the plugin itself, but at least now it is usable! :wob:

* Fixed DatCC so that images.dat's blank entry won't crash when loading SC. It's not a huge issue, unless you don't use the images you add.




Here is a demo of the dat extender plugin. The plugin lets you specify how many entries are in each dat file. It just creates some new memory and replaces all references to the existing dat arrays to the new memory, and replaces a few other things to make it work.

The only dats I have working currently are units.dat, flingy.dat, sprites.dat and images.dat.

There are some thing about Units.dat, though:
- Every property in the extended units.dat is available to every entry. DatEdit won't show them except for the ones they are normally limited to.
- The flingy.dat ID (graphics) is normally 1-byte, so there is a patch that expands it to 2 bytes so you can have more than 256 flingies. This probably should just always be done, but right now it's an option.
- Due to some of SC's code that reads several entries at a time (for some reason), units.dat must be a multiple of 12.
- IDs 228 to 250 are used for special cases (as seen in FireGraft), so instead of finding and changing all of them these IDs have been reserved as special use and should not be used.
- Because of how FireGraft works, I had to copy the button tables to some unused memory (unit count tables, kill count table, and death table) which gives a maximum limit of 912 units. I could probably find new memory if this is ever a problem, but I don't imagine it will be necessary.

Some of the other dats I haven't done yet might have similar things, but I haven't looked in to them fully yet.


Using the plugin currently is a bit awkward. There's not really an easy way to add entries to a dat file (if only there were a dat editor that would let you add entries...). DatEdit is pretty easy to change to read an expanded dat file, other than the previously mentioned problem with units.dat, by just changing the .def files, add entries to the data lists and add them to the origins lists. FireGraft is a bit tricky though. Increasing the unit counts and adding to the unit lists is easy enough (found in fgd file and unitlist.txt in the data.mpq), except FG reads data directly from the EXE so I had to make a modified EXE with the data copied to the corresponding location so it wouldn't break.

Once you actually have the .dat files, the plugin is pretty straightforward. Just add it to MPQDraft or FireGraft and hit Configure and it will let you enter the new entry counts (or save/load them from a config file). I would have just automatically counted them from the filesizes, except the plugin runs before the MPQs are loaded.. I could probably add a bunch of code in to SC to do it, but this is easier. :P

I attached a demo I've been using to test. It has 252 units (2 usable ones), +10 flingies, +1 sprite, and +1 image. The two new units are buildable at a supply depot. They're just random units I created to be usable and test it out.
If anybody wants, either just play around with it in games to see if anything crashes or if anyone can think of places unit counts or IDs are that I missed. One obvious one visible is the extended units don't have names. Other ones is I haven't modified triggers in any way to read them. Or tell me which files I should focus on next. :P

Attachments:
DatExtender Demo2.7z
Hits: 8 Size: 2745.97kb
DatExtend.qdp
Hits: 8 Size: 52kb
DatCCb.7z
Hits: 4 Size: 271.17kb

Post has been edited 12 time(s), last time on Oct 26 2022, 6:03 am by FaRTy1billion.



TinyMap2 - Latest in map compression! ( 7/09/14 - New build! )
EUD Action Enabler - Lightweight EUD/EPD support! (ChaosLauncher/MPQDraft support!)
EUDDB - topic - Help out by adding your EUDs! Or Submit reference files in the References tab!
MapSketch - New image->map generator!
EUDTrig - topic - Quickly and easily convert offsets to EUDs! (extended players supported)
SC2 Map Texture Mask Importer/Exporter - Edit texture placement in an image editor!
\:farty\: This page has been viewed [img]http://farty1billion.dyndns.org/Clicky.php?img.gif[/img] times!

May 22 2016, 9:07 am Pr0nogo Post #2



You're a real mother fucker, Farty. Thanks a ton. Keep us all updated on the improved usability and such.




May 22 2016, 4:00 pm Neiv Post #3



Played with it a bit, interceptors aren't working properly. (Or was it just due to this test mod?)
One array whose size depends on unit entry count and hasn't been handled is the ai strength array at 006BB210. Could that be redirected next? =)

Edit: Also 006902D8 for ai build limits.

Post has been edited 1 time(s), last time on May 22 2016, 4:10 pm by Neiv.



None.

May 22 2016, 5:38 pm KYSXD Post #4



What!

Hope something like this could be done to increase the iscript entries.

Nice work!




May 22 2016, 7:25 pm BeatMeistro Post #5



Now this is getting interesting.

Let's see what we can do with this bad boy.



None.

May 23 2016, 2:17 am O)FaRTy1billion[MM] Post #6

👻 👾 👽 💪

Quote from KYSXD
Hope something like this could be done to increase the iscript entries.
oshit, that might be necessary. :lol:

Quote from Neiv
Played with it a bit, interceptors aren't working properly. (Or was it just due to this test mod?)
One array whose size depends on unit entry count and hasn't been handled is the ai strength array at 006BB210. Could that be redirected next? =)

Edit: Also 006902D8 for ai build limits.
I'll look at interceptors, they may just be some random thing I missed. And thanks for the addresses! I'll look at those.



TinyMap2 - Latest in map compression! ( 7/09/14 - New build! )
EUD Action Enabler - Lightweight EUD/EPD support! (ChaosLauncher/MPQDraft support!)
EUDDB - topic - Help out by adding your EUDs! Or Submit reference files in the References tab!
MapSketch - New image->map generator!
EUDTrig - topic - Quickly and easily convert offsets to EUDs! (extended players supported)
SC2 Map Texture Mask Importer/Exporter - Edit texture placement in an image editor!
\:farty\: This page has been viewed [img]http://farty1billion.dyndns.org/Clicky.php?img.gif[/img] times!

May 26 2016, 5:20 am O)FaRTy1billion[MM] Post #7

👻 👾 👽 💪

Quote from Neiv
One array whose size depends on unit entry count and hasn't been handled is the ai strength array at 006BB210. Could that be redirected next? =)

Edit: Also 006902D8 for ai build limits.
Do you know how big these are, or their formats? I see their indexes multiplied by weird numbers, so I don't know what they mean. 006BB210 seems to make sense (int thing[228]), except somewhere the index is multiplied by 912 (sizeof(int) * 228) which makes me think there is more after it, or more than one table here? 006902D8 is [something * 1256 + something + 006902D8], and I don't know what 1256 means.

EDIT:
After looking at 0x006BB210 + 912 (which shares similar functions to 0x006BB210) it looks like there's just the two arrays, especially since 0x006BB210 + 912*2 is a single value. I still don't know what 1256 is.

Post has been edited 2 time(s), last time on May 26 2016, 5:35 am by FaRTy1billion.



TinyMap2 - Latest in map compression! ( 7/09/14 - New build! )
EUD Action Enabler - Lightweight EUD/EPD support! (ChaosLauncher/MPQDraft support!)
EUDDB - topic - Help out by adding your EUDs! Or Submit reference files in the References tab!
MapSketch - New image->map generator!
EUDTrig - topic - Quickly and easily convert offsets to EUDs! (extended players supported)
SC2 Map Texture Mask Importer/Exporter - Edit texture placement in an image editor!
\:farty\: This page has been viewed [img]http://farty1billion.dyndns.org/Clicky.php?img.gif[/img] times!

May 26 2016, 5:40 am Neiv Post #8



Oh, right. 006BB210 should be 2 * 228 32-bit ints, one for unit's strength vs. air and one for vs. ground.

006902D8 is a part of a larger structure. Guess it causes complications?
This structure starts at 0068FEE8, has 8 entries (1 for each player), and is 0x4e8 bytes large.
(So the unit build limit array is at offset 0x3f0 from the struct's beginning)



None.

May 26 2016, 5:47 am O)FaRTy1billion[MM] Post #9

👻 👾 👽 💪

Quote from Neiv
006902D8 is a part of a larger structure. Guess it causes complications?
This structure starts at 0068FEE8, has 8 entries (1 for each player), and is 0x4e8 bytes large.
(So the unit build limit array is at offset 0x3f0 from the struct's beginning)
So there's 157 entries per player? Or each entry is 0x4E8? I'm just not sure how 0x4E8 relates to the unit count, or how much I need to add to it.

EDIT:
So would it be 0x4E8 for each player, and then each unit has 1 byte, so I'd just add as many bytes as units added * 8, and then just adjust the size of the struct (the all of the *1256)? It looks like if this is true then there are 20 bytes at the end, but that shouldn't be a problem because those unit IDs shouldn't be used.



TinyMap2 - Latest in map compression! ( 7/09/14 - New build! )
EUD Action Enabler - Lightweight EUD/EPD support! (ChaosLauncher/MPQDraft support!)
EUDDB - topic - Help out by adding your EUDs! Or Submit reference files in the References tab!
MapSketch - New image->map generator!
EUDTrig - topic - Quickly and easily convert offsets to EUDs! (extended players supported)
SC2 Map Texture Mask Importer/Exporter - Edit texture placement in an image editor!
\:farty\: This page has been viewed [img]http://farty1billion.dyndns.org/Clicky.php?img.gif[/img] times!

May 26 2016, 5:58 am Neiv Post #10



There is a structure which is would be something like this:
Code
struct PlayerAi
{
   uint8_t other_ai_things[0x3f0];
   uint8_t build_limits[0xe4];
   uint8_t another_ai_thing[0x4];
};


That makes it total 0x4e8 bytes per struct, and there are 8 of them.

I don't think you should resize the entire struct, but figure a way to get the build limits to a separate location? Or if the entire struct is resized, you'd have to redirect everything that accesses something else from that struct.



None.

May 26 2016, 6:10 am O)FaRTy1billion[MM] Post #11

👻 👾 👽 💪

Ok, that's what I (eventually) thought. :P

Whichever works. The only problem with moving the part of it is if something references it from the struct's base address, such as when zeroing it. But that shouldn't be difficult.



TinyMap2 - Latest in map compression! ( 7/09/14 - New build! )
EUD Action Enabler - Lightweight EUD/EPD support! (ChaosLauncher/MPQDraft support!)
EUDDB - topic - Help out by adding your EUDs! Or Submit reference files in the References tab!
MapSketch - New image->map generator!
EUDTrig - topic - Quickly and easily convert offsets to EUDs! (extended players supported)
SC2 Map Texture Mask Importer/Exporter - Edit texture placement in an image editor!
\:farty\: This page has been viewed [img]http://farty1billion.dyndns.org/Clicky.php?img.gif[/img] times!

May 26 2016, 6:20 am Neiv Post #12



Quote from O)FaRTy1billion[MM]
Whichever works. The only problem with moving the part of it is if something references it from the struct's base address, such as when zeroing it. But that shouldn't be difficult.

Hm, good point. Zeroing should be simple, but the struct does get saved, so moving it partially requires more fixes in the function which saves it.



None.

May 26 2016, 6:29 am O)FaRTy1billion[MM] Post #13

👻 👾 👽 💪

It shouldn't be too hard to just resize and move the whole struct. All of the copy tables in the plugin code are pretty much designed exactly for that purpose. :P

EDIT:
I only found 487 references to the table, so it shouldn't be that bad. :P


EDIT2:
Ok, I think got those two tables. I don't really know how to test it (the game doesn't crash in the 30 seconds I tried it, and the AI was doing stuff).

There were a bunch of constants that might be significant that I kept seeing but didn't know what to do with, so I ignored them.
Ones I wrote down some addresses to were 0x2710 (remarkably similar to 0x4E8*8, which is 0x2740) and 0x26A0 (which is (0x4E8 - 20) * 8).


I didn't look in to the interceptor/scarab thing (other than it appears that they require a second order in order to acknowledge the order -- attacking/whatever once will launch them, but you must attack again for the launched interceptors to attack). I'll either do that next, or work on more .dat files. Hopefully I have some time this weekend. :P

Attachments:
test.exe
Hits: 4 Size: 634.55kb

Post has been edited 3 time(s), last time on Jun 4 2016, 11:50 pm by FaRTy1billion.



TinyMap2 - Latest in map compression! ( 7/09/14 - New build! )
EUD Action Enabler - Lightweight EUD/EPD support! (ChaosLauncher/MPQDraft support!)
EUDDB - topic - Help out by adding your EUDs! Or Submit reference files in the References tab!
MapSketch - New image->map generator!
EUDTrig - topic - Quickly and easily convert offsets to EUDs! (extended players supported)
SC2 Map Texture Mask Importer/Exporter - Edit texture placement in an image editor!
\:farty\: This page has been viewed [img]http://farty1billion.dyndns.org/Clicky.php?img.gif[/img] times!

May 27 2016, 7:42 am BeatMeistro Post #14



That's Great to know, Also as a suggestion, i guess add more units with their own grps? They might maybe share iscript entries for some of them? Just to start testing sprite data?

Post has been edited 1 time(s), last time on May 27 2016, 7:50 am by BeatMeistro.



None.

May 27 2016, 3:54 pm O)FaRTy1billion[MM] Post #15

👻 👾 👽 💪

I haven't added GRPs, but you should just be able to add lines to images.tbl. I tested sprites/images by making the added marine hallucinated. :P



TinyMap2 - Latest in map compression! ( 7/09/14 - New build! )
EUD Action Enabler - Lightweight EUD/EPD support! (ChaosLauncher/MPQDraft support!)
EUDDB - topic - Help out by adding your EUDs! Or Submit reference files in the References tab!
MapSketch - New image->map generator!
EUDTrig - topic - Quickly and easily convert offsets to EUDs! (extended players supported)
SC2 Map Texture Mask Importer/Exporter - Edit texture placement in an image editor!
\:farty\: This page has been viewed [img]http://farty1billion.dyndns.org/Clicky.php?img.gif[/img] times!

May 27 2016, 5:34 pm Neiv Post #16



Yeah, adding entries to images.tbl has always worked, as has adding new iscript.bin entries. You'll just have to type the numbers by hand in iamges.dat, or modify the dat editors's default data files. And if the .dat editors don't find .grps for their previews, they spit out annoying error messages (but work properly anyways)



None.

May 31 2016, 12:33 pm skff1234 Post #17



WOW! Very cool!!! By the way, It is compatible with GPTP ?



None.

Jun 1 2016, 6:28 am O)FaRTy1billion[MM] Post #18

👻 👾 👽 💪

Quote from skff1234
WOW! Very cool!!! By the way, It is compatible with GPTP ?
Probably not as it is. I was looking at GPTP to see if maybe I could make a custom header or something to make it work, and it is possible but I'll have to play with it later.



TinyMap2 - Latest in map compression! ( 7/09/14 - New build! )
EUD Action Enabler - Lightweight EUD/EPD support! (ChaosLauncher/MPQDraft support!)
EUDDB - topic - Help out by adding your EUDs! Or Submit reference files in the References tab!
MapSketch - New image->map generator!
EUDTrig - topic - Quickly and easily convert offsets to EUDs! (extended players supported)
SC2 Map Texture Mask Importer/Exporter - Edit texture placement in an image editor!
\:farty\: This page has been viewed [img]http://farty1billion.dyndns.org/Clicky.php?img.gif[/img] times!

Jun 4 2016, 11:51 pm O)FaRTy1billion[MM] Post #19

👻 👾 👽 💪

Uploaded a modified DatCC so you can decompile dat files to ini and recompile them to have more entries with a -n command-line option. Also updated the plugin, at clokr's request, to have sprites.dat have every value for every entry. No real progress on the plugin itself, but at least now it is usable! :wob:

DatCC download and stuff is in the first post.



An Example for expanding units.dat:

Code
datcc -d -u -n 264 .
Decompile default units.dat extended to 264 entries.

Code
datcc -c -u -n 264 -e "units.ini"
Re-compile units.dat to 264 entries and use 2-byte Flingy IDs.

See the readme or --help for the other dat files.

Note that the file loaded entries, the base dat file entries, the decompile -n value, and the compile -n value can all be totally different. Any extra entries are ignored, any missing entries are filled in with blank entries.


Also tell me if there are any bugs with the datcc. I didn't really test it. :P

Post has been edited 5 time(s), last time on Jun 5 2016, 12:02 am by FaRTy1billion.



TinyMap2 - Latest in map compression! ( 7/09/14 - New build! )
EUD Action Enabler - Lightweight EUD/EPD support! (ChaosLauncher/MPQDraft support!)
EUDDB - topic - Help out by adding your EUDs! Or Submit reference files in the References tab!
MapSketch - New image->map generator!
EUDTrig - topic - Quickly and easily convert offsets to EUDs! (extended players supported)
SC2 Map Texture Mask Importer/Exporter - Edit texture placement in an image editor!
\:farty\: This page has been viewed [img]http://farty1billion.dyndns.org/Clicky.php?img.gif[/img] times!

Jan 3 2017, 12:32 am Wertyu Post #20



I tried but maybe my mind its a bit little to understand; how exactly it works? I execute the DatCC.exe file and it opens and instantly closes. Perhaps the problem are my poor abilities programming or is there something wrong with another topic about the program?



"Colonel Tom Kazansky was the best Wraith pilot in the Terran Dominion"

Emperor Arcturus Mengsk :horde:

Options
  Back to forum
Please log in to reply to this topic or to report it.
Members in this topic: None.
[12:30 am]
ClansAreForGays -- When you join a pub lobby because you see 7/8 players, but then realize host is bating you with computers. :flamer: :flamer:
[11:48 pm]
O)FaRTy1billion[MM] -- :wob:
[2024-10-30. : 6:24 pm]
Ultraviolet -- :wob:
[2024-10-29. : 4:33 pm]
Vrael -- :wob:
[2024-10-29. : 1:32 pm]
Zoan -- :wob:
[2024-10-28. : 5:21 pm]
Ultraviolet -- :wob:
[2024-10-27. : 4:34 pm]
jjf28 -- :wob:
[2024-10-27. : 9:01 am]
Zycorax -- :wob:
[2024-10-27. : 3:31 am]
RIVE -- :wob:
[2024-10-26. : 7:12 pm]
Ultraviolet -- :wob:
Please log in to shout.


Members Online: Roy