Easy way to update Data mod when game updates?

Age of Empires II: DE II - Modding
1

I’m a bit new to modding. I see every time the game updates, it usually updates empires2_x2_p1.dat too. Is there an easy way to update or merge my empires2_x2_p1.dat with the latest?

As for copy pasting, the way I understand it is it only copy one single entry. And if I edit hundred entries and can’t remember which I edited exactly, then it is a big pain. Wonder if there’s a merge tool that detects the difference so to make merging easier.

If such solutions don’t exist, then I would be really surprised that this never been put on the table after all this time as the game updates pretty frequently.

2

This is the biggest challenge for full scale data modding.Someone really needs to figure out and create a tool that can do this easily.

3 Likes
3

It depends a lot on what you have to update infact.

If it’s vanilla unit stat modification, then eventually a batch script that translate your changes, to be saved in some kind of excell sheet structure, to inject into the dat could be usefull. However I think it’s not that easy, and would require some kind of “import-data” function if it’s not already there in AGE.

If it’s some new units added, there you will have no option other than redo it everytime the vanilla adds new ids because your content will be pushed back further and further, meaning not just your dataset must be updated but also all your scenarios which is kinda armageddon at this point.

XS scripting can only be usefull here as map-loaded-script when you’re doing some existing data edition, so it’s probably not the solution you need.

What I would advice you is to store in an external document infos about EVERY changes you made in your datamod, with vanilla id equivalence etc, but you will have to work a lot everytime an update is published to keep your content up to date
The “easiest” solution is to settle your datamod at one specific update version of the game, and do upgrade to the current version time to time, in a dual cycle of develeoppement. Like you start your work at Last Chieftain, and until your mod is not stable, as in usable stage, you don’t swap to the Viking Sagas datas. Then once your mod is stable enought to be upgraded to the latest version, you have to check if a dlc just came out, of if one is soon to come or if you’re inbetween that timeframe. If a dlc just came then wait for the next month patch before doing any upgrading. If a dlc is about to come, either choose to upgrade to the version before this dlc, or wait for the dlc to go live and the following month patch. If you’re more than 1 month before a release and more than 1 month after a release, upgrade as it is.
To do that upgrade you’ll then have to translate from your document all the datas, which will take a LOT of time. It could requiere a team because the more data you’ll have to affect, the more errors can occurs so the more testing you will need.

That’s one reason why now in the DE version I’m avoiding the datamods. When we were in the HD version, we had dlc very time spaced from each other so the datas stayed for longer time.

1 Like
4

Still waiting for a “blank” modifiable data mod. (That hopefully, would not interact with any of the existing data.) I have tried but no luck. Mainly because the game “expects” the normal game data. Even if you try to make a completely new one. Very frustrating to say the least but, I don’t think the devs will ever make one. (At least, not anytime soon. If ever.)

edit: Like, I can easily generate custom/editable string ID’s. I have done it with ease via programming generating simple text/code. Unfortunately, I am not… “skilled enough” to complete the task. (Mainly the data mod part. I have learned some bit and pieces but, not enough to make a custom “advanced genie editor” that I could understand how to generate a program to do so. There is an insane amount of byte data that would take me months/years to understand.)

edit: Even if I did, an update will probably end up breaking it again anyway. :frowning:

5

If you are good with Programming, you can use .dat Parsers to add all your changes to the most current .dat file and run a script that adds all your mod data.

There are primarily 3 .dat Parsers that I know off (these are links):

Genieutils :link: (for C++)
Genieutils-py :link: (for Python)

and the upgraded one (more orgarnised data structure instead of the nonsensical original one, but I use genieutils-py so I cant tell you more)

Genie Dat Parser :link: (also Python)

As I have already said I only use genieutils-py and it serves my needs, and to get started on that one I recommend this example repo (but if you are expierenced, Genie Dat Parser is overall better - I think):
genieutils-examples :link: (using genieutils-py with examples)

Essentially what all of these do is to change the .dat to add your modded and changed values, add units and what not.

I have used this to create my own Mod called “Better Land Warfare” - Promotional Video :link: and from there I always keep the mod updated (using github and git as version control) everytime a new patch comes around. If there is a new DLC its a bit harder (roughly like 2-3 days) till compatibility, but overall if everything goes right, I do everything that I wrote in the readme.md file, run the script and then upload the mod.

So here is my repo :link: - its a bit complex, I am not that good at coding, but there are lots of discords and other modders that can help you out (and if anything is too hard for me, I unfortunately have to use the VSCode AI Chat)

I used genieutils-examples as a very nice guide and then added additonal features on top of it. I also do json editing so my new units show up in the Tech Tree, as well as Tech Tree Preview, future unit UI and more.

Maintaining it, can be some work but if it is something you think is worth it, go for it. Unless the devs change some backend stuff, it is mostly smooth sailing. I will actually announce my newest Update for BLW fairly soon since I added the last new unit and Supplies back (I am procrastinating as I speak).

Since I wanted to give something back to the community I followed the lead of Krakenmeister (creator of genieutils-examples) and added comments to almost everything - but overall this is a step-up to his program.

My repo includes:

  • Editing Tech Tree json files
  • Adding Units and Techs (and effects of course)
  • Adding Tech Tree Info
  • Adding Icons
  • Adding Armor Classes
  • Using preexisting hotkeys for your units
  • Adding projectiles
  • Adding Unit Graphics
  • Adding Sounds
  • Adding Tasks
  • Adding modded strings.txt
  • Altering more json files needed for new units (techtreepreviewpanel, unitcategories.json, unitlines.json, icons.json and materials.json)
  • Some File management (moving files with Python)
  • Extending his helper functions

There is still room for improvement, like automatically grapping the game files instead of me pasting them in after every major update (mostly DLC), and more advanced stuff like auto-upload (everything auto :robot: )

But yeah, thats all - in theory if I stopped updating my mod, someone else could take over and continue my work.

P.S Dont ask me how to set up VsCode or you IDE with Python - I forgot and couldnt remember even if I wanted to.

1 Like
6

Wow I don’t know what to say. Thanks so much for typing this much information! This is so detailed and well explained, much appreciated! It’s a bit more complicated than I expected but at least we seem to have a solution.

I’m less experienced with programming so I can’t say I will get into it, but I may have a look when I got time. If I don’t, hopefully it’s still helpful to the others who pass by.

Maybe one day they will integrate this genieutils thing to AGE3?

7

Well, the dat-to-json command was very useful. Unfortunately, there is still an insane amount of data… I am a bit curious if I run this on my “blank” datamod and see if anything becomes of it. Thanks for that info Deathcounter. :slight_smile:

edit: I found one issue in my datamod. I was missing the “Terrain Table” graphic values. (I fixed those and it still crashes.) After looking though the json a bit, I noticed that there was some old data in there from the base datamod… I noticed that the “Automatic” checkbox somehow got checked on when I was editing it one day… So, it’s safe to assume it got corrupted because of this…

Anyway, I hope the devs can come up with a better data format. (Not a easy task, I know. But, if they can somehow reduce the data to a more realistic editable format, I would be glad to work with it more. But, at that point, AGE would probably be updated to reflect those changes.) Hell, even if you have to separate the data into multiple files would be perfect. (It would make more sense too for all the civs and whatnot. Editing one massive file is a pain. Unless, you are very organized/careful.)

1 Like