Shadow: Dummy folders, an obvious overflow, and a ROP chain

Intro

quick note from oct 25 ava: some of the sections where line breaks are important for legibility do not have the line breaks they should have. not sure why, i'll see if i can fix this when possible. until this note is removed, reading this article may be a little difficult in some places. sorry about that!

The recommended song for this article is Introspective by LAAKE. It accurately represents how fun working on this jailbreak was.

Shadow is my third, and most recent Kindle jailbreak. It's also by far the most complicated Kindle jailbreak to date. The reason the jailbreak is named Shadow is because it was, quite literally, going to be in the shadows for months. Unfortunately, the jailbreak was patched in the 5.20.1 update. Originally, I thought I was going to can the jailbreak, but I decided to release it for two reasons: number one, it's cool; number two, it's always good to have alternatives to working jailbreaks. As of now, Shadow is the second jailbreak to work on firmwares >=5.19.6.

Two final notes: all comments are preserved from the source, unless prefixed with "ava -", and all memory addresses are only from one run of the exploit. Have fun!

Kindle MTP Overview

Alright, so first, what even is MTP? Is it that "Multi-Token Prediction" thing that's apparently important for selfhosting ai or something? Or is it "Medical Termination of Pregnancy"..? Nope, don't think that's it either. OH! It's the Media Transfer Protocol!

Essentially, upon the release of the 2024 Kindles, Amazon switches Kindles from using usb-mass storage transfer (where your computer/phone sees the Kindle as a mass-storage device, similar to a USB stick), to Media Transfer Protocol (MTP). Specifically, it uses the tizen-mtp implementation of MTP. Fortunately for us, this just so happens to be open-source, from Collabora's fork of the Tizen mtp-responder. It can be found here, and will be referenced numerous times throughout this writeup.

Basically, rather than directly interfacing with the files, MTP acts as an intermediary between the file system and the client. It was made with data safety in mind, bla bla, we don't really care. What's important are these concepts:

  • Containers: Every request is sent with a container, which contains all the information about the request being made--container length in bytes, type (command, data, response, event), opcode, transaction ID, and then finally, the payload. When, for example, a file is transferred, there are separate containers for the command, and for data.
  • Objects and handles: All files and folders are called "Objects" by MTP. These objects are then referred to internally by their handles, rather than by their paths. These handles are 32 bits.

Now, here's what we're up against for exploiting MTP into executing arbitrary code:

  • Position Independent Executable (referred to as PIE throughout the rest of this guide) -- This, combined with ASLR (mentioned later) means that the tizen-mtp executable sits at a random point in memory every time it's ran. This is an obstacle that other jailbreaks, such as SpiderCat did not have to face, as the webreader binary sits at a fixed memory address. Essentially, we do not know the locations of the functions in tizen-mtp in memory, until we defeat the ASLR/PIE protections.
  • Address Space Layout Randomization (ASLR) -- Does exactly what it says on the tin, again, this means that we have to leak memory addresses and then derive the location of what we need. We can't hardcode locations of the information we need.
  • tizen-mtp runs as root. This means we do not need a separate privilege escalation exploit to run jb.sh as root, any code ran by tizen-mtp will run as root.
  • tizen-mtp carries it's own system() function. This means we do not need a primitive to read an arbitrary memory address, or even multiple memory addresses. Typically, you would leak the base of tizen-mtp, read the GOT to find an imported libc symbol, then find the libc base, then, finally, find the address of system() in memory. With this, finding the base of tizen-mtp in memory is enough, as we can then just add the offset of system() to get system()'s location in memory.

All in all, the last two things mentioned on their own are terrible security choices, but I'm not a developer. Who am I to judge?

The two vulnerabilities

Two vulnerabilities are chained together in the exploit, and fortunately, both were fairly easy to find on their own.

  1. Uninitialized timestrings

In src/entity/mtp_property.c, some properties for objects are initialized:

mtp_bool _prop_update_property_values_list(mtp_obj_t *obj)
{
	mtp_uint32 ii = 0;
	mtp_char guid[16] = { 0 };
	slist_node_t *node = NULL;
	slist_node_t *next_node = NULL;
	ptp_time_string_t create_tm, modify_tm;
	mtp_uint32 fmt_code = obj->obj_info->obj_fmt;

To anyone who's ever heard of the C programming language, one thing should blatantly stand out: create_tm (time a file was created) and modify_tm are not zero-initialized, unlike all of the other surrounding values. There is essentially no reason to do this, and it is the core of the memory leak bug.

We can get more information about this structure in ptp_datacodes.h, on line 764 and 815-820, respectively.

#define MAX_PTP_TIME_STRING_CHARS		20	/* eg: 20050526T171236 */
[...]
typedef struct {
	/* Num of chars in string including the NUL */
	mtp_uchar num_chars;
	/* Holds actual Unicode string with 2 bytes chars,'\0' terminated*/
	mtp_wchar str[MAX_PTP_TIME_STRING_CHARS]; /* ava - 20 wchars, 40 bytes */
} ptp_time_string_t;

Basically, the structure of the stack slot the struct is put into is as follows, in bytes: [1, Length][Padding][40, string buffer]. This is rounded to 44 bytes by the compiler for alignment purposes.

How this is actually exploited will be explored later, but that is the core bug that allows for the infoleak to happen.

  1. A container overwrites its own loop bound

As mentioned before, command containers are used to send commands to MTP. I hope that by this stage in life, you are familiar with what commands are. Now, let's check how this command container structure is defined in src/ptp_container.h

#define MAX_MTP_PARAMS			5
#define MTP_USB_HEADER_LENGTH		12
[...]
typedef struct {
	mtp_uint32 len;		/* the valid container size in BYTES */
	mtp_uint16 type;	/* Container type */
	mtp_uint16 code;	/* Operation, response or Event code */
	mtp_uint32 tid;		/* host generated transaction id */
	mtp_uint32 params[MAX_MTP_PARAMS];
	mtp_uint32 no_param;
} cmd_container_t;

Note that params are four bytes, so each container is meant to be structured as follows: [Bytes 0x00-0x0B: Header]-[Bytes 0x0C-0x1F: params[0-4]]-[no_param]]

Now, let's look at how this is actually validated in src/ptp_container.c

mtp_bool _hdlr_validate_cmd_container(mtp_uchar *blk, mtp_uint32 size)
{
	if (size < sizeof(header_container_t) || size > sizeof(cmd_container_t))
		return FALSE;

Now, looking at the second part of that if check, we can see something interesting: sizeof(cmd_container_t) is 36 bytes, but the last four bytes are meant to be no_param, not data that we send. What can we do with this?

Well, let's look at _hdlr_copy_cmd_container_unknown_params (the function to stage the container) in the same file:

("12" and "4" filled in by me)

	dst->no_param =
		(src->len - 12) / 4; /* ava - (36-12)/4, so 6 for a 36-byte container */

	for (ii = 0; ii < dst->no_param; ii++) /* ava - bound is re-read, ii = 5 overwrites it */
		dst->params[ii] = src->params[ii];

Basically, this calculation allows for us to insert six parameters. Fortunately for us, param[5] sits at the same offset within the container as no_param, allowing us to overwrite no_param in the for loop to whatever we want, allowing us to copy as many dwords as we want into the stack slot (eg, 0x10 if we want it to copy 16 dwords).

Chaining, exploiting, and complicating

Now, it's time to chain both of these vulnerabilities together to execute OUR code.

The Leak

So, above, we established that there's an uninitialized timestring, with a compiled struct size of 44 bytes. Let's look at how the timestring is actually grabbed, to see if we can get the MTP daemon to return the timestring when it doesn't exist. If we can do this, it will return raw stack, which is the leak we need to chain with the stack buffer overflow bug.

Firstly, we are led to the caller of the function that fills in create_tm in src/entity/mtp_property.c:

_entity_get_file_times(obj, &create_tm, &modify_tm); /* ava - return is ignored here */

This function is then defined in src/entity/mtp_object.c:

	if (FALSE == _util_get_file_attrs(obj->file_path, &attrs)) {
		ERR("_util_get_file_attrs Fail");
		return FALSE;
	}

Again, we go one level deeper, into _util_get_file_attrs_ in src/util/mtp_fs.c

	if (stat(filename, &fileinfo) < 0) {
		ERR_SECURE("%s : stat Fail errno [%d]\n", filename, errno);
		return FALSE;
	}

Now, the path is clear: if stat() fails, we can get raw, uninitialized stack back by asking an object when it was created. To make stat() fail, we have to fulfill two conditions:

  1. MTP believes that the object exists on the file system
  2. The object does not exist on the file system

Doing this is easier than it sounds: you simply send a SendObjectInfo request and then don't follow up with the upload, but we have to open a session first.

We send the following packet:

10 00 00 00 01 00 02 10 00 00 00 00 01 00 00 00

Translated, this is: len = 0x10, command, 0x1002 (OpenSession), tid =0, session = 1.

The daemon responds with:

0C 00 00 00 03 00 01 20 00 00 00 00

Again: len = 0x0C, response, 0x2001 (OK), tid = 0.

After opening the session, we could immediately make a SendObjectInfo request, but first, we must churn the allocator in order to get something useful into the slot that we read when we grab the object creation date.

We need to get something useful into the slot first: we perform a few operations in order to do this: (thanks to deepseek for finding this pattern, because good lord)

  1. Create 9 dummy folders in root (/mnt/base-us) each with 6-character random names:
  • I don't think the character names here are that strict? I was also able to reproduce the same leak with 8-character names too, but going to 1 or 100 would probably fail as well.
  1. Create 3 folders inside of each other, each with a length of 199 characters.
  • (name lengths in brackets) (A - {199})/(B - {199})/(C - {199}).
  1. Create a folder with name length 245 characters, then a folder with name length 200 characters inside of the 245 character folder.
  • (D - {245})/(E - {200})
  1. Create a 4-char folder, create another 4-char folder inside of that folder
  • (4char1 - {4})/(4char2 - {4})
  1. Move folder D (and consequentially, folder E) inside of folder C, such that the tree looks like:
  • (A - {199})/(B - {199})/(C - {199})/(D - {245})/(E - {200})

Then, we can send the SendObjectInfo request:

18 00 00 00 01 00 0C 10 TT TT TT TT 00 00 00 00 FF FF FF FF 00 00 00 00

Translated: len = 0x18, command, 0x100C (SendObjectInfo), tid, store = 0 (use the default store location), parent = root (highest level location in that store), reserved.

We follow up with a data phase packet containing some basic information about the object: storage type, object size, dates/flags/parent, filename, bla bla bla. Not that interesting. The important part is what we receive back from the daemon after sending the data phase packet:

18 00 00 00 03 00 01 20 TT TT TT TT 01 00 01 00 <parent> <HANDLE>

Translated: len = 0x18, response, OK, tid, store = 0x10001, parent, handle.

This handle is stored by Shadow, as it's what we subsequently reference to ask the ghost file for it's creation date. We could ask this file for it's creation date now, but we would still only get garbage. Instead, we send GetObjectHandles calls prior to asking for the creation date. Again, this is to ensure that the right garbage is left behind in the stack slot. The structure of those calls looks like this: GetObjectHandles(root) [all root-level objects], then GetObjectPropValue(ghost, 0xDC08) <- 0xDC08 is the code for creation date GetObjectHandles(4char1) [children of 4char1], then GetObjectPropValue(ghost, 0xDC08) GetObjectHandles(4char2) [handle of 4char2], then GetObjectPropValue(ghost, 0xDC08) GetObjectHandles(all) [ALL objects in the store], then GetObjectPropValue(ghost, 0xDC08)

The raw packet of that final GetObjectPropValue (and all of them) looks like this:

14 00 00 00 01 00 03 98 TT TT TT TT <ghost> 08 DC 00 00

Translated: len = 0x14, command, 0x9803 (GetObjectPropValue), tid, the handle of the ghost file, 0xDC08 (Creation date)

Then, the daemon responds back with the jackpot ('|' added in as a divider):

09 | 00 00 | 00 E0 62 7F | 00 00 00 00 | 08 3A 02 B4 | C0 88 7F B0

Translated: Length of response (9 wchars following the length byte), padding, 0x7F62E000 (bottom of GOT), slot + 8 (blank), 0xB4023A08 (unused), 0xB07F88C0.

Before explaining what all of that means, I want to say that 9 landing there is pure luck (deterministic, I mean that we are lucky it landed there), because it tells the daemon to send us 9 wchars. Were it to be, say, 4, we would be unable to get a sufficient leak. Also, this is why the exploit has a minimum version of 5.19.4: a pointer is left in the first word of the slot, so the low byte used as the number of characters is rejected due to being too high.

Firstly, the GOT is the global offset table. It can be used for exploits, but we do not use it here. All you need to know is that it sits at a fixed address inside the binary, which can be grabbed via readelf (or bin/elfscan in the Shadow repo). Additionally, the GOT is always page aligned (at a multiple of 4096 bytes) within memory. In the response above, we got 0x7F62E000--Shadow checks that this value is page aligned by doing an & operation with 0xFFF and ensuring that it's zero. Once that's done, we can subtract the fixed address of the GOT within the binary from the page-aligned address leaked, to get the PIE base of the daemon.

Simply put: PIE = pieWord - GOTbase PIE = 0x7F62E000 - 0x2B000 PIE = 0x7F603000

As for 0xB4023A08, it's a heap pointer. We don't really care about it. What we DO care about is 0xB07F88C0, which is a pointer on the stack. We need to use this pointer to figure out the location of the receiver frame (the stack frame of the function that receives the command packet). Again, this is done with simple subtraction: R - Receiver frame, P - pointer R = P - 0xCD8 R = 0xB07F88C0 - 0xCD8 R = 0xB07F7BE8

You are almost surely (rightfully!) asking "How did you get 0xCD8? Where did that come from?" Simply put, it was measured from the internals of the program (vibeslopped emulator). The way it works is as follows:

  1. Run the binary offline via qemu-arm, use the device libc and simulated FunctionFS transport (how tizen-mtp interfaces with the linux file system)
  2. Capture a leak hit and take the last four bytes of the output (0xB07F88C0 as shown above)
  3. Put a breakpoint (stopping point in the program) in gdb at the copy function mentioned before (the site of the overflow, _hdlr_copy_cmd_container_unknown_params) and read sp. Because the stack grows downward, this is the lowest memory address, so it is the base of this stack frame.
  4. Compute: delta = P - R, for KT6, this always comes out to 0xCD8.

Finally, we have enough information to exploit the stack buffer overflow.

The Overflow

Again, fortunately, this is much simpler. We already have all the information we need, now we just need to exploit it in the right way in order to gain code execution.

Welllll, actually, there is one more piece of information needed: the source of the copy for hdlr_copy_cmd_container. Essentially, this comes from a 1KB buffer which is used, freed and recycled for each MTP command sent. Once this buffer is used, it's recycled through the tcache, the next read gets the other buffer chunk back. There are two of these 1KB buffers, and they are recycled for each command.

Now, the phase variable is what determines how an incoming packet is processed. We need the phase to be IDLE (1) and have a valid container, otherwise no copy will happen. The phases work like this:

  • phase 1 + invalid container > phase 0, no copy
  • phase 0 + any container > phase 1, no copy
  • phase 1 + valid container > copy, this is what we need to exploit the stack buffer overflow

Since we've only sent valid containers so far (stays in phase 1), we send three filler packets. This goes as follows: phase 1 + invalid container > phase 0 + invalid container > phase 1 + invalid container > phase 0. The reason we send three containers instead of one is because we need BOTH buffers to have our payload, otherwise there would only be a 50% chance we land on the buffer container, AND because we need to be on phase 1 for the copy to happen successfully.

Now, the packets themselves:

  1. The filler

We make this packet invalid by giving it an invalid length (0x400/1024), but these bytes are still read into that RX buffer. The offset within this container maps to the offset within the stack frame according to the formula: S + 0x14 + o, so

  • 0x48 lands on the saved r4 at S+0x5C
  • 0x4C on saved r7
  • 0x50 on the LR
  • 0x74 at S+0x88

The packet is as follows:

0x48: 00 00 00 00 ; saved r4=0
0x4C: 3C 7C 7F B0 ; saved r7 = S+0x54
0x50: B1 7B 60 7F ; saved LR = PIE + 0x4BB1 = 0x7F607BB1
0x54: 00 00 00 00 ; r1
0x58: 01 2A 62 7F ; ava - r3 = PIE + 0x1FA01 = 0x7F622A01
0x5C: 00 00 00 00 ; r4
0x60: 00 00 00 00 ; r5
0x64: 00 00 00 00 ; r6
0x68: 03 DD 61 7F ; pc = PIE + 0x1AD03 = 0x7F61DD03
0x6C: 70 7C 7F B0 ; [r7 + 44] = string addr = 0xB07F7C70
0x74: "RUN_MODE=1 /bin/sh /mnt/base-us/jb.sh" 
  1. Phase flip

We need to flip the phase from 0 to 1 for the copy to occur, we can do this with a basic packet. Not much interesting about it.

24 00 00 00 01 00 01 10 71 (00 until offset 32) 40 00 00 00

Translated: len=0x24, command, 0x1001 (GetDeviceInfo), tid=0x71 (arbitrary tid)

  1. Overflow!

Finally, after a long, perilous journey, we can trigger the overflow, which will give us our ROP chain, which will give us code execution. Let's look at the packet:

24 00 00 00 01 00 01 10 78 00 00 00 (00 until offset 32) 24 00 00 00

This copies 36 dwords (0x24) from source offset 12 onto the stack, the locations are derived via the formula laid out in #1.

Assemblyland!

Well, what did the overflow actually do? How does it give us code execution? Was Ava lying to us and just using her Véra exploit in the background the whole time? Of course! Just kidding, of course not. Essentially, the function that receives the MTP packet is _receive_mq_data_cb. Its C code isn't important, so I won't link to it on github. What IS important is the assembly instructions for its epilogue, look at this:

8a28: adds r7, #0x54
8a2a: mov  sp, r7
8a2c: pop  {r4, r7, pc}

This does two things for us: firstly, our copy function then overwrote the saved LR due to our overflow-inducing packet, so the saved LR is now our pop gadget address (PIE + 0x4BB1). Secondly, it restores the r7 to R+0x54 (you'll see why this is used in a second). This causes the program to jump to that instruction:

pop {r1, r3, r4, r5, r6, pc}

Essentially, this acts as a checkpoint where we can load our call target r3 (the system() in the binary, mentioned long, long ago). Then, it lets us jump to another instruction by popping pc (jumps to PIE + 0x1AD03), that instruction is:

ldr r0, [r7, #44]; blx r3

This sets r0 to be [r7 + 44], which itself is a pointer to our string (0xB07F7C70), it then branches to r3 (PIE + 0x1FA01, system()). Since r0 is the first argument, it's essentially called like: system([r7 + 44]), which ends up being system(RUN_MODE=1 /bin/sh /mnt/base-us/jb.sh).

Conclusion

Well, that's it. system() being called with RUN_MODE=1 /bin/sh /mnt/base-us/jb.sh runs the jailbreak. Now, if you're into the KindleModding community, you may think "wait, don't jailbreaks usually curl jb.sh?" and the answer is yes! Shadow physically cannot do that, because Wi-Fi is disabled while MTP is active. Instead, Shadow stores jb.sh on-device, and then has the user replug in order to get a fresh daemon for the exploit to run. If jb.sh is transferred within the same session, then the memory layout will not be suitable for the infoleak to occur.

Lastly, I'd like to thank a person in the KindleModding server for researching MTP back in January. He was the reason I learned about the stack buffer overflow, as well as the reason I started looking into this jailbreak in the first place. I did not tell him I was researching MTP until I finished my chain, so I did not use any code from him. Without knowing that there even were vulnerabilities in MTP, I would've thought MTP was a dead end.

I'm very glad that this jailbreak is able to see the light of day, I'll look back on it fondly.

chunk diffing, shell scripts, and pain

Intro

A recommended song for reading this article is Forgotten Friend by Mammal Hands, as it's what I listened to during a large portion of the events described.

I play a Minecraft modpack with a close friend of mine, Shea. The modpack in specific is Better MC 5 NeoForge, which is on the Minecraft version 1.21.1. We started playing this modpack together approximately 5 months ago, around Christmas time, and have played it frequently since. When we started, the modpack version was v45(.0). Also, the server we played on was on a VPS located in ██████.

Over time, we would gradually see the version number on an update alert increase, 46, 47, 48... Eventually, Shea and I agreed that I would update the modpack once version 50 was released—I was hesitant to do so due to a fear of potential issues arising from updating the modpack, as there is and was no built in server-side updating mechanism (though updating the clients is like, four clicks and forty seconds in prism launcher).

Around a week ago, I had nothing to do while on a call with Shea, and I decided to finally update the server, as version 50 had released. My worries would be proven valid.

Updating

Updating itself wasn't that difficult, nor scary, as I take backups every 72 hours using Backrest (though I took another simple backup of the modpack folder with the copy command, just in case). I simply shut down the server, downloaded the most recent version with wget, unzipped, copied over the mods and config folders from the newest version, and turned on the server. There were no problems in the slightest, except for one catastrophic issue:

Every single pale oak thing—a type of block used in almost every single one of our builds—in our world was gone. What possibly could have gone wrong?

Naturally, I looked at the logs, and found out why almost immediately:

ResourceKey[minecraft:root / minecraft:block]: vanillabackport:pale_oak_trapdoor -> using default);

This was repeated for every pale oak thing in the world, only one log message is shown here for brevity.

Essentially, because Vanilla Backport no longer had the pale oak blocks, and the pale oak blocks were all under the vanillabackport namespace, the game could not find the pale oak blocks, so they were all transformed into air. To this day, I'm not sure why this happened: the modpack was still on 1.21.1, and pale oak trees were only added in a 1.21.2 snapshot, but somehow the pale oak trees were moved to the minecraft namespace. All I could find was this blurb in the v48 changelog for BMC5:

Added all content from Minecraft versions 1.21.2 - 1.21.5, alongside hundreds of bug fixes and improvements

How were we playing with pale oak planks on v45???

Well, now what? I still had a backup of the world at this point, but I did not have much time to figure out what may have gone wrong. It was around 10pm by this point.

Research

Firstly, I had a vague idea of what to do: somehow, modify the chunk files to bring the blocks back to normal. I couldn't get any coordinates from the log files, so I had to somehow get the coordinates of the pale oak blocks from the old chunk (mca) files, and then somehow put them into the most recent version of the server.

Thankfully, most of this was documented fairly well on the minecraft wiki. I started by looking at Anvil File Format and Region File Format, as well as finding NbtExplorer around this time. Thankfully, finding the chunk files in question for the specific areas of the world was not hard in the slightest, because the F3 menu tells the exact position of the current chunk within a certain file.

NBTExplorer, the program of all time

After using NBTexplorer for some time, I realized that it would be essentially useless, as most blocks do not have any NBT data associated with them, as NBT data is used for storing, well data. This data may include orientation, content (such as on a sign), et cetera. Simple blocks just don't have anything to store.

Following this, I went down a deep, deep rabbit hole searching for other potential programs for editing individual blocks (I still didn't know how I was going to extract the coordinates of the missing blocks). Sadly, most world editing software is paid, not available on linux, or simply for outdated versions of Minecraft. I did find something that appeared to be what I was looking for (though I have no idea if this would have actually worked): MCA2Nbt, but this software is seriously outdated, and only works on FreeBSD systems—you know you're in some deep shit when you stumble upon software only for FreeBSD—I was not in the mood for editing C code, so I moved on.

xkcd 1350

Since I hit a dead end with software, I decided to ask the server for the modpack, to which I was ignored once, and then told the Minecraft equivalent of "Just take tylenol, I'm sure your chest pain is nothing." Then, I contacted the developer of the Vanilla Backport mod, and received a hopeful response:

From here, I simply decided to wait it out. As of writing this on May 30th, no patch has been released, nor have there been any pertinent commits on the Vanilla Backport github, and I was ignored upon asking for an update in the Vanilla Backport discord server.

A few days later, in Jason's anatomy class, I had an idea that would go on to form the basis for solving this problem.

Diffing

The diff command, in my opinion, is one of the most powerful linux commands. I'd already used it a couple times in the past for various tasks, but I had a real-world use for it here. Firstly, I went into minecraft and grabbed the file name of a chunk in which I was certain pale oak blocks were turned into air: r.1.5.mca. I copied this file from both the pre-update world and post-update hellscape to my primary machine, and ran the diff command. Obviously, I was not expecting straight text stating "(x,y,z) was sent to the cognitive realm!", but I only got gibberish. I didn't have too much time that day, so I decided to try again another day.

In my deep software rabbit hole, I found a python library which would go on to be crucial in this journey: anvil-parser2. I originally brushed it off as being useless, but around this time, I realized the role it could play in diffing the relevant mca files. I vibecoded a quick python script for diffing (specifically, checking if a block was filled in the old file, but turned to air in the new file), let it run until my room was hot, and it just hung indefinitely (realistically, I should've added verbose output here, but that ship has sailed). Subsequently, I decide to try again another day.

Around two days later, I realize how bad the original script was: it ran a read_block check on every single block in every single chunk. Each chunk contains 16 * 384 * 16 (98304) blocks, each mca file contains 32^2 or 1024 chunks, which means 201,326,592 (~100m multiplied by two because we're diffing two MCA files) get_block calls were made for one run of diffing. That alone would take ages, as the language in question is python, but that doesn't even include the actual comparisons being made. There was no way that script would ever have finished in any reasonable amount of time. Optimizations had to be made.

For reference, whenever I write "first (mca) file", I am referring to the old MCA file, with the pale oak blocks—whenever I write "second (mca) file", I am referring to the newer MCA file, without the pale oak blocks.

Firstly, NVME SSD read speeds are blazing fast: around 35 gigabytes per second in my case. We can simply read the raw chunk data (without any python parsing) and compare it to that of the same chunk in the second MCA file, if the two are identical, we can move along. This alone saves an incredible amount of time. Next, we can skip any blocks that are air in the first file (cutting off ~50% of comparisons), as we know that there's not any pertinent changes in the second file.

After this part, most of the optimizations were written by LLMs, but I understand them regardless. First, each section (16x16x16 cubes that form chunks) is transformed from 3d into a 1d 4096 number array, where we can decode if a certain block is filled in the first file, and then decode the same block in the second file, checking if it's air as we do so. Decoding is fairly simple, as a palette is located at the start of the file, assigning certain blocks to certain codes for compression purposes. Additionally, math is kept as simple and straightforward as possible here.

An example of a line of output from the script:

1008 65 3183: pale_oak_planks -> air

The full script can be found on pastebin.

Now, with this god-tier script, I was able to diff 1024 chunks in ~20 seconds on my Ryzen 5 7600X. This gave me an output of altered blocks for the file r.1.5.mca, but I was not close to being done yet.

Shell Scripts

This was by far the most boring section to go through. Firstly, I needed to somehow find all files with differences between the old and the new version. Because Shea and I hadn't played at all after the update, these differences would presumably only be in relation to the disappearance of the pale oak blocks. This was actually very easy to do, as I was just able to run

diff regions/region_old/ regions/region_new/ > diff_files.txt

This command gave an output of

Binary files regions/region_old/r.-1.-2.mca and regions/region_new/r.-1.-2.mca differ
Binary files regions/region_old/r.1.5.mca and regions/region_new/r.1.5.mca differ
Binary files regions/region_old/r.1.6.mca and regions/region_new/r.1.6.mca differ
Binary files regions/region_old/r.2.5.mca and regions/region_new/r.2.5.mca differ
Binary files regions/region_old/r.2.6.mca and regions/region_new/r.2.6.mca differ

For some reason, r.2.* and the first file listed did not have any blocks that were transformed to air, I'm honestly not sure why they had differences, but I'm not really concerned about that.

After running the diffing python script on all of the files, I concatenated them all into a master.txt by using the following:

cat x.txt >> compiled.txt

From here, I needed to somehow turn these outputs into setblock commands, and then somehow parse those commands into the server. I wrote a simple shell script to turn these outputs into setblock commands, using compiled.txt. The shell script is reproduced below:

#!/bin/bash

FILE=$1

declare -a blocks
declare -a coords

addblocks() {
    while IFS= read -r line; do
        blocks+=("$line")
    done < <(
        sed 's| *->.*||' "$FILE" \
        | tr -cd '[:alpha:]\n[=_=]' \
        | sed 's|^|minecraft:|'
    )
}

addcoords() {
    while IFS= read -r line; do
    coords+=("$line")
    done < <(
        tr -cd '0-9[:space:]\n' < "$FILE"
    )
}


formcommands() {
    for i in "${!coords[@]}"; do
    echo "setblock ${coords[i]} ${blocks[i]}"
    done > commands.txt
}

addblocks
addcoords
formcommands

An example line from commands.txt:

setblock 412 67 184 minecraft:pale_oak_planks replace

Slight tangent, for some reason, 3 spaces are always there before minecraft, and I'm not smart enough to figure out why. I just made a simple sed command to fix this issue and moved on:

sed 's/.\{3\}\(minecraft\)/\1/g' commands.txt > commands_fixed.txt

Now, I needed some way to pipe these commands into the Minecraft server. Thankfully, with a quick search, I was able to find the RCON protocol, which essentially acts as a sort of API for sending commands to a Minecraft server. After a few quick changes to server.properties, I enabled rcon. I downloaded ARRCON as a client to use for this project, as it was the first result online.

Then, I wrote a quick shell script to use commands.txt as an input for rcon:

#!/usr/bin/env bash
set -euo pipefail

HOST=localhost
PORT=25575
PASS="do you think i am stupid?"
COMMAND_FILE=$1

while IFS= read -r command || [[ -n "$command" ]]; do
  if [[ -z "$command" || "$command" == \#* ]]; then
    continue
  fi
  echo "Sending: $command"
  arrcon -H "$HOST" -P "$PORT" -p "$PASS" "$command"
done < "$COMMAND_FILE"

Finally, it was time to try this. I thought that there was absolutely no way this could fail, that the script was so simple it could not break, right?

That position is not loaded.

Okay, great, awesome. Thankfully, there is a forceload command which should be able to fix this issue pretty easily. I wrote another quick shell script to load chunks, it didn't work, I vibecoded a shell script to load chunks, it didn't work, I almost went insane, until I realized:

412 67 184

Those coordinates are not even remotely close to where the pale oak blocks are actually located (around 1000, 67, 3000). This was caused by an error near the end of diffs.py, where Claude didn't use the correct formulas (reproduced below), for some reason:

region_x * 512 + cx * 16 + x = world_x region_z * 512 + cz * 16 + x = world_z

In which, cx and cz are the chunk x and z coordinates. Y coordinate does not matter, as chunks only have the coordinates (x, z), and the mca files store the region x and z in their file names.

Sadly, I don't have the original formulas used, so you can't laugh at them.

Now, I had to run the diffing process again for r.1.5.mca and r.1.6.mca in order to fix these coordinates (I probably could've written a third shell script to modify commands.txt accordingly, but someone else can do that). I ran the command formation shell script again, and then finally, I was ready to do another attempt at running the setblock commands.

After a handful of small tweaks to the send_commands.sh file (which I sadly don't remember, nor do I want to remember), it finally worked! (With one large issue)

Here, you can see both my happiness and disappointment unfold simultaneously:

Now, it is apparent that all the stairs were rotated the wrong way, but I honestly didn't care at this point. I was just happy that all of our builds were back. There's still a few other weird issues: slabs are only in the top position, some slabs are missing, there's a cluster of like 10 blocks that I can't get back no matter what, but I could not force myself to go back down the NBT rabbit hole to fix those issues. A simple wrench and a little bit of teamwork in the game will allow us to fix those issues easily.

Conclusion

In conclusion, the developers of BMC5 are... interesting, to say the least. I don't want them in jail for this issue, since I know how hard developing a modpack can be, but this is gross negligence at best. I am shocked that Shea and I were the only two people to ever encounter this error in the modpack. If anyone can explain how this occurred, please let me know, and I will be incredibly grateful.

Additionally, let this be a call to all modpack developers to simply add an update mechanism within the server pack for the pack. It's honestly shocking that out of all the modpacks I've played, I have never found even ONE with this feature.

In the end, I am not a developer, but I do my best to think like one when I have to.