Trouble Recording Inputs on ATmega328P

Hello all! I'm a second year Computer Engineering Student and my summer project is creating a modifiable NES controller using only the chip on the Arduino UNO. I was able to get my controller to interface with the console by connecting the two external interrupts to the NES's LATCH and PULSE lines.

For those who may not be familiar, the NES uses a shift register that holds the inputs from the controller. (I simulated this in the registers of the CPU) 60 times a second, the LATCH lines grabs the current inputs, and outputs the first of 8 bits. Then the PULSE line goes high 8 times, and on each rising-edge the next bit of the output byte is sent to the console until the final pulse signifies the end of the sequence.

What I'm trying to do is add the ability to record, then play back a run of inputs to the game. I have added a button and indicator LED to enter record/playback mode, and that part is working fine. I keep track of time by using one byte to count the number of latches. After latch 255, it resets back to 0 and adds a "NO-INPUT" entry to the recording if there were no input changes for the 0 - 255 cycle. I'm recording inputs by only recording input CHANGES during the LATCH ISR, and the latch count that the change occurred on. This then gives me a time and a mask that I can EX-OR to a starting "NO-INPUT".

The problem I'm having is with the recording, it's very inconsistent. Sometimes every input is recorded, but the lengths of time between changes are very wrong. Sometimes, inputs are missed altogether. If I playback the recording, it executes the same way every time, so I don't think it's a problem with the playback function.

I've tried using the chip's USART and a terminal on my computer to get some data from the chip that I can use to debug, but the multitude of interrupts seems to be messing with both the data being sent to the terminal and the data sent to the NES console.

My two questions are this: What could be causing this issue with (seemingly) recording the data. And what can I do to debug this issue that doesn't require USART?

Here's a GitHub repo with all of my code. I included the USART files I wrote even though they aren't used in the application. Thanks!