Looking at this again, it seems to be able to read from the 4 Background ROM's correctly, but it's getting the wrong data sometimes.
I'm thinking it's whatever drives A13 on the EPROM's for banking, but the schematic is completely missing whatever drives that circuit (BKA13).
If you follow it...