I find there are too many threads that end with the OP finally resolving the problems and not following up.
You and me both, brother. That's a pet peeve of mine: threads with long discussions of problems, and no resolution posted. But every forum I've seen is plagued by it.
This guy on you tube has a very similar problem with his BZ, his text and "2d" components are great (mine are ok) but his moving "3D" images are really fubar, mine aren't quite as bad but his video pretty well sums up my problems exactly. Looks like he has tried a known good aux board without any results. I can't help but wonder why there are 3 DACs and what each does ( and why did they use 2 different types?).
I'd prefer a video of *your* output (even if you think it looks comparable, minor differences can be important clues) but I'll take what I can get
In general, I'd say issues with 3D objects (but 2D fine) tend to imply the aux board and/or the interconnect to it. You've already re-flowed the header connectors on both ends of the interconnect (IIRC), and you've replaced all of the bit-slice processors on the aux board (also IIRC). This still doesn't put the aux board "in the clear". You said the you tube guy swapped in a known-good aux board... If at all possible, you should try the same. If (like in his case) it doesn't resolve the problem, then of course the issue lies on the main PCB.
OK, to answer your question about the DACs... Jed Margolin was one of the designers of BZ at Atari. He has a website with a good write up explaining vector generators (both the earlier digital type, used in Asteroids etc., and the later analog ones like BZ has). The HTML version is here:
http://www.jmargolin.com/vgens/vgens.htm There's also a PDF version of it on his website somewhere. As it explained by Jed: Two of the DACs (the two that are alike, AM6012s) are the "main" DACs. They are 10-bit, and there is one for the X-axis and another for the Y. The third DAC (the DAC-08) is an 8-bit DAC used for linear scaling. It allows the hardware to draw the same vector 'pattern' at vastly different sizes. If we were to suspect a DAC as having possibly causing your problem, it would be the DAC-08. This is because the 3D object presumably get scaled (due to how close or far they are from you) using this DAC; whereas text and other 2D object may not require scaling.
Another possibility is that it has something to do with vector "normalization", which is done by the AVG. Unfortunately, Jed doesn't discuss this in depth. Looking at the schematics, there is a "/NORM" line generated in the "vector timer control" block, by a 74574 @ J10. It then goes to the "vector generator memory shifter" block, where it is fed through some logic gates (74S00 @ C6) which in turn go to all the shift registers. You may want to look at those guys with a logic probe to see if anything appears "stuck"...
BTW, many of the "hard core" vector-heads are on an old-school mailing list called vectorlist. You can search the archives, and join the list, at vectorlist.org There are some guys on there who have forgotten more about the inner workings of vector games that I'll ever know.