It must have been a big factor. I have been checking release dates and two things are apparent.
1) Home ports were not a factor. Pac-Man for the 2600 wasn't even released until March 1982. The Colecovision wasn't strong even then, it was released in August and would get half a million units sold in Christmas, meaning it still couldn't have had any sizable impact given the huge size of the arcade industry. The PC boom didn't exist yet and the intellivision didn't have many
2) The genres in 1982 were limited but given the success of many titles, I wonder if it's really a factor. We all know that Q*Bert, Joust, Dig Dug, Pole Position and Robotron sold well. That said, I doubt any of those titles could be compared to the huge success that Pac Man and Defender were. If there was a fad, it ended here.
So unemployment must have played a big part. The market was also oversatured by that point as I heard many places saw arcades as a quick buck. I read Ottumwa had 4 arcades, that looks like overkill for such a small place.