Abstract: The ideal superconductor provides a pristine environment for the delicate states of a quantum computer: because there is an energy gap to excitations, there are no spurious modes with which the qubits can interact, causing irreversible decay of the quantum state. As a practical matter, however, there exists a high density of excitations out of the superconducting ground state even at ultralow temperature; these are known as quasiparticles. Observed quasiparticle densities are of order 1~$\mu$m$^{-3}$, tens of orders of magnitude greater than the equilibrium density expected from theory. Nonequilibrium quasiparticles extract energy from the qubit mode and can induce dephasing. Here we show that a dominant mechanism for quasiparticle poisoning is direct absorption of high-energy photons at the qubit junction. We use a Josephson junction-based photon source to controllably dose qubit circuits with millimeter-wave radiation, and we use an interferometric quantum gate sequence to reconstruct the charge parity of the qubit. We find that the structure of the qubit itself acts as a resonant antenna for millimeter-wave radiation, providing an efficient path for photons to generate quasiparticles. A deep understanding of this physics will pave the way to realization of next-generation superconducting qubits that are robust against quasiparticle poisoning. The single flux quantum (SFQ) digital superconducting logic family has been proposed for the scalable control of next-generation superconducting qubit arrays. In the initial implementation, SFQ-based gate fidelity was limited by quasiparticle (QP) poisoning induced by the dissipative on-chip SFQ driver circuit. In this work, we introduce a multi-chip module architecture to suppress phonon-mediated QP poisoning. Here, the SFQ elements and qubits are fabricated on separate chips that are joined with In bump bonds. We use interleaved randomized benchmarking to characterize the fidelity of SFQ-based gates, and we demonstrate an error per Clifford gate of 1.2(1)%, an order-of-magnitude reduction over the gate error achieved in the initial realization of SFQ-based qubit control. We use purity benchmarking to quantify the contribution of incoherent error at 0.96(2)%; we attribute this error to photon-mediated QP poisoning mediated by the resonant mm-wave antenna modes of the qubit and SFQ-qubit coupler. We anticipate that a straightforward redesign of the SFQ driver circuit to limit the bandwidth of the SFQ pulses will eliminate this source of infidelity, allowing SFQ-based gates with fidelity approaching theoretical limits, namely 99.9% for resonant sequences and 99.99% for more complex pulse sequences involving variable pulse-to-pulse separation.