Abstract: We investigate the near-critical dynamics of atomic density fluctuations in the non-equilibrium self-organization transition of an optically driven quantum gas coupled to a single mode of a cavity. In this system cavity-mediated long-range interactions between atoms, tunable by the drive strength, lead to softening of an excitation mode recently observed in experiments. This phenomenon has previously been studied within a two-mode approximation for the collective motional degrees of freedom of the atomic condensate which results in an effective open-system Dicke model. Here, including the full spectrum of atomic modes we find a finite lifetime for a roton-like mode in the Bogoliubov excitation spectrum that is strongly pump-dependent. The corresponding decay rate and critical exponents for the phase-transition are calculated explaining the non-monotonic pump-dependent atomic damping rate observed in recent experiments. We compute the near-critical behavior of the intra-cavity field fluctuations, that has been previously shown to be enhanced with respect to the equilibrium Dicke model in a two-mode approximation. We highlight the role of the finite size of the system in the suppression of it below the expectations of the open Dicke model.