In a realistic model of magneto-elastic oscillations in magnetars, we find that crustal shear oscillations, often invoked as an explanation of quasi-periodic oscillations (QPOs) seen after giant flares in soft gamma-ray repeaters (SGRs), are damped by resonant absorption on timescales of at most 0.2s, for a lower limit on the dipole magnetic field strength of $5\times 10^{13}$G. At higher magnetic field strengths (typical in magnetars) the damping timescale is even shorter, as anticipated by earlier toy-models. We have investigated a range of equations of state and masses and if magnetars are dominated by a dipole magnetic field, our findings exclude torsional shear oscillations of the crust from explaining the observed low-frequency QPOs. In contrast, we find that the Alfv\'en QPO model is a viable explanation of observed QPOs, if the dipole magnetic field strength exceeds a minimum strength of about several times $10^{14}$G to $10^{15}$G. Then, Alfv\'en QPOs are no longer confined to the fluid core, but completely dominate in the crust region and have a maximum amplitude at the surface of the star.