I have had structual damage to one wall were all my subs have sit over the past 8 years, the house was brand new when I moved in and was built by a quality builder. I think older houses are constructed a little bit better but if it was me, build the big sub and fix it later.
I guess it would depend on what your setup is. It's definitely possible if you take it to the extreme. Experimenting with test tones I've broken crystal glasses, had the suspended ceiling tiles fall down, and made nails pop out of wall board. Watching movies has never caused an issue.