I have an Onkyo TX-NR809 and suffer an oddity in relation to the sub out but not quite the same as yours.
I noticed that when playing the Onkyo's subwoofer level tone or the Audyssey set up tone there seems to be noise introduced into the signal. This is with the frequency contour on the sub set to bypass. If I back off the frequency control until it clicks, so now it is at 120dB the noise goes away. Another symptom with the frequency contour on bypass is that on a very few films the sub misbehaves, the driver making a whack whack noise but with the contour at 120dB this does not happen.
At this point I have no idea whether the frequency control on the sub is to blame or the line output on the Onkyo. Making an assumption that maybe the line output on the Onkyo is too high I did the following. (Not sure about this but the 809 being THX certified the line out has to be a certain level which can be higher than some subs like. My sub is not THX)
Setting the subs output to 75dB using Audyssey the level setting in the Onkyo is about 0dB. Setting the level with Audyssey to 82dB the setting in the Onkyo is about -6dB so the line out signal is 6dB lower. This improves things considerably. The final thing I did was to set the subs frequency contour to 80dB. the rationale for this is that THX subs have as an option a fixed THX 80dB crossover. This has stopped any films misbehaving even the dreaded 'Edge of Tomorrow' which I can play at all levels up to reference level without the subwoofer driver throwing a wobbly.
My sub is a BK Monolith Plus. 'BK Subwoofers Direct' is a UK internet direct company.
http://www.bkelec.com/HiFi/Sub_Woofers/monolith-FF_Plus.htm
I have no idea if any of this will be any help to you.
I noticed that when playing the Onkyo's subwoofer level tone or the Audyssey set up tone there seems to be noise introduced into the signal. This is with the frequency contour on the sub set to bypass. If I back off the frequency control until it clicks, so now it is at 120dB the noise goes away. Another symptom with the frequency contour on bypass is that on a very few films the sub misbehaves, the driver making a whack whack noise but with the contour at 120dB this does not happen.
At this point I have no idea whether the frequency control on the sub is to blame or the line output on the Onkyo. Making an assumption that maybe the line output on the Onkyo is too high I did the following. (Not sure about this but the 809 being THX certified the line out has to be a certain level which can be higher than some subs like. My sub is not THX)
Setting the subs output to 75dB using Audyssey the level setting in the Onkyo is about 0dB. Setting the level with Audyssey to 82dB the setting in the Onkyo is about -6dB so the line out signal is 6dB lower. This improves things considerably. The final thing I did was to set the subs frequency contour to 80dB. the rationale for this is that THX subs have as an option a fixed THX 80dB crossover. This has stopped any films misbehaving even the dreaded 'Edge of Tomorrow' which I can play at all levels up to reference level without the subwoofer driver throwing a wobbly.
My sub is a BK Monolith Plus. 'BK Subwoofers Direct' is a UK internet direct company.
http://www.bkelec.com/HiFi/Sub_Woofers/monolith-FF_Plus.htm
I have no idea if any of this will be any help to you.