Only use the analog output with REW. The digital output isn't correct since you aren't able to loop it back to the input and create a soundcard.cal file (which you'll need to do). The soundcard calibration routine creates a file that is the inverse of the soundcards poor response and is applied to the measured responses to make them accurate.
Anyway, can you post a jpg of the REW soundcard mixer, so we can see what you have selected?
It will look like this............
Check Levelsxxxxx clear.jpg
brucek