There's a lot of factors I can think of that might cause this.
One is that your memory of the song will be in a particular key (unless you've never heard a song before).
Another is that higher pitches generally sound more exciting, friendly, cute, etc etc. And low notes sound full and rich. Bass players (like me, hehe) are especially aware of this, and can use it to influence the mood of the song.
Then another is the "perfect pitch" effect - certain notes have different colours, and these have a subtle effect on the sound of the song.
And I think timbre also affects it. All instruments have subtle (and some less subtle) changes in timbre as the pitch changes. The only exception might be a synthesizer, but even then the signal is still going through speakers, resonating in different ways in the environment, and all that is making a difference.