I think it might feel quieter, although not by a large amount. Think about it, smaller drums will move less air. Of course, there's a lot of other factors too, like skin, tuning, shell material, how hard you hit it, and so on, but all other things being equal, I would expect a smaller kit to be less disturbing. Still, it would be loud enough to disturb people. For example, for a period of time I was using a 13" bass drum. With the right choice of skins and similar amount of muffling so it's more bass-y, I managed to make it sound quite a lot like my 20" bass drum, just more punchy and only slightly higher in pitch. However, comparing them side by side, it is definitely much softer than the 20" bass drum. This is simply because the skin is so much smaller, so the air it moves is much less. It's just less powerful. Like comparing an amp stack with small computer speakers.
And besides the exact volume it is playing at, there's also the "feeling" when you listen to it. I think long, boomy or sloshy drums or cymbals feel louder psychologically than short, sharp sounds, when played at the same volume. Smaller equipment usually has a shorter, more punchy sound, which hits you but doesn't surround you.