Yeah, that is one reason some of our special forces guys are having "stopping power" issues with the really short barreled M4's, like the ones with 10 inch barrels. You lose a lot of velocity out of a shorter barrel, that is one reason the bullpup concept is good, short weapon with a regular length barrel, except in the carbine variants.
Most people think of the 9mm as "weak"(in real life), but your average 9mm JHP is a 115 or 125 grain bullet traveling at around 1100 FPS. The .357 mag is a 125 grain JHP traveling at 1600 FPS. Bullet speed does make a big difference. That is basically the reason that rifle rounds are so devastating. A 7.62x39 bullet weight is 122 grains, but it is averaging 2400 FPS.
I did see some guys in Iraq take 3-4 9mm rounds before going down, but that is mainly because we had to use ball. I carry a Glock 19 now with hollowpoints, and 9mm in HP is fine. All my training mentions that to never expect a one shot stop with a defensive handgun, and that is how the failure drill came into practice.
This is also the biggest reason pocket pistols suck. If you have a .380 with a 2 inch barrel, you aren't getting much of anything. It is still a lot better than nothing, though.