I think it should be toned down for all perks except the Sharpshooter. As many have said, when a fleshpound shows up the situation shouldn't be "Oh, a fleshpound. Hey Bob, can you whip our your crossbow and take it out? Thanks." Where in this example Bob could be anything from a level six Sharpshooter to a level zero Firebug who finds himself in possession of a crossbow.
As it is now, if any competent teammate has a crossbow, fleshpounds may as well be the equivalent of a clot (in terms of a threat). By that do I mean that fleshpounds will never be a nuisance? Of course not. Murphy's law. But in most current cases they aren't.
In the hands of a Sharpshooter the crossbow is rightly a devastating weapon. It certaintly does have the perfect limitations, as you have stated. It has a limited ammo supply and it doesn't have a fast rate of fire (it is accurate outside the scope, that all depends on the user). But it becomes a problem when Bob, our friendly level zero Firebug, can pick it up and in most cases be just as devastating as a level six Sharpshooter can be.