Challenge in video games (and games in general honestly) is clearly the spice that keeps you craving for more... if it's balanced the way you desire! I guess it does depend on both the player (maybe you're simply willing to chill after a long day at work) and the genre of course (I wouldn't expect a game driven by its story to be necessarily challenging... it can be ! But it could also focus on the "experience")
I always found it curious how I absolutely CAN'T get into souls-like, but absolutely crave roguelikes for example. Both genres are very punishing, and require a deep learning curve that might require you to fetch external resources (wikis !! But also other players) to even realize some of the games' mechanics, items or secrets. I would guess that I'm less annoyed by the idea of starting a whole new playthrough than crying bloody tears at a single boss for an entire weekend...
It's also pretty different for competitive multiplayer games. The difficulty is usually measured by the skills of the people you're facing... and your own. And it's also why I'm not really giddy at trying my luck at games like Counter Strike or Call of Duty for example. I've played them a good dozen years ago of course... but I feel like players latched onto such games for just as long, while I went on to play other things. It's definitely NOT very welcoming for newcomers, even if you get access to the best tools the game has to offer.
But Killing Floor is obviously a little different, considering it's 95% coop rather than competitive (I wouldn't want to piss off the four players still playing Versus mode...) But I think it still warrants challenges in its own right. Hell, even solo games should offer a challenge to whoever plays them. A lot of RPGs are single-player only, a lot of FPS too. But would something like Doom Eternal by truly enjoyable if you could just steamroll through everything? I don't really think so. And I also think it's one of the best modern examples of a game that makes you feel both VERY POWERFUL, and VERY VULNERABLE. Being quick on your feet and mastering combos are not just the best way to beat the game... But absolutely necessary at harder levels.
As I've said multiple times lately though, I do admit that gauging KF's difficulty is probably not the easiest thing in the world. You have to change the difficulty depending on the setting picked (obviously... normal should be easier than suicidal), but also the amount of players on the team. I would assume that a lot of mechanics are at work behind the scene to offer something that is both manageable, but still hard enough to promote good teamwork and quick reflexes.
I compared KF's difficulty settings to L4D's in the past, and especially the now infamous AI Director and how sadistic or lenient it could be. But having played the latest TF2 community event, I realize that we could also compare it to that game's Mann vs Machine game mode... I feel like both aren't that different.
-Multiple difficulty settings (although linked to specific "missions" rather than being available for every possible maps and configurations)
-Six players being able to choose between a set of various classes with highly specific roles (but the missions won't be any easier if there's only 4-5 of you)
-An upgrade system to make you sturdier against the robot hordes (akin, yet different to the "tier system" of KF, as you don't pick up better weapons... you just upgrade them, and yourself !)
And similar strategies do apply, at least in their basics. You gotta fill your class role. You ought to have a well-balanced team. You got to know what might be the most useful (if need mandatory) upgrades for the coming wave, but also what each opponent might do to you, and how you could dispatch them quickly.
And the funniest thing is that, for a mode that hasn't been updated by Valve for TEN YEARS (unless we count the handful of weapons they've added since November 2013), it's still somewhat more balanced than KF2. It's entirely possible to goof around and use some ridiculous loadouts or team comps on lower difficulties... And beat lower difficulties with one or more players missing ! But once Advanced missions (or Expert for that matter) are on the plate : you'll WANT to have an experienced and well-balanced team that knows what they're doing. And I think it's perfectly understandable, reasonable... and obvious even? And something that, as you said, Killing Floor 2 lost along the way. I do believe that balancing stuff for both SOLO and MULTIPLAYER playthroughs might be partially to blame (it's totally impossible to beat even the easiest mission on MvM alone, and I do believe that Killing Floor should maybe stop considering solo entirely for its balance... but that's probably controversial).
But it's also, as you mentioned, a simple lack of care with a lot of the new content on offer. It's unfair to fully compare the two games obviously : Team Fortress 2 is essentially a PvP game, where Mann vs Machine is mostly an interesting distraction... While Killing Floor's PvE is its bread and butter. But while loadouts are often limited to a certain "meta" in Mann vs Machine (hell, Scout and Pyro players should pretty much NEVER change their own single viable loadout...), I think it's almost the opposite in Killing Floor... Where nearly every single option, even the silliest ones, is at least viable thanks to how lenient the game became. And while I do believe that in a perfect world, every single addition should be meaningful even if its in a few rare circumstances... In KF2, it became overlap instead. Rather than having different tools for different roles, you get multiple tools to do exactly the same thing. And it doesn't help that perk roles have been diluted into oblivion as well. Most of them are now husks of their former selves, being able to deal with nearly anything... Again, it's ok for a SOLDIER (essentially TF2's equivalent of the "jack-of-all-trade" class) to be able to do a bit of everything (team support, crowd control, nuking, tank busting...). But not for a Heavy or a Sniper for example. And I think that rule should apply to Killing Floor as well.
All in all, I'm not fully against those "rare high moments" (

) when they are sparse. Of course I like the odd "all clots" wave where I can just freely mow through weaker zeds. Just like it's mad funny to get your hands on the Super Shotgun in Doom and start blasting... But these should be kept sparse, as a way to "blow off some steam" before coming back for an even bigger threat. When the entire game becomes a simple walk in the park... Where's the joy of improving and beating the odds anymore?