I think you are mostly right, but the issue is not altchans as much as the fact that people always refer to the big centralized site. I have no idea how this can be fixed, but while in the past you had like 1000 small sites that were genuinely insular with their own culture, right now it's almost impossible to have that. Almost every single user will come to your tiny alt imageboard with a baggage of news and memes from the big centralized sites, and other people will latch onto those because they're familiar, and what you end up with is just a slower echo of the Reddit/Twitter/4chan amalgam everyone participates to. Also everyone is an attention seeker nowadays so the worst people will inevitably seize all the attention on the small website and consume it from the inside.
I know for sure that this is the problem, but I also have no idea how to actually fix it, it's something much bigger even than people's individual decision making: I've stayed isolated from social media since forever but I'm still up to date with the memes and general internet zeitgeist simply because it's impossible not to absorb it by just listening in to any frequency. I could join a random Discord and pick up the conversation as if I've always been there. How do you even create a community that is truly insular and independent? Even offline isn't safe from interconnectedness.
And if you explicitly distance yourself from this situation you'll be ripped apart because it's like telling people to stop eating. Or you'll attract the attention of people who like the idea but will never quit feeding into their FOMO.