I mean, if I create a militant group and use a rotated swastika as the emblem, I don't think I'd be saying "only 20% of us max are Nazis, we're not a Nazi group." If I didn't want to be associated with Nazis, I wouldn't use their insignia, and I'd expel from the group anyone who is a Nazi.

If you use Nazi emblems and provide a safe haven for Nazis, you're a Nazi group.