How do i stop fusion from like auto installing everyone’s avatar packs

When i join a lobby it auto installs everyone’s avatar pack to my mods. This would be great if it didnt take up like a GB of storage every time i join a lobby. I see the setting to stop downloading them but will this make it so i cant see peoples avatars? Just confused