Are non-food nuts and seeds (e.g. shea, jojoba) also to be avoided on baby skin prior to weaning?

I understand that there may be reason to avoid applying food ingredients to babies' skin prior to them starting on solid food (especially common allergens like peanut oil, sesame oil, almond oil, coconut oil), because they might cause the child to become allergic if they encounter it through the skin before the gut.

But what about other nuts and seeds (for example) that we don't eat (like shea butter or jojoba oil), are these thought to be able to cause allergies to other edible nuts and seeds, or does the allergy trigger only apply to the specific foodstuff applied (like sesame oil could only cause a sesame allergy)? Hope that makes sense, I haven't been able to find out the answer myself!