Is designing for accessibility "really worth it"? Or is it best to focus on what the experience for the larger majority of people?

To be very clear, I think that accessibility is very important. As a matter of fact I am specifically really interested in design for different disabilities, both in the sense of physical products, and stuff related to software.

Despite me of course thinking that design for accessibility should be a thing, I can't seem to figure out "why" that is the case. I don't know much about how accessible design acts in this way when it comes to for instance wheelchair ramps, however at least when it comes to software, designing things to work well for many different disabilities can require a ton of work to make the software compatible with different features, like theming so that a high-contrast mode can be added, making logic for screen reader support, so that buttons and text and stuff can be properly read, and a bunch of other stuff (that I embarrassingly can't think of right now). Due to (to my understanding) only a very small minority of the population needing these features, wouldn't it at the very least according to Utilitarianism thus be better to focus on the overall software instead of complicating the software with a bunch of features that need upkeep for future updates, as that is going to affect the majority of the user base?

Again, I personally think that accessibility is important, but I can't specifically pinpoint why, as at least according to the just-discussed view, it could be seen as more worthwhile to not work on accessibility features.