How to deal with an unethical work enviroment?

The company I'm working at has built their entire business model around providing services to 3rd party clients. Exchange, SharePoint, Microsoft CRM, as well as some other applications. We have all the clients logging into the software and using the email etc. and everything is licensed with retail licenses.

I'm fairly new to the industry and have never had a position where I would have had exposure to the various licensing models provided. Right now I'm not in charge of any of the licensing that is all handled by my boss, I simply manage accounts and help users log into their stuff. But I've done some reading and I can't see any scenario where we are within compliance of the EULA. At least with Microsoft, I know they have the SPLA option which I suspect we "might" be able to get into, but other applications we are using don't seem to have an equivalent option.

On top of all of this it is still unclear if upper management will be willing to address the issue correctly. This is a small company doing a decent amount of business, so my department boss is besties with the owner of the company. The same boss is also literally about to retire. Its myself, my supervisor, (10yr industry vet who actually spotted the issue, and was hired with the intent to replace the CTO) and our boss the CTO. The boss CTO is away and has made a brief call and argued that we are within eula because we are not technically hosting crm websites or anything like that, which neither myself nor my supervisor accept as a reasonable answer. still insisting that we are in breach of licensing.

So now that I've tried to summarize that whole debacle:

1. Does anyone actually know the best process/path to getting back into compliance with Microsft/other vendor?

2. How can I, as clearly as possible, convince my 40 year industry veteran boss, who has gotten us into this situation, that he is wrong and we need to basically redo everything?

3. If managment decides they want to keep riding dirty, what is the best course of action for me to take? Should I walk off the job? Should I give 2 weeks notice? Should I fly under the radar until I line up another job (say a few weeks + 2 weeks notice)