How strong is too strong of a wind to ride in?

the walk to the bar is 20 minutes but the ride is only 5. The wind is strong enough that I worry it could wobble me on my bike, but I KNOW it's too strong for me to not be freezing at the end of 20 minutes.

how fast is too fast a wind?