r/vmware•
Latch

NSX: Be VERY careful rebooting vSphere Server instance when running NSX (for now).

I thought I'd throw out a warning in case this can bite other NSX users (especially if they're running it in prod).

We hit a bug (which turned out to be a known problem and is being worked on) where if you restart your vCenter Server services (for an instance that's using NSX), random hosts will 'lose track' of their NSX VTEP(s) and refuse to do any virtual network traffic (it tries to create new VTEPs multiple times, but fails, as well). This seems to have something to do with how vCenter Server pushes out the VDS config to the host when it comes back up? I don't know. I just know it bit us (although thankfully just on our test instance), and any VMs on the host that were going over a vxlan network dropped off the air, as expected when VTEPs/NSX are broken.

NSX's Installation area will show the affected host(s) broken for VXLAN.

I'm not sure how wide-spread it is. I didn't see it mentioned in the release notes so I'm guessing not too big, but still. It took VMware Support a couple hours to figure it out, but then determined it was a known issue and being looked at. Be careful ;)

Environment, in case it's based off that and you're similar:
NSX 6.2.1
vSphere 6.0
NSX Config: Load Balanced/SRCMAC (switching to SRCID later), 2 VTEPs/host.

Edit: Got word back from support, it's a known issue that will be fixed in 6.2.2 (which has no ETA).

Posted Feb 10, 2016, 9:44 PM UTC · edited

4052
jadedargyle333•

I hit a huge bug with NSX. Pricing.

20
LiamGP•

We hit the same bug. We're not a small company, but until the pricing comes down, it's WAY to expensive for us.

3
bongthegoat•

Funny we hit that bug too, it's KB:6Kx2

3
vDocks[VCIX-NV]•

What is your use case for NSX and what alternative did you end up using in it's place?

1
MrDogers•

This little piggy had none :(

We have tons of VLANs with firewalls dotted all over the shop managed by a network team. We've got devs wanting to throw up environments left right and centre and another team looking heavily at AWS. We either need to bite the bullet and go NSX or look at a parallel setup of OpenStack in the coming year, IMO..

1
vDocks[VCIX-NV]•

Definitely a prime use case for NSX. Automated provisioning of networks, firewall rules and a simplified physical network. I asked the OP what their use case was specifically because I see a lot of folks barking about the cost of the product. But in most circumstances, they either didn't understand OpEx or have a valid enough use case to make the shift to SDN in the first place. That's most cases, there will always be those that are just driven away to the CapEx to set it up in the first place too.

1
MrDogers•

Yep, but I just got pricing (literally today) and it'd cost us close to $800k to implement across all our environments. NOT gonna happen :(

1
jadedargyle333•

We plan on rolling it out for 5 different labs. Price for vdi is a fraction of the price for our server cluster. It's 5 times more expensive to use it for half the machines.

1
vDocks[VCIX-NV]•

We plan on rolling it out for 5 different labs.

Static lab environments? Or are you talking 5 different internal private clouds that will be used for automatic provisioning of labs? If it's the former, you wouldn't use 90% of what makes NSX expensive, and I can understand why it would turn you off. If it's the latter, I'm curious what you'll be using in it's place as an alternative.

Price for vdi is a fraction of the price for our server cluster. It's 5 times more expensive to use it for half the machines.

This makes me think you're talking about 5 static lab environments as opposed to a self service internal cloud. To this point, you have alternatives in the VDI market to accomplish roughly the same thing. NSX is in a league of it's own in many ways. It sounds like you may have just wanted bits and pieces of what NSX offers instead of the full suite. And to that point, I wouldn't expect the price of NSX to be remotely justifiable.

Not trying to be combative btw. I just see a lot of folks (and obviously you have a ton of people that agree with you) comment about how the price tag of NSX turns them off. But more often than not I find that there wasn't a valid use case to justify the cost to begin with.

1
jadedargyle333•

We'll be using NSX. The labs will be auto provisions based on customer needs, but they are going to be VDI labs. That's why I feel like there is a bug in pricing. The vsphere servers are all pretty static, but cost 5x as much to license. The benefits make sense in R&D, but don't fit the bill for production at the current price point.

1
Dave2SSRS•

How did you get it back online - recreate all the VTEPs?

4
LatchOP•

Yep, remove the host from cluster, reboot to clear anything. Manually remove the old vmkernel interfaces (since it 'loses track', it doesn't remove the vmk interfaces when you remove it from the cluster), put the host back in the cluster. Reboot again.

I haven't tested, but I may be able to just remove the interfaces manually and reboot the host (without the cluster swapping) and NSX/vSphere may re-create the VTEPs on boot. Not sure.

3
cmbg78[VCDX-NV]•

Did they have you try to force resync first? And did they have you do it via the UI or API?

Were all VMs on logical switches impacted? Could two VMs on the same host still communicate?

I've bounced vCenter many times without issue, but i haven't tried on 6.2.1 NSX yet. Looks like a lab experiment is in order this weekend!

2
LatchOP•

I did a resync through the GUI before I called support, no go. They didn't attempt it in the WebEx (or have me attempt it).

We only have one logical switch, but if the two VTEPs are gone (in NSX's eyes), I'm guessing any logical switch would have been down. Not sure about two VMs on the same host.

Not sure 6.2.1 is the issue. Right before I tore down my 6.2.0 instance to install 6.2.1 freshly (was doing it for more experience), I think I saw this same thing one one of my clusters. Might happen to you, but I'm not sure version is the common string in the problem record. Good luck.. I guess? ;)

2
cmbg78[VCDX-NV]•

Out of curiosity, if you only have one logical switch, why are you using VXLAN at all?

2
Dave2SSRS•

Ugh!!

1
BeechesLoveLeaves•

Is there a KB for the bug?

3
LatchOP•

No KB that I know of (at least, support didn't give me one). My ticket got attached to the PR though, apparently.

1
BeechesLoveLeaves•

Damn. You have me scared. We're running the same stuff in production. Have not got that yet, though.

1
imp3131•

Haha. Shit. This has happened to me twice now. I can relate. The problem lies with a vCenter service called 'EAM' essentially losing connection to the host/VDS way in the back end processes. Nothing shows otherwise. Support struggles with the issue. The workaround you posted is the same I've had to use. The most important step is to delete the vmk vtep port so that NSX can recreate it after the reboot and cluster rejoin.

However... Someone informed me that since VMware and NSX first integrated, they have been intertwined with this 'EAM' service. Apparently, the NSX engineering team is completely fed up with these failed integration attempts and will be completely removing the integration in the next major NSX release. It should make it much more reliable.

2
LatchOP•

Ahhh interesting, thanks for the info! I hope the next major version comes soon, then. I'm very hesitant to use anything production with this bug around. Thanks!

1
bongthegoat•

ouch. we are just currently building out a test environment for an nsx pilot. I'll have to add that to our test plans and see if we can recreate it.

1
LatchOP•

Get it all up and running, and have stuff on a virtual network (not sure this part matters), then just reboot the vCenter Server instance (or restart the services). On our test cluster it reliably causes a random host to break on every restart.

3
bongthegoat•

Man i was trying my little heart out to get this to happen then i revisited the post and saw you updated it with the it's fixed in 6.2.2 which is what we deployed in our pilot. Thanks for the update!

1
LatchOP•

I didn't realise 6.2.2 was out, thanks!

On a quick glance at the release notes I didn't see the issue mentioned, but good news that you weren't able to reproduce it! I'll get that installed on my dev instance soon.

1
bongthegoat•

6.2.3 is out too ;) just a couple days ago

1
cloudmech•

Ouch. Thanks for sharing this info.

1
bongthegoat•

so much for that management plane separation huh?

1
Coldpawz[VCP]•

Just to clarify, you upgraded successfully to 6.2.1, then restarted the vCenter Service and the problem occurred?
The reason I ask is I'll look to test this on my current vCenter 5.5, and NSX 6.2 environment.

1
LatchOP•

I had a 6.2.0 install on vSphere 6.0. I tore 6.2.0 down and installed 6.2.1 freshly. A while (couple weeks) later I had to reboot the vCenter Server VM, which caused this to pop up. When I was decomming a host in the 6.2.0 environment a few weeks ago, I saw this issue pop up as well. I didn't chase it as I was getting rid of 6.2.0 anyway.

Give your vCenter Server instance a few reboots and see if VXLAN is broken for any of your hosts after reboot (NSX->Installation area). I'm guessing this only affects vSphere 6, but I have no real idea about it. Good luck!

1
Coldpawz[VCP]•

What ESXi version are you running on the hosts?

1
laggedreaction•

Do you have a KB (or iKB) or SR #???

1
LatchOP•

No KB that I know of (at least, support didn't give me one). My ticket got attached to the PR though, apparently. I have an SR number.

1
CptCervix•

Are these hosts with internal boot or stateless hosts?

1
LatchOP•

Not stateless. "Internal" boot, although the root disks are LUNs over fiberchannel.

1
CptCervix•

I would be anything this is a vSphere 6 and/or dual VTEP specific problem.

1
Zadnak•

What hardware are you running, Cisco UCS?

1
LatchOP•

Yep. B200 M2 blades atm (test cluster).

1
Zadnak•

I experienced a similar issue recently in regards to the VTEPs not accepting traffic. What happened was the VM on an NSX port group would not route past the DLR unless the Global Edge was on the same host as that VM. With only 1 VTEP and two vNICs per host, it turned out the traffic did not pin to the NIC coming back into the host, so we had to tear down the NSX install, and configure to have 1 VTEP per NIC.

The other thing I will say is make absolutely sure the correct Cisco UCS Enic and Fnic drivers are installed on each ESXi host. This was a problem as well. I swear by the Cisco UCS compatibility matrix instead of VMware's for this.

3
t0xie•

Upvoting for 'I swear by the Cisco UCS compatibility matrix instead of VMware's' We haven't installed NSX yet, but worked with Cisco quite a bit at another company and they emphasized always always always check the Cisco matrix before every VMware update.

2
LatchOP•

Thanks for the info! We're doing 1 VTEP/nic for that reason too, I think. Read it in some NSX+UCS design doc that VMware put out a while ago.

We've had (VMware) professional services down to look at our configuration while we got set up and they seemed pretty happy with most of it. He did list some ucs-enic-specific stuff, and we're all good there now. I hate how the UCS and VMware compatibility matrices don't always say the same thing. So annoying. I also hate how Cisco won't put RSS or vxlan offload into the enic driver for VMware. I get it, with their ACI product sort of trying to compete with NSX, but it's dumb.

1
vDocks[VCIX-NV]•

I also hate how Cisco won't put RSS or vxlan offload into the enic driver for VMware. I get it, with their ACI product sort of trying to compete with NSX, but it's dumb.

Talk to your Cisco rep. Probably for the same reasons you mention it won't be a priority for Cisco until customers make it a priority.

2
Zadnak•

My other advice is take your time getting it implemented and tested. Bugs are just growing pains, but they will cause major stress on everyone if production is moved into the environment too soon.

1
JFCronus[VCAP]•

This may sound strange but do you have VUM connected to this vCenter instance?

I ask because we experienced similar behavior during our POC testing. We just deployed a fresh VCSA appliance and NSX 6.2 and ran into these issues. Reboots of any hosts or vCenter with NSX cause EAM/VUM to check or update the VTEP database and fix this. We had to use the API to clear out two VTEPs manually out of the pool but it's been OK since.

There is a MOB value you can modify in vCenter if you need to change the behavior but essentially make sure VUM is installed and functional before installing NSX.

1
LatchOP•

Thanks for the idea! VUM is in and working fine. I'm thinking there must be some weird legacy configuration somewhere in the system. The vCenter server database is from 3.5 or 4.0 days, so who knows what sort of random legacy cruft is in there.

1