Source
r/paloaltonetworks•
demonlag

Globalprotect disconnects on 10.1.6

Hoping someone has seen an issue like this before because PAN support is not making entirely great suggestions to help troubleshoot.

We have a pair of 5220s in active/standby HA. We are currently running 10.1.5-h1. We have a mix of Globalprotect client versions (5.1, 5.2 and 6.0). This past weekend we upgraded the firewalls to 10.1.6-h3.

Post upgrade everything seemed fine until I got dropped from Globalprotect with a keep-alive timeout. I blamed Verizon and kept working until 16 minutes later I got dropped again. I popped into the monitor tab and looked and every Globalprotect connection was doing the same thing. Connect for 16 minutes, disconnect, reconnect. Client OS version didn't matter, Globalprotect version didn't matter.

I tried a downgrade to 10.1.6 with the same issue. Reverted back to 10.1.5-h1 and all is well.

GP clients are all using IPSec, no SSL. I can't imagine it's a connectivity issue as there are about 200 active clients at any time using multiple ISPs and the problem starts as soon as the active firewall is on 10.1.6 or 10.1.6-h3 and stops as soon as the active firewall is on 10.1.5 again.

PanGPS.log shows every 16 minutes:

Too many outstanding keepalive and no response from GP gateway,disconnect tunnel
Tunnel downtime after keep-alive timeout 51xxx ms
VPN timeout due to keepalive, get out of ProcMonitor
In timeout handling, tunnel downtime is 51xxx ms

Posted Jun 30, 2022, 4:32 PM UTC

1126
br0tatochips•

So we just upgraded this evening to 10.1.6 during the upgrade I mentioned to TAC regarding this issue. At which point he performed some research and concluded that yes there is a bug/issue with 10.1.6 (we are running 5220's). related to GP's "inactivity TTL" displaying negative values after an upgrade. The fix was to physically power off the devices and power them back on.

Below is from TAC Case:

>We performed the upgrade of PA-5220 pair to 10.1.6 which went successfully.

>Post upgrade we found it hitting known issue PAN-196005 (negative Inactivity TTL timer values). As per the available workaround, we shut down the devices and boot them up again. The times values are showing correct now.

​

I'm crossing my fingers this workaround holds. our Inactivity value is set to 180 minutes and per TAC customers were experiencing disconnects at the halfway mark of this value. In our case that would be at 1 1/2 hours. Curious of those who were getting disconnected at 30 mins had their their Inactivity Timeout set to 60 minutes?

But thus far I've been connected for under 2 hours.

Before workaround:

Login Time : Jul.08 21:52:50

Logout/Expiration : Aug.07 21:52:50

TTL : 2591452

Inactivity TTL : -21525693

After workaround:

Login Time : Jul.08 22:20:13

Logout/Expiration : Aug.07 22:20:13

TTL : 2591958

Inactivity TTL : 10767

3
demonlagOP•

So I checked today, the inactivity timer on the firewall that users were constantly disconnecting from is 30 minutes, so disconnecting every ~15 minutes definitely lines up for us.

2
nukulaar•

So the workaround would be not to reboot the devices but shutdown and start again?

1
e_nitz•

Yes, I ran into this over the weekend and TAC had two options.

  1. Power off the units completely or
  2. Downgrade to 10.1.5-h2

Powering them down and back up worked for us

1
xXNorthXx•

Could you alternate them or did they both need to go down at the same time to correct?

1
e_nitz•

I alternated, failing over/back between the two.

1
dfctr•

3220 HA-Pair.

Same issue. But users disconnect after 5-8 minutes.

in 10.1.6, Inactivity TTLs are negative. Rolled back to 10.1.5-h2 and values returned to positive.

TAC Recommended this:

I am suspecting the same issue. Confirming the logs if it is matching with the known issue.

In meantime, could you try the below workaround on the passive device -

- Double the time configure under "inactivity logout"

- Power off/Power on the device to reset the tick count register

- Do failover and let me know the behavior

We couldn't do the workaround as our local partner decided it was best to downgrade and I don't have anyone on site.

Hopefully this fixes it.

2
quanx22•

do you have panorama? we have two PA-3250s and did the same (10.1.6>10.1.5-h2) and the active FW had now issues downgrading, but the passive FW won't connect to Panorama after the downgrade.

1
technicalityNDBO•

I upgraded to 10.1.6 on a pair of PA-850s this past weekend. I just checked the active GP users and most have been logged in for at least 4 hours (since this morning). There's a handful that have been logged in since Monday. We're running 5.2.9 for the client.

1
rh681•

See my thread here: https://www.reddit.com/r/paloaltonetworks/comments/v5goni/ipsec_issues_on_101/

Mine are not GP clients per se, but Avaya IPSec phones that use the GP Portal. They died at 25 minutes on the nose. The TAC was not helpful so went all the way back to 10.0.5. I have not used <10.1.6 on these firewalls so I can't verify if 10.1.5.h1 would fix it.

1
demonlagOP•

That's an interesting data point. We only use GP, no third-party clients but our first attempt at upgrading to 10.1 way back we immediately hit PAN-181116 and had all our GP clients fail over to SSL. Feels similar enough, though PAN support still is pushing that our issue is related to either data center internet or seemingly every client we have's internet.

1
rh681•

This shouldn't matter, but part of me thinks there may be a specific order of upgrade to avoid this issue since it's not widespread. What was your first 10.1.x version?

1
demonlagOP•

On this particular pair we moved from 10.0.7 straight to 10.1.5. We also have another firewall serving globalprotect clients that are running 10.1.6 without issue, though it's a virtual platform and not a 5220. The working firewall started at 10.1 from deployment (10.1.0, 10.1.2, 10.1.4, 10.1.5, 10.1.6).

3
DotrenACE•

Are you still having the issue with them failing down to SSL? I've been seeing this as well.

1
demonlagOP•

Nope, IPSec only still. Even when the 10.1.5 clients fail they connect right back up over IPSec and truck along until failing again.

1
nukulaar•

We have the same issues. Updated Yesterday to 10.1.6 and have disconnects every 15 minutes. No matter if ipsec or ssl connection.

1
demonlagOP•

What hardware? Our issue is with a 5220 cluster, but we have an Azure firewall running 10.1.6 without this issue, just curious if it could be platform specific.

1
nukulaar•

Yes 5220 ha-cluster.
We ugpraded from the 10.0.10 release and already hat issues with dualstack ipsec clients. We had to reduce the MTU size of the Tunnel interface (to 1350 in our case). Don't know if this has something to do with it.

EDIT: We just did Downgrade to 10.1.5-h2 and the GP Clients are still connected after 30minutes....

EDIT2: We are using Dualstack IPv4+IPv6 on our Globalprotect Gateway. Could this be the reason why not everyone is having this issue?

2
databeestjegdh•

We have dual stacked our gateway as well. And we even have icmp, icmpv6 and ping, ping6 allowed to both the v4 and v6 addresses. But clearly this isn't enough.

1
rh681•

I reopened my case. It was closed because I thought I was doing something wrong. I referenced this Reddit thread in the notes. So basically:

PA-800 series = good

VM's = good

PA-3220 or higher = possibly bad

1
demonlagOP•

We're similar. We have a VM series in Azure that's fine. We have some PA-850s that are fine. Only our 5220s start dropping clients when upgraded.

2
quanx22•

can confirm - PA-3250s are affected

2
demonlagOP•

As a data point we are IPv4 only still (we have IPv6 address space, company just doesn't want us to deploy it cause change is scary).

1
databeestjegdh•

3220 here, also

1
dfctr•

+1 here. Rollback to 10.1.5-h2 to see if it's fixed.

1
rh681•

I just upgraded a PA-820 and IPSec phone disconnect issue isn't affected. So so far, it only does it on my PA-3220 cluster.

1
[deleted]•

[deleted]

1
Thornton77•

I’m running 4 pairs of pa-5220 on GlobalProtect We have some disconnect complaints . But we had a few issues before we upgraded also. We have about 4000 across 2 pairs and 1000 on the other set.
When we first upgrade our 1000 user pair we had a the morning after about mass disconnects . 300 user at 8:30 am got disconnected. While we were looking into it a 2nd set came in at 10:08. But this time it was only 60 users . The first one at 8:30 was attributed to an Adjacency drop with one of our ISP’s. The second one at 10:08 Palo Alto linked it to a client side uses problem. Yeah the recommended a total uninstall of the client. And also for some unexplained reason deleting the wmi database . When we push back on the wmi delete the tech couldn’t explain why it was nessary. Or why a client side issue would cause a disconnect experienced by independent clients . I’ll go find the case and copy pasta what they said in another reply .

I wrote a script to copy off the event log from every connected clients and for a few days I was grep’ing disconnects . I had data going back to November when we switch over . The disconnect rate was crazy high. But it was always high. Over a day we have a disconnect about every minute. But that is across all 4000-6500 clients.
Before and after the 10.1.6 upgrade .

1
[deleted]•

[removed]

1
tm001tm001•

I had exactly the same issue with upgrading a HA pair of 5250s to 10.1.6-h3. Almost rolled back, but another emergency request came in that required a new rule to be deployed/burned in. After that, as a side effect, the disconnections have disappeared and everything works as it should now.

1
MoDj_K•

Interesting! So may be a new push/commit after the upgrade is necessary to resolve the issue as a workaround (?) ....

2
rh681•

Very interesting...

1
rotearc•

We have a very small Global Protect deployment, the firewall is running 9.1.14. One user with a Windows 10 laptop, it will disconnect every 30 minutes only when the laptop is on office network. No disconnect issue at all while the user is working from home. TAC reviewed the pangps.log and found these recevied an IP packet with a non-tunnel source IP <the GP host interface> and the packet received from virtual interface is discarded.

We ended up replace the user laptop since that laptop was running Windows 10 version 1803. So far so good.

1
demonlagOP•

This is very interesting. I had tried restarting the firewalls but not powering them off. Support closed my case because I couldn't arrange taking the company down for a few hours with limited notice but I'll see if I can reopen and ask if this bug number applies to us. I think our inactivity timer is 30 minutes also, which would line up with your math.

1
martinsysadmin•

Hey, PA-5220 HA Pair - same story. Timeout is usually set to 15 minutes. After the update there is a disconnect after 7.5 minutes. In any case, increasing the timer increased the time until disconnect. I will notify you after the shutdown and boot.

1
quanx22•

we ran into the same issue, upgrading HA Pair from 10.0.8 to 10.1.6. GP users disconnected after 5 mins. We downgraded to 10.1.5-h2 on our passive FW, failed over to make it active and tested GP which worked. Then we attempted to downgrade the newly passive FW and after downgrading, the passive FW no longer connects to panorama. I found this linkand tried it but still no dice. happy friday to me

1
xXNorthXx•

For those who did the power off method in an HA pair, did you need to do both at the same time or could you do it to the passive/failover/passive so there wasn't an outage?

1
Electrical_Gear_5532•

We ran into this with the 10.1.5-h2 and the 10.1.6-h6 updates. We have some 3220s and 800 series units out there. Only the 3220s were impacted. Completely powercycling the firewalls (shutdown + someone to push the button in the datacenter) solved it in both instances.

1
Fldaboy•

The preferred release is 10.1.6-h6. Anyone seeing any major issues for this?

1
McKeznak•

10.1.6-h6 has the same "Half the time-out issue" PAN-196005

1
Fldaboy•

I don't see a PAN-196005 under 10.1.6-h6 known issues?

1
McKeznak•

Ya they they don't have it up there yet it would seem

1
McKeznak•

Anyone that did the power cycle, have the issues returned for you since?

1
McKeznak•

10.1.7 Released, trying it Friday

PAN-196005 (PA-3200 Series, PA-5200 Series, and PA-5400 Series firewalls only) Fixed an issue where GlobalProtect IPSec tunnels disconnected at half the inactivity logout timer value.

1
stacca_stacca•

Hi All,

Same issues with 10.1.6-h6
I resolved downgrade it to 10.1.4-h4 and after few weeks PA support replied with this:

----------
As per our analysis, this is behavior is matching a known issue PAN-196005 and is resolved in PAN OS 10.1.7. Please find below article on the addressed issues in 10.1.7 and search for PAN-196005.

https://docs.paloaltonetworks.com/pan-os/10-1/pan-os-release-notes/pan-os-10-1-7-known-and-addressed-issues/pan-os-10-1-7-addressed-issues
----------

PAN-196005

(PA-3200 Series, PA-5200 Series, and PA-5400 Series firewalls only) Fixed an issue where GlobalProtect IPSec tunnels disconnected at half the inactivity logout timer value.

From firewall logs: "reason user session expired"

from the GP client Logs
rasmgr_sslvpn_client_terminate space GP_GW_NAME domain yyyyyy user aaaaaa.bbbbbb computer PC123 reason user session expired

1
bakerds•

I have not encountered this issue with my PA-5250s on 10.1.6 and a couple thousand GlobalProtect users

0
Fun_Artichoke8765•

Yeah, me neither on PA-5250, maybe this model is not affected :)

1