A direct flight? What?

A story about some very shady software vendors before I worked in IT.

All week we've been experiencing problems with our data entry software that seems to just get worse and worse. We outsource the software from another company and the software is hosted in a different state and runs through a web app. It's pretty basic software and we haven't ran into too many problems until now.

Lately, the software lags to a crawl every so often when saving work, just navigating or when signing in. Everyday I get calls and complaints regarding this and am constantly working to get it resolved. Upon constantly checking our internet connection and seeing it's fine, the first call with our software vendor support goes like this:

$me = me $sup = vendor support

$me: Good morning, the web app on our end seems to be very laggy and spotty when users are entering information. This happens very randomly when users are signing in, when users enter information, when the users are navigating throughout the app etc.

$sup: everything looks okay on our end, please check with your IT department.

So I go and visit with IT and they say everything is fine and show me the UTM and the monitoring software as well. They assure me everything is fine on our end and I believe them.

So I call support again

$me: I checked with IT, everything is fine on our end, they are asking you to please check your internet connection and also to check on your server connectivity.

$sup: everything is fine here

$me: have you checked with your IT department? I looked at everything and the problem seems to be on your side. Our other software and browser doesn't seem to have this problem, ruling out an issue with our ISP

$sup: okay I will talk with IT and get back to you.

They don't get back to me for the rest of the day. Although the software is spotty the data entry staff are able to work but it gets progressively worse. I try calling back everyday with them telling me they are still looking at it and I get the same run around about how I need to check on our network connectivity which of course is still fine.

Later that week I get a call from a couple supervisors over in the data entry side including upper management asking WHY the web app software is so laggy to the point they can now not work. I inform them of my problems with the software support team and how they are looking into it.

Coincidentally, we have a meeting regarding several changes within this same web app software with their upper management. I figured I'd bring this up to them so we can get this fixed. I run in to talk to our IT department just in case they found something but they tell me everything is still running fine. They run a trace route off the vendors server and find a timeout right before it hits the vendors server. They screen cap it and give me a printout.

I go into the meeting with this information and the conference call is just beginning. They ask if any outstanding issues need addressed before they begin, management eyes me down and I begin

$vm = Vendor Management

$me: We're still experiencing issues with the webapp.

$vm: Ah yes I was on the email this morning with our CIO and they are still saying things are running just fine on our end.

$me: our IT dept ran a trace route off your server and found a drop in connection in your area, not sure that this is the problem but I thought I'd share.

$vm: that's odd, lets get our CIO on the phone here and tell him this.

So they get this genius CIO who spills out a bunch of jargon regarding network connectivity (this guy is new btw) and basically says that we need to call our ISP. I wanted to scream out "WE"VE DONE THAT ALMOST EVERY SINGLE DAY THE PROBLEMS ON YOUR SIDE!" but I bit my tongue and let him continue to BS this situation down. At the end of his manifesto, I ask him to call me directly so they can continue on with the meeting.

So I get a call some hour or so later and have to leave the meeting. I suggest to him to reboot the server. I ran a ping off their dns and it runs okay but occasionally cuts off. He gives me some BS line that the ping is only affecting my local machine, what? that doesn't make sense! He asks me to email him a screen shot of the ping command, so I do and then go back into the meeting.

Hour later or so into the meeting the subject comes up again regarding the web app. I say that our users are still experiencing lag but your CIO is "looking into it" after I sent him screen caps. They get him on the phone to "inform" everyone what's going on and yet again he spews out another load of BS technical crap to throw people off about how it couldn't be possibly on their end. I then ask to reboot the server while management is on the phone but he absolutely refuses and is hell bent on the problem being on our side and then asks to speak to our IT dept.

So he gets off the phone and the meeting continues while I track down IT to get him on the phone. Finally, I have our net admin talk to him and tell him everything's working just fine on our end etc.

Honestly at this point its just getting ridiculous.

So i go back into the meeting and near the end the vendor-management asks if everything is working okay now and what the status is. Frustrated at this point, I finally let loose that this whole ordeal is getting worse and nothing is getting fixed, and if you could just "humor" me and have your guys reboot the servers! The line gets quiet, they get the CIO on the phone and it goes like this :

$vm: Hi $cio, this is $vm, we need you to reboot their server

$cio: but i'm looking at our monitoring and its...

$vm: just do it.

$cio: ohhhh kay

So i get a call from the supervisor in our data entry department asking: "what did you do?? its working flawlessly now!"

I get a call from their CIO asking if everythings working. I say "yes, its working perfectly, what did you do?" He says, "Well other than reboot the server, I looked into your network connectivity..."etc.

OH FFS

So i walk back into the meeting and everyone asks if the situation is resolved now just as their CIO joins the call. I tell them yes everything is working just fine. They ask what he did. He said this.

$cio:"Well after we reboot the server, we looked into your ISP in comparison to ours and saw that it jumped through several servers while making its way to you, sort of like connecting flights, we changed it to a direct flight, so now it works"

Everyone listened and almost applauded while I sat there, ready to slam my head on the table over and over...

TL:DR Software vendor CIO tries to turn a simple server reboot into him being some kind of ISP networking god.

*edit - Actually I remember what the CIO was BS-ing about the first time. He said that he had several techs test their connectivity and even had someone out of London, as our server hosted is in the US, check the connectivity with no problems. What I found out later was that he was having them ping THEIR dns and obviously being able to have no issue.