Crashing/frozen Fortigate 60F's

For a couple of months now I keep experiencing Fortigate 60F models which in course of time go offline because they enter a crashloop. Yesterday I was finally able to capture the console output (in contrast to powercycling by the customers).

The weird thing is I've experienced this with several 60F's in the field but when googling nothing yields any result which match what I am experiencing. Fortinet support is useless as a handbrake on a canoe so I won't be purchasing support anymore for future purchases.

The situation is quite simple; A solitairy Fortigate 60F is deployed without any fancy configuration and after a certain time it just goes offline. The time is between crashloops is somewhere weeks and months. The units were running the 7.2.x and the 7.4.x trains.

This is the part of crashloop which is barfed out of the console port:

pc: 0x0<00000> Backtrace:
pid=1 get sig=11 fault:0x7f90c06000
pc: 0x0pid=1 get sig=11 fault:0x7f90c06000
pc: 0x0000000000d810d8 sppid=1 get sig=11 fault:0x7f90c06000
pc: 0x0000000000d810d8 sppid=1 get sig=11 fault:0x7f90c06000
pc: 0x0000000000d810d8 sppid=1 get sig=11 fault:0x7f90c06000
pc: 0x0000000000d810d8 sppid=1 get sig=11 fault:0x7f90c06000
pc: 0x0000000000d810d8 sppid=1 get sig=11 fault:0x7f90c06000
pc: 0x0000000000d810d8 sppid=1 get sig=11 fault:0x7f90c06000

The final part of the crashlog is as follows:

825: 2024-07-28 10:20:10 <15320> Node.JS restarted: (unhandled rejection)
826: 2024-07-28 10:20:10 <15320> Error: kill ESRCH
827: 2024-07-28 10:20:10 <15320> at process.kill (node:internal/process/per_thread:232:13)
828: 2024-07-28 10:20:10 <15320> at /node-scripts/chunk-449c6eed240ab919355e.js:4:484599
829: 2024-07-28 10:20:10 <15320> at Array.forEach (<anonymous>)
830: 2024-07-28 10:20:10 <15320> at stopWorkers (/node-scripts/chunk-449c6eed240ab919355e.js:4:484572)
831: 2024-07-28 10:20:10 <15320> at async CronSchedule.httpsdHealthCheck (/node-scripts/chunk-449c6eed24
832: 2024-07-28 10:20:10 0ab919355e.js:4:477006)
833: 2024-07-28 10:20:10 <15320> at async Cron._trigger (/node-scripts/chunk-0238041ac4439f9b2c08.js:4:4
834: 2024-07-28 10:20:10 8619)
835: 2024-07-29 04:12:33 the killed daemon is /bin/sflowd: status=0x0
836: 2024-07-29 04:50:43 <16166> Node.JS restarted: (unhandled rejection)
837: 2024-07-29 04:50:43 <16166> Error: kill ESRCH
838: 2024-07-29 04:50:43 <16166> at process.kill (node:internal/process/per_thread:232:13)
839: 2024-07-29 04:50:43 <16166> at /node-scripts/chunk-449c6eed240ab919355e.js:4:484599
840: 2024-07-29 04:50:43 <16166> at Array.forEach (<anonymous>)
841: 2024-07-29 04:50:43 <16166> at stopWorkers (/node-scripts/chunk-449c6eed240ab919355e.js:4:484572)
842: 2024-07-29 04:50:43 <16166> at async CronSchedule.httpsdHealthCheck (/node-scripts/chunk-449c6eed24
843: 2024-07-29 04:50:43 0ab919355e.js:4:477006)
844: 2024-07-29 04:50:43 <16166> at async Cron._trigger (/node-scripts/chunk-0238041ac4439f9b2c08.js:4:4
845: 2024-07-29 04:50:43 8619)
846: 2024-07-30 07:45:11 <16396> Node.JS restarted: (unhandled rejection)
847: 2024-07-30 07:45:11 <16396> Error: kill ESRCH
848: 2024-07-30 07:45:11 <16396> at process.kill (node:internal/process/per_thread:232:13)
849: 2024-07-30 07:45:11 <16396> at /node-scripts/chunk-449c6eed240ab919355e.js:4:484599
850: 2024-07-30 07:45:11 <16396> at Array.forEach (<anonymous>)
851: 2024-07-30 07:45:11 <16396> at stopWorkers (/node-scripts/chunk-449c6eed240ab919355e.js:4:484572)
852: 2024-07-30 07:45:11 <16396> at async CronSchedule.httpsdHealthCheck (/node-scripts/chunk-449c6eed24
853: 2024-07-30 07:45:11 0ab919355e.js:4:477006)
854: 2024-07-30 07:45:11 <16396> at async Cron._trigger (/node-scripts/chunk-0238041ac4439f9b2c08.js:4:4
855: 2024-07-30 07:45:11 8619)
856: 2024-08-14 10:48:30 the killed daemon is /bin/sflowd: status=0x0
857: 2024-08-14 13:03:49 the killed daemon is /bin/sflowd: status=0x0
858: 2024-08-14 20:24:46 the killed daemon is /bin/iked: status=0x0
Crash log interval is 3600 seconds
Max crash log line number: 16384

The only thing I can imagine is some kind of issue with SSL-VPN which was active on the units until I upgraded to 7.6.0 (which in fact removes SSL-VPN). Now I'm waiting to see if the 7.6.0 upgraded models craps out.

Is anyone experiencing this kind of behaviour? I'd like to know before disemminating the problem further.

Posted Aug 15, 2024, 8:04 AM UTC

68
Loading comments…