r/Juniper•6 yr. ago

SRX cluster problems. Im sure someone has seen this before

Hi, before i start off, I'd like to say these boxes are out of support and end of life and i have no help from juniper so I can't call them. I'm really hoping someone here can shed some light on this issue.

I'm bringing up a new SRX to the cluster.

I've followed the steps from the juniper KB and everything goes smooth. until the very end.

https://kb.juniper.net/InfoCenter/index?page=content&id=KB21134&actp=METADATA

After the cluster comes up, i get this strange dump on the secondary and it causes the cluster to break. a few seconds later the cluster comes back up, and then a few seconds after that it breaks again with the same dump. it just loops. Please see my screenshot below.

the black screenshot is from the secondary console. The green is the ssh output from the primary.

https://imgur.com/a/o20ui03

I've been dealing with this all night, so i'm going to go to bed. I'm really hoping i wake up to some awesome news. Sorry in advance if i don't up-vote you or reply to you tonight. Also, if your company gets your proper juniper support, give them a hug tomorrow, because some of us out here are all alone :{

Posted Jan 22, 2020, 6:58 AM UTC

112

Sort by:

shedgehog•6 yr. ago

Hard to say without seeing a full config plus some more show commands. What type of SRXs are they? What version of Junos?

Here’s some thoughts though:

Make sure both nodes are running the same Junos version
how are they connected together? You need at least two ports for chassis cluster and control. They should be directly connected , and some SRXs have dedicated ports for these.
lost status generally indicates a hardware issue. Try this https://kb.juniper.net/InfoCenter/index?page=content&id=KB20672&actp=METADATA
post your config here also

Along with:

show chassis cluster interfaces
show chassis cluster information
show configuration chassis cluster

DaggerHashimotoOP•6 yr. ago

Hard to say without seeing a full config plus some more show commands. What type of SRXs are they? What version of Junos?

Model: srx240h2

JUNOS Software Release [12.3X48-D80.4]

I verified that they are both the same model and running the same firmware.

>how are they connected together? You need at least two ports for chassis cluster and control. They should be directly connected , and some SRXs have dedicated ports for these.

0/0/0/ is fxp0 mgmt

0/0/1 if fabric

0/0/2 is control

0/0/04 - 6 are the trunk ports

show chassis cluster interfaces
Control link status: Down

Control interfaces:
    Index   Interface   Monitored-Status   Internal-SA
    0       fxp1        Up                 Disabled

Fabric link status: Down

Fabric interfaces:
    Name    Child-interface    Status
                               (Physical/Monitored)
    fab0    ge-0/0/2           Down / Down
    fab0

Redundant-ethernet Information:
    Name         Status      Redundancy-group
    reth0        Up          1
    reth1        Down        Not configured
    reth2        Down        Not configured
    reth3        Down        Not configured
    reth4        Down        Not configured
    reth5        Down        Not configured
    reth6        Down        Not configured
    reth7        Down        Not configured
    reth8        Down        Not configured
    reth9        Down        Not configured

Redundant-pseudo-interface Information:
    Name         Status      Redundancy-group
    lo0          Up          0

Interface Monitoring:
    Interface         Weight    Status    Redundancy-group
    ge-5/0/4          255       Down      1
    ge-5/0/5          255       Down      1
    ge-5/0/6          255       Down      1
    ge-0/0/6          255       Up        1
    ge-0/0/5          255       Up        1
    ge-0/0/4          255       Up        1

This is the output from the primary node 0 and secondary node 1 is currently isolated and

show chassis cluster information
node0:
--------------------------------------------------------------------------
Redundancy Group Information:

    Redundancy Group 0 , Current State: primary, Weight: 255

	Time            From           To             Reason
        May  4 04:40:44 hold           secondary      Hold timer expired
        May  4 04:41:03 secondary      primary        Better priority (100/1)

    Redundancy Group 1 , Current State: primary, Weight: 255

	Time            From           To             Reason
        May  4 04:40:44 hold           secondary      Hold timer expired
        May  4 04:41:03 secondary      primary        Remote yield (0/0)

Chassis cluster LED information:
    Current LED color: Red
    Last LED change reason: Peer node: node1 is not present
Control port tagging:
    Disabled

Failure Information:

    Fabric-link Failure Information:
        Fabric Interface: fab0
          Child interface   Physical / Monitored Status
          ge-0/0/2              Down / Down

    Control-link Failure Information:
        Link Status: Monitor Failed
        Dual Control Link Status: Unsupported

        Interface        Physical / Monitored Status
           fxp1              Up   / Up

{primary:node0}



show configuration chassis cluster
reth-count 10;
redundancy-group 0 {
    node 0 priority 100;
    node 1 priority 1;
}
redundancy-group 1 {
    node 0 priority 100;
    node 1 priority 1;
    preempt;
    interface-monitor {
        ge-0/0/4 weight 255;
        ge-0/0/5 weight 255;
        ge-0/0/6 weight 255;
        ge-5/0/6 weight 255;
        ge-5/0/5 weight 255;
        ge-5/0/4 weight 255;
    }
}

{primary:node0}

The config should be the same as I copied the latest version of the config and placed in on the secondary /var/tmp and load override it so they should match.

Thank you so much for your attempt to help.

dwolcot1JNCIP•6 yr. ago

Shouldn't you have 2 fab interfaces? One for each node? I have fab0 for node 9 and fab1 for node 1 assigned to interfaces.

DaggerHashimotoOP•6 yr. ago

you only need one fab interface as far as i'm aware. This config was working before the unit failed originally and had to be replaced.

studiox_swe•6 yr. ago

Model: srx240h2

This will not solve the problem but it looks like this is a PoC/Lab Enviroment right? Or surplus wanna-learn?

Have you read the deployment guides for the setup? Some of your interfaces are down. Did you run the first chassi cluster commands and rebooted both nodes?

DaggerHashimotoOP•6 yr. ago

I followed the instruction for replacing a RMA srx into the cluster. I tried it with my lab setup and everything went smooth. Currently the interfaces are down because the secondary unit is powered off as it was joining and leaving the cluster every minute.

shedgehog•6 yr. ago

Fabric interfaces:
Name Child-interface Status
(Physical/Monitored)
fab0 ge-0/0/2 Down / Down
fab0

there's your problem... fix that :)

DaggerHashimotoOP•6 yr. ago

i think that's because currently the secondary is powered off and not connected. fab and control ports are disconnected at the moment.

mydogfartsa-lot•6 yr. ago

That particular message happens during/right after Junos kicks off a bunch of processes. We typically notice it when our interfaces are coming up right after boot. Could be that your reth interfaces are coming trying to come online and something is erroring out.

Easy things to check for:

Are they the same model device, same firmware

Post interface and cluster config

set traceoptions on the chassis cluster hierarchy and post the output

Other than the cluster config did you wipe the secondary node before trying to build the cluster? (weve have issues with two devices with different configs joining a cluster)

DaggerHashimotoOP•6 yr. ago

That particular message happens during/right after Junos kicks off a bunch of processes. We typically notice it when our interfaces are coming up right after boot. Could be that your reth interfaces are coming trying to come online and something is erroring out.

Easy things to check for:

Are they the same model device, same firmware

Hi there, thanks for your reply. Yes, they are both same model and on same firmware.

Post interface and cluster config

show configuration chassis cluster

reth-count 10;

redundancy-group 0 {

node 0 priority 100;

node 1 priority 1;

}

redundancy-group 1 {

node 0 priority 100;

node 1 priority 1;

preempt;

interface-monitor {

ge-0/0/4 weight 255;

ge-0/0/5 weight 255;

ge-0/0/6 weight 255;

ge-5/0/6 weight 255;

ge-5/0/5 weight 255;

ge-5/0/4 weight 255;

}

{primary:node0}

set traceoptions on the chassis cluster hierarchy and post the output

I would have to set that up when the cluster is in action, but right now it's disconnected. Unfortunately this pair of FW is on the other side of the world and I have to try to reconnect it again later when i get a chance with my local hands.

Other than the cluster config did you wipe the secondary node before trying to build the cluster? (weve have issues with two devices with different configs joining a cluster)

I copied the current config from primary and placed it on secondary /var/tmp and loaded it. Before that i had done a factory reset and zeroized. the only thing on there was the root password. then i did the load override. config, code, model all matched. I also verified that the PO on the switch side had come up and it was in "trunk" mode.  Then it got stuck in the loop of going down and coming back up (the cluster that is) and the secondary would spew this on console every 20 seconds or so. as soon as it would happen, the primary node 0 would show connection lost and then it would come back. everything would be fine for about 20 seconds or so and then crap out again. 



secondary:node1}
root@RKCORP-FW02>
Message from syslogd@RKCORP-FW02 at Jan 22 06:11:14  ...
RKCORP-FW02 SCHED: Thread 4 (Module Init) ran for 1639 ms without yielding

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:14  ...
RKCORP-FW02 Scheduler Oinker

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:14  ...
RKCORP-FW02 Frame 00: sp = 0x520a3550, pc = 0x2024dbb8

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:14  ...
RKCORP-FW02 Frame 01: sp = 0x520a35f8, pc = 0x20208c2c

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:14  ...
RKCORP-FW02 Frame 02: sp = 0x520a3668, pc = 0x20761b78

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:14  ...
RKCORP-FW02 Frame 03: sp = 0x520a36b8, pc = 0x204e4150

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:14  ...
RKCORP-FW02 Frame 04: sp = 0x520a3738, pc = 0x21583b94

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:14  ...
RKCORP-FW02 Frame 05: sp = 0x520a37d8, pc = 0x2077a230

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:14  ...
RKCORP-FW02 Frame 06: sp = 0x520a3820, pc = 0x2141093c

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:14  ...
RKCORP-FW02 Frame 07: sp = 0x520a4460, pc = 0x2077a01c

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:14  ...
RKCORP-FW02 Frame 08: sp = 0x520a4478, pc = 0x2077a3e0

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:14  ...
Message from syslogd@RKCORP-FW02 at Jan 22 06:11:59  ...
RKCORP-FW02 SCHED: Thread 4 (Module Init) ran for 1617 ms without yielding

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:59  ...
RKCORP-FW02 Scheduler Oinker

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:59  ...
RKCORP-FW02 Frame 00: sp = 0x520a3550, pc = 0x2024dbb8

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:59  ...
RKCORP-FW02 Frame 01: sp = 0x520a35f8, pc = 0x20208c2c

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:59  ...
RKCORP-FW02 Frame 02: sp = 0x520a3668, pc = 0x20761b78

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:59  ...
RKCORP-FW02 Frame 03: sp = 0x520a36b8, pc = 0x204e4150

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:59  ...
RKCORP-FW02 Frame 04: sp = 0x520a3738, pc = 0x21583b94

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:59  ...
RKCORP-FW02 Frame 05: sp = 0x520a37d8, pc = 0x2077a230

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:59  ...
RKCORP-FW02 Frame 06: sp = 0x520a3820, pc = 0x2141093c

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:59  ...
RKCORP-FW02 Frame 07: sp = 0x520a4460, pc = 0x2077a01c

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:59  ...
RKCORP-FW02 Frame 08: sp = 0x520a4478, pc = 0x2077a3e0

Message from syslogd@RKCORP-FW02 at Jan 22 06:11:59  ...

{secondary:node1}

techyb46•6 yr. ago

I would make sure the 2 nodes are the same hardware. I've seen clusters not able to form because one member has more memory than the other.

DaggerHashimotoOP•6 yr. ago

They sure are. same model and same code.