SDN : Mismatch AFI with BGP controller IPv6 session?

Hello,

I try to setup a IPv6 plage into my EVPN zone (shared into 3 PVE in my cluster).
I configure the IPv6 subet on the Vnet and it's working. I can ping the gateway from VM/LXC with this Vnet associted.

My 3 PVEs are exit-node and share prefixes with my network via a BGP controller (to route incoming traffic to the exit-node and enter the EVPN).
I've added an IPv6 session to the BGP controllers and while the GUI accepts it, the configuration generated behind it doesn't seem to take IPv4/IPv6 differences into account.

It's impossible to connect the BGP sessions with my Juniper, because the AFI proposed by PVE is IPv4 unicast when I'm expecting IPv6 unicast.

Code:

Dec 18 09:53:36  re1.routeur.par rpd[21588]: BGP_NLRI_MISMATCH: bgp_process_caps: mismatch NLRI with 2001:XXXX:2:103::80:2 (External AS YYYY): peer: <inet-unicast>(1) us: <inet6-unicast>(16)
Dec 18 09:53:36  re1.router.par rpd[21588]: bgp_process_caps:3907: NOTIFICATION sent to 2001:XXXX:103::80:2 (External AS YYYY): code 2 (Open Message Error) subcode 7 (unsupported capability) MP capability afi 1, safi 1 <inet-unicast>

After looking at the FRR configuration, I think it could be that it is placed in the “BGP” group and only seems to be defined in “ipv4 unicast”.
Note : Some particular configurations (route-map on session) come from my frr.conf.local

Code:

router bgp YYYY
 bgp router-id X.Y.Z.A
 no bgp default ipv4-unicast
 bgp disable-ebgp-connected-route-check
 coalesce-time 1000
 bgp bestpath as-path multipath-relax
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor BGP ebgp-multihop 10
 neighbor VTEP peer-group
 neighbor VTEP remote-as 31367
 neighbor VTEP bfd
 neighbor VTEP update-source A.B.C.D
 neighbor X.Y.A.IPv4 peer-group BGP
 neighbor X.Y.B.IPv4 peer-group BGP
 neighbor 2001:XXXX:2:103::80:1 peer-group BGP <- IPv6 new session
[...]
 neighbor 172.31.0.2 peer-group VTEP
 neighbor 172.31.0.3 peer-group VTEP
 !
 address-family ipv4 unicast
  network 172.31.0.1/32
  network X.Y.Z.A
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  neighbor BGP allowas-in
  neighbor X.Y.A.IPv4 route-map PUBLIC_RING0_IN in
  neighbor X.Y.B.IPv4 route-map PUBLIC_RING1_IN in
  neighbor 172.31.1.2 route-map EVPN_RING0_IN in
  neighbor 172.31.2.2 route-map EVPN_RING1_IN in
[...]
  import vrf vrf_public
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_public
 exit-address-family
 !

Result :

Code:

BGP neighbor is 2001:XXXX:2:103::80:1, remote AS AAAA, local AS YYYY, external link
  Local Role: undefined
  Remote Role: undefined
 Member of peer-group BGP for session parameters
  BGP version 4, remote router ID Y.X.Z.Z, local router ID Y.X.Z.A
  BGP state = Idle
  Last read 18:35:19, Last write 00:01:09
  Hold time is 9 seconds, keepalive interval is 3 seconds
  Configured hold time is 9 seconds, keepalive interval is 3 seconds
  Configured conditional advertisements interval is 60 seconds
  Graceful restart information:
    Local GR Mode: Helper*

    Remote GR Mode: NotApplicable

    R bit: False
    N bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 120
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                563        563
    Notifications:        563          0
    Updates:                0          0
    Keepalives:             0          0
    Route Refresh:          0          0
    Capability:             0          0
    Total:               1126        563
  Minimum time between advertisement runs is 0 seconds

 For address family: IPv4 Unicast <- IPv4 Unicast
  BGP peer-group member
  Not part of any update group
  Inbound soft reconfiguration allowed
  Local AS allowed in path, 3 occurrences
  Community attribute sent to this neighbor(all)
  0 accepted prefixes

  Connections established 0; dropped 0
  Last reset 18:35:19,  Notification sent (OPEN Message Error/Unsupported Capability)
  Message received that caused BGP to send a NOTIFICATION:
    FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
    003F0104 201A005A 53A73F12 22020601
    04000200 01020280 00020202 00020440
    02407802 06410400 00201A02 024700
  External BGP neighbor may be up to 10 hops away.
Local host: 2001:1b48:2:103::80:2, Local port: 38180
Foreign host: 2001:1b48:2:103::80:1, Foreign port: 179
Nexthop: Y.X.Z.Z <- Next-hop IPv4
Nexthop global: 2001:XXXX:2:103::80:2 <- Next-hop IPv6
Nexthop local: fe80::1270:fdff:fe85:b2da
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 10
Next start timer due in 51 seconds
Read thread: off  Write thread: off  FD used: -1
  Capability error for: Multi protocol AFI IPv6, SAFI Unicast

  BFD: Type: multi hop
  Detect Multiplier: 3, Min Rx interval: 300, Min Tx interval: 300
  Status: Unknown, Last update: never

I'm not too comfortable with FRR groups yet, so I don't know if adding some "neighbor BGP activate" in unicast IPv6 will make it switch automatically, or if I have to manually create an IPv6 group and match it in unicast ipv6 (that it advertises/receives the right prefixes).

I don't know if it's a limitation of the current design (case not too expected) or if it's a bug.
If anyone has a tip, otherwise I would probably do a BGP6 group manually via my frr.conf.local, hoping it works well with the PVE parser

Thanks!

An update : I managed to get my route exchange to work by creating a new group dedicated to IPv6
Note : Of course, I have removed the IPv6 neighbor in the BGP controller in GUI before.

/etc/frr/frr.config.local

Code:

router bgp XXXX
 neighbor BGP6 peer-group
 neighbor BGP6 remote-as external
 neighbor BGP6 bfd
 neighbor BGP6 ebgp-multihop 10
 neighbor 2001:XXXX:103::80:1 peer-group BGP6
 address-family ipv6 unicast
  neighbor BGP6 activate
  neighbor BGP6 soft-reconfiguration inbound
 exit-address-family
exit

Code:

BGP neighbor is 2001:XXXX:2:103::80:1, remote AS AAAA, local AS YYYY, external link
  Local Role: undefined
  Remote Role: undefined
 Member of peer-group BGP6 for session parameters
  BGP version 4, remote router ID Y.X.Z.Z, local router ID Y.X.Z.A
  BGP state = Established, up for 02:38:02
  Last read 00:00:02, Last write 00:00:02
  Hold time is 9 seconds, keepalive interval is 3 seconds
  Configured hold time is 9 seconds, keepalive interval is 3 seconds
  Configured conditional advertisements interval is 60 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    Extended Message: advertised
    AddPath:
      IPv6 Unicast: RX advertised
    Long-lived Graceful Restart: advertised and received
      Address families by peer:
    Route refresh: advertised and received(old & new)
    Enhanced Route Refresh: advertised
    Address Family IPv6 Unicast: advertised and received
    Hostname Capability: advertised (name: prox1,domain name: n/a) not received
    Graceful Restart Capability: advertised and received
      Remote Restart timer is 120 seconds
      Address families by peer:
        none
  Graceful restart information:
    End-of-RIB send: IPv6 Unicast
    End-of-RIB received: IPv6 Unicast
    Local GR Mode: Helper*

    Remote GR Mode: Helper

    R bit: False
    N bit: True
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 120
    IPv6 Unicast:
      F bit: False
      End-of-RIB sent: Yes
      End-of-RIB sent after update: No
      End-of-RIB received: Yes
      Timers:
        Configured Stale Path Time(sec): 360
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  1          1
    Notifications:          0          0
    Updates:                6          2
    Keepalives:          3161       3476
    Route Refresh:          0          0
    Capability:             0          0
    Total:               3168       3479
  Minimum time between advertisement runs is 0 seconds

 For address family: IPv6 Unicast
  BGP6 peer-group member
  Update group 2, subgroup 2
  Packet Queue length 0
  Inbound soft reconfiguration allowed
  Community attribute sent to this neighbor(all)
  1 accepted prefixes

  Connections established 1; dropped 0
  Last reset 02:38:06,  Waiting for peer OPEN
  External BGP neighbor may be up to 10 hops away.
Local host: 2001:XXXX:2:103::80:2, Local port: 48074
Foreign host: 2001:XXXX:2:103::80:1, Foreign port: 179
Nexthop: Y.X.Z.Z
Nexthop global: 2001:XXXX:2:103::80:2
Nexthop local: fe80::1270:fdff:fe85:b2da
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 10
Estimated round trip time: 100 ms
Read thread: on  Write thread: on  FD used: 34

  BFD: Type: multi hop
  Detect Multiplier: 3, Min Rx interval: 300, Min Tx interval: 300
  Status: Down, Last update: 0:02:38:02

The only funny thing I see is that there's still an IPv4 next-hop and a version of BGP v4 in this show. But at least the family is the right one.
So it seems that it's more a question of Proxmox's configuration generation, which accepts IPv6, but doesn't seem to handle it correctly/isn't aware that it needs to handle it differently.

Hi,
for the evpn peers,
currently, we can't haveipv6 vxlan tunnel for evpn (technically possible, but ifupdown2 which manage /etc/network/interfaces don't manage it).

But for bgp peers, I think it's a bug in config generation.

I'l test with splitting ipv4/ipv6 peers group like your configuration, thanks for sharing !

I was doing some test on my side, and if I define ipv6 bgp peers, the current code seem also to use ipv6 for vxlan tunnel.

can you check in /etc/network/interfaces.d/sdn ,

the value:

vxlan-local-tunnelip ....

it's should be ipv4 (like the evpn peers), as ipv6 is not supported currently.

if you want to test, I can send you a .deb

I can do the test on my own to see if there's any different behavior/side effects between the new generated configuration and the old one+the one that's currently working. Feel free to send me the .deb url in DM or on Discord

Edit : Deleted my next message. I misread the patch with a night of maintenance o_O

Last edited: Dec 20, 2024

Hello,

Do you know when the patch will be commit?
I checked and seems be not merged at now on PVE 8.3.4 :-)

Just to be know when I need to check & cleanup my configuration.
Thanks!

Hi, I would like to ask if anyone knows the recent status of this issue.
Activating the peer-group BGP for address-family ipv6 unicast solves the problem.
However, once you apply the SDN config in the GUI, the settings are regenerated/reverted.
Thanks and best regards
Wolfgang

We have found a bug with loopback with the patch I have submitted, I need to fix it first before applying it.

Thanks @spirit!
Please let me know when you pushed the fix to the community no-subscription repo, I'll check it then.

@wolfgang-h2JEqEz : If you really need IPv6 in the meantime, here's a configuration you can add to /etc/frr/frr.conf.local
This file is automaticaly read and apply on FRR configuration generated by Proxmox (like a "merge") during SDN regeneration.

Activation of ipv6 unicast on BGP group (not tested on my side, but it should be work):

Code:

router bgp XXXX
 address-family ipv6 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
 exit-address-family
exit

Separation IPv4/IPv6 sessions (tested and used on my clusters):

Code:

router bgp XXXX
 neighbor BGP6 peer-group
 neighbor BGP6 remote-as external
 neighbor BGP6 bfd
 neighbor BGP6 ebgp-multihop 10
 neighbor 2001:XXX:YYY::1 peer-group BGP6
 address-family ipv6 unicast
  neighbor BGP6 activate
  neighbor BGP6 soft-reconfiguration inbound
 exit-address-family
exit

The session won't be visible in the GUI, and you'll probably have to remove this patch when it's available, but in the meantime it allows you to have an IPv6 session between your host and your router.

Last edited: Mar 24, 2025

@wolfgang-h2JEqEz : If you really need IPv6 in the meantime, here's a configuration you can add to /etc/frr/frr.conf.local
This file is automaticaly read and apply on FRR configuration generated by Proxmox (like a "merge") during SDN regeneration.
Activation of ipv6 unicast on BGP group (not tested on my side, but it should be work):
Code:
router bgp XXXX
 address-family ipv6 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
 exit-address-family
exit
Separation IPv4/IPv6 sessions (tested and used on my clusters):
Code:
router bgp XXXX
 neighbor BGP6 peer-group
 neighbor BGP6 remote-as external
 neighbor BGP6 bfd
 neighbor BGP6 ebgp-multihop 10
 neighbor 2001:XXX:YYY::1 peer-group BGP6
 address-family ipv6 unicast
  neighbor BGP6 activate
  neighbor BGP6 soft-reconfiguration inbound
 exit-address-family
exit
The session won't be visible in the GUI, and you'll probably have to remove this patch when it's available, but in the meantime it allows you to have an IPv6 session between your host and your router.

Thanks for your instructions. Combining IPv4 and IPv6 neighbours in the same peer-group works.

I have a similar problem, but with ipv4.

I've setup a test cluster, containing five pve nodes, connected via evpn ::100 - ::500). A bgp controller is being used to connect the cluster to two external routers ::a, ::b)

SDN controllers:

Code:

evpn: evpn
        asn 65000
        bgp-mode internal
        nodes pve1,pve2,pve3,pve4,pve5
        peers 2001:db8:10::100,2001:db8:10::200,2001:db8:10::300,2001:db8:10::400,2001:db8:10::500

bgp: bgppve1
        asn 65000
        node pve1
        peers 2001:db8:10::a,2001:db8:10::b
        bgp-multipath-as-path-relax 0
        ebgp 0

bgp: bgppve2
        asn 65000
        node pve2
        peers 2001:db8:10::a,2001:db8:10::b
        bgp-multipath-as-path-relax 0
        ebgp 0

bgp: bgppve3
        asn 65000
        node pve3
        peers 2001:db8:10::a,2001:db8:10::b
        bgp-multipath-as-path-relax 0
        ebgp 0

bgp: bgppve4
        asn 65000
        node pve4
        peers 2001:db8:10::a,2001:db8:10::b
        bgp-multipath-as-path-relax 0
        ebgp 0

bgp: bgppve5
        asn 65000
        node pve5
        peers 2001:db8:10::a,2001:db8:10::b
        bgp-multipath-as-path-relax 0
        ebgp 0

Generated frr configuration

Code:

!
frr version 10.6.1
frr defaults datacenter
hostname pve2
log syslog informational
service integrated-vtysh-config
!
ip prefix-list only_default seq 1 permit 0.0.0.0/0
!
ipv6 prefix-list only_default_v6 seq 1 permit ::/0
!
route-map MAP_VTEP_IN deny 1
 match ip address prefix-list only_default
exit
!
route-map MAP_VTEP_IN deny 2
 match ipv6 address prefix-list only_default_v6
exit
!
route-map MAP_VTEP_IN permit 3
exit
!
route-map MAP_VTEP_OUT permit 1
exit
!
vrf vrf_evpn
 vni 10000
exit-vrf
!
router bgp 65000
 bgp router-id 17.77.185.189
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor BGP peer-group
 neighbor BGP remote-as 65000
 neighbor BGP bfd
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65000
 neighbor VTEP bfd
 neighbor 2001:db8:10::a peer-group BGP
 neighbor 2001:db8:10::b peer-group BGP
 neighbor 2001:db8:10::100 peer-group VTEP
 neighbor 2001:db8:10::300 peer-group VTEP
 neighbor 2001:db8:10::400 peer-group VTEP
 neighbor 2001:db8:10::500 peer-group VTEP
 use-underlays-nexthop-weight
 !
 address-family ipv4 unicast
  import vrf vrf_evpn
 exit-address-family
 !
 address-family ipv6 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_evpn
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  neighbor VTEP route-map MAP_VTEP_IN in
  neighbor VTEP route-map MAP_VTEP_OUT out
  advertise-all-vni
 exit-address-family
exit
!
router bgp 65000 vrf vrf_evpn
 bgp router-id 17.77.185.189
 no bgp hard-administrative-reset
 no bgp graceful-restart notification
 use-underlays-nexthop-weight
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
exit
!
end

As you can see, this works for ipv6, but the peer-group bgp isn't being activated for ipv4 unicast.

I can fix this in vtysh using the following config:

Code:

router bgp 65000
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
 exit-address-family
exit

So I tried to add the above to /etc/frr/frr.conf.local, but after restarting the service it hasn't been added to the configuration. Even restarting the whole node didn't merge the configurations properly.

Is the /etc/frr/frr.conf.local still being read and apply to the frr configuration?

Last edited: 8 minutes ago

推荐订阅源

Proxmox Support Forum