Mesh to Leaf/Spine Migration

A simple approach to migrating from a mesh architecture to a leaf/spine architecture.

This post outlines a network migration I did at a large university. This migration was the “core refresh” component of a larger network refresh effort.

Current State - Partial Mesh

The legacy meshed network was built with simplicity in mind – OSPF for generic transport, and VPLS for ad-hoc tunneling to a central firewall (a topic for another post). The VPLS component was an opt-in security service provided to university departments.

There were four core routers on main campus and two more in a (nearby) remote campus.

• Buildings A and B each represent a group of ~100 building edge routers that are dual-homed to a pair of Core routers. Building C represents a group of ~20 building edge routers.
• Cores 5 and 6 are single homed to the mesh due to fiber constraints.
• Other network modules like Data Center, Wireless, and VoIP, are dual homed to Cores 1 and 3.
• Cores are high density 10g/1g Brocade MLX-12s. All transit interfaces are 10g.

Final State - Leaf Spine

We wanted to keep things simple. Only use what you need, nothing more (12 networking truths, #12). We decided to keep the OSPF underlay for generic transport, move to single area 0, and do overlay tunneling with iBGP/EVPN. The building edge routers were EVPN VTEPs. All transits were 100g and OSPF reference bandwidth was set to 400g.

The method we used for migrating the overlays from VPLS to EVPN is outside the scope of this post.

Migration Strategy

One of our main requirements around this network refresh was minimal network impact. Thankfully, most interconnects were L3 and dual-homed… but how did we move active building routers to the new core infrastructure while maintaining connectivity to all other network tiers?

We simply stood up the new gear in parallel to the legacy gear and directly interconnected the two core networks.

In this drawing, we have the mesh portion of the legacy network on bottom and the new leaf spine network on top. Notice the building edges A and B are still dual homed to their respective legacy cores. The most important links in this drawing are the transits between each legacy core and their links to the new spines. Let’s label these links “infrastructure transits” and focus on them for a moment.

I wanted to get the new stuff online before actually moving traffic through it. To do this, the infrastructure transits needed to be less preferred than existing traffic paths. We were using OSPF as our IGP; therefore, I turned the OSPF cost up on both sides of these links to 1000. The high cost guaranteed no east/west traffic would flow through the new core infrastructure. This gave us plently of time to import the gear into our monitoring systems and watch them for a couple of weeks.

After this initial two week period, we were ready to move forward with pushing all east-west traffic through the new gear. The method? Change OSPF cost on the infrastructure transits from 1000 to 4. This creates a new best east/west path through the leaf/spine network. The path from Building-Edge-A to Building-Edge-B is now traversing the new cores (see the path arrows).

The only remaining work was to cut existing buildings and network tiers to the new gear! Building routers were moved to the new spines (aggregation layer) and other tiers moved directly to the super spines (core layer). Since everything was reachable from anywhere on the network during the transition, the cuts could take place at a reasonable pace. No long weekends for teams trying to move hundreds of links at once.

There are factors to keep in mind for this type of transition. The first is OSPF’s reference bandwidth – we used the same value (400g) amongst the integrated core environments to make traffic patterns predictable. Second, be cognizant of encapsulation overhead if you aren’t already doing jumbo frames in the core. For example, if VPLS traffic will be flowing from building A to building B through the new infrastructure, you must allow for the MPLS overhead on all off of those links; otherwise, you’ll get packet drops. Lastly, try this in a lab first! I spent many hours ironing out this process in our on-prem lab. Having a good lab is priceless.

Overall, the project was a success, and I’d do it the same way again.