Credited Vs Uncredited Buffers
One of the biggest issues we ran into in 2017 was finding out that Multicast was buffered differently on the Cisco Nexus platform. This took a while to troubleshoot because you really have to know where to look to find a problem.
When Cisco came out with the F3 cards we jumped all over those for the Nexus 7700 and 7000 platform because they were cheaper and promised to be line rate. Which they are . . . if you are using TCP based applications. Those applications go into a credited buffer which was large. Unfortunately multicast video being UDP based went into the MUCH smaller uncredited buffer. On the 7700 I believe the uncredited buffer was 512k. When we tried to push too much multicast out the Asic egress buffer would basically start dropping packets. Unfortunately finding that isn’t easy. You can’t just do a show interface to find the problem.
For the Nexus 7700 if you are running multiple VDC’s you need to log into the admin VDC and run the following command on your module that has the potentially overloaded ASIC to see if you are having drops.
show hardware queuing drops egress module 1
The above image shows a drop that occurred at some point in the egress buffers. This could have happened at any point. To make sure that these are still going on and causing a problem run the command again to see if you are still seeing the drops continue to increment. If it is you definitely have a problem on the asic groups that are being shown.
There is also a difference in how layer 2 vs layer 3 traffic is handled by the egress buffer. When it is a layer 2 VLAN you may face a replication issue where you have even less bandwidth to play with then you would if it was a layer 3 interface. In general on these asics I think we were able to get about 60 Gigs of traffic out in total when dealing with layer 3 interfaces, but far less when it was all layer 2. On a 20 gig port channel due to replication issues we were only able to do 5 gigs of traffic before we started having some problems.
In general its my opinion that if you are doing lots of Multicast on a Nexus 7k platform, stay away from the F3 line card. We learned that the hard way.