#Vmware Esxi 6.7 Nic Teaming Arp Driver Control Plane

It seems to me that Kubernetes has finally reached the momentum and everybody isIn a NIC teamed environment where multiple uplinks are configured for a virtual switch and a port channel or LACP is not configured on the physical switch, the vSwitch will receive a multicast or broadcast packet from the physical network on each vSwitch uplink in the NIC team.Trying to jump into the wagon. In vSphere Distributed Switch you can configure the NIC Teaming on the Port Group level to provide the redundancy of the Uplinks.Today I will write a first blog about Kubernetes. NIC Teaming groups the multiple NIC into one single logical NIC which will help when one NIC fails to load or network fails.

Nevertheless, if you believe there is something totally wrong, speak up in comments below the article.First thing first, I'm the visual guy therefore let's start with overall solution architecture.The DevOps process to create persistent volume is following In this article, I would like to describe the architecture overview of vSphere CSI Driver and some process flow behind the scene.Disclaimer: Please note that this is just my personal understanding how it works and some things can be inaccurate or at very high detail. Therefore, if you want to get some info about VMware Tanzu, I highly recommend Cormac's blog which is available at. Cormac Hogan is my favorite Tanzu/Kubernetes blogger, probably because in the past he was blogging about vSphere and storage related topics. I do not want to describe Tanzu platform from product perspective because there are plenty of such blog posts across the blogosphere. VMware hasRecognized this trend very early and integrated Kubernetes into VMwareVSphere platform, also known as Tanzu.

Vmware Esxi 6.7 Nic Teaming Arp Driver Control Plane

CSI driver control plane via vCenter API asks vSphere to create storage volume. CSI driver control plane is integrated with vCenter server via vSphere API DevOps Admin request (claim) of persistent volume is sent to CSI driver control plane CSI driver has control plane in K8s supervisor and CSI Driver agents on all K8s worker nodes

Note 2: CSI driver can add additional PVSCSI adapters to VM dynamically Note 1: block persistent volumes are attached to virtual machines via PVSCSI driver as it supports higher number (64) of disks and as virtual machine supports up to four (4) SCSI adapters, single VM (K8s worker node) can have up to 256 volumes. And it also knows in which K8S Worker Node (linux guest os on top of virtual machine) the K8s pod is running, therefore, it dynamically attach the volume (it leverages hot-plug/hot-add capability) to particular virtual machine. CSI driver control plane can leave such storage volume unattached (aka FCD - First Class Disk) or it can attach the storage volume into particular ESXi host because eventually it knows into which K8s pod (container) such volume should be attached. vCenter will create such storage volume via some ESXi host

Storage Class is just a mapping between vSphere Storage Policy and K8s Storage Class object (aka kind). The first of all, K8s cluster must know K8s Storage Class which is later used for persistent volume claims. Finally, CSI driver agent detects new storage volume within K8s worker node (linux guest os) and because it knows into which K8s pod (linux container / chroot) the particular volume should be attached, it will attach it to the desired container (pod).Hope I did not forget something in the automated workflow vSphere CSI driver is doing :-)I guess now you would ask me, how DevOps admin issues persistent volumes claims into K8s cluster, right?Well, it is two step process.

original ESXi host, where VM was running before the restart, was isolated (network isolation)Well, for those understanding how vSphere HA Cluster works it is pretty simple diagnosis. VM was restarted in different ESXi host After further troubleshooting, they realized their particular NICs (Network Adapters) do not propagate link down state to the physical link, when vmnic is administratively disabled by command ' esxcli network nic down -n vmnic2'.It is worth to mention, that such network outage does not mean 2-node vSAN outage because that's the reason why we have vSAN witness, however, vSAN is in degraded state and cannot provide mirror (RAID1) protection of vSAN objects.Such network behavior is definitely strange and we have opened discussion and root cause analysis with hardware vendor, however, we have also started the internal discussion about design alternatives we have to mitigate such weird situations and increase resiliency and the overall availability of vSAN system.Here are three design options how to implement direct connect networking between two ESXi hosts.Design Option 1 - Switch independent teaming with explicit fail-overI've just finished a root cause analysis of VM restart in customer production environment, so let me share with you the symptoms of the problem, current customer's vSphere design and recommended improvement to avoid similar problems in the future.After the further discussion with customer we have identified following symptoms: The resiliency test was done by administrative shutdown of one vmnic (physical NIC port) on one vSAN node. Generally they have very good experience with vSAN, but recently they have experienced vSAN Direct Connect outages when testing the network resiliency. One of many reasons of using 2-node vSAN is the possibility to leverage existing 1 Gb network and use 25 Gb Direct Connect between ESXi hosts (vSAN nodes) without the need of 25 Gb Ethernet switches.

) and Fault Domain Manager (aka FDM) log message about the host isolation and VM restart.Cool, we know the Root Cause, but what options do we have to avoid such situation?Well, the issue described above is called Network Port Flapping and in such single port issue, in our case with vmnic3, the vmk3 (vSAN, HA heart beat) interface was originally pinned to vmnic3 and when vmnic3 went down, vmk3 was failed over from vmnic3 to vmnic1. We have found vmnic3 flapping (link up, down, up, down. customer has vSAN, which is a IP storage, therefore such setting makes perfect senseThat having said, this was the reason VM was restarted and it is expected behavior to achieve higher VM availability in cost of some small unavailability because of VM restart.Why was ESXi isolated from network when there is network teaming (vmnic1 + vmnic3) configured?The customer environment is depicted on design drawing below.Customer is using VMware LogInsight (syslog + data analytics) for central log management, therefore troubleshooting was a piece of cake. this is recommended setting for IP storage, because when network is not available, there is a huge probability, the storage is not available and VM is in trouble

Some switches supports profile configuration which can have positive impact on manageability. Source: Each option above has their own benefits and drawbacksLet's go option by option and discuss pluses and minuses.Option 1: Physical Ethernet switch Link Dampening+ per physical switch port setting, therefore not too much places to set, but still some effort. I was blogging about ' Dell Force10 Link Dampening' few years ago, which should help in these situations.(2) There is VMware vSwitch 'Teaming and failover' option Failback=No available through GUI.(3) And there is ESXi advanced setting 'Net.teampolicyupdelay' which is something like 'Link Dampening' described above.