1 / 80

Top Support issues and how to solve them – Part II

Darren Burnett Senior Technical Support Engineer. Top Support issues and how to solve them – Part II. Unable to connect to the Service Console. Rebuild Networking. Network Connection Problem. Deleting the vSwitch that vSwif0 is connected Connecting the wrong NICs to vSwitch0 Upgrade issues

knox
Download Presentation

Top Support issues and how to solve them – Part II

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Darren Burnett Senior Technical Support Engineer Top Support issues and how to solve them – Part II

  2. Unable to connect to the Service Console

  3. Rebuild Networking Network Connection Problem. Deleting the vSwitch that vSwif0 is connected Connecting the wrong NICs to vSwitch0 Upgrade issues Incorrect IP Address External Network Changes

  4. Rebuild Networking At this stage you can no longer connect to your ESX server using VI Client or SSH. You can connect to the Service Console remotely if you have ILO, DRAC an IP KVM or something similar. Otherwise, it’s time to use some shoe leather and walk to the server room.

  5. Rebuild Networking The following procedure will work. However it is a quick and inelegant way to get your VI client connected. Other options include, Crossover cable connected to a laptop Adding or removing NICs to a vSwitch

  6. Rebuild Networking Use the esxcfg-vswitch –l command to list all of you vSwitches

  7. Rebuild Networking [root@newross root]# esxcfg-vswitch -l Switch Name Num Ports Used Ports Configured Ports Uplinks vSwitch0 32 4 32 vmnic0 PortGroup Name Internal ID VLAN ID Used Ports Uplinks VM Network portgroup1 0 0 vmnic0 Service Console portgroup0 0 1 vmnic0 VMkernel portgroup7 0 1 vmnic0 Switch Name Num Ports Used Ports Configured Ports Uplinks vSwitch1 64 2 64 vmnic1 PortGroup Name Internal ID VLAN ID Used Ports Uplinks vlan100 portgroup10 100 0 vmnic1 install VLAN 310 portgroup6 0 0 vmnic1 Switch Name Num Ports Used Ports Configured Ports Uplinks vSwitch2 64 2 64 vmnic6 PortGroup Name Internal ID VLAN ID Used Ports Uplinks crossovercable portgroup9 0 0 vmnic6

  8. Rebuild Networking Delete all vSwitches esxcfg-vswitch –d vSwitch0 esxcfg-vswitch –d vSwitch1 esxcfg-vswitch –d vSwitch2

  9. Rebuild Networking Create a vSwitch esxcfg-vswitch -a vSwitch0 Create the Service Console portgroup esxcfg-vswitch -p "Service Console" vSwitch0 Add a NIC to the vSwitch esxcfg-vswitch –L vmnic0 vSwitch0 Add a vswif interface and configure esxcfg-vswif –a vswif0 -p "Service Console" i 10.10.10.3 –n 255.0.0.0

  10. Rebuild Networking Check if you can connect. Use PING both to and from the ESX server Try SSH Try VI Client

  11. Rebuild Networking If you still can’t connect. Use “esxcfg-nics –l” to list available NICs.

  12. Rebuild Networking [root@newross root]# esxcfg-nics -l Name PCI Driver Link Speed Duplex Description vmnic0 03:0c.00 e1000 Up 1000Mbps Full Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) vmnic1 03:0c.01 e1000 Down 0Mbps Half Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) vmnic8 07:00.00 tg3 Up 1000Mbps Full Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet vmnic6 08:00.00 tg3 Down 0Mbps Half Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet

  13. Rebuild Networking Remove and add NICs to vSwitch0 Remove esxcfg-vswitch –U vmnic0 vSwitch0 Add esxcfg-vswitch –L vmnic8 vSwitch0

  14. Rebuild Networking It might also need a VLAN ID esxcfg-vswitch –v 101 –p “Service Console” vSwitch0

  15. Rebuild Networking To avoid this issue, be careful when configuring the Service Console virtual NIC or its parent virtual switch property that can affect the Service Console virtual NIC connectivity, for example, the uplink. If possible, before updating the Service Console virtual NIC, create another independent working Service Console NIC so that, in the event the configuration brings the console NIC down, the second Service Console NIC is still available to repair the configuration.

  16. Rebuild Networking As stated, depending on your environment it may be easier not to delete all your configurations. It may just require reassigning a NIC or changing VLAN ID. Check the following guide. http://www.rtfm-ed.eu/docs/vmwdocs/esx3.x-vc2.x-serviceconsole-guide.pdf

  17. Network Bond NICs in a Bond not on the same Broadcast Domain

  18. Network Bond NICs in a Bond should be in the same broadcast domain. Determine what NICs are in each Bond.

  19. Network Bond [root@cork root]# esxcfg-info|grep -i -A 2 VirtualSwitchImpl \==+VirtualSwitchImpl : |----Name.........................................vSwitch0 |----Uplinks......................................vmnic0 -- \==+VirtualSwitchImpl : |----Name.........................................vSwitch1 |----Uplinks......................................vmnic4 -- \==+VirtualSwitchImpl : |----Name.........................................vSwitch2 |----Uplinks......................................vmnic1,vmnic3 --

  20. Next we need to determine which networks each NIC is connected.

  21. [root@cork root]# esxcfg-info |grep -i -B 5 hint \==+PnicImpl : |----_name..............................................vmnic3 |----_bus...............................................6 |----_slot..............................................3 |----_function..........................................1 |----Network Hint.......................................0 10.16.157.00/255.255.255.192 -- \==+PnicImpl : |----_name..............................................vmnic0 |----_bus...............................................11 |----_slot..............................................7 |----_function..........................................0 |----Network Hint.......................................0 10.16.156.00/255.255.255.00 -- \==+PnicImpl : |----_name..............................................vmnic1 |----_bus...............................................12 |----_slot..............................................8 |----_function..........................................0 |----Network Hint.......................................0 10.16.156.00/255.255.255.00

  22. HA and STP

  23. HA and STP Spanning Tree Protocol When there is a network change STP can cause a temporary outage on your Network

  24. HA An ESX Server will determine that it is isolated after 15 Seconds Depending on the “Isolation Response” that you have set, all your VMs may power down

  25. It is therefore worth checking your network to determine if STP can be configured to reduce the temporary network outage

  26. Expanding the size of a VMDK with an existing Snapshot

  27. Expanding VM with a Snapshot You can NOT expand a VM’s VMDK file while it still has snapshots. e.g. #ls * important.vmdk important-000001-delta.vmdk #vmkfstools –X 20G important.vmdk

  28. Expanding VM with a Snapshot If you do, you will now have a VM that won’t boot

  29. Expanding VM with a Snapshot Tricking ESX into seeing the expanded VMDK as the original size. In this example we have a test.vmdk that we expand from 5GB to 6GB #vmkfstools -X 6G test.vmdk

  30. Expanding VM with a Snapshot If we check test.vmdk we see # Disk DescriptorFile version=1 CID=3f24a1b3 parentCID=ffffffff createType="vmfs" # Extent description RW 12582912 VMFS "test-flat.vmdk" # The Disk Data Base #DDB ddb.virtualHWVersion = "4" ddb.geometry.cylinders = "783" ddb.geometry.heads = "255" ddb.geometry.sectors = "63" ddb.adapterType = "buslogic"

  31. Expanding VM with a Snapshot Original - RW 10485760 VMFS "test-flat.vmdk“ New - RW 12582912 VMFS "test-flat.vmdk“

  32. Expanding VM with a Snapshot If we have no “BACKUPS” how do we get the original value? #grep -i rw test-000001.vmdk RW 10485760 VMFSSPARSE “test-000001-delta.vmdk"

  33. Expanding VM with a Snapshot We change test.vmdk RW value. # Disk DescriptorFile version=1 CID=3f24a1b3 parentCID=ffffffff createType="vmfs" # Extent description RW 10485760 VMFS "test-flat.vmdk" # The Disk Data Base #DDB ddb.virtualHWVersion = "4" ddb.geometry.cylinders = "783" ddb.geometry.heads = "255" ddb.geometry.sectors = "63" ddb.adapterType = "buslogic"

  34. Expanding VM with a Snapshot Commit The snapshot(s) #vmware-cmd /pathtovmx/test.vmx removesnapshots

  35. Expanding VM with a Snapshot Grow the VMDK file #vmware-cmd –X 6GB test.vmdk

  36. Expanding VM with a Snapshot If needed add a snapshot #vmware-cmd pathtovmx/test.vmx createsnapshot <name> <description>

  37. Corrupted .VMSD file

  38. Corrupted Snapshots In this example we will deal with a corrupt .VMSD file. Let’s first look at a working .VMSD file, separated into 3 slides for the people at the back

  39. Corrupted Snapshots snapshot.lastUID = "4" snapshot.numSnapshots = "3" snapshot.current = "4" snapshot0.uid = "2" snapshot0.filename = "VC1.3to201_standard-Snapshot2.vmsn" snapshot0.displayName = "myfirst" snapshot0.description = "My first test snapshot" snapshot0.createTimeHigh = "273684" snapshot0.createTimeLow = "942632403" snapshot0.numDisks = "1" snapshot0.disk0.fileName = "VC1.3to201_standard.vmdk" snapshot0.disk0.node = "scsi0:0"

  40. Corrupted Snapshots snapshot.needConsolidate = "FALSE" snapshot1.uid = "3" snapshot1.filename = "VC1.3to201_standard-Snapshot3.vmsn" snapshot1.parent = "2" snapshot1.displayName = "second" snapshot1.description = "My second test snapshot" snapshot1.createTimeHigh = "273684" snapshot1.createTimeLow = "980947483" snapshot1.numDisks = "1" snapshot1.disk0.fileName = "VC1.3to201_standard-000001.vmdk" snapshot1.disk0.node = "scsi0:0"

  41. Corrupted Snapshots snapshot2.uid = "4" snapshot2.filename = "VC1.3to201_standard-Snapshot4.vmsn" snapshot2.parent = "3" snapshot2.displayName = "third" snapshot2.description = "My third test snapshot" snapshot2.createTimeHigh = "273684" snapshot2.createTimeLow = "1088942286" snapshot2.numDisks = "1" snapshot2.disk0.fileName = "VC1.3to201_standard-000002.vmdk" snapshot2.disk0.node = "scsi0:0"

  42. Corrupted Snapshots After corruption of the .VMSD file the file now looks like this.

  43. Corrupted Snapshots

  44. Corrupted Snapshots However we see that the snapshots still exist [root@newross VC1.3to201_standard]# ls VC1.3to201_standard-000001-delta.vmdk VC1.3to201_standard-Snapshot4.vmsn VC1.3to201_standard-000001.vmdk VC1.3to201_standard.vmdk VC1.3to201_standard-000002-delta.vmdk VC1.3to201_standard.vmsd VC1.3to201_standard-000002.vmdk VC1.3to201_standard.vmx VC1.3to201_standard-000003-delta.vmdk VC1.3to201_standard.vmxf VC1.3to201_standard-000003.vmdk vmware-1.log VC1.3to201_standard-flat.vmdk vmware-2.log VC1.3to201_standard.nvram vmware-3.log VC1.3to201_standard-Snapshot2.vmsn vmware.log VC1.3to201_standard-Snapshot3.vmsn

  45. Corrupted Snapshots At this stage rename the .VMSD file to .VMSD.OLD

  46. Corrupted Snapshots We are going to create a new .VMSD file What kind of magic is required to build a new VMSD file?

  47. Corrupted Snapshots Create another snapshot to automatically recreate a .VMSD file #vmware-cmd VC1.3to201_standard.vmx createsnapshot addedforrecovey "Hope it works"

  48. Corrupted Snapshots You wont be able to selectively rollback to a particular snapshot. You will have to commit them all.

  49. Corrupted Snapshots Commit the Snapshots #vmware-cmd VC1.3to201_standard.vmx removesnapshots All a bit too easy 

  50. Corrupted Snapshots Corrupted Snapshot

More Related