Skip to main content

Cancelling a Failed VxRail 7.0 Upgrade

We’ve all been there. You’re working to stage your VxRail upgrade and something goes terribly, horribly wrong. You’re beside yourself because you’ve got a deadline, and your upgrade is dead in the water. You’ve tried cancelling the upgrade in the GUI, but it JUST. KEEPS. FAILING.

Well, here’s what you do to get back on track.

  1. SSH into your VxRail Manager as mystic.
  2. Elevate to root using the su command.
  3. Run the following string of commands.
psql -U postgres vxrail -c "delete from system.operation_status where (state='FAILED' or state='IN_PROGRESS') and owner like 'Lcm%';"

psql -U postgres vxrail -c "update system.operation_status set state='FAILED' where owner='CustomizeComponentScan' and state='STARTED';"

echo '{"state":"NONE","deployed_for_public_api":false}' > /var/lib/vmware-marvin/bundle_state.jsonvxrail:/home/mystic # mv /var/lib/vmware-marvin/composite-upgrade.json /var/lib/vmware-marvin/composite-upgrade.json.old

curl -X GET --unix-socket /var/lib/vxrail/nginx/socket/nginx.sock "http://127.0.0.1:5000/rest/vxm/internal/lockservice/v1/lock"

curl -X POST --unix-socket /var/lib/vxrail/nginx/socket/nginx.sock "http://127.0.0.1:5000/rest/vxm/internal/lockservice/v1/lock/release"  -H "Content-Type: application/json" -d '{"locked_by": "UPGRADE"}'

systemctl restart runjars
systemctl restart vmware-marvin

The result of the command above should resemble this:

DELETE 1
UPDATE 0
mv: cannot stat '/var/lib/vmware-marvin/composite-upgrade.json': No such file or directory
{"error_code": 1001, "message": "Failed to get the lock information due to the lock does not exist."}
{"error_code": 1001, "message": "Failed to release the lock due to the lock does not exist."}

If you’re not comfortable with executing that command, open …

Read More

Cancelling a Failed VxRail 4.7 Upgrade

We’ve all been there. You’re working to stage your VxRail upgrade and something goes terribly, horribly wrong. You’re beside yourself because you’ve got a deadline, and your upgrade is dead in the water. You’ve tried cancelling the upgrade in the GUI, but it JUST. KEEPS. FAILING.

Well, here’s what you do to get back on track.

  1. SSH into your VxRail Manager as mystic.
  2. Elevate to root using the su command.
  3. Run the following string of commands.
psql -U postgres mysticmanager -c "delete from operation_status where state='FAILED';" ; psql -U postgres mysticmanager -c "delete from operation_status where state='IN_PROGRESS';" ; psql -U postgres mysticmanager -c "update virtual_appliance SET upgrade_status='HAS_NEWER' where component_id='VXRAIL_SYSTEM';" ;psql -U postgres mysticmanager -c "update virtual_appliance SET upgrade_status_response=null where component_id='VXRAIL_SYSTEM';" ;psql -U postgres mysticmanager -c "DELETE FROM composite_upgrade;" ;psql -U postgres mysticmanager -c "DELETE FROM operation_lock;" ;echo "Reset the Database Table and now will clean up LCM Folder" ;rm -rf /data/store2/lcm/* ;echo "This is above 4.7.300, so will reset the bundle_update.json" ;echo "{\"state\":\"NONE\",\"alerts\":[]}" > /var/lib/vmware-marvin/bundle_state.json ;echo "Restarting Services" ; systemctl restart vmware-marvin;systemctl restart runjars

The result of the command above should resemble this:

If you’re not comfortable with executing that command, open up a support ticket with Dell EMC …

Read More

VxRail Composite Package Fails to Upload or Unpack

We’ve all been there. You’re staging your VxRail upgrade and the composite package fails to upload or unpack. You can use the commands below to find temporary files on the VxRail Manager appliance and remove them if you don’t have enough space to unpack them during the upgrade process.

This command will find all temporary files larger than 20MB.

find /tmp -type f -size +20000k -exec ls -lh {} \; | awk '{print $9":"$5}'

These commands will give you the total space used by logs, and find all log files older than 30 days.

du -sh /var/lib/pgsql/data/log

find /var/lib/pgsql/data/log -mtime +30 -exec ls -lh {} \; | awk '{print $9":"$5}'

Once you’ve found all of the logs older than 30 days, you can use this command to remove those logs.

find /var/lib/pgsql/data/log -mtime +30 -exec rm {} \; | awk '{print $9":"$5}'

This command will find all files larger than 200MB.

find / -type f -size +200000k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'

Of course, if you’re uncomfortable doing any of this – contact Dell VxRail support and have them walk through it with you. Don’t get caught deleting something accidentally.…

Read More

How to Find a Dell EMC Model and PSNT on VxRail Appliances

You can find the Model and PSNT on the luggage tag at the front of a VxRail Appliance. If you’re looking for a way to get the Model or PSNT remotely, check out one of the following methods.

You’ll find the Model and PSNT for each VxRail Appliance in the VxRail Plugin for vCenter. Navigate to Hosts & Clusters > Cluster > Configure > VxRail > Hosts.

Finding the Model and PSNT using the VxRail Manager Console

Once you’ve connected to your VxRail Manager as mystic, run the following bit of code.

psql -U postgres mysticmanager -c “select sn,name,psnt from node;”

Finding the Model and PSNT using the ESXi Console

Once you’ve consoled to the VxRail appliance, run the following command if you’re looking for both the Model and PSNT.

opt/dell/DellPTAgent/tools/ipmitool_static fru print 17

If, for some reason, the location of the ipmitool changes in a future version, run this handy command to find the new location.

find /* -name ipmitool_static

If you’re only looking for the PSNT, the first section of the Local Datastore name on each VxRail Appliance includes the 14 character PSNT. Run the following command if you’re just looking for the PSNT.

df -h

Read More

How to Find a Dell EMC Service Tag on PowerEdge Servers and VxRail Appliances

You can find the Service Tag on the luggage tag at the front of a PowerEdge or VxRail Appliance. If you’re looking for a way to get that Service Tag remotely, check out one of the following methods.

Finding the Service Tag in the vCenter UI

PowerEdge Servers

You’ll find the service Tags for each Power Edge Server by navigating to Hosts & Clusters > Cluster > Configure > Hardware > Processors.

VxRail Appliances

You’ll find the Service Tags for each VxRail Appliance in the VxRail Plugin for vCenter. Navigate to Hosts & Clusters > Cluster > Configure > VxRail > Hosts.

Finding the Service Tag using PowerCLI

Once you’ve connected to your VxRail vCenter, run the following bit of code. Get-Hosts -Server VxRailManager_IP -Username administrator@vsphere.local -Password <Password> |ft sn, hostname

Finding the Service Tag using the ESXi Console

Once you’ve consoled to the PowerEdge Server or VxRail appliance, run the following command. /opt/vxrail/tools/ipmitool fru print 17

Read More

Access a Dell EMC 14G iDRAC Using the USB Console Interface

Deploying infrastructure over the years has exposed some pretty useful shortcuts. It’s tedious to program an iDRAC using the front LED. Sometimes, unconfigured network switches hold up the entire configuration process. This method will allow you to hook up a USB console cable and control your iDRAC directly, which can be a life saver and makes configuration and deployment much easier and without the need for a network connection.

  1. Connect your laptop to the USB Management Port on the front of the PowerEdge or VxRail.
  2. Wait for your laptop to install the appropriate drivers.
  3. The USB interface on your laptop will be assigned an IP address of 169.254.0.4.
  4. Open the browser of your choice.
  5. Browse to https://169.254.0.3, which is the IP address assigned to the iDRAC USB interface.
  6. Log into the iDRAC as root.

Read More

Dell EMC VxRail 4.7 Default Password List

Recently, I’ve had to factory reset a few VxRail nodes for some testing I’ve been doing and I thought it might be a good idea to publish the defaults that Dell EMC uses for VxRail running 4.7 code.

ComponentNetworking ConfigurationUsernameDefault Password
BIOSN/AN/Aemcbios
iDRACDHCProotcalvin
ESXi rootDHCProotPassw0rd!
vCenter Server ApplianceDefined at deploymentrootvmware
Platform Services ControllerDefined at deploymentrootvmware
vRealize Log InsightDHCProotPassw0rd!
VxRail Manager192.168.10.200rootPassw0rd!
VxRail Manager PluginDefined at deploymentrootPassw0rd!
VxRail Manager mysticVxRailManager@201602!
VxRail Manager Plugin mysticVBManager201415!

Hopefully this helps someone out there!…

Read More

Dell EMC VxRail 4.5 Upgrade Process: The Aftermath

Part 1: Dell EMC VxRail 4.5 Upgrade Process: Before We Begin
Part 2: Dell EMC VxRail 4.5 Upgrade Process: Upgrading the Cluster
Part 3: Dell EMC VxRail 4.5 Upgrade Process: The Aftermath

In the final post in this 3-part series, we cover some of the things that need to be done post upgrade for a VxRail 4.5 cluster.

Cleanup Process After a Successful Upgrade

  1. Log into the vSphere Web Client.
  2. Navigate to Hosts and Clusters.
  3. Delete all snapshots from the VxRail Manager service VM.
    1. Right-click the VxRail Manager VM, select Snapshots > Delete All Snapshots.
  4. Delete all snapshots from the vCenter Server Appliance service VM.
    1. Right-click the vCenter Server Appliance VM, select Snapshots > Delete All Snapshots.
  5. Delete all snapshots from the Platform Services Controller service VM.
    1. Right-click the Platform Services Controller VM, select Snapshots > Delete All Snapshots.
  6. If changes were made to avoid vSAN timeout issues during the upgrade process, change the following Advanced System Setting when the upgrade is complete.
    1. Navigate to Hosts and Clusters.
    2. Select the first node in the cluster.
    3. Select Configure > System > Advanced System Settings.
    4. Locate the VSAN.ClomRepairDelay setting.
    5. Change the current value of

Read More

Dell EMC VxRail 4.5 Upgrade Process: Upgrading the Cluster

Part 1: Dell EMC VxRail 4.5 Upgrade Process: Before We Begin
Part 2: Dell EMC VxRail 4.5 Upgrade Process: Upgrading the Cluster
Part 3: Dell EMC VxRail 4.5 Upgrade Process: The Aftermath

In Part 2 of this 3-part series, we’ll cover the actual upgrade process for your 4.5 VxRail cluster and how to monitor that upgrade so you don’t end up pulling your hair out.

Upgrading the VxRail Cluster

Local Upgrade Process

The local upgrade process is preferred, because it stages the upgrade to the VxRail Manager before proceeding with the upgrade.

  1. Log into the VxRail Manager.
  2. Navigate to the CONFIG page, and click on the System tab.
  3. Click the Local Upgrade button.
  4. In the Open File dialog, navigate to the VxRail Composite Bundle previously downloaded and saved locally, and then click Open to upload it to the VxRail Manager.
  5. Once the upgrade is staged to the VxRail Manager, click on the Continue button to begin the upgrade.
  6. When prompted, enter the required credentials for the vCenter Server local administrator account, the VxRail Manager root account, the vCenter Server root account, and the Platform Services Controller root account, and click the Submit button.

Internet

Read More

Dell EMC VxRail 4.5 Upgrade Process: Before We Begin

Part 1: Dell EMC VxRail 4.5 Upgrade Process: Before We Begin
Part 2: Dell EMC VxRail 4.5 Upgrade Process: Upgrading the Cluster
Part 3: Dell EMC VxRail 4.5 Upgrade Process: The Aftermath

In Part 1 of this 3-part series, we’ll address what to do prior to kicking off an upgrade of your 4.5 VxRail cluster. There are a few “gotchas” I’ve run into in the past that I address here. Hopefully it’s helpful.

Generating the SolVe Upgrade Procedure

  1. Log into Dell EMC SolVe Online.
  2. Select All Products from the menu.
  3. Click the VxRail Appliance button.
  4. Click the Software Upgrade Procedures link.
  5. Select the model of the VxRail appliances being upgraded, and click the Next button.
  6. Select the VxRail software version currently installed on the VxRail appliances being upgraded, and scroll down.
  7. Select the VxRail software version targeted for installation on the VxRail appliances being upgraded, and click the Next button.
  8. Click the Generate button to generate an upgrade procedure.
  9. Save, open and read the procedure thoroughly once it has been generated.
  10. Download the VxRail Composite Bundle for the version of software targeted in the upgrade.

Before Upgrading the VxRail Cluster

  1. Log into the vCenter Server Appliance Management Interface using

Read More