We’ve all been there. You’re working to stage your VxRail upgrade and something goes terribly, horribly wrong. You’re beside yourself because you’ve got a deadline, and your upgrade is dead in the water. You’ve tried cancelling the upgrade in the GUI, but it JUST. KEEPS. FAILING.
Well, here’s what you do to get back on track.
- SSH into your VxRail Manager as mystic.
- Elevate to root using the su command.
- Run the following string of commands.
psql -U postgres vxrail -c "delete from system.operation_status where (state='FAILED' or state='IN_PROGRESS') and owner like 'Lcm%';"
psql -U postgres vxrail -c "update system.operation_status set state='FAILED' where owner='CustomizeComponentScan' and state='STARTED';"
echo '{"state":"NONE","deployed_for_public_api":false}' > /var/lib/vmware-marvin/bundle_state.jsonvxrail:/home/mystic # mv /var/lib/vmware-marvin/composite-upgrade.json /var/lib/vmware-marvin/composite-upgrade.json.old
curl -X GET --unix-socket /var/lib/vxrail/nginx/socket/nginx.sock "http://127.0.0.1:5000/rest/vxm/internal/lockservice/v1/lock"
curl -X POST --unix-socket /var/lib/vxrail/nginx/socket/nginx.sock "http://127.0.0.1:5000/rest/vxm/internal/lockservice/v1/lock/release" -H "Content-Type: application/json" -d '{"locked_by": "UPGRADE"}'
systemctl restart runjars
systemctl restart vmware-marvinThe result of the command above should resemble this:
DELETE 1
UPDATE 0
mv: cannot stat '/var/lib/vmware-marvin/composite-upgrade.json': No such file or directory
{"error_code": 1001, "message": "Failed to get the lock information due to the lock does not exist."}
{"error_code": 1001, "message": "Failed to release the lock due to the lock does not exist."}If you’re not comfortable with executing that command, open up a support ticket with Dell EMC and let them make the changes for you.
To watch your upgrade stage in real time, run the following command and then watch it fly as your composite package unpacks.
tail -f /var/log/mystic/lcm-web.log- Accolades
Scott is an experienced professional and recognized leader specializing in the design and implementation of enterprise virtualization, enterprise storage, server infrastructures and operations. A self-starter able to work both independently and in a team-oriented environment, Scott has a proven ability to create and automate solutions to improve productivity, reliability and performance.
Throughout his professional career, Scott has proven successes implementing technology and service improvement initiatives and has a demonstrated ability to think strategically about business, create technical definition around objectives in complex situations, develop solution strategies, motivate and mobilize resources, and deliver end-to-end technology solutions.
Scott has achieved multiple industry recognized certifications in virtualization, networking, cloud, storage, converged infrastructure, hyperconverged infrastructure and other technologies, and is honored to be a VMware vExpert in 2018-2022, and vExpert EUC 2020-2022.
Achievements
VMware vExpert EUC 2020-2022
VMware vExpert 2022
VMware vExpert 2021
VMware vExpert 2020
VMware vExpert 2019
VMware vExpert 2018
Double VCP – Data Center Virtualization & Desktop Mobility
Double VCP – Data Center Virtualization & Cloud Management
Double VCP – Data Center Virtualization & Network Virtualization
Double VCP – Network Virtualization & Desktop Mobility
Double VCP – Network Virtualization & Cloud Management
Double VCP – Cloud Management & Desktop Mobility
Dell Technologies Proven Professional Exam Developer – 2021
Certifications
VMware Certified Associate – Digital Business Transformation 2020
VMware Certified Professional – Desktop and Mobility 2020
VMware Certified Professional – Data Center Virtualization 2020
VMware Certified Professional 6 – Desktop and Mobility
VMware Certified Professional 6 – Cloud Management and Automation
VMware Certified Professional 6 – Network Virtualization
VMware Certified Professional 6 – Data Center Virtualization
VMware vSphere 6 Foundations
Dell Technologies Cloud Platform Administrator 2020
Dell Technologies Virtual Desktop Infrastructure 2019
Dell Technologies Specialist – Infrastructure Security Version 1.0
Dell Technologies Specialist – Systems Administrator, VxRail Appliance Version 2.0
Dell Technologies Specialist – Systems Administrator, VxRail Appliance Version 1.0
Dell Technologies Associate – Converged Systems and Hybrid Cloud Version 2.0
Dell Technologies and VMware Co-Skilled Associate – Converged Systems and Hybrid Cloud
IBM Enterprise Design Thinking Practitioner
IBM Enterprise Design Thinking Co-Creator
SolarWinds Certified Professional
