I recently encountered a situation where an issue with a vCenter Server Appliance 6.0 required logging into the shell as the “root” user, but either the password was recorded incorrectly, or the password which was set was typed incorrectly (twice). Regardless, it was not possible to log in as root, nor was the grub password known (most likely the same password as root when the appliance was initially configured), so we were stuck between a rock and a hard place.
VMware has a KB article that details how to reset the VCSA root password, however and unfortunately, this required entering the grub boot loader password to edit the boot file, so it was kind of a “chicken before the egg” scenario. Luckily, I found a blog post on UnixArena.com that detailed using a Red Hat Enterprise Linux ISO to boot into recovery and gain access to a file that allows you to bypass the grub password which in turn allowed me to change the root password. However, once the root password was changed, the grub boot loader was still unprotected by a password which is no bueno. With some assistance from VMware Support, I was able to set a new grub boot loader password on the VCSA and all was good with the world again.
This post aggregates information from several different sources, and I’ve added in some material of my own to tie the whole process together and it a little easier to follow. Thanks to UnixArena.com, VMware Support, and Tecmint.com for the resources.
Now, on to the good stuff…
First, you need to download a Red Had Enterprise Linux .ISO – you are required to create an account to request an evaluation, which allows you to download the .ISO. The version I used for this post was RHEL 7.4.
Upload the .ISO to a vSphere datastore and mount it in the CD-ROM drive of your VCSA. Power down the VCSA, take a snapshot of it, and then edit the “boot options” to “Force BIOS Setup” so that you can enter the VCSA’s BIOS and modify the boot order.
Once you’re in the BIOS, change to the “Boot” tab and use the “+” key to move “CD-ROM Drive” to the top of the boot order list. Use the “right arrow” key to move to the “Exit” tab and choose “Exit Saving Changes”. The VCSA should reboot and boot to the RHEL .ISO.
Use the “down arrow” to select “Troubleshooting”.
Use the “down arrow” key or “R” to select the “Rescue a Red Hat Enterprise Linux system” line, then press “Enter”.
The next screen will prompt you to mount the file system in “read-write mode” by selecting option 1.
When prompted, press “Enter” (or Return) to get a shell.
Once the shell is loaded, you should see this:
Change to the “mnt/sysimage/boot” directory (cd /mnt/sysimage/boot), view the contents (ls –lrt) and you should see the “grub” folder.
Change to the “grub” folder (cd grub) and view the contents (ls –lrt) and you should see a “menu.lst” file.
This next step is optional, and if you’ve taken a snapshot of the VCSA before making any changes (which I hope you did) you could always just roll back, but I like to make a backup of the file I’m about to modify, which in this case is “menu.lst”. Enter the command “cp menu.lst menu.lst.bak” and a copy of the “menu.lst” file will be made named “menu.lst.bak” which could be used to recover the file if you make a mistake in the next step.
Use the “vi” editor to modify the “menu.lst” file by entering the command “vi menu.lst”
The hashed grub password is highlighted below – use the “down arrow” key to move to the line beginning with “password” and type “dd” to remove the line. Then, enter the command “:wq” to exit and save the file.
Note that the “password” line is removed.
Exit the shell by entering the commands “cd” and then “exit”. Be sure to unmount the RHEL .ISO or you will boot back into it.
When the grub boot menu appears, press “space”. Now that the grub password has been removed, you should see that the instructions to enter “p” to “unlock additional options” is no longer present, and you can proceed to edit mode immediately.
Make sure that the “SUSE Linux Enterprise Server…” line is selected, and press “e”.
On the next screen, select the line beginning with “kernel” and press “e” again to edit the boot command.
Append “init=/bin/bash” to the line below, and then press the “enter”.
With the below line highlighted, press “b” to boot into the shell.
When you get a shell, type “passwd root” to change the root password.
Once you’ve entered a matching password twice, you should get a success message that the password has been changed. Apparently it thinks the password I used is too simple, but whatever, lab.
Reboot the VCSA by issuing the “reboot” command or “Power > Reset” VM option.
When you see the boot menu again, you should notice that there is still no grub password set, meaning that anyone who gains access to the console of the VM and can reboot it can change the root password. Obviously, if someone is crafty enough to mount your RHEL image and go through the process we just followed, they could still remove the grub password and then change root, so it’s important to have “least privileged” role based access, shield your management network from user facing subnets, and that sort of thing.
The next portion of this post will focus on putting a grub password back in place.
Once the VCSA has booted, press “Alt + F1” to gain console access, then enter the “root” username and your recently set root password.
Once you’ve authenticated, enter the command “ssh.get” to verify that SSH is currently disabled. If the status returned is “True” skip to the next section. If the status returned is “False”, enter the command “ssh.set –enabled true” to enable SSH. Verify that SSH is now enabled by entering the “ssh.get” command again.
Alternatively, you can use a web browser to the VCSA’s “VAMI page” by going to https://vcenterhostnameorip:5480, logging in as root, selecting “Access” from the navigation menu, and enabling SSH and bash shell. Since I already had the VCSA VM console open, I did it there.
The next steps we are going to use an SSH client like PuTTY instead of the VCSA VM console so that we can use copy and paste functions easier, which will help ensure the MD5 hash gets entered correctly. Connect to your VCSA using SSH and login using the root account and the newly set password.
Once logged in to the SSH session, enable shell by entering the commands “shell.set –enabled true” and then “shell”.
At the shell, enter the command “grub-md5-crypt”, and then correctly enter a matching grub password twice. You will need to copy the md5 hash to clipboard, so highlight it and then paste into Notepad or another text editor for safe keeping.
Next, we will need to modify the “menu.lst” file that we removed the previous hashed password from earlier. Edit the menu.lst file by entering the command “vi /boot/grub/menu.lst”.
Once in the text editor, press the “insert” key, “down arrow” to the line underneath “timeout” and press “enter”. “up arrow” once to the newly create blank line, type “password –md5 “ (space after –md5) and then paste in the copied md5 hash.
Once the hash has been pasted into the “menu.lst” file, press “Esc” to exit “edit mode”, then enter “:wq” to save and quite. Reboot the system and verify that the grub boot loader is once again password protected. You should see it prompt for “p” instead of “e” if the menu.lst file modifications were successful. If “e” is still displayed, verify the contents of the menu.lst file are correct and there aren’t any missing characters or anything like that.
Press “p” to enter your new grub password to ensure everything is good to go – if it unlocks the option to edit boot commands, your job is done. Don’t forget to remove the VM snapshot once you’ve determined your changes are successful.