Linux

Oh noes! I’ve lost my vCenter appliance root and/or grub password, halp!

I recently encountered a situation where an issue with a vCenter Server Appliance 6.0 required logging into the shell as the “root” user, but either the password was recorded incorrectly, or the password which was set was typed incorrectly (twice).  Regardless, it was not possible to log in as root, nor was the grub password known (most likely the same password as root when the appliance was initially configured), so we were stuck between a rock and a hard place.

halp

VMware has a KB article that details how to reset the VCSA root password, however and unfortunately, this required entering the grub boot loader password to edit the boot file, so it was kind of a “chicken before the egg” scenario.  Luckily, I found a blog post on UnixArena.com that detailed using a Red Hat Enterprise Linux ISO to boot into recovery and gain access to a file that allows you to bypass the grub password which in turn allowed me to change the root password.  However, once the root password was changed, the grub boot loader was still unprotected by a password which is no bueno.  With some assistance from VMware Support, I was able to set a new grub boot loader password on the VCSA and all was good with the world again.

This post aggregates information from several different sources, and I’ve added in some material of my own to tie the whole process together and it a little easier to follow.  Thanks to UnixArena.com, VMware Support, and Tecmint.com for the resources.

Now, on to the good stuff…

First, you need to download a Red Had Enterprise Linux .ISO – you are required to create an account to request an evaluation, which allows you to download the .ISO.  The version I used for this post was RHEL 7.4.

Upload the .ISO to a vSphere datastore and mount it in the CD-ROM drive of your VCSA.  Power down the VCSA, take a snapshot of it, and then edit the “boot options” to “Force BIOS Setup” so that you can enter the VCSA’s BIOS and modify the boot order.

vcsa1

Once you’re in the BIOS, change to the “Boot” tab and use the “+” key to move “CD-ROM Drive” to the top of the boot order list.  Use the “right arrow” key to move to the “Exit” tab and choose “Exit Saving Changes”.  The VCSA should reboot and boot to the RHEL .ISO.

vcsa2vcsa3

Use the “down arrow” to select “Troubleshooting”.

vcsa4

Use the “down arrow” key or “R” to select the “Rescue a Red Hat Enterprise Linux system” line, then press “Enter”.

vcsa5

The next screen will prompt you to mount the file system in “read-write mode” by selecting option 1.

vcsa6

When prompted, press “Enter” (or Return) to get a shell.

vcsa7

Once the shell is loaded, you should see this:

vcsa8

Change to the “mnt/sysimage/boot” directory (cd /mnt/sysimage/boot), view the contents (ls –lrt) and you should see the “grub” folder.

vcsa9

Change to the “grub” folder (cd grub) and view the contents (ls –lrt) and you should see a “menu.lst” file.

vcsa10

This next step is optional, and if you’ve taken a snapshot of the VCSA before making any changes (which I hope you did) you could always just roll back, but I like to make a backup of the file I’m about to modify, which in this case is “menu.lst”.  Enter the command “cp menu.lst menu.lst.bak” and a copy of the “menu.lst” file will be made named “menu.lst.bak” which could be used to recover the file if you make a mistake in the next step.

vcsa11

Use the “vi” editor to modify the “menu.lst” file by entering the command “vi menu.lst”

The hashed grub password is highlighted below – use the “down arrow” key to move to the line beginning with “password” and type “dd” to remove the line.  Then, enter the command “:wq” to exit and save the file.

vcsa12

Note that the “password” line is removed.

vcsa13

Exit the shell by entering the commands “cd” and then “exit”.  Be sure to unmount the RHEL .ISO or you will boot back into it.

When the grub boot menu appears, press “space”.  Now that the grub password has been removed, you should see that the instructions to enter “p” to “unlock additional options” is no longer present, and you can proceed to edit mode immediately.

Make sure that the “SUSE Linux Enterprise Server…” line is selected, and press “e”.

vcsa14

On the next screen, select the line beginning with “kernel” and press “e” again to edit the boot command.

vcsa15

Append “init=/bin/bash” to the line below, and then press the “enter”.

vcsa16

With the below line highlighted, press “b” to boot into the shell.

vcsa17

When you get a shell, type “passwd root” to change the root password.

vcsa18

Once you’ve entered a matching password twice, you should get a success message that the password has been changed.  Apparently it thinks the password I used is too simple, but whatever, lab.

vcsa19

Reboot the VCSA by issuing the “reboot” command or “Power > Reset” VM option.

When you see the boot menu again, you should notice that there is still no grub password set, meaning that anyone who gains access to the console of the VM and can reboot it can change the root password.  Obviously, if someone is crafty enough to mount your RHEL image and go through the process we just followed, they could still remove the grub password and then change root, so it’s important to have “least privileged” role based access, shield your management network from user facing subnets, and that sort of thing.

The next portion of this post will focus on putting a grub password back in place.

Once the VCSA has booted, press “Alt + F1” to gain console access, then enter the “root” username  and your recently set root password.

vcsa20

Once you’ve authenticated, enter the command “ssh.get” to verify that SSH is currently disabled.  If the status returned is “True” skip to the next section.  If the status returned is “False”, enter the command “ssh.set –enabled true” to enable SSH.  Verify that SSH is now enabled by entering the “ssh.get” command again.

Alternatively, you can use a web browser to the VCSA’s “VAMI page” by going to https://vcenterhostnameorip:5480, logging in as root, selecting “Access” from the navigation menu, and enabling SSH and bash shell.  Since I already had the VCSA VM console open, I did it there.

vcsa21

The next steps we are going to use an SSH client like PuTTY instead of the VCSA VM console so that we can use copy and paste functions easier, which will help ensure the MD5 hash gets entered correctly.  Connect to your VCSA using SSH and login using the root account and the newly set password.

Once logged in to the SSH session, enable shell by entering the commands “shell.set –enabled true” and then “shell”.

vcsa22

At the shell, enter the command “grub-md5-crypt”, and then correctly enter a matching grub password twice.  You will need to copy the md5 hash to clipboard, so highlight it and then paste into Notepad or another text editor for safe keeping.

vcsa23

Next, we will need to modify the “menu.lst” file that we removed the previous hashed password from earlier.  Edit the menu.lst file by entering the command “vi /boot/grub/menu.lst”.

Once in the text editor, press the “insert” key, “down arrow” to the line underneath “timeout” and press “enter”.  “up arrow” once to the newly create blank line, type “password –md5 “ (space after –md5) and then paste in the copied md5 hash.

vcsa24

Once the hash has been pasted into the “menu.lst” file, press “Esc” to exit “edit mode”, then enter “:wq” to save and quite.  Reboot the system and verify that the grub boot loader is once again password protected.  You should see it prompt for “p” instead of “e” if the menu.lst file modifications were successful.  If “e” is still displayed, verify the contents of the menu.lst file are correct and there aren’t any missing characters or anything like that.

Press “p” to enter your new grub password to ensure everything is good to go – if it unlocks the option to edit boot commands, your job is done.  Don’t forget to remove the VM snapshot once you’ve determined your changes are successful.

vcsa25

whew

Resources:

http://www.unixarena.com/2016/04/reset-grub-root-password-vcsa-6-0.html

https://kb.vmware.com/s/article/2069041

https://www.tecmint.com/password-protect-grub-in-linux/

Advertisements

Password Protect the Pi-hole Admin Page

I recently got Pi-hole configured on my Raspberry Pi 3 to block ads on my home network.  So far the Pi-hole has worked great and the amount of ads it has blocked is impressive.  I have about 13 devices that all connect wirelessly to my home network including several TV’s, and it’s blocking several thousand ads per day, with a significant bandwidth savings to boot.

There is an admin page where you can view all these interesting stats (http://%5Bpi-hole-IP-address%5D/admin).  There is one problem with this page though – it’s not password protected, so anyone that knows the IP address of your Pi-hole (AKA, anyone who can connect and view their client’s IP info) and also knows it’s a Pi-hole and the admin page is /admin can reach it.

In addition, in the “/admin” page is a section called “Query Log”, and as the name indicates, it’s a log of all the DNS lookups performed for devices on your network.  While I don’t particularly have anything to hide, it’s also not information I want freely available for anyone to review either.

This post will detail how to configure authentication on the Pi-hole admin page.  One of the Pi-hole developers (Jacob Salmela) has a pretty detailed set of instructions on how to enable this (kudos for the info), but I found that with my Linux/Pi-hole newbness, there were some gaps I had to fill in, and figured maybe someone else will find this useful as well.

  1. Open an SSH session to your Raspberry Pi.  The first step in this process is to create a password file in a hidden directory.  This password file will be hashed in a later step.Enter the command “sudo mkdir /etc/lighttpd/.htpasswd”
    rbppw1
  2. Change to the hidden directory by entering the command “cd /etc/lighttpd/.htpasswd”.
  3. This step creates a script that will hash a user’s password.  Enter the command “sudo touch [filename.sh]”.  Then, enter the command “ls” to verify the script exists in the directory.  I called my file “hashme.sh”.
    rbppw2
  4. Now, we will need to add the following content into the script file by entering the command “sudo nano hashme.sh” to modify it in nano (text editor).
    rbppw3
    #!/bin/sh
    user=$1
    realm=$2
    pass=$3
    hash=`echo -n “$user:$realm:$pass” | md5sum | cut -b -32`
    echo “$user:$realm:$hash”
    rbppw4
    After you’ve pasted in the script content, enter “Ctrl+X”, then “Y” to save the changes, then hit “Enter” to accept the “File Name to Write”.
  5. Now we need to make the file executable by entering the command “sudo chmod 755 [filename.sh]
    rbppw5
  6. In this step we will run the script with three arguments (user, realm, password) which will then get hashed.  Enter the command “sudo ./[filename.sh] ‘[username]’ ‘[realm]’ ‘[password]’”.  The output will look something like “username:realm:[string of numbers and letters].  Copy the output to your text editor of choice (outside of the SSH session) as we will need it in the next step.
    rbppw6
  7. Now we will create the password file.  You will paste the output of the previous command into nano after issuing this command “sudo nano /etc/lighttpd/.htpasswd/lighttpd-htdigest.user”.
    rbppw7rbppw8
    Once you’ve pasted in the output from the previous step, enter “Ctrl+X”, then enter “Y” to save the changes, and then “Enter” to accept the file name to write to.
  8. I found the next couple of steps to be a bit hard to understand in the developer’s blog post (mainly, where exactly the code had to be inserted).  It took a few tries to get it right, so I recommend backing up the lighttpd.conf file prior to making any changes – it makes recovering from a problem easy.  Because we are still in the “hidden” .htpasswd directory, enter the command “cd ..” to go up one directory.First, we will back up the lighttpd.conf file by entering the command “sudo cp lighttpd.conf lighttpd.conf.bak”.
    rbppw9
    Then enter the command “ls” to verify the backup file exists.
    rbppw10
    If you need to rollback the changes made to the lighttpd.conf file, just enter the command “sudo cp lighttpd.conf.bak lighttpd.conf” and the unmodified file will be restored.
  9. Now that we’ve made our backup of the lighttpd.conf file, it’s time to modify the original.  Enter the command “sudo nano /etc/lighttpd/lighttpd.conf”.The highlighted section below is where we will be pasting in the additional content.  Hit “Enter” at the arrow.
    rbppw11
    Copy the following text and place it in the blank space created by your “enter” key strikes:backend = “htdigest”
    auth.backend.htdigest.userfile = “/etc/lighttpd/.htpasswd/lighttpd-htdigest.user”auth.require = ( “/path/to/protect/” =>
    (
    “method”  => “digest”,
    “realm”   => “myrealm”,
    “require” => “valid-user”
    ),
    )Change the “auth.require = ( “/path/to/protect/” =>” field to “auth.require = (“/admin/” =>
    rbppw12
    Then hit “Ctrl + X”, then “Y”, then “enter” to save the changes.
  10. Now we need to restart the lighttpd service by entering the command “sudo service lighttpd restart”.
    rbppw13
  11. If your changes to lighttpd.conf were successful, you should receive no errors and go right back to the command prompt.
    rbppw14
  12. Now, you need to go to your admin page and see if you are prompted for credentials.  If you’re currently logged into the admin page, hit “Ctrl + F5” or try opening the page in a private/incognito window.  Enter the username and password configured in the previous step, and you should log right into the admin page.
    rbppw15

wwpd.png

Nuke Ads from Orbit with Raspberry Pi and Pi-hole [it’s the only way to be sure]

The Problem:

Earlier this week, I turned on my new Samsung “smart TV” and was greeted with a notification that I needed to accept “new terms and conditions”, and that I would “be sent targeted ads” [i.e. they’re collecting your data and selling it (you)]. Not only that, but if I declined, “certain smart features of the TV may no longer work”.  After “declining”, none of my apps worked.  Say what now?  You mean to tell me you’re going to neuter the TV I paid for if I don’t agree to being pimped out?  Bad move, Samsung.  This has obviously ruffled a lot of feathers, and rightly so.  There is a “mega thread” on Reddit that goes into great detail about it.

Imagine watching Netflix and having an ad or “commercial” pop up thanks to your smart TV?  Yeah no.  What’s likely occurring is that Samsung is subsidizing the cost of their TV’s with the revenue generated by their advertisements.  The fact I just bought a mid-upper range 40″ Samsung 4K smart TV for only $275, which just a few years ago would’ve cost over $1,000, is not lost on me.  A clever idea on paper, but horrible in practice, and whichever executive signed off on that idea should be reassigned to the toaster division.

While I don’t like the idea of what Samsung is doing at all, I can either disconnected my TV from my local network and break all the handy smart TV functionality like Netflix, Amazon Prime Video, Youtube, etc. or “deal with it”.  For now, I’ve chosen to “deal with it” by using something called “Pi-hole”, which essentially turns a Raspberry Pi into a DNS server for your local network, which intercepts advertisements being sent to your client devices and replaces them with white space.  While I suppose this may not stop Samsung from collecting the data, it prevents it from disrupting my use of the TV.

While there are browser based plugins like AdBlock that work quite well for computers and mobile devices, that doesn’t help me much with my TV.  There is only one option – nuke the ads from orbit…it’s the only way to be sure.

nuke-from-orbit

The Gear:

I ordered a Raspberry Pi 3 kit from Amazon for about $50, which came with a clear case, power supply, two heat sinks, and the Raspberry Pi board itself.  In addition, I purchased a 16 GB class 10 micro SD card for about $6 bucks (side note, can’t believe how cheap high capacity removable flash media is now).

I won’t bother with the assembly instructions as it’s pretty straight forward, but figured I would create a post that detailed the steps required to install the Raspberry Pi operating system (Raspbian Jessie Lite which is the GUI / desktop-less version), configure basic settings, and install Pi-hole.

The Solution:

  1. Download the latest version of Raspbian Jessie Lite from https://www.raspberrypi.org/downloads/raspbian/ and extract the .IMG file from the .ZIP archive.  The light version contains no GUI, which is probably unnecessary anyway for the way this Raspberry Pi will be used.
  2. Download Win32 Disk Imager from https://sourceforge.net/projects/win32diskimager/ and install it.
  3. Download SD Card Formatter from https://www.sdcard.org/downloads/formatter_4/ and install it.
  4. Plug in the SD card to your computer and launch the SDFormatter App.  Click the “Format Option – Option” button and set “Format Size Adjustment” to “ON”, and then click “Format”.
    rbp1
  5. Next, run Win32 Disk Imager.  Click the “Browse” button and browse to Raspbian Jesse .IMG file you extracted from the .ZIP archive.
    rbp2
    Ensure the correct drive letter for your SD card is selected under “Device” and then click “Write” to burn the .IMG file to it.  This may take several minutes to complete.
    rbp3
    Once it is done writing the image to your SD card, you should receive a success message.
  6. Eject your SD card, insert it in the Raspberry Pi, and power the unit up.  It will be easiest if you hard wire the Raspberry Pi to your router or switch using the ethernet port.  You will need to attach a monitor and keyboard as well so that you can enable SSH for remote administration.
  7. Login to the Raspberry Pi with the default credentials – username = pi and the password = raspberry
  8. Perform the initial configuration by entering the command “sudo raspi-config”.
  9. The “Raspberry Pi Software Configuration Tool” window will open, and a variety of options for configuration will be displayed.The first step we will do is to enable SSH so that we can access the Raspberry Pi from a remote system for the remainder of the configuration.Select “7 Advanced Options” and then “A4 SSH”.  Answer “Yes” when asked “Would you like the SSH server to be enabled?”.

    Select “Finish” from the raspi-config menu and answer “Yes” when asked to “Reboot now?”.  From this point on, you should be able to perform the rest of the configuration from a remote system at the comfort of your own desk.

  10. Assuming you have DHCP enabled on your router or switch, your Raspberry Pi should’ve received an IP address. You will need to know this IP address to connect to use SSH from another system.  I logged into my router’s admin portal and found a device named “raspberrypi”, which is the default name given, and then noted its IP address.
  11. Open an SSH session to the Raspberry Pi’s IP address and login using the default credentials.
    rbp4
  12. Once you’re logged in to the SSH session, you will see a warning that SSH is enabled but the default credentials have not been changed, which poses a security risk.  We will be changing this, among other settings, by running the raspi-config wizard again.  Enter the command “sudo raspi-config”.
    rbp5
  13. First, we will choose “1 Expand Filesystem” to utilize the remainder of the SD card.  Mine is 16 GB and it’d be nice to have it all available for use.  You will see a message that the root partition has been resized and it’ll require a reboot to complete.  No need to reboot yet though.
    rbp6
  14. Next, choose “2 Change User Password”.  Click “Ok” on the next window, and then you will be prompted to enter and re-enter a new password.  Assuming you entered the password correctly twice, you should see a success window.
    rbp7
  15. Next, choose “4 Internationalisation Options”.  We will be configuring our timezone and Wi-fi County here.  Select “I2 Change Timezone” and select your major region, then your applicable timezone.
    rbp8rbp9
  16. Next, select “7 Advanced Options” and then “A2 Hostname”.  This will allow us to change the default hostname from “raspberrypi” to something custom.  Perhaps you have a naming convention on your network that you need to adhere to, or you don’t want it to be blatantly obvious what the device is used for based on the name.  Enter your hostname then select “Ok”.
    rbp10
  17. Now that most of the basic configuration items have been addressed, return to the main raspi-config menu and choose “Finish”. A reboot will be required to apply the changes.
  18. Open a new SSH session to the Raspberry Pi and login with your new credentials.  You will notice that the terminal now shows the customized hostname you selected in the previous steps.
    rbp11
  19. Now that we have basic configuration and network connectivity, it is time to download and install any updates available for your Raspberry Pi.  The first command you need to enter is “sudo apt-get update”.  You should see the updates begin downloading.
    rbp12
  20. When you see the message “Reading package lists… Done”, the download is complete and it’s time to install the updates.  Enter the command “sudo apt-get dist-upgrade”.  You will be notified that some amount of additional disk space will be required, and you will need to answer “Y” to continue.  Depending on the amount of updates available, it could take a few minutes to complete.  Once it’s done installing updates, enter “sudo reboot” for good measure.
    rbp13
  21. Now we will install the Pi-hole software.  Log back into the Raspberry Pi with an SSH session and enter the command “curl –L https://install.pi-hole.net | bash”.  Alternatively, if you don’t want to pipe to bash, you can use the “alternative semi auto installation” instructions located here.
    rbp14
    The install script will launch, which performs various prerequisite checks and downloads the necessary files before launching the “Pi-hole automated installer” wizard.
    rbp15
  22. The first message you will see is a notification that “the installer will transform your device into a network-wide ad blocker”. Well, that is why we’re here after all.  The next window notifies you that the software is free and open sourced, but “powered by your donations”.  If you like the results, kick a little money their way.
  23. The next window states that you need to use a STATIC IP address, since it is after all a server and if its IP were to change, it’d break your Pi-hole DNS service.  Because we never set a static IP address in any of the previous steps, we will have the chance to do so now.
    rbp16
  24. Now you will be asked to select an interface.  “eth0” is the Ethernet port on your Raspberry Pi, and “wlan0” is the WiFi adapter.  I am going to hard wire my Pi-hole to my router for the simplest and most reliable service, so I have selected “eth0”.
    rbp17
  25. The next window asks you to select which protocol(s) to use – since I am not using IPv6 on my network, I’ve left just IPv4 selected.
    rbp18
  26. The next window displays the current IP information and asks if you’d like to use that as your static address.  Since I currently have a DHCP address leased, I do not want to configure a static IP with this address, so I’ve selected “No” and will enter the new information on the next window.  If you decide to reuse the IP address issued to you by DHCP instead of one outside the DHCP pool, it’s possible that a duplicate IP address could be issued (depending on how smart your router/switch is) and cause an issue.
    rbp19
  27. Enter your IP address in “CIDR notation” – meaning that instead of specifying an IP and subnet mask like “255.255.255.0”, you’d enter the IP like “192.168.1.254/24”, implying that it’s a 24 bit mask.
    rbp20
  28. Most likely, the default gateway you received from DHCP will be the same one you want to use when configuring the static IP.
    rbp21
  29. Verify the information is correct and if so, answer “Yes”.
    rbp22
  30. The next screen asks you to select an “Upstream DNS Provider”.  I use OpenDNS currently and have selected that for use by the Pi-hole.  OpenDNS may give you some additional filtering flexibility as opposed to your local internet service provider’s DNS service.
    set-opendns
  31. You will be shown a summary of your DNS configuration – if everything looks correct, select “Yes”.
  32. You will see some commands execute in the background, and then the “Installation Complete” window should appear.  It tells you that in order for the Pi-hole to do its job, the devices on your network need to use it as their DNS server.  Also, since we did assign a new IP address as part of the configuration, a reboot will be required.  Select “Ok”, and then enter the “sudo reboot” once you’re back at the terminal prompt.
    rbp23
  33. At this point, the Pi-hole is ready for use by your clients.  If you’re using DHCP on your router or switch, the easiest way to accomplish this is probably to modify your DHCP options so that the Pi-hole’s IP address is handed out as the DNS server.  If you have static IP addresses set on any of your devices, you will have to modify their DNS server information manually.The option in red below “inserts” my routers IP address into the DHCP config as an available DNS server.  For purposes of verifying that the Pi-hole is doing its job, I have disabled this setting which forces the clients to only use the Pi-hole and nothing else.  For long term use, you could either configure a public DNS server like OpenDNS or Google in the “DNS Server 2” field, or set “Advertise router’s IP in addition….” To “Yes”.
    rbp24png
  34. Now, we will test that the Pi-hole is doing its job and blocking ads.  Before doing this testing, be sure to disable any browser-based ad blockers like “AdBlocker” so that we don’t mask the results of the test.  Also, since you’ve already set your DHCP options to use the Pi-hole, you will need to manually override your device’s IP settings to use the router, or something other than the Pi-hole, as your DNS server.Now, pick a site that is chock full of ads – while I’m not the Hollywood gossip type, I figure they spam you pretty well, so I went to www.thehollywoodgossip.com and sure enough, several Amazon ads were there (I guess someone has been shopping for pastel colored Yeti tumblers in this house).
    rbp25
  35. Now that we have our “control”, go ahead and revert to using the Pi-hole as the DNS server on your device.  Refresh the page (Ctrl + F5 if using Windows) and you shouldn’t see any ads this time.rbp26
  36. Also, the Pi-hole hosts a webpage that gives you ad blocking statistics, which I find really interesting.  In your browser, navigate to http://%5Byour-pihole-ip%5D/adminIn just a few minutes of time, I can see that 16.5% of my internet traffic has been for ads – not an insignificant amount.  It’s also worth noting that this page is open for access by default, so anyone that knew the IP of your Pi-hole and was aware of how to pull up the “admin” page could open your query log and see everything you’ve accessed on the internet.pi-hole-stats

    I found a good blog post that details how to add authentication to force password protection to a page hosted on your Raspberry Pi.  This is probably a good idea to do, for obvious reasons.  I haven’t had a chance to try yet, but will update this post when I get it setup.