Note: This is a critical RHCSA 7 exam objective (if you can’t take control of a VM through a reboot at the beginning of the exam, you will fail it entirely).
Presentation
In RHEL 7, the procedure to get access to a system during the boot process and modify the root password has changed because of the adoption of Systemd.
There were several procedures floating around to recover the root password. Some were working with physical servers but not with virtual machines, some the other way around.
The following procedure works all the time.
Procedure
At the beginning of the boot process, at the GRUB 2 menu, type the e key to edit.
Then, go to the kernel line (the line starting with linux16) and add the following statements at the end:
rd.break enforcing=0
Caution: The keys to press are those of a US keyboard (querty).
Note: rd.break asks for a break at an early stage of the boot process. enforcing=0 puts the system into SELinux Permissive mode. Don’t confuse with selinux=0 that completely disables SELinux.
Press Ctrl x to resume the boot process.
Then, mount the /sysroot partition as read/write:
switch_root:/# mount –o remount,rw /sysroot
Execute the chroot command on the /sysroot partition:
switch_root:/# chroot /sysroot
Change the root password:
sh-4.2# passwd root Changing password for user root. New passwd:mypasswordRetype new password:mypasswordpasswd: all authentication token updated successfully. sh-4.2# exit exit switch_root:/# exit logout
Connect to your server at the console (don’t reboot now!) with the root user and the new password:
... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Crash recovery kernel arming. [ OK ] Reached target Multi-User System. CentOS Linux 7 (Core) Kernel 3.10.0-229.14.1.el7.x86_64 on an x86_64 vm login: root Password:mypassword
Then type:
# restorecon /etc/shadow # reboot
If you strictly follow this procedure, you don’t need to force a SELinux relabel (# touch /.autorelabel or # fixfiles onboot) or load the SELinux policy (# /usr/sbin/load_policy -i).
You don’t even need to reboot at the end! In this case, type # setenforce enforcing
For the RHCSA exam, you need to intensely practice this procedure.
Thanks to salvador and hunter86_bg for their precious comments.
Note: When dealing with boot problems, the following options can be added to the kernel command line, bringing additional information:
rd.debug rd.udev.debug systems.log_level=debug
Additional Resources
Ralph Nyberg‘s video about recovering root access (15min/2015) explains the procedure very well.
Fedora Documentation‘s got a page about Resetting the root password.
There is also a documentation available for the Anaconda Boot Options.
As there is more than one way to do it, a page describes several ways to reset the root password.
Also, there is a Serverfault question discussing SELinux problems that may happen when resetting the root password.
Use all available tips and paper writer contact our team of professional academic writers to get help.
I thought that the whole thing about relabeling was due to a new file being created when you run passwd.
If you just edit the /etc/passwd with ed, then the current file is kept and you can just clear root’s password, go multi user and then re-establish selinux and change the (by now null) password to something else. I’ve done this and AFAIK it’s the fastest way. Any downside ?
Your solution seems pretty interesting but I don’t think many people understand how to use the ed editor.
Thanks for putting this article together. Red Hat documentation states to remount the root directory to read only after setting the password for root: “mount -o remount,ro /”. Perhaps this is related to security of the system. Anyway, it seems to me also the most logical action to take. Plus, Red Hat suggest to remove rhgb and quiet parameters in order to enable system messages.
Your remarks are very interesting in a production environment where security matters.
However, during an exam, you need to be quick and these instructions increase the number of tasks to execute without great added value in this specific context.
Sorry if I’m asking a noob question… but what is the purpose of restorecon on /etc/shadow?
This is not a noob question but a very good one.
In the procedure, you disable SELinux, then you change the root password with the passwd command.
The passwd command modifies the content of the /etc/shadow file where passwords are stored.
If you don’t run the restorecon command on the /etc/shadow file, the risk is to trigger a complete SELinux relabel when rebooting which could take some time.
It is a specific of “passwd” command work.
When “passwd” command changes a user’s password it creates new shadow file. Next applies SELinux label to new file, copies the content of old shadow file and writes new password. Finally renamed this new file to /etc/shadow.
“rd.break” interrupts the boot process and at this phase SELinux doesn’t work and “passwd” command can’t correctly relabel file.
As a result when system will be booted, /etc/shadow file will have incorrect SELinux label.
Thanks for these interesting details.
If you do not relabel /etc/shadow, your system which is in Enforcing Selinux mode can not access to /etc/shadow, and you will not log in.
Below you have got two crucial lines from the audit logs:
type=AVC msg=audit(1471855249.615:42): avc: denied { open } for pid=2056 comm=”unix_chkpwd” path=”/etc/shadow” dev=”dm-1″ ino=1112495 scontext=system_u:system_r:chkpwd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
type=AVC msg=audit(1471855249.615:42): avc: denied { read } for pid=2056 comm=”unix_chkpwd” name=”shadow” dev=”dm-1″ ino=1112495 scontext=system_u:system_r:chkpwd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
I thought that the whole thing about relabeling was due to a new file being created when you run passwd.
If you just edit the /etc/passwd with ed, then the current file is kept and you can just clear root’s password, go multi user and then re-establish selinux and change the (by now null) password to something else. I’ve done this and AFAIK it’s the fastest way. Any downside ?
The risk is to trigger a SELinux autorelabel at the next reboot.
What would cause that ? I thought that was a manual process that you could trigger by creating a file (/.autorelabel)
According to the SELinux documentation: “To prevent incorrectly labeled and unlabeled files from causing problems, file systems are automatically relabeled when changing from the disabled state to permissive or enforcing mode.”
There are two options to trigger a relabel: creating an .autorelabel file or changing the SELinux mode.
However, for a unknown reason, changing the SELinux mode doesn’t always trigger a relabel, it isn’t done so automatically!
Well, makes sense if going from disabled to enabled, either permissive or enforcing. If you do not disable selinux, then I see no risk. May be I’m missing something.
Just to be clear, here’s my proposed recovery procedure:
1- Interrupt Grub, edit load line, insert rd.break, boot.
2- Remount /sysroot rw
3- Edit in place /sysroot/etc/shadow (e.g. /bin/ed) and remove root password
4- Continue to multiuser, log in as root w/o password and use passwd to set a password.
No selinux issues, no reboots, no hassles.
At the step 3), I don’t find the ed command!
What command are you using?
ed is, at this stage, at /sysroot/bin/ed.
The way I do it:
# /sysroot/bin/ed /sysroot/etc/shadow
1s/:.*/::16799:0:99999:7:::
w
q
# exit
and off you go.
You can use “1p” to print the first line to have a reference of what the line looks like.
Your solution works in my tests and seems perfectly fine.
However, the syntax of the line to send to ed remains slightly complicated.
Thank you tron. Your procedure is working very good for Vbox virtual machines also. Can we use Vi editor instead of ed?
You can use any editor that is available and that does an in-place edit, i.e., it does not create a new file and then swaps it in.
I know ed fits the bill, I’m also proficient in vi so I guess it does not, I really don’t remember. Try! And post back. You can just try outside this procedure with any file and check the file inode after an edit to see if it is the same file or not.
It’s exactly what we think:) vi change the inode number if I delete the crypted root password part. But with ed editor inode is not changing. This is the trick.
This didn’t work for me on a minimal install of centos 7 as the ed command doesn’t appear to be available. It did work on a server with GUI installation…
Does the exam expect us to only use this method, or as long as the password gets changed we are good? I have found a quicker method, adding systemd.debug-shell to kernel parameters adds an open root shell on vt9(ctrl-alt-f9). You can then use passwd without worrying about having to remount sysroot or selinux context issues, and just switch back to vt1 to login.
This is a very good question. Unfortunately, I don’t have the answer!
Your method seems very interesting. However, I will stay with the main method because nobody knows!
hi Certdepot.
Ive been practicing RHCSA objectives based on everything from REDHAT Official and Anything from the net Google gives, just wanna validate this procedure on breaking root…
BTW i will take my RHCSA this year (2016)
at GRUB menu add rd.break
# mount -o remount,rw /sysroot
# chroot /sysroot
#passwd root
# “newpass”
#touch /.autorelabel
#ctrl + d
#ctrl + d
Your procedure is the official Red Hat procedure. It obviously works but there is at least another procedure quicker (don’t require touch /.autorelabel). As you are judged on the result and not on the procedure, it’s your call.
hi sir,
you mean this one (got from this discussion)?
Then type:
# restorecon /etc/shadow
# reboot
If you strictly follow this procedure, you don’t need to force a SELinux relabel (# touch /.autorelabel) or load the SELinux policy (# /usr/sbin/load_policy -i).
You also have to add enforcing=0 to the kernel line in GRUB.
You can even avoid rebooting by typing # setenforce enforcing at the end.
hi Certdepot,
thanks for this,very helpful and more faster (i tested it)
may i just confirm on the process below
rd.break enforcing=0 on GRUB menu (delete rhgb quite)
#mount -o remount,rw /sysroot
# chroot /sysroot
#passwd root
# “newpass”
# restorecon /etc/shadow
#ctrl + d
#ctrl + d
correct?
yes, it should work fine.
thank you very much… and yes this more more faster than the 1st approach ..very important on the exam since we have limited time
Hmm, no, you can not do restorecon at the initial single user shell. That’s the whole issue of dealing with selinux !
Yes, you are perfectly right. I missed this point.
If you strictly follow this procedure, you don’t need to force a SELinux relabel (# touch /.autorelabel) or load the SELinux policy (# /usr/sbin/load_policy -i).
Hi CertDepot,
Since you have mentioned the above statement can I use this procedure without having any issues?
Thanks
-remove rhgb quiet
-rd.break
-mount -o remount ,rw /sysroot
-chroot /sysroot
-passwd (to change the pw)
-touch /.autorelabel
-ctrl + d
-ctrl + d
Follow exactly the procedure highlighted in the tutorial and it will be fine.
Hi CertDepot,
My procedures comes from redhat 7 manual.
It just made me think what steps should I follow on the exam. 🙂
Thanks..
To pass the exam you may need to understand what’s going on, rather than remember a set of procedures.
Yes, the procedure that you state does work but it has a time penalty that will impact on your time for the rest of the exam.
Other procedures stated in this thread are faster. Go with whatever you feel safe, and do practice in advance if you can.
Thanks Tron.
Hi CertDepot,
I have tried the procedure you have here. and after i tried running #grep sealert /var/log/messages and I have got this error message:
localhost setroubleshoot: SELinux is preventing /usr/sbin/unix_chkpwd from getattr from read access on the file /etc/shadow. run sealert -l
I have tried restorecon /etc/shadow then reboot.
But when i tried my own procedure I don’t get any sealart message.. Thanks
I don’t think this should matter.
Thanks CertDepot. I think I’m just afraid if my system will not run smoothly if I got a selinux error message. 🙂
hello Bro,
This is the simple way to change the root passwd.
rd.break console=tty1
mount -o remount,rw /sysroot/
chroot /sysroot/
passwd root
type passwd
touch /.autorelabel
ctrl+d
ctrl+d
It works, don’t need to type enforcing or restorecon.
Your solution works and is the official solution. However, it can take some time to make an autorelabel.
The solution shown here avoids this autorelabel and gives the same result. After, it’s your call …
What if I skip the rd.break and instead modify the kernel line from ro to “rw init=/sysroot/bin/sh”
then chroot /sysroot
passwd root
touch /.autolabel
exit
reboot
Would this work?
I don’t know and I don’t plan to test all the possible combinations.
Try your instructions and report back.
Link to description of system boot against ramdisk and different options like rd.break etc.
https://www.kernel.org/pub/linux/utils/boot/dracut/dracut.html#dracutbootup7
Interesting. Thanks.
Here is another great method.. at GRUB menu add rd.break
mount -o remount ,rw /sysroot
# chroot /sysroot
#passwd root
# “newpass”
# load_policy -i
#chcon -t shadow_t /etc/shadow
#exit
#reboot
reboot is required at end not exit. It saves time.
Fine. Thanks.
Hi certdepot,
can we do this ??
at GRUB menu add rd.break
# mount -o remount,rw /sysroot
# chroot /sysroot
#passwd root
# “newpass”
#load_policy -i
#chcon -t shadow_t /etc/shadow
#exit
#reboot
I don’t know. I propose one method that works and is efficient.
Hi CertDepot,
I find strange situation when trying your method (sorry not mean to be rude)
when the root using the xfs filesystem it’s working, but when using the ext3 file system after reboot and then to login root failed
I’m using virtualbox environment maybe is this some kind of bugs or something?
This could be a VirtualBox bug because the method explained in the tutorial is tried and tested.
I don’t know about the ext3, but I tried with the ext4 and VirtualBox 5.0 – it works without any issues.
I dont know much about Virtual box, however can you mount the image, and take a look at the log files. This may point you in the correct direction!
I have just returned from the ex200 exam… fell foul right at the first hurdle… now that I now the new rhel7 procedure, I’ll know next time… but it is a waste of 500 euros + vat … which I have to pay myself. It reminds me of that special darts game: double-in, double-out … ie you don’t score anything until you have hit a double. Not only is it a waste of money and time, but I was not able to answer the rest of the questions for which I trained quite hard. Oh well… c’est la vie
I feel sorry for you but you are not the only one.
There are many people who were stuck in front of their console, not able to reset the root password.
I’m sorry you haven’t passed the exam, but the RHEL7 procedure was mentioned several of times on this site…
I do realise that this procedure is very handy for a Red Hat Systems Admins to know, but it should not be a show-stopper. After all, we all use various help facilities … and Google pointed me at the answer… very much like other technical queries I have had in the past. It would be useful to get Red Hats own official feedback on this… because without that, it ‘seems’ (I use that word carefully), they are cashing in nicely on everyone having to do this exam at least twice.
Sorry to here that. It dose feel that way, that RedHat sets up the exam, as I have been through this.
On the other side of the coin, RedHat makes the exam hard for the reason of being professional. This is so an employer knows what they are getting. I can only suggest practice, practice, practice. Pay attention to ALL of the exam objectives. There are usually nuggets of info in the comments.
Side note: Some European Countries offer partial tax back for education.
If you can afford it, look at getting one of the books at http://www.pmsas.pr.gov.br/wp-content/?id=certdepot-EX200&exam=rhel7-book-review/ . There can be good questions there.
There were cases when I got blades down in a DC and was forced to reset the root account password via console to be able to troubleshoot. I could’ve used Google or any other mean of information, however, that would’ve cost me time (and money).
I agree with you that we all use various means to obtain information, but I also think that resetting a root password from memory is a vital skill of any Linux admin.
As you say, RedHat may be cashing in on everyone having to re-take the exam, but that’s down to the fact that you haven’t learnt the exam objectives. What is the point of doing the rest of the exam tasks if you cannot get access to the system? If we were given unlimited time and Internet resources, we would likely solve every single exam question.
FYI … redhat were unwilling to change anything or to help with any of the costs. I have, however, asked icttrainingen.nl to review their self-study course to add a specific section for this. Here another link that also worked for me: http://www.tecmint.com/reset-forgotten-root-password-in-centos-7-rhel-7/
Thanks.
On RHEL 7, I would use rd.break rather than init=/bin/bash as per official RedHat documentation.
Hello everyone.
Mr. Certdepot, your root password reset method works at 100%. Thank you. But there is another way to reset password. Go to grub menu with pressing “e”. Look for “ro” word. Replace it with phrase “rw init=/sysroot/bin/sh”. CTRL+X. chroot /sysroot. passwd root. exit. reboot. Has anybody use this way? What is the preferable way of password reset at the exam?
There isn’t one unique solution for this problem. Use the one you prefer!
Thank you for this procedure CertDepot. Also tron’s procedure is working very good for Vbox virtual machines. I was using tron’s procedure for my Solaris Unix system at work 2 or 3 years ago. Vbox doesn’t boot up because of SELinux policy change with CertDepot’s procedure. Without disabling SELinux, it’s impossible to boot up in Vbox after this procedure. I was busy with this all day 🙂 I have to use KVM or real machine I think. But it’s ok now.
I found the following procedure would hardly take 1 minute and wouldn’t cause selinux relabeling
1- edit the kernel option by adding the init=/bin/sh “remove rhgb quiet”
2- load the selinux /usr/sbin/load_policy -i
3- remount -o remount,rw /
4- passwd root
5- remount -o remount,ro /
6- exec /sbin/init 6
And that’s it. This is another easier and less time-consuming method.
Interesting. Thanks.
Update from RedHat support portal:
Note that the above-mentioned steps may drop you to a prompt without access to a USB keyboard and do not work in a VM like KVM or VirtualBox. To reset the root password in these environments:
So the solution is :
1- interrupt boot process by adding rd.break enforcing=0 ‘editing the enforcing will help you not to relabel the whole disk by SELinux as its time consuming and depends on the disk size’
2- mount -o remount,rw /sysroot
3- chroot /sysroot
4- passwd
5- mount -o remount,ro /
6- exit
7- exit
After the system boot:
restorecon /etc/shadow
setenforce 1
I tested this on KVM VM and it works and doesn’t prompt you to relabel the disk.