This is a transcript for a video linked here: Hacking the Linux passwd and shadow files.
00:00:00.480 --> 00:00:05.840 hello this video is a guide to how linux stores usernames and passwords 00:00:06.640 --> 00:00:11.520 this will include instructions on how you can perform password recovery on a linux system 00:00:12.240 --> 00:00:17.840 using raspberry pi as an example and we're going to show how you can reset the password 00:00:17.840 --> 00:00:23.200 without knowing it in the first place this video is also an introduction for a video 00:00:23.200 --> 00:00:29.840 i plan to make in the future which will show how hackers try and crack the system passwords 00:00:31.440 --> 00:00:35.440 in my earlier video i explained about the importance of passwords in this video i'm 00:00:35.440 --> 00:00:40.080 going to look at these in a bit more depth on how they are stored on the linux system 00:00:41.840 --> 00:00:47.360 and then the future video will show some of the vulnerabilities of how they are stored i'm going 00:00:47.360 --> 00:00:52.320 to start with a little history just because this will make the explanation both little simpler 00:00:52.320 --> 00:00:58.960 and it'll explain the reason that the password field is configured the way it is linux is based 00:00:58.960 --> 00:01:05.600 on the unix operating system back in the 1980s and earlier as my children would say the olden days 00:01:06.240 --> 00:01:11.840 the password information was stored in the password file 00:01:12.960 --> 00:01:17.360 the file is often known as the etc password or passwd file 00:01:17.920 --> 00:01:25.360 and it's available to anybody on the local system to be able to read this has some 00:01:25.360 --> 00:01:32.640 security implications which i'll explain about later the file is plain text so anybody can view 00:01:32.640 --> 00:01:39.120 it although you need to be root to be able to edit it and just go through the fields here the 00:01:39.680 --> 00:01:46.160 top line shows the names of the fields and now i'm going to show a working example here so 00:01:46.160 --> 00:01:50.960 the first field is the username typically this represents the real name of the user 00:01:52.320 --> 00:01:56.640 or it may refer to a system account such as root which is the admin account 00:01:59.760 --> 00:02:05.840 the next field is the password if we put the plain text password here then it would be readable for 00:02:05.840 --> 00:02:13.040 anybody which is obviously bad instead this is a hash value of the password and this particular 00:02:13.040 --> 00:02:20.160 example is an md5 hash of the word password i'll explain hashing and its vulnerabilities later in 00:02:20.160 --> 00:02:27.280 the video on newer systems the password is moved to the shadow file and instead you just get an x 00:02:27.280 --> 00:02:33.840 in this position in the password file this has an extra level of security and i'll explain all 00:02:33.840 --> 00:02:41.040 about why this came about you may also see an asterisk or an exclamation mark which indicate 00:02:41.040 --> 00:02:46.560 that the account will not accept logins although typically that will be in the shadow file as well 00:02:47.840 --> 00:02:52.880 but if the password field is blank then a user can log in without needing supply password 00:02:53.520 --> 00:02:58.560 which is usually a bad idea but useful for password recovery as i'll show later 00:03:01.680 --> 00:03:09.520 next are the user id and group id the uid is a unique number for this particular user and the gid 00:03:09.520 --> 00:03:16.720 is id of the default group for this user on modern systems they'll normally be a group name which 00:03:16.720 --> 00:03:22.640 is the same as a username and these are created automatically when you use the add user command 00:03:23.600 --> 00:03:31.680 and the uid and gid will be the same for user accounts the uid and gid typically starts at 1000 00:03:33.120 --> 00:03:40.400 the lower numbers are reserved for the special accounts now here that i'm showing that line 00:03:40.400 --> 00:03:49.280 wrap into the next line this would actually all be on a single line the next field is shown is 00:03:49.280 --> 00:03:59.520 known as the gcos field i'll refer to above as the user info field and this is where you include 00:03:59.520 --> 00:04:05.040 information about the user such as their name possibly their phone number or other information 00:04:08.480 --> 00:04:13.360 next is home directory normally each user has their own home directory which 00:04:13.360 --> 00:04:17.440 is in the slash home directory and is named the same as their username 00:04:20.400 --> 00:04:25.680 and the final field is the shell this is the command line user interface that you get when 00:04:25.680 --> 00:04:34.480 logging in through a text terminal or using a remote terminal such as ssh for most users this 00:04:34.480 --> 00:04:41.280 will be the best shell which is slash bin bash for system accounts that do not allow logins then this 00:04:41.280 --> 00:04:48.720 is normally set to slash usr slash s bin no login which will not allow interactive shell sessions 00:04:51.600 --> 00:05:00.560 as i mentioned previously this is the password field and you can't put the plain text password 00:05:00.560 --> 00:05:04.800 in that field as it obviously everyone then would be able to read it and know what your password was 00:05:05.840 --> 00:05:11.120 so we need a way that the computer can know if you've entered the correct password but so 00:05:11.120 --> 00:05:16.640 that anyone else that looks at it can't tell what your password is this is where a hashing algorithm 00:05:16.640 --> 00:05:24.640 comes in i'm going to show this using md5 this was used for passwords in the past but the algorithms 00:05:24.640 --> 00:05:30.080 used today are usually a lot more secure md5 is still quite widely used for other purposes 00:05:33.040 --> 00:05:41.040 the md5 stands for the message digest algorithm an md5 is one implementation it's not necessarily the 00:05:41.040 --> 00:05:46.720 fifth version as the name might suggest because some of the version numbers were never actually 00:05:46.720 --> 00:05:55.200 used i'm going to walk through how the md5 hashing algorithm works we start with the password here 00:05:56.480 --> 00:06:01.120 and here i've used the word password obviously you'd use a more secure password than this 00:06:03.440 --> 00:06:11.680 this is processed by the md5 hashing algorithm and this results in a 32 character long string 00:06:12.640 --> 00:06:17.600 note that this is actually a 16 bit hexadecimal number that represented as text 00:06:18.960 --> 00:06:22.560 the important thing is the direction of the arrows in the diagram 00:06:23.840 --> 00:06:29.200 so now that whilst you can go from the password and get the output it is not 00:06:29.200 --> 00:06:34.320 possible to reverse this and get back to the original password if you have the 00:06:35.520 --> 00:06:42.720 hashed output if that was possible it would break the whole concept of the password hash 00:06:43.280 --> 00:06:52.000 and it would render it unsafe i won't go into the details of how the hashing algorithm works this 00:06:52.000 --> 00:06:59.040 diagram gives an approximation the important thing is not to be able to go in the reverse direction 00:07:00.160 --> 00:07:06.640 now this is one of the key differences between encryption which can be reversed versus hashing 00:07:06.640 --> 00:07:13.040 which can't obviously for encryption you do need to know the key and the algorithm to be able to 00:07:13.040 --> 00:07:21.040 reverse it but even if you know how the algorithm works for a hashing algorithm you can't reverse it 00:07:23.360 --> 00:07:28.560 so only the output of the hash is ever saved on the computer and after storing it in the password 00:07:28.560 --> 00:07:36.320 file the computer cannot find out what your original password was as it cannot be reversed the 00:07:36.320 --> 00:07:41.520 way the computer is able to validate the password is that when you enter your password it performs 00:07:41.520 --> 00:07:47.600 the same hash algorithm against it and checks to see if it matches the saved password hash 00:07:49.520 --> 00:07:54.960 now md5 is now considered to be broken it's not that it's possible to go in the reverse direction 00:07:56.240 --> 00:08:01.600 but it may be possible to create a collision instead which is another string that gives the 00:08:01.600 --> 00:08:06.960 same md5 hashtag value so whilst you might not be able to get back to the original password 00:08:06.960 --> 00:08:12.160 what you might be able to do is think of another password that would give you the same value 00:08:13.520 --> 00:08:18.000 it's still quite complicated but it is theoretically possible 00:08:21.040 --> 00:08:25.520 even before the algorithm was broken there were other ways of defeating the clashing algorithm 00:08:25.520 --> 00:08:31.280 method of creating passwords and some of these are still possible with some of the other algorithms 00:08:31.280 --> 00:08:36.400 i've discussed some of these in my earlier video where i discussed ways of guessing the 00:08:36.400 --> 00:08:42.640 password using dictionary attacks and brute force attacks and these both rely on trying different 00:08:42.640 --> 00:08:47.200 combinations of possible passwords and just passing them through the same hashing algorithm 00:08:50.080 --> 00:08:56.720 an additional risk with a hashing algorithm such as md5 is that it's possible to create 00:08:56.720 --> 00:09:04.640 rainbow tables these are often used by online md5 decryption tools and you're able to have a 00:09:04.640 --> 00:09:11.920 go one of those if you want they're a form of brute force attack but instead of needing to 00:09:12.560 --> 00:09:19.120 re run the algorithm for every permutation they run once and compute it into a large table 00:09:20.000 --> 00:09:23.040 and then that table can be efficiently searched afterwards 00:09:25.200 --> 00:09:30.880 a way of defeating this is by adding a salt to the password and this salt is a known string and 00:09:30.880 --> 00:09:35.920 it's normally added to the start of the password before the hashing and it prevents the tables 00:09:35.920 --> 00:09:43.520 being used so as long as different salts are used for different systems it effectively makes it 00:09:43.520 --> 00:09:50.880 unfeasible to be able to create sufficient rainbow tables to be able to break it using this method 00:09:52.800 --> 00:10:01.920 there's a few of the hashing algorithms that are used for linux passwords and when you look at the 00:10:01.920 --> 00:10:11.360 password string that is generated they start with a few characters and these are telling 00:10:11.360 --> 00:10:18.720 you the algorithm that's used so that the system knows what algorithm to apply to your password 00:10:21.600 --> 00:10:26.640 and this choice is normally made by your distribution but if you have a choice of 00:10:26.640 --> 00:10:34.720 hashing algorithm then i would recommend the two bottom ones are the sha 512 or yes crypt 00:10:37.360 --> 00:10:43.520 and those are the ones that are commonly used on most linux systems so if you look at this it's 00:10:43.520 --> 00:10:50.320 showing you that md5 the first few letters of the password entry would say dollar one dollar 00:10:51.280 --> 00:11:00.080 and for yes crypt it's dollar y dollar and you'll be able to see those if you look in your 00:11:00.080 --> 00:11:05.760 password file or technically your shadow file will come on to that in a minute so although 00:11:05.760 --> 00:11:12.560 the new hashing algorithms are much safer they have still got risk of dictionary or brute force 00:11:12.560 --> 00:11:18.800 attacks especially if the user chooses a poor password and this is why the password is no longer 00:11:18.800 --> 00:11:28.640 stored in the etc password file instead it's moved into a different file called /etc/shadow 00:11:30.320 --> 00:11:34.720 which is a bit of a compromise because the ideal solution would be to make the /etc/passwd 00:11:34.720 --> 00:11:40.800 file so that only the admin root user could read it and then you can just have it all in one file 00:11:41.520 --> 00:11:48.080 however that would have broken other programs which were relying on being able to access the 00:11:48.880 --> 00:11:53.920 safer information that's contained in that file so the solution they came up with was 00:11:53.920 --> 00:12:04.080 to remove the password from etc passwd file and instead store that into the file etc shadow and 00:12:04.080 --> 00:12:11.200 if you look at the passwd file the password field has been replaced by an x lowercase x character 00:12:11.200 --> 00:12:18.320 which tells the system to look at the shadow file instead of the password being in the password file 00:12:19.120 --> 00:12:24.480 this shows the format of the shadow file and the username is the same as the password file and 00:12:24.480 --> 00:12:30.800 then the password which is stored in the second field and again this is the hashed password field 00:12:32.160 --> 00:12:36.400 they also added other fields which store details about when the password 00:12:36.400 --> 00:12:41.200 was last changed the minimum and maximum days that the password is valid for 00:12:42.400 --> 00:12:46.080 the number of days to warn before forcing a password change 00:12:47.120 --> 00:12:54.480 the number of days after it's inactive before it gets disabled and finally when the account expires 00:12:57.920 --> 00:13:02.800 as promised i'm now going to show how you can use what we've just learned about the 00:13:02.800 --> 00:13:07.280 password and shadow files if you find yourself locked out of a linux system 00:13:08.480 --> 00:13:14.560 this is based on using a raspberry pi it'll also work with other linux systems but in that case 00:13:14.560 --> 00:13:20.960 you will either need to boot into a system remote or what's known as recovery mode or perhaps use 00:13:20.960 --> 00:13:29.520 a live boot disk to be able to access the system and then onto the drives alternatively you could 00:13:29.520 --> 00:13:36.480 always remove the drives from the computer but that's of a bit more hands-on physically 00:13:36.480 --> 00:13:42.000 with a computer compared to the raspberry pi where it's just a case of pulling out your micro sd card 00:13:42.800 --> 00:13:52.000 here i am i've got a raspberry pi system and i've set a very long password and 00:13:53.040 --> 00:14:00.080 i don't know what the password is so effectively i'm logged in i can try guessing a password 00:14:01.600 --> 00:14:10.160 i know that the user is pixel server but if i try the password hit enter i'll click log in incorrect 00:14:10.160 --> 00:14:17.760 password please try again and without knowing that password obviously i can't get on the system 00:14:19.760 --> 00:14:27.760 however the one thing about linux systems in fact most computer systems if you can physically 00:14:27.760 --> 00:14:34.800 access the drive and the data is not encrypted then there's ways of resetting the password 00:14:34.800 --> 00:14:41.440 so that's what we're going to do i'm going to shut this computer down and then remove the sd card 00:14:42.480 --> 00:14:50.080 bring that into another computer and show you how i can basically remove the passwords 00:14:52.320 --> 00:14:59.040 so i've taken the sd card out of the raspberry pi i'm going to pop this onto my linux computer here 00:14:59.920 --> 00:15:09.520 so you need to put this into a computer that's able to read ext4 file systems which is what is 00:15:09.520 --> 00:15:20.320 used by linux so as you can see the elements is is my external drive so we can ignore that we've 00:15:20.320 --> 00:15:28.320 got two drives appeared one's boot and the other is called rootfs boot is the fat32 file system 00:15:28.320 --> 00:15:35.440 that holds the boot information but the root fs is the entire root file system on the raspberry 00:15:35.440 --> 00:15:42.240 pi so this is the one we're interested in and we're going to click on that and 00:15:42.240 --> 00:15:48.080 we're going to go into etc and the password file and the shadow files are both in here 00:15:51.600 --> 00:15:59.840 there we are we've got the password and the shadow files you'll see that we're actually 00:15:59.840 --> 00:16:06.320 unable to edit this at the moment and we don't have permissions to view that that's what this red 00:16:06.320 --> 00:16:13.600 circle is so we're gonna have to do this as root i can do is launch terminal and then we'll change to 00:16:13.600 --> 00:16:23.120 that so it's it's mounted into media then under my own username and then it's called root fs 00:16:27.520 --> 00:16:36.160 and it's then in the etc and it's the shadow so you have to use sudo to edit this as root and we 00:16:36.160 --> 00:16:43.680 can edit i'll use the via editor you could use nano or you could use any other editor you want 00:16:47.440 --> 00:16:53.920 and now i'm going to go through this to find the password of the 00:16:53.920 --> 00:16:58.960 user that i wanted to get on so you can see says pixel server is the username 00:16:59.600 --> 00:17:05.760 and as i said the dollar y dollar indicates that this is a 00:17:08.160 --> 00:17:20.320 yes crypt a hash value and basically we can just remove that entire thing up to the next colon 00:17:21.760 --> 00:17:26.160 and now this has got no password and that means you can just log in without any password 00:17:28.160 --> 00:17:34.000 we'll save that i'm going to just run sync that's just going to make sure that it's written to the 00:17:35.440 --> 00:17:43.440 sd card and it's not just kept in a buffer anywhere and then i can switch back to the 00:17:44.560 --> 00:17:52.800 window we can just eject that busy because i'm still in it so just eject that now 00:17:54.720 --> 00:18:01.920 it said it can now be safe to remove so i'll pop that out pop that back into the raspberry pi 00:18:01.920 --> 00:18:09.200 and we'll see if that's worked so i've taken that sd card after i've modified the shadow file and 00:18:09.200 --> 00:18:17.280 put that back into the raspberry pi and booted up now without a password in find a lot of the 00:18:17.920 --> 00:18:22.480 default network services won't allow you to log in anyway they'll only let you log in if there 00:18:22.480 --> 00:18:31.360 is actually a password set so in this case i'm actually physically on the raspberry pi here 00:18:32.080 --> 00:18:42.560 i've got a console connected directly to the hdmi output so i can log in directly here 00:18:44.080 --> 00:18:51.440 the username's pixel server just leave password blank and click login and as you can see it's 00:18:51.440 --> 00:18:58.000 taking me straight in there's no password needed and that's thanks to editing that password file 00:18:58.880 --> 00:19:01.440 very important that you change that password straight away 00:19:02.720 --> 00:19:05.280 so we can just use the passwd command 00:19:07.360 --> 00:19:10.160 now i'm going to use a really simple password here 00:19:12.160 --> 00:19:15.440 just going to use the word password obviously you wouldn't use that 00:19:18.560 --> 00:19:23.840 but this will now allow me to have a look at that shadow file 00:19:28.880 --> 00:19:30.080 that we edited earlier 00:19:32.160 --> 00:19:35.520 and you'll see that this has put that password in so this 00:19:35.520 --> 00:19:40.320 is the yes crypt version of the word password 00:19:43.680 --> 00:19:50.240 and that's how you can recover a server where you've logged and lost the password 00:19:50.240 --> 00:19:54.800 the important thing really is that it's the physical security that prevents you doing this 00:19:55.360 --> 00:20:02.720 if you can physically access a device then you're going to be able to get to the data 00:20:02.720 --> 00:20:06.720 unless it's all encrypted and that's the only way you can really protect 00:20:07.280 --> 00:20:12.640 the data if somebody can physically get hold of the disks that it's stored on 00:20:14.560 --> 00:20:18.000 so obviously i'm going to change this password now to a nice secure one 00:20:19.760 --> 00:20:24.240 hopefully this video has given you an insight into how the linux password and shadow files work 00:20:24.960 --> 00:20:32.400 remember the md5 example used was only for example purposes there are much better hash algorithms 00:20:32.400 --> 00:20:37.280 available today if you learned something new from this video please give it a thumbs up 00:20:37.920 --> 00:20:42.720 if you've not already subscribed please do so click the notification icon to get notified when 00:20:42.720 --> 00:20:49.760 i create my next video thanks for watching and i look forward to seeing you on a future video