Although I spend a lot of time on Linux system on my daily work, my main focus is on application side. Yesterday a colleague brought a user login issue to me: he’s been working on a secured Oracle Linux server and all of sudden he could not login into the system with his service user via ssh.
Fortunately he still had one ssh session live with the user root logged in so that we could do some investigation.
First thought I had was that the password somehow didn’t work. So I tried to change the password for this user and after that still the user could not login.
It seemed the user has been locked somehow. I chcked the password uisng:
# passwd -S <username> user PS 2020-11-18 1 60 7 0 (Password set, SHA512 crypt.) # passwd --status <username>
to see if it’s locked. The password for the user was just fine — no “Password locked” in the output of the command.
Or just check the corresponding user entry in the file /etc/shadow to see if the encrypted password field has leading “!!” or “!“. Normally if there is leading double “!!”, the user is locked by “passwd -L“. And if there is just one leading “!”, the user is locked by “usermod -L“.
The next thing I tried was to delete(userdel)/recreate(useradd) the user. Still after the recreation of the user, it could not login.
I then “su” to this user from the user “root”, but this user could not “su” or “sudo” which it used to be able to. I started to think to put “sudo” in debug with the following two lines in /etc/sudo.conf:
Debug sudo /var/log/sudo_debug.log all@debug Debug sudoers.so /var/log/sudo_debug.log all@debug
both sudo debug log and ssh entries from the log file /etc/log/secure pointed to PAM.
Checked /etc/pam.d/sudo and /etc/pam.d/sshd, they both eventually point to the same PAM module (within /etc/pam.d/system-auth and /etc/pam.d/password-auth) which has the following lines:
auth required pam_faillock.so preauth silent audit deny=3 auth required pam_faillock.so preauth silent audit deny=3 even_deny_root root_unlock_time=900 unlock_time=604800 fail_interval=900 auth [default=die] pam_faillock.so authfail audit deny=3 even_deny_root root_unlock_time=900 unlock_time=604800 fail_interval=900 auth sufficient pam_faillock.so authsucc audit deny=3 even_deny_root root_unlock_time=900 unlock_time=604800 fail_interval=900 account required pam_faillock.so
That’s it — the module pam_faillock.so (pam_tally and pam_tally2 are deprecated) is used. With such configuration, if there are 3 number of consecutive authentication failures within 15 minutes (fail_interval=900) for a user, this user is locked for 7 days (unlock_time=604800)!
The tally (recording number of failed login attempts) files using the user name as file names are stored at the default location: /var/run/faillock. Each user has a file there and files are not removed by the userdel command. That’s why recreation of the user didn’t work.
The command “faillock” can be used to see the status of each user and clear failure records.
faillock --user <username> faillock --user <username> --reset
As this Red Hat Linux 6 Security Guide indicates:
“To prevent the system from locking users out even after multiple failed logins, add the following line just above the line where pam_faillock is called for the first time in both /etc/pam.d/system-auth and /etc/pam.d/password-auth. Also replace user1, user2, user3 with the actual user names.“
auth [success=1 default=ignore] pam_succeed_if.so user in user1:user2:user3
Also note “When modifying authentication configuration using the authconfig utility, the
password-auth files are overwritten with the settings from the authconfig utility. This can be avoided by creating symbolic links in place of the configuration files, which authconfig recognizes and does not overwrite.” and adding necessary lines in files /etc/pam.d/system-auth-local and /etc/pam.d/password-auth-local as indicated in the Red Had Security Guide. Then custom settings in the configuration files and authconfig can be used simultaneously.
-rw-r--r--. 1 root root 1309 Oct 27 11:22 system-auth-local lrwxrwxrwx. 1 root root 28 Oct 27 11:22 system-auth -> /etc/pam.d/system-auth-local -rw-r--r--. 1 root root 1309 Oct 27 11:22 password-auth-local lrwxrwxrwx. 1 root root 30 Oct 27 11:22 password-auth -> /etc/pam.d/password-auth-local
Some other places or commands might be useful when troubleshooting a user login issue:
- /etc/login.defs — default settings for password aging
root@joetest:~# cat /etc/login.defs|grep -v "^$"|grep -v "^#" MAIL_DIR /var/spool/mail PASS_MAX_DAYS 60 PASS_MIN_DAYS 1 PASS_MIN_LEN 5 PASS_WARN_AGE 7 UID_MIN 1000 UID_MAX 60000 SYS_UID_MIN 201 SYS_UID_MAX 999 GID_MIN 1000 GID_MAX 60000 SYS_GID_MIN 201 SYS_GID_MAX 999 CREATE_HOME yes UMASK 077 USERGROUPS_ENAB yes ENCRYPT_METHOD SHA512 FAIL_DELAY 4 CREATE_HOME yes
2. /etc/default/useradd — can be used to define inactive & expire
root@sadvlprpadbs01:~# cat /etc/default/useradd # useradd defaults file GROUP=100 HOME=/home INACTIVE=0 EXPIRE= SHELL=/bin/bash SKEL=/etc/skel CREATE_MAIL_SPOOL=yes
3. the command chgage. The option “-l” to check (just as “passwd -S“). The option “-M” to change Maximum number of days between password change. The option “-m” to change Minimum number of days between password change.
chgage basically reads data from /etc/shadow. And this file contains lines for users, each line has 9 fields to control things about users’ password such as minium password age, maxium password age. For the 3rd field — date of last password change, if it has a value 0, it means the user should change their password the next time she/he will log in the system. So if you want to disable this behaviour, set it to empty to disable password aging feature. Use “man 5 shadow” to get more details.
root@joetest:~# chage -l service Last password change : Dec 11, 2020 Password expires : Dec 06, 2021 Password inactive : never Account expires : never Minimum number of days between password change : 1 Maximum number of days between password change : 360 Number of days of warning before password expires : 7 root@joetest:~# chage -l root Last password change : Dec 14, 2020 Password expires : never Password inactive : never Account expires : never Minimum number of days between password change : 0 Maximum number of days between password change : 99999 Number of days of warning before password expires : 7 root@joetest:~# chage -M 180 service root@joetest:~# chage -m 0 service
One more thing to note,
pam_lastlog.so might be used in /etc/pam.d/password-auth or /etc/pam.d/postlogin or /etc/pam.d/system-auth etc.
pam_lastlog.so is a PAM module to display date of last login and perform inactive account lock out. So even with faillock to reset the number of failed attempts, if an account has not logged in for a certain amount of time (days normally), the login of such account will still be blocked.
You will see similar log messages in /var/log/secure:
Oct 10 10:41:03 joetest sshd: pam_lastlog(sshd:auth): user user_1 inactive for 247 days - denied Oct 10 10:41:05 joetest sshd: Failed password for user_1 from 10.71.12.169 port 54346 ssh2 Oct 10 10:42:01 joetest sshd: pam_lastlog(sshd:auth): user user_1 inactive for 247 days - denied Oct 10 10:42:03 joetest sshd: Failed password for user_1 from 10.71.12.169 port 54356 ssh2 Oct 10 10:44:41 joetest sshd: pam_lastlog(sshd:auth): user user_1 inactive for 247 days - denied Oct 10 10:44:43 joetest sshd: Failed password for user_1 from 10.71.12.169 port 54370 ssh2 root@joetest:~# lastlog -u user_1 Username Port From Latest user_1 pts/1 10.71.12.169 Wed Dec 10 10:50:12 -0500 2019
From the man page of pam_lastlog “man 8 pam_lastlog“, there is an option for setting inactive days:
This option is specific for the auth or account phase. It specifies the number of days after the last login of the user when the user will be locked out by the module. The default value is 90.
A simple way to activate a user is to use “su -” from root. For example:
su - user_1
This will update lastlog database (/var/log/lastlog file actually) to activate the user. If for whatever reasons, the lastlog database is not updated with “su -“, you might have to use:
cat /dev/null > /var/log/lastlog
Which will wipe out all records for all users.