How I fixed a Linux user login issue

Although I spend a lot of time on Linux system on my daily work, my main focus is on application side. Yesterday a colleague brought a user login issue to me: he’s been working on a secured Oracle Linux server and all of sudden he could not login into the system with his service user via ssh.

Fortunately he still had one ssh session live with the user root logged in so that we could do some investigation.

First thought I had was that the password somehow didn’t work. So I tried to change the password for this user and after that still the user could not login.

It seemed the user has been locked somehow. I chcked the password uisng:

# passwd -S <username>
user PS 2020-11-18 1 60 7 0 (Password set, SHA512 crypt.)

# passwd --status <username>

to see if it’s locked. The password for the user was just fine — no “Password locked” in the output of the command.

Or just check the corresponding user entry in the file /etc/shadow to see if the encrypted password field has leading “!!” or “!“. Normally if there is leading double “!!”, the user is locked by “passwd -L“. And if there is just one leading “!”, the user is locked by “usermod -L“.

The next thing I tried was to delete(userdel)/recreate(useradd) the user. Still after the recreation of the user, it could not login.

I then “su” to this user from the user “root”, but this user could not “su” or “sudo” which it used to be able to. I started to think to put “sudo” in debug with the following two lines in /etc/sudo.conf:

Debug sudo /var/log/sudo_debug.log all@debug
Debug sudoers.so /var/log/sudo_debug.log all@debug

both sudo debug log and ssh entries from the log file /etc/log/secure pointed to PAM.

Checked /etc/pam.d/sudo and /etc/pam.d/sshd, they both eventually point to the same PAM module (within /etc/pam.d/system-auth and /etc/pam.d/password-auth) which has the following lines:

auth required pam_faillock.so preauth silent audit deny=3 auth required pam_faillock.so preauth silent audit deny=3 even_deny_root root_unlock_time=900 unlock_time=604800 fail_interval=900
auth [default=die] pam_faillock.so authfail audit deny=3 even_deny_root root_unlock_time=900 unlock_time=604800 fail_interval=900
auth sufficient pam_faillock.so authsucc audit deny=3 even_deny_root root_unlock_time=900 unlock_time=604800 fail_interval=900
account required pam_faillock.so

That’s it — the module pam_faillock.so (pam_tally and pam_tally2 are deprecated) is used. With such configuration, if there are 3 number of consecutive authentication failures within 15 minutes (fail_interval=900) for a user, this user is locked for 7 days (unlock_time=604800)!

The tally (recording number of failed login attempts) files using the user name as file names are stored at the default location: /var/run/faillock. Each user has a file there and files are not removed by the userdel command. That’s why recreation of the user didn’t work.

The command “faillock” can be used to see the status of each user and clear failure records.

faillock --user <username>
faillock --user <username> --reset

As this Red Hat Linux 6 Security Guide indicates:

To prevent the system from locking users out even after multiple failed logins, add the following line just above the line where pam_faillock is called for the first time in both /etc/pam.d/system-auth and /etc/pam.d/password-auth. Also replace user1, user2, user3 with the actual user names.

auth [success=1 default=ignore] pam_succeed_if.so user in user1:user2:user3

Also note “When modifying authentication configuration using the authconfig utility, the system-auth and password-auth files are overwritten with the settings from the authconfig utility. This can be avoided by creating symbolic links in place of the configuration files, which authconfig recognizes and does not overwrite.” and adding necessary lines in files /etc/pam.d/system-auth-local and /etc/pam.d/password-auth-local as indicated in the Red Had Security Guide. Then custom settings in the configuration files and authconfig can be used simultaneously.

-rw-r--r--. 1 root root 1309 Oct 27 11:22 system-auth-local
 lrwxrwxrwx. 1 root root   28 Oct 27 11:22 system-auth -> /etc/pam.d/system-auth-local
 -rw-r--r--. 1 root root 1309 Oct 27 11:22 password-auth-local
 lrwxrwxrwx. 1 root root   30 Oct 27 11:22 password-auth -> /etc/pam.d/password-auth-local

Some other places or commands might be useful when troubleshooting a user login issue:

  1. /etc/login.defs — default settings for password aging
root@joetest:~# cat /etc/login.defs|grep -v "^$"|grep -v "^#"
MAIL_DIR        /var/spool/mail
PASS_MAX_DAYS     60
PASS_MIN_DAYS     1
PASS_MIN_LEN    5
PASS_WARN_AGE   7
UID_MIN                  1000
UID_MAX                 60000
SYS_UID_MIN               201
SYS_UID_MAX               999
GID_MIN                  1000
GID_MAX                 60000
SYS_GID_MIN               201
SYS_GID_MAX               999
CREATE_HOME     yes
UMASK           077
USERGROUPS_ENAB yes
ENCRYPT_METHOD SHA512
FAIL_DELAY      4
CREATE_HOME     yes

2. /etc/default/useradd — can be used to define inactive & expire

root@sadvlprpadbs01:~# cat /etc/default/useradd
# useradd defaults file
GROUP=100
HOME=/home
INACTIVE=0
EXPIRE=
SHELL=/bin/bash
SKEL=/etc/skel
CREATE_MAIL_SPOOL=yes

3. the command chgage. The option “-l” to check (just as “passwd -S“). The option “-M” to change Maximum number of days between password change. The option “-m” to change Minimum number of days between password change.

chgage basically reads data from /etc/shadow. And this file contains lines for users, each line has 9 fields to control things about users’ password such as minium password age, maxium password age. For the 3rd field — date of last password change, if it has a value 0, it means the user should change their password the next time she/he will log in the system. So if you want to disable this behaviour, set it to empty to disable password aging feature. Use “man 5 shadow” to get more details.

root@joetest:~# chage -l service
Last password change                                    : Dec 11, 2020
Password expires                                        : Dec 06, 2021
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 1
Maximum number of days between password change          : 360
Number of days of warning before password expires       : 7

root@joetest:~# chage -l root
Last password change                                    : Dec 14, 2020
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 7

root@joetest:~# chage -M 180 service
root@joetest:~# chage -m 0 service

One more thing to note,

pam_lastlog.so might be used in /etc/pam.d/password-auth or /etc/pam.d/postlogin or /etc/pam.d/system-auth etc.

pam_lastlog.so is a PAM module to display date of last login and perform inactive account lock out. So even with faillock to reset the number of failed attempts, if an account has not logged in for a certain amount of time (days normally), the login of such account will still be blocked.

You will see similar log messages in /var/log/secure:

Oct 10 10:41:03 joetest sshd[3185]: pam_lastlog(sshd:auth): user user_1 inactive for 247 days - denied
Oct 10 10:41:05 joetest sshd[3185]: Failed password for user_1 from 10.71.12.169 port 54346 ssh2
Oct 10 10:42:01 joetest sshd[3256]: pam_lastlog(sshd:auth): user user_1 inactive for 247 days - denied
Oct 10 10:42:03 joetest sshd[3256]: Failed password for user_1 from 10.71.12.169 port 54356 ssh2
Oct 10 10:44:41 joetest sshd[3411]: pam_lastlog(sshd:auth): user user_1 inactive for 247 days - denied
Oct 10 10:44:43 joetest sshd[3411]: Failed password for user_1 from 10.71.12.169 port 54370 ssh2

root@joetest:~# lastlog -u user_1
Username         Port     From             Latest
user_1      pts/1    10.71.12.169     Wed Dec 10 10:50:12 -0500 2019

From the man page of pam_lastlog “man 8 pam_lastlog“, there is an option for setting inactive days:

inactive=<days>
This option is specific for the auth or account phase. It specifies the number of days after the last login of the user when the user will be locked out by the module. The default value is 90.

A simple way to activate a user is to use “su -” from root. For example:

su - user_1

This will update lastlog database (/var/log/lastlog file actually) to activate the user. If for whatever reasons, the lastlog database is not updated with “su -“, you might have to use:

cat /dev/null > /var/log/lastlog

Which will wipe out all records for all users.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s