Managing Windows servers with Rundeck?

I’ve been using rundeck (an automation platform) to manage Linux servers for quite a while. Recently I’ve expanded the practice to Windows servers and have had some difficulties with authentication because either OS packages or Python modules were missing.

Rundeck supports 4 types of authentication — basic, credssp, ntlm, kerberos for the Windows system. It uses the pywinrm plugin which in turn uses the python library pywinrm.

The official guide from rundeck doesn’t really say how to make authentication work. It only mentions you need to install pywinrm with pip. Here is the lesson I’ve learned.

  • Use python3 and pip3
  • When installing a python module, install it as the rundeck user with “–user” switch. For example
pip3 install --user --proxy http://10.13.21.23:3128 pywinrm
  • To make pywinrm work with credssp, ntlm, kerberos, you need to install additional packages as OS level (depends on the Linux distribution). For example, with Oracle Linux 8, I had to install the following packages:
gssntlmssp, krb5-workstation, krb5-libs, krb5-devel, gcc, python36-devel
  • python modules needed in addition to pywinrm
ntlm-auth
requests-ntlm
gssapi
requests-kerberos
pykerberos
requests-credssp

If you miss one of them, you might see all kinds of issues. For ntlm to work, the username needs a domain as the prefix, for example: domain\username.

But for kerberos, don’t use the domain prefix becasue the domain is defined in /etc/krb5.conf. One thing about this configuration file is that realm needs to be all uppercase as “EXAMPLE.COM” showed below. Or you will see the error like “KDC reply did not match expectations while getting initial credentials“

root@joetest:~# cat /etc/krb5.conf
# To opt out of the system crypto-policies configuration of krb5, remove the
# symlink at /etc/krb5.conf.d/crypto-policies which will not be recreated.
includedir /etc/krb5.conf.d/

[logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log

[libdefaults]
    dns_lookup_realm = false
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
    rdns = false
    pkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crt
    spake_preauth_groups = edwards25519
    default_realm = EXAMPLE.COM
    default_ccache_name = KEYRING:persistent:%{uid}

[realms]
EXAMPLE.COM = {
    kdc = kerberos.example.com
    admin_server = kerberos.example.com
}

[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM

You can use kinit command to verify your krb5.conf. For example:

root@joetest:~# kinit testuser
Password for testuser@EXAMPLE.COM:

If you don’t see anything returned after input the password, it means your kerberos configuration is working.

Once you have all necessary OS packages and python moduels installed, rest of things are just rundeck level configuration which can be found in the guide mentioned above.

To verify if your configuration is working or not, after adding a Window server node, you can run a command or define a job against this node. rundeck also has a default workflow step “WinRM Check Step” which can be used for verification.

For Windows server management, there are some limitations:

  1. speed of transfer files is terrible
  2. when copying files with directories using “Copy File” step (WinRM Python file copier), it won’t be able to create directories on the destination even “Recursive copy” is selected. The workaround could be coping zipped file instead, then unzip it on the destination server. But again, the speed is a concern.
  3. When using robocopy from a network share to the destination node, I get the “access denied” error even the user has the right permissions and the command succeeds on the server itself with this user. The workaround is to use “net use” first to map the share to a drive letter because “net use” can be provided with the username and password. Then robocopy the folder over. It’s interesting that the mapped drive is deleted automatically after. Put commands together as an inline script. With this way, software packages can be pushed to the multiple Windows nodes.

Leave a comment