Setting up a Kerberised NFS Server on RHEL 7

Posted on 04/06/2016 by Lisenet

We are going to set up a Kerberised NFSv4 server.

[UPDATE]: instructions have been tested on RHEL 7.0, 7.1 and 7.2. Notes for different versions have also been added, where necessary.

Requirements

To establish a Kerberised session between NFS client and host, a few things are required (credit goes to Sander van Vugt).

/etc/krb5.keytab: This file contains the security principals for both the NFS server as the NFS client, and it is required to join the Kerberos realm. In the /etc/krb5.keytab file, the host principal, the NFS principal, or both principals are contained.

A Kerberised user session: After configuring remote authentication against a Kerberos server, the login procedure is already Kerberised and no additional action is required.

The sec=method option, both in the share definition and in the mount options. This enables the desired Kerberos method. The following ones are available: none, sys, krb5, krb5i and krb5p.

The nfs-server as well as the nfs-secure-server services must be active on the NFS server. We have to manually load these services and enable them on RHEL 7.0.

The nfs-secure service must be loaded on the client. We have to manually load and enable this service on RHEL 7.0.

The Lab

We have three RHEL 7.0 servers (with no updates installed) available in our lab:

ipa.rhce.local (10.8.8.70) – FreeIPA server, provides DNS, LDAP, NTP and Kerberos services,
srv1.rhce.local (10.8.8.71) – will be configured as a Kerberised NFS server,
srv2.rhce.local (10.8.8.72) – will be configured as an NFS client.

All three servers have SELinux set to enforcing mode. Our FreeIPA server setup can be found here.

Servers srv1 and srv2 have been configured for LDAP and Kerberos authentication using sssd, check this blog post for details.

On a RHEL 7.0 with no updates installed, we have to import the Kerberos key tabs on both srv1 and srv2 servers, run nfs-secure-server and nfs-secure and mount the share with sec=krb5p.

Keep Time in Sync

To get Kerberos running, NTP synchronisation and hostname resolution must be working. This is required for both the NFS server srv1 and the NFS client srv2.

Our FreeIPA server provides NTP and DNS services. Time synchronisation can be achieved by using ntpd, or chronyd. We use the latter in this article.

Install package on both srv1 and srv2:

# yum install -y chrony

Open /etc/chrony.conf and add the line pointing to our NTP server:

server ipa.rhce.local iburst

Enable and start the service:

# systemctl enable chronyd && systemctl start chronyd

Enable NTP based network time synchronisation:

# timedatectl set-ntp true

Check:

# chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* ipa.rhce.local                3   6    17    41  -1761ns[  +10us] +/-   83ms

Base NFS Server Setup

All commands in this section are run on the server srv1.

The nfs-utils package version used in the article is 1.3.0.

Packages, Services and Firewall

# yum install -y nfs-utils
# systemctl enable nfs-server && systemctl start nfs-server

We can optionally enable the rpcbind daemon to convert RPC program numbers into addresses, this is used by NFSv3, but not required for NFSv4.

# systemctl enable rpcbind && systemctl start rpcbind

On RHEL 7.0, we have to enable the nfs-secure-server service:

# systemctl enable nfs-secure-server

These are nfs related services available on RHEL 7.0:

# ls -lon /usr/lib/systemd/system/|grep " nfs-"
-rw-r--r--. 1 0 286 Mar 26 2014 nfs-blkmap.service
-rw-r--r--. 1 0 265 Mar 26 2014 nfs-idmap.service
-rw-r--r--. 1 0 491 Mar 26 2014 nfs-lock.service
lrwxrwxrwx. 1 0 16 Jun 3 21:02 nfslock.service -> nfs-lock.service
-rw-r--r--. 1 0 228 Mar 26 2014 nfs-mountd.service
-rw-r--r--. 1 0 263 Mar 26 2014 nfs-rquotad.service
-rw-r--r--. 1 0 332 Mar 26 2014 nfs-secure-server.service
-rw-r--r--. 1 0 279 Mar 26 2014 nfs-secure.service
-rw-r--r--. 1 0 650 Mar 26 2014 nfs-server.service
lrwxrwxrwx. 1 0 18 Jun 3 21:02 nfs.service -> nfs-server.service
lrwxrwxrwx. 1 0 18 Jun 3 21:02 rpcgssd.service -> nfs-secure.service
lrwxrwxrwx. 1 0 17 Jun 3 21:02 rpcidmapd.service -> nfs-idmap.service
lrwxrwxrwx. 1 0 25 Jun 3 21:02 rpcsvcgssd.service -> nfs-secure-server.service

On RHEL 7.1 and RHEL 7.2, the nfs-secure-server service becomes a static service and cannot be enabled. It is started by nfs-server service automatically assuming the file /etc/krb5.keytab is present on the system.

These are nfs related services that are available on RHEL 7.1 and RHEL 7.2:

# ls -lon /usr/lib/systemd/system/|grep " nfs-"
-rw-r--r--. 1 0 345 Jan 23 2015 nfs-blkmap.service
-rw-r--r--. 1 0 215 Jan 23 2015 nfs-blkmap.target
-rw-r--r--. 1 0 477 Jan 23 2015 nfs-client.target
-rw-r--r--. 1 0 144 Jan 23 2015 nfs-config.service
-rw-r--r--. 1 0 240 Jan 23 2015 nfs-idmapd.service
lrwxrwxrwx. 1 0 18 Jun 23 12:41 nfs-idmap.service -> nfs-idmapd.service
lrwxrwxrwx. 1 0 17 Jun 23 12:41 nfs-lock.service -> rpc-statd.service
-rw-r--r--. 1 0 300 Jan 23 2015 nfs-mountd.service
lrwxrwxrwx. 1 0 19 Jun 23 12:41 nfs-secure-server.service -> rpc-svcgssd.service
lrwxrwxrwx. 1 0 16 Jun 23 12:41 nfs-secure.service -> rpc-gssd.service
-rw-r--r--. 1 0 881 Jan 23 2015 nfs-server.service
lrwxrwxrwx. 1 0 18 Jun 23 12:41 nfs.service -> nfs-server.service
-rw-r--r--. 1 0 567 Jan 23 2015 nfs-utils.service

Notice that the nfs-idmap service becomes a symbolic link to nfs-idmapd.

On RHEL 7.0, don’t start the nfs-secure-server just yet, as there is no keytab file created. Secure NFS server will fail without it.

Configure firewall:

# firewall-cmd --permanent --add-service=nfs
# firewall-cmd --reload

Be advised that rpc-bind and mountd services are only required by NFSv3, and can be skipped when setting up an NFSv4 server.

# firewall-cmd --permanent --add-service={mountd,rpc-bind}
# firewall-cmd --reload

Configure NFS Exports

We are going to create three different exports as explained below:

/srv/nfs_pub – a public NFS share with ‘all_squash’,
/srv/nfs_group – a share for group collaboration,
/srv/nfs_secure – a secure Kerberised NFS share.

Create directories:

# mkdir -m 0755 /srv/nfs_{pub,group,secure}

Change ownership for the public NFS share. All users will be squashed. We will map all uids and gids to the NFS anonymous user, in this case the nfsnobody account:

# chown nfsnobody:nfsnobody /srv/nfs_pub

Configure collaboration for the group share:

# groupadd devops
# chgrp devops /srv/nfs_group
# chmod 2770 /srv/nfs_group

For the kerberised NFS share, we are going to make an LDAP user alice the owner of the share so that we can easily test authentication:

# chown alice /srv/nfs_secure

Let us check the default SELinux context:

# ls -Z /srv
drwxrws---. root      devops    unconfined_u:object_r:var_t:s0   /srv/nfs_group
drwxr-xr-x. alice     root      unconfined_u:object_r:var_t:s0   /srv/nfs_secure
drwxr-xr-x. nfsnobody nfsnobody unconfined_u:object_r:var_t:s0   /srv/nfs_pub

For the public share, we want to the public_content_rw_t context type. For other shares, the nfs_t type should suffice.

# yum install -y policycoreutils-python
# semanage fcontext -a -t public_content_rw_t "/srv/nfs_pub(/.*)?"
# semanage fcontext -a -t nfs_t "/srv/nfs_group(/.*)?"
# semanage fcontext -a -t nfs_t "/srv/nfs_secure(/.*)?"

Note that files labeled with the public_content_t type allow them to be read by FTP, Apache, Samba and rsync. Files labeled with the public_content_rw_t type require booleans to be set before services can write to files labeled with the public_content_rw_t type.

To allow the NFS server to modify public files used for public file transfer services, we must turn on the nfsd_anon_write boolean:

# setsebool -P nfsd_anon_write=1

Restore SELinux context:

# restorecon -Rv /srv/nfs_*

Verify:

# ls -Z /srv
drwxrws---. root      devops    unconfined_u:object_r:nfs_t:s0   nfs_group
drwxr-xr-x. nfsnobody nfsnobody unconfined_u:object_r:public_content_rw_t:s0 nfs_pub
drwxr-xr-x. alice     root      unconfined_u:object_r:nfs_t:s0   nfs_secure

Open the file /etc/exports and add the following lines:

/srv/nfs_pub     10.8.8.0/24(sec=sys,rw,sync,root_squash,all_squash)
/srv/nfs_group   10.8.8.0/24(sec=sys,rw,sync,root_squash)
/srv/nfs_secure  srv2.rhce.local(sec=krb5p,rw,sync)

We can see that access to the secure share is allowed from the server srv2 only. Access to the other two shares is allowed from the whole LAN.

Export the shares:

# exportfs -rav
exporting srv2.rhce.local:/srv/nfs_secure
exporting 10.8.8.0/24:/srv/nfs_group
exporting 10.8.8.0/24:/srv/nfs_pub

Check what NFS protocols are enabled:

# cat /proc/fs/nfsd/versions
-2 +3 +4 +4.1 -4.2

By default, NFS mounts have the SELinux context nfs_t independently of the SELinux context they have on the NFS server. The NFS server can be forced to properly export the SELinux context of a share by switching to NFS version 4.2. Note that this option needs to be turned on explicitly.

Enable NFS v4.2 by adding the following line to the file /etc/sysconfig/nfs:

RPCNFSDARGS="-V 4.2"

On the client side, mount -o vers=4.2 must be specified as the mount option.

One other thing to check, by default the nfs_export_all_ro and the nfs_export_all_rw SELinux booleans are both enabled:

# getsebool -a | grep nfs_ex
nfs_export_all_ro --> on
nfs_export_all_rw --> on

This allows the NFS daemon to read and write almost any file. Disable the booleans to lock down the capabilities.

Kerberos Keytab File

There are a couple of ways of creating the keytab file /etc/krb5.keytab.

If we know the IPA admin credentials, we can obtain a Kerberos ticket, and use ipa-getkeytab command.

# yum install -y ipa-client

Obtain a Kerberos ticket and verify:

# kinit admin
# klist

Get a keytab for our Kerberos NFS principal:

# ipa-getkeytab -s ipa.rhce.local -p nfs/srv1.rhce.local -k /etc/krb5.keytab
Keytab successfully retrieved and stored in: /etc/krb5.keytab

Verify with:

# klist -k
   2 nfs/[email protected]
   2 nfs/[email protected]
   2 nfs/[email protected]
   2 nfs/[email protected]
   2 nfs/[email protected]
   2 nfs/[email protected]

Now if we don’t have Kerberos admin credentials, we can alternatively download the keytab file for the server via FTP (assuming it was made available on the Kerberos server):

# wget -O /etc/krb5.keytab ftp://ipa.rhce.local/pub/srv1.keytab
# chmod 0600 /etc/krb5.keytab
# restorecon -v /etc/krb5.keytab

Verify:

klist -k
   3 nfs/[email protected]
   3 nfs/[email protected]
   3 nfs/[email protected]
   3 nfs/[email protected]
   3 nfs/[email protected]
   3 nfs/[email protected]

Once the keytab is created, start the nfs-secure-server service:

# systemctl start nfs-secure-server

Check the NFS server status for any errors.

For verbose logging, open the file /etc/sysconfig/nfs, assign the -vvv string to the RPCGSSDARGS variable and restart the nfs-secure-server service.

Reboot the NFS server before proceeding to the client configuration. After restart, ensure that both services are running:

# systemctl status nfs-server nfs-secure-server

NFS Client Setup

All commands in this section are run on the server srv2.

Packages and Services

# yum install -y nfs-utils

On RHEL 7.0, enable the nfs-secure service:

# systemctl enable nfs-secure

On RHEL 7.1 and RHEL 7.2, the nfs-secure is a static service and cannot be enabled. It is started by the nfs-client.target service assuming the file /etc/krb5.keytab is present on the system:

# systemctl enable nfs-client.target

Mount a Kerberised NFS Share

If the NFSv3 server has firewall configured to allow mountd and rpcbind traffic, we can use the showmount command to verify that the client can see the shares:

# showmount -e srv1.rhce.local
Export list for srv1.rhce.local:
/srv/nfs_group  10.8.8.0/24
/srv/nfs_pub    10.8.8.0/24
/srv/nfs_secure srv2.rhce.local

Because NFSv4 does not use the mountd daemon, showmount will not return information about version 4 mounts.

As with the NFS server, if we know the Kerberos admin credentials, we can obtain the keytab with ipa-client:

# yum install -y ipa-client

Obtain a Kerberos ticket and verify:

# kinit admin
# klist

Get a keytab for our Kerberos NFS principal:

# ipa-getkeytab -s ipa.rhce.local -p nfs/srv2.rhce.local -k /etc/krb5.keytab
Keytab successfully retrieved and stored in: /etc/krb5.keytab

Verify with:

# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   4 nfs/[email protected]
   4 nfs/[email protected]
   4 nfs/[email protected]
   4 nfs/[email protected]
   4 nfs/[email protected]
   4 nfs/[email protected]

As we can see above, our keytab does not contain host credentials as we haven’t created them when setting up a FreeIPA server. Therefore it’s vital that our NFS client is configured for Kerberos authentication (with authconfig-tui f.e.), otherwise we won’t be able to mount the kerberised NFS share, and rpc.gssd will complain a lot:

ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry found in keytab /etc/krb5.keytab for connection with host srv1.rhce.local

If the Kerberos credentials are not available, we can get the keytab from FTP (as we have configured it for our system). Download the keytab file from the Kerberos server:

# wget ftp://ipa.rhce.local/pub/srv2.keytab -O /etc/krb5.keytab
# chmod 0600 /etc/krb5.keytab 
# restorecon -v /etc/krb5.keytab

# klist
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_0)

# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   5 nfs/[email protected]
   5 nfs/[email protected]
   5 nfs/[email protected]
   5 nfs/[email protected]
   5 nfs/[email protected]
   5 nfs/[email protected]

Start the nfs-secure service:

# systemctl start nfs-secure

For verbose logging, open the file /etc/sysconfig/nfs, assign the -vvv string to the RPCGSSDARGS variable and restart the nfs-secure service. You may need it if you run into problems.

Mount the share:

# mkdir /mnt/nfs_secure
# mount -t nfs4 -o vers=4.2,sec=krb5p srv1.rhce.local:/srv/nfs_secure /mnt/nfs_secure

If you get the error message “mount.nfs: an incorrect mount option was specified“, check that you started nfs-secure daemon on the client and nfs-secure-server on the server.

Verify:

# mount | grep secure
srv1.rhce.local:/srv/nfs_secure on /mnt/nfs_secure type nfs (rw,relatime,seclabel,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=10.8.8.72,local_lock=none,addr=10.8.8.71)

To mount on boot, add the following to /etc/fstab:

srv1.rhce.local:/srv/nfs_secure /mnt/nfs_secure nfs4 rw,vers=4.2,sec=krb5p 0 0

Note that the _netdev option should not be required to mount the NFSv4 share.

It is my understanding that under NFSv3 (mount type nfs), the _netdev option tells the system to wait to mount until the network is available. With a mount type of nfs4 the _netdev option is ignored as remote mounts are pulled by remote-fs.target.

However, Sander van Vugt recommends that all remote file systems that need to be mounted through /etc/fstab include the _netdev and the x-systemd.automount mount options.

The _netdev mount option ensures that the mount is delayed until the network is fully available. The x-systemd.automount option ensures optimal integration with systemd and will ensure that the mount is made a lot faster.

Test Kerberos, change to user alice to access the share:

# su - alice
Last login: Wed May 18 16:06:51 BST 2016 on pts/2
su: warning: cannot change directory to /home/alice: No such file or directory
id: cannot find name for group ID 1219400005
$ touch /mnt/nfs_secure/file
touch: cannot touch ‘/mnt/nfs_secure/file’: Permission denied
$ kinit
Password for [email protected]:
$ touch /mnt/nfs_secure/file
$ ls -l /mnt/nfs_secure/file
-rw-r--r--. 1 alice 1219400005 0 Jun  3 20:57 /mnt/nfs_secure/file
$ kdestroy
$ exit

Mount Non-Kerberised NFS Shares

Create mountpoints:

# mkdir /mnt/nfs_{pub,group}

Mount the public and group shares as these require no Kerberos configuration:

# mount -t nfs4 -o vers=4.2,sec=sys srv1.rhce.local:/srv/nfs_pub /mnt/nfs_pub 
# mount -t nfs4 -o vers=4.2,sec=sys srv1.rhce.local:/srv/nfs_group /mnt/nfs_group

Problems With Key Version Numbers

Sometimes the key version number (KVNO) used by the KDC and the service principal keys stored in /etc/krb5.keytab for services hosted on the system do not match. The KVNO can get out of synchronisation when a new set of keys are created on the KDC without updating the keytable file with the new keys.

To diagnose, list the keytab entries:

# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   3 nfs/[email protected]
   3 nfs/[email protected]
   3 nfs/[email protected]
   3 nfs/[email protected]
   3 nfs/[email protected]
   3 nfs/[email protected]

Acquire Kerberos ticket by using the host key:

# kinit -k

Or use admin credentials:

# kinit admin

Check the KVNO that is used by the KDC:

# kvno nfs/srv2.rhce.local
nfs/[email protected]: kvno = 4

References

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/kerb-nfs.html

159 thoughts on “Setting up a Kerberised NFS Server on RHEL 7”

Martin Chamambo says:

21/07/2016 at 9:14 am
Good day

I have been reading your site alongside the certdeport one after finishing the sander’s vedios and jangs book on RHEL 7…and i am really impressed…I will be writing my RHCE on the 23rd of Septemeber and i think i have managed to prepare very ,well thanks to all those sources and mainly this one for being specific.

I have a few grey areas ,the kerberos server can be the redhat ipa or the standalone server installed from stand alone packages…

my questions this config below when configuring the redhat cleint
```
[libdefaults]
default_realm = EXAMPLE.COM
dns_lookup_realm = false
dns_lookup_kdc = false
rdns = false
forwardable = yes
ticket_lifetime = 24h

[realms]
EXAMPLE.COM = {
      kdc = ipaserver.example.com:88
      admin_server = ipaserver.example.com:749
      }
[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM
```
the above is for the redhat IPA configuration
and for a standalone configuration there are no ports 88 and port 749
my question is ,how do you then know how to configure a client without knowing the ports if its a redhat IPA server or not ?
Reply
- Lisenet says:
  
  21/07/2016 at 9:20 am
  
  I’m glad you found it helpful. To configure a Kerberos client I use authconfig-tui. You need to know the REALM and have DNS configured to point to the Kerberos server. No manual configuration of krb5.conf is required.
- Ami says:
  
  29/10/2018 at 7:07 am
  
  Hello Martin,
  I’m also impressed from your detail level, however somewhat confused…(Regarding keytab).
  I’ve also seen the krb nfs lesson from Sander, ITFountains and more….
  Both,
  demonstrated they are generating the keytab only for the principal of nfs/server1.example.com and copied the SAME file to BOTH ,server1 AND server2 /etc/krb5.keytab location,
  While,
  you’ve generated two different keytabs, one for server1 and the other for server 2 and place the keytab file in each /etc folder…..
  So,
  how server2 will be able to mount server1:/exports if it does not have its principal service embedded in it keytab?
  Thanks in adv.
Martin Chamambo says:

21/07/2016 at 3:52 pm

@tomas so basically what you are saying is the kerberos server should provide DNS services for the client to pick it up? i have a KDC configured but with no DNS service running and it seems i have to configure the clients via authconfig-tui but i have to manually place the server name and ports when it asks for the kdc and the kdc_admin server

Reply
- Lisenet says:
  
  21/07/2016 at 4:41 pm
  
  You need to ensure the hostname resolution is working. It can be a DNS server, or it can be static entries in /etc/hosts, up to you really.
Martin Chamambo says:

21/07/2016 at 11:35 pm

another final question i have is ,why shouldnt i use ipa-client-install

Reply
- Martin Chamambo says:
  
  21/07/2016 at 11:38 pm
  
  I have also noticed that the /etc/sssd/sssd.conf file does get automatically created which makes the whole process complex
- Lisenet says:
  
  22/07/2016 at 7:59 am
  
  You are free to choose between nslcd and sssd. Use the one that you are comfortable with.
- Lisenet says:
  
  22/07/2016 at 7:58 am
  
  To use ipa-client-install, you need to know the Kerberos admin password, otherwise you cannot join the domain. I doubt such information will be supplied during the RHCE exam.
Martin Chamambo says:

22/07/2016 at 9:12 am

Thank you @tomas , i had actually overlooked the fact that there is a password needed to join the IPA server and its mostly like not going to be supplied during the exam and that makes sense to use the other method…

I luv sssd because its newer so i will stick to sssd.

Reply
Jarrod says:

17/08/2016 at 2:06 pm

Interesting information regarding _netdev, I spoke to a Red Hat trainer who advised that it should not be required for SMB/NFS mounting, because when you mount the file system is either specified or automatically detected as either Samba or NFS, and as such it knows that these mounts should come up after the network. Despite this I think I’ll still add it in just to be safe!

Reply
- Lisenet says:
  
  17/08/2016 at 3:49 pm
  
  It’s not required indeed as under systemd remote mounts are pulled by remote-fs.target.
Martin Chamambo says:

20/08/2016 at 8:08 am

@tomas confirm something on this nfs share below

/srv/nfs_group – a share for group collaboration,

I am sure we need the devops groups on the client also ,with users in it to be able to create files and directories

Reply
- Lisenet says:
  
  20/08/2016 at 10:08 pm
  
  Yep.
Rex says:

27/09/2016 at 1:16 pm

Omg! Because of your tutorial I’ve finally got my Kerberized NFS setup to work using CentOS 7.2. I could finally move on with the rest of my labs. Thank you very much!

Reply
- Lisenet says:
  
  27/09/2016 at 2:46 pm
  
  Welcome!
Alex says:

22/11/2016 at 7:54 pm

Hey Tomas, I cannot connect to a Kerberized share in my CentOS 7.2 environment. The service is registered and the key is downloaded from IPA server, LDAP clients are configured correctly. nfs-server service is started on both server and client, but on client, I cannot start nfs-client service:

[root@server2 /]# systemctl start nfs-client
Failed to start nfs-client.service: Unit nfs-client.service failed to load: No such file or directory.

So, getting this:

[root@server2 /]# mount -o sec=krb5p server1.example.com:/nfs-secureshare /nfssecure
mount.nfs: an incorrect mount option was specified
[root@server2 /]# mount -o sec=sys server1.example.com:/nfs-secureshare /nfssecure
mount.nfs: access denied by server while mounting server1.example.com:/nfs-secureshare

Reply
- Lisenet says:
  
  22/11/2016 at 7:58 pm
  
  The error is self-explanatory, the service file does not exist.
  
  You need a target, not a service. It’s explained in the blog post.
Alex says:

23/11/2016 at 9:14 pm

Yes, thank, Tomas! I have started the target, now I have this:
[root@server2 ~]# mount -o sec=krb5p -V 4.2 server1.example.com:/nfs-secureshare /nfssecure
mount from util-linux 2.23.2 (libmount 2.23.0: selinux, debug, assert)
[root@server2 ~]# echo $?
0
[root@server2 ~]# mount | grep nfssecure
[root@server2 ~]#

/var/log/messages shows nothing

Reply
- Lisenet says:
  
  23/11/2016 at 9:16 pm
  
  I’m glad you figured it out.
- Alex says:
  
  23/11/2016 at 10:27 pm
  
  yes, the command was executed successfully, but the share is still not mounted ((
- Lisenet says:
  
  23/11/2016 at 10:28 pm
  
  What does the journal say? Any issues?
Alex says:

23/11/2016 at 10:29 pm

[root@server2 ~]# mount | grep nfs
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
[root@server2 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 9.4G 1.1G 8.3G 12% /
devtmpfs 483M 0 483M 0% /dev
tmpfs 493M 0 493M 0% /dev/shm
tmpfs 493M 6.7M 486M 2% /run
tmpfs 493M 0 493M 0% /sys/fs/cgroup
/dev/sda1 497M 166M 332M 34% /boot
//server1.example.com/smb 9.4G 1.3G 8.1G 14% /smb
tmpfs 99M 0 99M 0% /run/user/0

Reply
- Lisenet says:
  
  23/11/2016 at 10:32 pm
  
  This is useless information I’m afraid and doesn’t help at all.
  
  Try mounting in verbose mode and post the logs.
Alex says:

23/11/2016 at 11:00 pm

-v option for “mount” did not show anything. And I dont know where to see the logs (/var/log/messages shows nothing) Here is strace output (you may not to post it all here, just have a look):

[root@server2 ~]# strace -vts1024 mount -o sec=krb5p -V 4.2 server1.example.com:/nfs-secureshare /nfssecure

Reply
- Lisenet says:
  
  23/11/2016 at 11:16 pm
  
  See “man mount” for the -V parameter that you use. You should figure it out.
  
  Did you read the blog post? There is an example on how to mount a Kerberised NFS share.
Alex says:

24/11/2016 at 12:44 pm

yes, I have read it more carefully now:

[root@server2 ~]# mount -t nfs4 -o sec=krb5p,vers=4.2 server1.example.com:/nfs-secureshare /nfssecure –verbose
mount.nfs4: timeout set for Thu Nov 24 02:08:11 2016
mount.nfs4: trying text-based options ‘sec=krb5p,vers=4.2,addr=10.39.92.67,clientaddr=10.39.92.77’
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting server1.example.com:/nfs-secureshare

I have a few issues here: time is not synced between server and client (chronyd is configured to sync with IPA server, ports are opened and tcpdump shows traffic on port 123, timedatectl set-ntp true)

[root@server2 ~]# systemctl status nfs-secure
Б≈▐ rpc-gssd.service – RPC security service for NFS client and server
Loaded: loaded (/usr/lib/systemd/system/rpc-gssd.service; static; vendor preset: disabled)
Active: active (running) since Wed 2016-11-23 23:25:31 +07; 50min ago
ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry found in keytab /etc/krb5.keytab for connection with host

Key version is not 4:

[root@server2 ~]# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
—- ————————————————————————–
1 NFS/[email protected]
1 NFS/[email protected]
1 NFS/[email protected]
1 NFS/[email protected]

I will dig into it all….

Reply
- Lisenet says:
  
  24/11/2016 at 1:23 pm
  
  “no usable keytab entry found in keytab /etc/krb5.keytab for connection with host” – what’s the KVNO that is used by the KDC (FreeIPA)?
Alex says:

24/11/2016 at 3:54 pm

[root@ipa ~]# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
—- ————————————————————————–
2 host/[email protected]
2 host/[email protected]
2 host/[email protected]
2 host/[email protected]
2 host/[email protected]
2 host/[email protected]

[root@server1example ~]# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
—- ————————————————————————–
1 NFS/[email protected]
1 NFS/[email protected]
1 NFS/[email protected]
1 NFS/[email protected]

[root@server2 ~]# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
—- ————————————————————————–
1 NFS/[email protected]
1 NFS/[email protected]
1 NFS/[email protected]
1 NFS/[email protected]

Reply
- Lisenet says:
  
  24/11/2016 at 4:53 pm
  
  Your KVNO looks out of sync. Can you verify with the “kvno” command?
- Alex says:
  
  24/11/2016 at 5:16 pm
  
  [root@ipa ~]# kvno NFS/[email protected]
  NFS/[email protected]: kvno = 1
- Alex says:
  
  24/11/2016 at 5:23 pm
  
  updating the keytable file, isn’t it? how to do it?
- Lisenet says:
  
  24/11/2016 at 5:26 pm
  
  By using kadmin.local, see here for details. You can also use ipa-getkeytab to generate the keys but do not save them in the host keytab.
Alex says:

24/11/2016 at 4:07 pm

time is synced in this environment

Reply
Alex says:

24/11/2016 at 6:28 pm

kadmin.local
ktadd NFS/server1.example.com
ktadd NFS/server2.example.com
quit

ipa-getkeytab -s ipa.example.com -p NFS/server1.example.com -k /root/server1.nfs.keytab
ipa-getkeytab -s ipa.example.com -p NFS/server2.example.com -k /root/server2.nfs.keytab

copy them to server1 and server2 to /etc/krb5.tab

Is everything correct?

Reply
- Lisenet says:
  
  25/11/2016 at 12:22 am
  
  Does it say that a keytab was successfully retrieved? If so, then looks good.
- Alex says:
  
  25/11/2016 at 1:18 am
  
  yes, it does, I will copy the keys to the servers and will let you know it the issue is fixed
Alex says:

25/11/2016 at 1:33 am

[root@server2 ~]# mount -o vers=4.2,sec=krb5p server1.example.com:/nfs-secureshare /nfssecure –verbose
mount.nfs: timeout set for Fri Nov 25 08:31:13 2016
mount.nfs: trying text-based options ‘vers=4.2,sec=krb5p,addr=192.168.0.102,clientaddr=192.168.0.104’
mount.nfs: mount(2): Invalid argument
mount.nfs: an incorrect mount option was specified

[root@server2 ~]# systemctl status nfs-secure
● rpc-gssd.service – RPC security service for NFS client and server
Loaded: loaded (/usr/lib/systemd/system/rpc-gssd.service; static; vendor preset: disabled)
Active: active (running) since Fri 2016-11-25 08:26:42 +07; 5s ago
Process: 3139 ExecStart=/usr/sbin/rpc.gssd $GSSDARGS (code=exited, status=0/SUCCESS)
Main PID: 3140 (rpc.gssd)
CGroup: /system.slice/rpc-gssd.service
└─3140 /usr/sbin/rpc.gssd

[root@server1example ~]# systemctl status nfs-secure-server
● rpc-svcgssd.service – RPC security service for NFS server
Loaded: loaded (/usr/lib/systemd/system/rpc-svcgssd.service; static; vendor preset: disabled)
Active: inactive (dead)
Condition: start condition failed at Fri 2016-11-25 08:28:00 +07; 24s ago
none of the trigger conditions were met

This is CentOS 7.2 and nfs-secure-servver should be autostarted by nfs-server, right?

Reply
- Lisenet says:
  
  25/11/2016 at 9:40 am
  
  Yeah, it should be, providing the file /etc/krb5.keytab is accessible on the system.
- Lisenet says:
  
  25/11/2016 at 10:31 am
  
  I followed the procedure described in the blog post, and mounted a Kerberised NFS share on RHEL 7.2 without any issues.
Alex says:

25/11/2016 at 2:59 pm

mount.nfs4: trying text-based options ‘sec=krb5p,vers=4.2,addr=192.168.0.102,clientaddr=192.168.0.104’
mount.nfs4: mount(2): Invalid argument
mount.nfs4: an incorrect mount option was specified

[root@server1 ~]# cat /proc/fs/nfsd/versions
-2 +3 +4 +4.1 +4.2

rolled-back the whole environment and reinstalled the services and the keys. now getting this.
knvo shows the same key versions.
investigating further. from the IPA server installation instructions, I haven’t configured DNS zone transfer
investigating this further…

Reply
- Lisenet says:
  
  25/11/2016 at 3:19 pm
  
  Assign verbose logging (-vvv) to the RPCGSSDARGS and check logs for clues.
- Eryk says:
  
  10/12/2016 at 6:00 pm
  
  vers=4.2 does not work for RHEL7.0
  Therefore if you use 7.0 do not edit /ets/sysconfig/nfs at all. I will use nfs4 and fall-back to nfs3 if necessary
- Lisenet says:
  
  10/12/2016 at 6:08 pm
  
  Please check your facts, NFS vers=4.2 does work on RHEL 7.0.
Alex says:

25/11/2016 at 5:46 pm

set it, /var/log/messages shows nothing.
its strange: kvno version is changes randomly:

ipa
2 host/[email protected]
1 nfs/[email protected]
1 nfs/[email protected]

server1
2 nfs/[email protected]
server2
2 nfs/[email protected]

previously, with the same actions done, the versions on the servers were 1
still getting
mount.nfs: mount(2): Invalid argument
mount.nfs: an incorrect mount option was specified

Reply
- Lisenet says:
  
  25/11/2016 at 7:07 pm
  
  I find it hard to believe, there should be something logged for rpc.gssd.
Lisenet says:

25/11/2016 at 7:09 pm

That explains it. Although this is a requirement, I’ve updated the article to make it clear.

Reply
Alex says:

25/11/2016 at 7:30 pm

YES! nfs-secure service can be started on client sustem only if proper keytab is installed

Reply
- Lisenet says:
  
  25/11/2016 at 7:57 pm
  
  Indeed, and it’s mentioned in the blog post to make readers aware.
Alex says:

25/11/2016 at 7:31 pm

the following command can be used during troubleshooting to get the creds manually:

# kinit -k -t /etc/krb5.keytab nfs/[email protected]

I tried and got an error, something like cannot contact to KDC

Reply
Alex says:

25/11/2016 at 7:48 pm

when I used authconfig-tui to configure LDAP client, I set “User DNS to find servers”, something like that
as I can see, specifying ipa.example.com in /etc/hosts was not enough

so, to sum-up:

1. principals were not added to keytab
2. nfs version was not specified in /etc/sysconfig/nfs
3. DNS was not configured correctly

Thank you very much!

Reply
- Lisenet says:
  
  25/11/2016 at 7:58 pm
  
  There are two links mentioned for how to configure LDAP auth, one with SSSD and another with nslcd. I take if you haven’t checked them? :)
Alex says:

25/11/2016 at 8:14 pm

I was using instructions from itfountain.com to configure IPA server and LDAP client, sorry ))

Reply
- Lisenet says:
  
  25/11/2016 at 11:26 pm
  
  No worries. I’m really glad that you finally got it all working.
Alex says:

26/11/2016 at 2:39 pm

in your sample exam a keytab for srv1 is already provided on FTP
how will it be on the exam? both keytabs for srv1 and srv2 will be on FTP or I will make these files using ipa-getkeytab?

Reply
- Lisenet says:
  
  26/11/2016 at 7:39 pm
  
  Sorry Alex, I cannot tell you what’s on the exam.
Alex says:

26/11/2016 at 2:40 pm

if one keytab is already provided, where is the second one? if I kame it, the kvno versions can differ

Reply
Alex says:

06/12/2016 at 8:56 am

Tomas, I have Permission denied error accessing /srv/nfsssec share:

[root@server1 ~]# ls -Zd /srv/nfssec/
drwxr-xr-x. vince vince unconfined_u:object_r:nfs_t:s0 /srv/nfssec/

[root@server1 ~]# exportfs -avr
exporting 192.168.0.104/32:/srv/nfssec
[root@server1 ~]# cat /etc/exports
/srv/nfssec 192.168.0.104/32(rw,sync,sec=krb5p)

[root@server2 ~]# su – vince
Last login: Tue Dec 6 15:46:53 +07 2016 on tty1
su: warning: cannot change directory to /home/vince: No such file or directory
-sh-4.2$ cd /protected/
-sh: cd: /protected/: Permission denied
-sh-4.2$ exit
logout
[root@server2 ~]# mount | grep nfs
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
server1.example.com:/srv/nfssec on /protected type nfs4 (rw,relatime,seclabel,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=192.168.0.104,local_lock=none,addr=192.168.0.102)

locally, the /srv/nfssec dir is accessible:

[root@server1 ~]# su – vince
Last login: Tue Dec 6 15:47:17 +07 2016 on tty1
su: warning: cannot change directory to /home/vince: No such file or directory
-sh-4.2$ cd /srv/nfssec/
-sh-4.2$ touch vincelocal2
-sh-4.2$ ll
total 0
-rw-r–r–. 1 nfsnobody nfsnobody 0 Dec 3 10:16 dffdg
-rw-r–r–. 1 nfsnobody nfsnobody 0 Dec 5 19:52 rootfile
-rw-rw-r–. 1 vince vince 0 Dec 6 15:47 vincelocal
-rw-rw-r–. 1 vince vince 0 Dec 6 15:55 vincelocal2

Will check the logs…

Reply
- Lisenet says:
  
  06/12/2016 at 1:50 pm
  
  You forgot kinit after logging in with the LDAP user, that’s why you get that permission denied error.
Alex says:

08/12/2016 at 2:07 pm

Perfect!!!! Thanks alot ))

Reply
Alex says:

30/12/2016 at 6:31 pm

Hey Tomas, I am stuck again at this. Please advice:

mount.nfs4: timeout set for Sat Dec 31 01:29:26 2016
mount.nfs4: trying text-based options ‘vers=4.2,sec=krb5p,addr=192.168.0.102,clientaddr=192.168.0.104’
mount.nfs4: mount(2): Operation not permitted
mount.nfs4: Operation not permitted

log:
Dec 31 05:51:04 system2 rpc.gssd[4070]: WARNING: Failed to create krb5 context for user with uid 0 for server [email protected]
Dec 31 05:51:04 system2 rpc.gssd[4070]: WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5ccmachine_EXAMPLE.COM for server system1.example.com
Dec 31 05:51:04 system2 rpc.gssd[4070]: WARNING: Failed to create machine krb5 context with any credentials cache for server system1.example.com

Reply
- Alex says:
  
  06/01/2017 at 10:31 am
  
  fixed
- Muhammad Asif says:
  
  31/10/2017 at 3:27 pm
  
  I am also getting the same error..
  
  Please share the fix.
  
  thanks
Brian says:

04/01/2017 at 12:12 am

Hi, Tomas
Thanks for the sharing.
I got some problems when trying to start nfs-secure-server
[root@ipa~]# systemctl status nfs-secure-server
● rpc-svcgssd.service – RPC security service for NFS server
Loaded: loaded (/usr/lib/systemd/system/rpc-svcgssd.service; static; vendor preset: disabled)
Active: inactive (dead)
Condition: start condition failed at Fri 2016-11-25 08:28:00 +07; 24s ago
none of the trigger conditions were met

I’ve tried several times…. it failed again and again either on ipa server or srv1 ……
I followed the steps on you posting and everything seems fine until I tried to active nfs-secure-server on both ipa and srv1..
On ipa server, why my nfs-secure-server status is “static”, it won’t work even when I try to enable it , right?
What could be the common cause or do you have any suggestion?
Thanks for the help.

Reply
- Lisenet says:
  
  04/01/2017 at 2:57 pm
  
  On RHEL 7.1 and RHEL 7.2, the nfs-secure-server service becomes a static service and cannot be enabled. It is started by nfs-server service automatically assuming the file /etc/krb5.keytab is present on the system.
- Michael says:
  
  12/05/2017 at 5:07 pm
  
  This seems to be a bug in the status output. I’m seeing the exact same status output on my server however I can mount the share using nfs v4.2 and kerberos authentication.
  
  mount -o sec=krb5p,nfsvers=4.2,rw srv1.rhce.local:/srv/nfs_secure /nfssecure
  
  You should be able to then manage the files in the directory using the alice account.
- Lisenet says:
  
  19/05/2017 at 1:41 pm
  
  Thanks for your feedback Michael.
nihongo says:

16/01/2017 at 1:25 am

Hi, Tomas,

On RHEL 7.0 with no update, I found out that it is not enough to enable ‘nfs-secure’ service alone on the client side.
Because the ‘nfs-secure’ service can not start on reboot. The fix is to enable “nfs-server” as well.

This Youtube tutorial confirmed that.
https://www.youtube.com/watch?v=876AWmYQFOE&t=18s

Reply
- Lisenet says:
  
  16/01/2017 at 11:22 am
  
  It seems to be enough, are you sure that your issue isn’t somewhere else? I tested instructions numerous times, they work. The nfs-secure service does start on reboot on the client, I get no issues with that.
  
  I don’t have to enable the nfs-server service on the NFS client machine because I don’t need an NFS server on it.
Alex says:

17/01/2017 at 1:56 pm

it is enough, but it is not stable as hell: the Kerberized share can have “Failed to create machine krb5 context” error, then it automatically fixes, out of the blue. Then “operation not permitted” error can appear, and be fixed the same way.

Reply
- Lisenet says:
  
  17/01/2017 at 2:42 pm
  
  Although I haven’t encountered the issue you described myself, I agree that there may be bugs in software. If you think you found a bug, you may have more luck getting in touch with Red Hat and perhaps filling a bug report.
Alex says:

29/01/2017 at 7:40 pm

I have bad experience with thier support: they redirect the ticket to another team and then colse it in a week without any answer)

Reply
- Lisenet says:
  
  29/01/2017 at 7:45 pm
  
  Thanks for sharing.
hunter86_bg says:

15/04/2017 at 10:23 am

Hello friends,

While I was doing a sample exam I have noticed that the nfs-secure.service required a restart in order to mount the kerberized nfs.
Please note that this is RHEL 7.0 DVD repo and as I have not met this in higher versions – it’s probably fixed.

HOW TO DEBUG: As Tomas has mentioned enable “-vvv” flag for
RPCGSSDARGS=”-vvv”
RPCSVCGSSDARGS=”-vvv”
in both CLIENT and SERVER “/etc/sysconfig/nfs”

QUICK FIX: check the hostname is FQDN (static section) like this:
####################
[root@server2 ~]# hostnamectl
Static hostname: server2.example.com
####################

The reason , which I found 30 min later is the following:
1. I’m using hostnames like “ipa” , “server1” and “server2″ , while dns resolution still works (until you dig the short name) :
#############################
[root@server2 ~]# hostnamectl
Static hostname: server2
#############################
Of course dns resolves with short and FQDN , but this caused the issue that was logged as this:
#############################
[root@server2 ~]# systemctl status nfs-secure
nfs-secure.service – Secure NFS
Loaded: loaded (/etc/systemd/system/nfs-secure.service; enabled)
Active: active (running) since сб 2017-04-15 11:28:48 EEST; 25s ago
Process: 654 ExecStart=/usr/sbin/rpc.gssd $RPCGSSDARGS (code=exited, status=0/SUCCESS)
Main PID: 662 (rpc.gssd)
CGroup: /system.slice/nfs-secure.service
└─662 /usr/sbin/rpc.gssd -vvv

apr 15 11:28:48 server2 systemd[1]: Starting Secure NFS…
apr 15 11:28:48 server2 rpc.gssd[662]: beginning poll
apr 15 11:28:48 server2 systemd[1]: Started Secure NFS.
apr 15 11:28:55 server2 rpc.gssd[662]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt0)
apr 15 11:28:55 server2 rpc.gssd[662]: handle_gssd_upcall: ‘mech=krb5 uid=0 service=* enctypes=18,17,16,23,3,1,2 ‘
apr 15 11:28:55 server2 rpc.gssd[1405]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt0)
apr 15 11:28:55 server2 rpc.gssd[1405]: process_krb5_upcall: service is ‘*’
apr 15 11:28:55 server2 rpc.gssd[1405]: Name or service not known while getting full hostname for ‘server1.example.com’
apr 15 11:28:55 server2 rpc.gssd[662]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt0)
apr 15 11:28:55 server2 rpc.gssd[662]: handle_gssd_upcall: ‘mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ‘
apr 15 11:28:55 server2 rpc.gssd[1406]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt0)
apr 15 11:28:55 server2 rpc.gssd[1406]: process_krb5_upcall: service is ”
apr 15 11:28:55 server2 rpc.gssd[662]: Closing ‘gssd’ pipe for /var/lib/nfs/rpc_pipefs/nfs/clnt0
apr 15 11:28:55 server2 rpc.gssd[662]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt1
apr 15 11:28:55 server2 rpc.gssd[662]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt0
[root@server2 ~]# mount -a
mount.nfs4: access denied by server while mounting server1.example.com:/srv/secure

[root@server2 ~]# systemctl status nfs-secure
nfs-secure.service – Secure NFS
Loaded: loaded (/etc/systemd/system/nfs-secure.service; enabled)
Active: active (running) since сб 2017-04-15 11:30:04 EEST; 6s ago
Process: 2515 ExecStart=/usr/sbin/rpc.gssd $RPCGSSDARGS (code=exited, status=0/SUCCESS)
Main PID: 2516 (rpc.gssd)
CGroup: /system.slice/nfs-secure.service
└─2516 /usr/sbin/rpc.gssd -vvv

apr 15 11:30:06 server2 rpc.gssd[2536]: using FILE:/tmp/krb5ccmachine_EXAMPLE.COM as credentials cache for machine creds
apr 15 11:30:06 server2 rpc.gssd[2536]: using environment variable to select krb5 ccache FILE:/tmp/krb5ccmachine_EXAMPLE.COM
apr 15 11:30:06 server2 rpc.gssd[2536]: creating tcp client for server server1.example.com
apr 15 11:30:06 server2 rpc.gssd[2536]: DEBUG: port already set to 2049
apr 15 11:30:06 server2 rpc.gssd[2536]: creating context with server [email protected]
apr 15 11:30:06 server2 rpc.gssd[2536]: DEBUG: serialize_krb5_ctx: lucid version!
apr 15 11:30:06 server2 rpc.gssd[2536]: prepare_krb5_rfc4121_buffer: protocol 1
apr 15 11:30:06 server2 rpc.gssd[2536]: prepare_krb5_rfc4121_buffer: serializing key with enctype 18 and size 32
apr 15 11:30:06 server2 rpc.gssd[2516]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt8
apr 15 11:30:06 server2 rpc.gssd[2516]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt7
#############################

Reply
samuel sappa says:

29/06/2017 at 4:09 am

hi tomas,
for nfs client with user, when the system reboot the user must must execute the kinit command again to have access in the shared dir, is this behaviour normal ? since the exam require all the configuration must persist after reboot
can we make it persistent ?

Reply
- Lisenet says:
  
  02/07/2017 at 10:04 pm
  
  A user has to have a valid Kerberos ticket, therefore it is normal behaviour.
ZsZs says:

29/07/2017 at 9:47 pm

Hi Tomas,
Thanks for your great articles.
Just wanted to contribute with following info.
Mounting a kerberized share on stock RHEL 7.2 (both server and client) results following output:
mount.nfs4: access denied by server while mounting srv1.rhce.local:/srv/nfs_secure

In srv1.rhce.local:/var/log/messages you’ll see:
sqlite_insert_client: insert statement prepare failed: table clients has 2 columns but 3 values were supplied

This is a known bug access.redhat.com/solutions/2065873 and has been fixed in:
– nfs-utils-1.3.0-0.33.el7_3.x86_64.rpm
– libtirpc-0.2.4-0.8.el7_3.x86_64.rpm
Of course one need an active subscription to download them separately, or extract them from RHEL7.3 ISO.
HTH

Reply
- Lisenet says:
  
  30/07/2017 at 11:17 am
  
  Thanks for your feedback, much appreciated.
  
  You can get a no-cost RHEL Developer Subscription by registering with the Red Hat Developers Program. It includes RHEL 7, High Availability Add-On etc as well as provides access to Red Hat knowledgebase, forums and updates. I found it useful when studying for RHCE exam.
Sultan says:

21/08/2017 at 9:22 am

Hello,
Thank you very much for this well-explained article. I have one issue and need to know if there is any workaround of it.
I am using CentOS 7.0 and have been able to set-up complete Kerberised NFS environment using ipa-client without any problem following this article as well as Sander’s videos. Sander recommend ipa-client procedure which assume that IPA admin credentials will provided in the exam. However, this is not sure. Therefore, I have tried using authconfig-tui procedure. In this procedure, there is no problem in nfs-server side set-up. However, on nfs-client side, nfs-secure failed to start in its first attempt and gives below error.

rpc.gssd[17099]: segfault at 0 ip 00007f9812d63f0a sp 00007fffabe086e8 error 4 in libc-2.17.so[7f9812c32000+1b6000]
rpc.gssd[747]: WARNING: forked child was killed with signal 11

However, if I restart nfs-secure after that fail, it works fine afterwards. When I include the mount in /etc/fstab, nfs-secure crash during boot time and give above error message and if I don’t include in /etc/fstab, this error comes at first mount time. After that, if I restart nfs-secure service, everything works fine afterwards.

Keeping everything same on nfs-server side, if I use ipa-client-install procedure on nfs-client side, again everything goes perfect but I have to have IPA admin credentials.

This was a bug ( https://bugzilla.redhat.com/show_bug.cgi?id=1032379) which has been resolved in the later release. However in exam, if we’ll have 7.0 release and will not have IPA admin credentials, then what would be the workaround?

Reply
- Lisenet says:
  
  21/08/2017 at 9:57 am
  
  It is my understanding that you will get a version of RHEL which has this bug resolved.
Alex says:

21/08/2017 at 11:22 am

Hi Tomas,
Maybe you can point me to my mistake with NFS? I’ve encountered an error that an LDAP user cannot access mounted krb5p share on the system2. It shows permission denied. And on system1 in /var/log/messages, nslcd service reported something like that the user is not found in validusers for nfs/…..
this user was an owner of the share on system1 and permissions were 777. the error was the same with disabled firewalld/selinux.

Reply
- Lisenet says:
  
  21/08/2017 at 3:52 pm
  
  You need to have a valid kerberos ticket for the LDAP user.
Sultan says:

21/08/2017 at 4:35 pm

Thank you very much for your quick reply. I also believe that such bugs will at least be fixed in the version of RHEL provided in the exam. I spent one whole day to figure out the exact problem which I summarized in my previous message but this bug has led me a new and very useful learning and got the solution this morning.

Actually, the IPA admin credentials are only required by ipa-client-install to generate the kytab file and if we already have keytab file, there is an option in ipa-client-install to give that keytab file i.e

-k KEYTAB, –keytab=KEYTAB path to backed up keytab from previous enrollment

After giving this option, ipa-client-install don’t ask admin credentials.
Therefore, if you have keytab file, use below command instead of authconfig-tui
ipa-client-install -k KEYTABFILE

I have tested on the same (fresh) RHEL7.0 system without any update or patch fix and it works and also have no issue by putting the two (public and secure krb5p) mounts in /etc/fstab file.

Conclusion:
No need to use authconfig-tui utility at all.
If you have keytab file provided in the exam, use “ipa-client-install -k KEYTABFILE”.
Otherwise, you should have IPA admin credentials and use ipa-client-install without any option and provide IPA admin credentials when asked by ipa-client-install.

I have not seen this info anywhere on the internet and if anyone test this on his/her system, please let us know if it works or not.

Reply
- Lisenet says:
  
  21/08/2017 at 5:24 pm
  
  Thanks for your valuable input. If you have a keytab file, it should be sufficient to copy it to /etc/krb5.keytab providing that remote authentication against a Kerberos server is configured.
Timi says:

06/10/2017 at 6:38 pm

Hi i am setting up the LAB but when i need to get the keytabs is give me this message:

[root@server1 ~]# ipa-getkeytab -s ipa.rhce.local -p nfs/server1.rhce.local -k /etc/krb5.keytab
Unable to initialize STARTTLS session
Failed to bind to server!
Retrying with pre-4.0 keytab retrieval method…
Unable to initialize STARTTLS session
Failed to bind to server!
Failed to get keytab
[root@server1 ~]

Reply
- Timi says:
  
  06/10/2017 at 6:40 pm
  
  [root@server1 ~]# cat /etc/sssd/sssd.conf
  [domain/default]
  
  autofs_provider = ldap
  cache_credentials = True
  ldap_search_base = dc=rhce,dc=local
  id_provider = ldap
  auth_provider = krb5
  chpass_provider = krb5
  ldap_uri = ldap://ipa.rhce.local
  ldap_id_use_start_tls = True
  ldap_tls_cacertdir = /etc/openldap/cacerts
  ldap_tls_reqcert = never
  krb5_realm = RHCE.LOCAL
  krb5_server = ipa.rhce.local
  krb5_store_password_if_offline = True
  [sssd]
  services = nss, pam, autofs
  
  domains = default
  [nss]
  homedir_substring = /home
  
  [pam]
  
  [sudo]
- Lisenet says:
  
  06/10/2017 at 7:07 pm
  
  Which version of the FreeIPA server?
  
  Can you capture the exit status code of the ipa-getkeytab command and post it here?
  
  Is the LDAPS port open on the FreeIPA server and is there a service that listens on it?
  
  Did you configure the FreeIPA server as per these instructions:
  
  https://www.lisenet.com/2016/freeipa-server-on-rhel-7-centos-7/
  
  Make sure that the ipa-server version that you have matches the ones the blog post was written for, as otherwise it may be a configuration issue that’s relevant to your version.
Timi says:

17/10/2017 at 5:10 am

[root@ipa ~]# rpm -qa | grep ipa
ipa-common-4.5.0-21.el7.centos.1.2.noarch
ipa-server-dns-4.5.0-21.el7.centos.1.2.noarch
ipa-client-common-4.5.0-21.el7.centos.1.2.noarch
python2-ipalib-4.5.0-21.el7.centos.1.2.noarch
sssd-ipa-1.15.2-50.el7_4.2.x86_64
python-iniparse-0.4-9.el7.noarch
python-libipa_hbac-1.15.2-50.el7_4.2.x86_64
python-ipaddress-1.0.16-2.el7.noarch
python2-ipaclient-4.5.0-21.el7.centos.1.2.noarch
ipa-client-4.5.0-21.el7.centos.1.2.x86_64
python2-ipaserver-4.5.0-21.el7.centos.1.2.noarch
ipa-server-4.5.0-21.el7.centos.1.2.x86_64
ipa-server-common-4.5.0-21.el7.centos.1.2.noarch
[root@ipa ~]#
[root@server1 cacerts]# ipa-getkeytab -s ipa.rhce.local -p NFS/server1.rhce.local -k /etc/krb5.keytab
Unable to initialize STARTTLS session
Failed to bind to server!
Retrying with pre-4.0 keytab retrieval method…
Unable to initialize STARTTLS session
Failed to bind to server!
Failed to get keytab
[root@server1 cacerts]# echo $?
9
[root@server1 cacerts]#

i didnt invoke the ipa-client-install

Reply
- Lisenet says:
  
  17/10/2017 at 8:24 pm
  
  I didn’t use ipa-server 4.5.0. It may be a configuration issue that is relevant to this particular version. Let us know if you get this resolved as it may help others.
Timi says:

17/10/2017 at 9:04 pm

Hi i redone the client install for the server1 and server2.

490 ipa-client-install –mkhomedir –enable-dns-updates –force-ntpd
491 cd /etc/openldap/cacerts/
492 ls
493 elinks ftp://ipa.rhce.local/pub and got the ca.crt to /et/openldap/cacerts/
494 rm cacert.p12
495 su – alice
496 kinit admin
497 klist
498 klist -l
499 ipa-getkeytab -s labipa.example.com -p nfs/
500 ipa-getkeytab -s ipa.rhce.local -p nfs/server1.rhce.local -k /etc/krb5.keytab
and it got it.

the issue now is that after exporting the shares as:

[root@server1 ~]# cat /etc/exports
/srv/nfs_pub 10.1.1.0/24(sec=sys,rw,sync,root_squash,all_squash)
/srv/nfs_group 10.1.1.0/24(sec=sys,rw,sync,root_squash)
/srv/nfs_secure server2.rhce.local(sec=krb5p,rw,sync)

when i mount it on the server2 like:
[root@server2 ~]# mount -t nfs4 -o vers=4.2,sec=krb5p server1.rhce.local:/srv/nfs_secure /mnt/nfs_secure/
mount.nfs4: mounting server1.rhce.local:/srv/nfs_secure failed, reason given by server: No such file or directory
[root@server2 ~]#

i am lost now the OS is CentOS 7.3

Reply
Timi says:

17/10/2017 at 9:10 pm

[root@server2 ~]# mount -o sec=krb5p server1.rhce.local:/srv/nfs_secure /mnt/nfs_secure
mount.nfs: access denied by server while mounting server1.rhce.local:/srv/nfs_secure
[root@server2 ~]# mount -t nfs -o sec=krb5p server1.rhce.local:/srv/nfs_secure /mnt/nfs_secure
mount.nfs: access denied by server while mounting server1.rhce.local:/srv/nfs_secure
[root@server2 ~]# mount -t nfs4 -o sec=krb5p server1.rhce.local:/srv/nfs_secure /mnt/nfs_secure
mount.nfs4: mounting server1.rhce.local:/srv/nfs_secure failed, reason given by server: No such file or directory
[root@server2 ~]# mount -t nfs4 -o v4.2 sec=krb5p server1.rhce.local:/srv/nfs_secure /mnt/nfs_secure

Reply
- Lisenet says:
  
  17/10/2017 at 9:55 pm
  
  Does the NFS mount point exist? I’d suggest to enable verbose logging (-vvv) for the RPCGSSDARGS, then try mounting with the verbose mount option and see what gets recorded.
Timi says:

18/10/2017 at 12:33 am

SERVER1:
[root@server1 ~]# ls -lZHtas /srv
total 0
dr-xr-xr-x. root root system_u:object_r:root_t:s0 ..
drwxr-xr-x. root root system_u:object_r:var_t:s0 .
drwxrws—. root devops unconfined_u:object_r:nfs_t:s0 nfs_group
drwxr-xr-x. nfsnobody nfsnobody unconfined_u:object_r:public_content_rw_t:s0 nfs_pub
drwxr-xr-x. alice root unconfined_u:object_r:nfs_t:s0 nfs_secure
[root@server1 ~]#

SERVER2:
[root@server2 ~]# ls -lartshZ /mnt/
total 0
drwxr-xr-x. root root unconfined_u:object_r:mnt_t:s0 nfs_secure
drwxr-xr-x. root root system_u:object_r:mnt_t:s0 .
dr-xr-xr-x. root root system_u:object_r:root_t:s0 ..
[root@server2 ~]#

tried: mount -t nfs4 -o vers=4.2,sec=krb5p server1.rhce.local:/srv/nfs_secure /mnt/nfs_secure

Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: handle_gssd_upcall: ‘mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ‘ (nfs/clnte)
Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: krb5_use_machine_creds: uid 0 tgtname (null)
Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: Full hostname for ‘server1.rhce.local’ is ‘server1.rhce.local’
Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: Full hostname for ‘server2.rhce.local’ is ‘server2.rhce.local’
Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: No key table entry found for [email protected] while getting keytab entry for ‘[email protected]’
Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: No key table entry found for [email protected] while getting keytab entry for ‘[email protected]’
Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: No key table entry found for root/[email protected] while getting keytab entry for ‘root/[email protected]’
Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: Success getting keytab entry for ‘nfs/[email protected]’
Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: INFO: Credentials in CC ‘FILE:/tmp/krb5ccmachine_RHCE.LOCAL’ are good until 1508368319
Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: INFO: Credentials in CC ‘FILE:/tmp/krb5ccmachine_RHCE.LOCAL’ are good until 1508368319
Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: creating tcp client for server server1.rhce.local
Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: creating context with server [email protected]
Oct 17 19:27:44 server2.rhce.local rpc.gssd[885]: doing downcall: lifetime_rec=71521 [email protected]
[root@server2 ~]# date
Tue Oct 17 19:28:33 EDT 2017
[root@server2 ~]#
now i am clueless

Tim

Reply
Timi says:

18/10/2017 at 12:54 am

i nailed it down:

[root@server2 ~]# mount -t nfs4 -o vers=4.2,sec=krb5p server1.rhce.local:/srv/nfs_secure /mnt/nfs_secure
mount.nfs4: mounting server1.rhce.local:/srv/nfs_secure failed, reason given by server: No such file or directory
[root@server2 ~]# man mount.nfs4
[root@server2 ~]# mount -t nfs4 -v -o vers=4.2,sec=krb5p server1.rhce.local:/srv/nfs_secure /mnt/nfs_secure
mount.nfs4: timeout set for Tue Oct 17 19:49:46 2017
mount.nfs4: trying text-based options ‘sec=krb5p,vers=4.1,addr=192.168.4.101,clientaddr=192.168.4.102’
mount.nfs4: mount(2): No such file or directory
mount.nfs4: mounting server1.rhce.local:/srv/nfs_secure failed, reason given by server: No such file or directory

i had setup the IPA with the reverse DNS entries for both my networks 192.168.4.0/24 and the 10.1.1.0 but i exported it only for server2.rhce.local

and the reverse DNS that its doing is 192.168.4.101 instead of 10.1.1.101 as here:

[root@server2 ~]# mount -t nfs4 -v -o vers=4.2,sec=krb5p 10.1.1.101:/srv/nfs_secure /mnt/nfs_secure
mount.nfs4: timeout set for Tue Oct 17 19:50:19 2017
mount.nfs4: trying text-based options ‘sec=krb5p,vers=4.1,addr=10.1.1.101,clientaddr=10.1.1.102’
[root@server2 ~]#

[root@server2 ~]# findmnt | grep secure
└─/mnt/nfs_secure 10.1.1.101:/srv/nfs_secure nfs4 rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=10.1.1.102,local_lock=none,addr=10.1.1.101
[root@server2 ~]#

i tracked the issue that when i did the IPA from start i had put the internet eth in manual and when installed IPA it regiestered it in DNS.
so the total point was to use the ca.crt

Tim

Reply
- Lisenet says:
  
  18/10/2017 at 9:16 am
  
  Well done! Exactly the reason I suggested to try to mount with the verbose mount option, as now you can see the clientaddr.
Muhammad Asif says:

18/10/2017 at 2:43 pm

I know this question has already been asked on this page, but i guess i need help on this topic.
I am constantly getting these errors:

# showmount -e srv01.example.local
Export list for srv01.example.local:
/srv/nfs_secure *
/srv/nfs_group *
/srv/nfs_pub *

Error while mounting: ( mouting a root)

# mount -t nfs4 -o vers=4.2,sec=krb5p srv01.example.local:/srv/nfs_secure /mnt/
mount.nfs4: access denied by server while mounting srv01.example.local:/srv/nfs_secure

#tail -f /var/log/messages
Oct 18 18:27:21 localhost rpc.gssd[3587]: ERROR: No credentials found for connection to server srv01.example.local
Oct 18 18:27:21 localhost rpc.gssd[3587]: ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry found in keytab /etc/krb5.keytab for connection with host srv01.example.local
Oct 18 18:27:21 localhost rpc.gssd[3587]: ERROR: No credentials found for connection to server srv01.example.local
Oct 18 18:27:21 localhost rpc.gssd[3587]: ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry found in keytab /etc/krb5.keytab for connection with host srv01.example.local
Oct 18 18:27:21 localhost rpc.gssd[3587]: ERROR: No credentials found for connection to server srv01.example.local

#systemctl status nfs-secure -l
Oct 18 18:28:58 srv02.example.local rpc.gssd[3587]: ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry found in keytab /etc/krb5.keytab for connection with host srv01.example.local
Oct 18 18:28:58 srv02.example.local rpc.gssd[3587]: ERROR: No credentials found for connection to server srv01.example.local4

From srv01.example.local
# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
—- ————————————————————————–
2 nfs/[email protected]
2 nfs/[email protected]

From srv01.example.local
#klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
—- ————————————————————————–
2 nfs/[email protected]
2 nfs/[email protected]

Both the servers have NOT joined ipa domain. could this be the issue ?
I have tried 3 times and getting the same error on all machines.

if you have a youtube video for this topic, that will be helpfull for me.

Reply
- Lisenet says:
  
  18/10/2017 at 6:22 pm
  
  I don’t do youtube videos I’m afraid.
  
  You may want to check the answers that have been provided on this page as the solution may already be there.
- Timi says:
  
  20/10/2017 at 5:46 pm
  
  join it. with ipa-client i think. check on the web gui on ipa that the hosts are registered. you have to add the hosts to the IPA as on the guide here.
  
  Tim.
Muhammad Asif says:

31/10/2017 at 1:48 pm

I have setup this ipa and nfs things from scratch. using centos 7.0.
authentication works fine . when I copy the keytab files, i receive following error.

From server2,example.local
[root@server2 ~]# klist -k
Keytab name: FILE:/etc/krb5.keytab
klist: Unsupported key table format version number while starting keytab scan

From server ( ipa.example.local )
[root@ipa ~]# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
—- ————————————————————————–
2 host/[email protected]
2 host/[email protected]
2 host/[email protected]
2 host/[email protected]
1 nfs/[email protected]
1 nfs/[email protected]
1 nfs/[email protected]
1 nfs/[email protected]
1 nfs/[email protected]
1 nfs/[email protected]
1 nfs/[email protected]
1 nfs/[email protected]
[root@ipa ~]# kvno nfs/server2.example.local
nfs/[email protected]: kvno = 3

how to solve this error ???
thanks

Reply
Dennis Creemer says:

02/11/2017 at 9:55 am

In the description above we first have to change the file /srv/nfs_secure to alice on srv1:
chown alice /srv/nfs_secure
How is that possible if we don’t have first made the Kerberized connection with the ipa server?

Maybe i’m confused….or I don’t see it anymore!!!

Properly you have to do first on ipa-server:
kinit admin
ipa user-add alice
ipa passwd alice
After that on srv1 all commands for “Kerberos Keytab File”.
Then you are able to chown the /srv/nfs_secure file to alice and continue with “Configure NFS Exports”.

Thx for all Dennis

Reply
- Lisenet says:
  
  02/11/2017 at 10:09 am
  
  If you read the article again, you will notice the following line under “The Lab” section:
  
  “Servers srv1 and srv2 have been configured for LDAP and Kerberos authentication using sssd”.
eduardo says:

10/11/2017 at 4:47 pm

Hi,
I’ve followed all the steps and can’t start nfs-secure-server:

[root@server1 log]# systemctl -l status nfs-secure-server
● rpc-svcgssd.service – RPC security service for NFS server
Loaded: loaded (/usr/lib/systemd/system/rpc-svcgssd.service; static; vendor preset: disabled)
Active: inactive (dead)
Condition: start condition failed at Fri 2017-11-10 11:36:20 EST; 8min ago
none of the trigger conditions were met
Nov 10 11:18:21 server1.example.com systemd[1]: Started RPC security service for NFS server.
Nov 10 11:25:21 server1.example.com systemd[1]: Started RPC security service for NFS server.
Nov 10 11:25:25 server1.example.com systemd[1]: Started RPC security service for NFS server.
Nov 10 11:27:33 server1.example.com systemd[1]: Started RPC security service for NFS server.
Nov 10 11:36:20 server1.example.com systemd[1]: Started RPC security service for NFS server.

I’ve checked the files, permissions, SELinux settings, firewall settings, logs and also googled it and still can’t figure out what’s the problem. What logs can I check?

Reply
- Lisenet says:
  
  10/11/2017 at 7:30 pm
  
  Which version of RHEL are you using? The output on your OS indicates that nfs-secure-server is a static service and therefore cannot be enabled.
EDUARDO says:

10/11/2017 at 9:56 pm

Hi, it’s a centos 7.0. I downloaded the IPA server from Sander van Gundt’s website. Should I setup my own?

Reply
Barbie says:

14/02/2018 at 1:43 am

Hi Tomas, I’m having problem to mount sub directory in client even though shared from the server.Permission is correct.It says no /data/mount fount. Please Here’s the line from /etc/exports:
/data 192.168.10.200/24(rw,sync,no_root_squash)

Reply
- Lisenet says:
  
  14/02/2018 at 12:07 pm
  
  The export file looks good, how do you mount it?
Barbie says:

15/02/2018 at 2:13 am

My fstab entry is as below:
server1:/data /mnt/share nfs defaults 0 0

I tried this way too:
192.168.10.201:/data/mount /mnt/share nfs defaults 0 0

Reply
- Lisenet says:
  
  15/02/2018 at 12:57 pm
  What happens when you run the following command, where server1 is your NFS sever?
```
# mount -v -t nfs4 server1:/data /mnt/share
```
  Change nfs4 to nfs if you use NFSv3.
Andersson says:

28/02/2018 at 9:07 am

Hello,
it will be better if:
1) in the line “groupadd devops” we define the free GID number, for example “groupadd -g 9999 devops”
2) on an nfs client we should add a group with the same GID (“groupadd -g 9999 devops”) and add some users there: “usermod -G devops vince”
In other case, you’ll probably get “Permission denied” when you try to test access to this mount, because gids of the devops group may be different on a server and clients.

Reply
- Lisenet says:
  
  02/06/2018 at 12:43 pm
  
  Thanks for your feedback.

Nfs-secure-server is giving me some trouble only with SELinux enabled on NFS Kerberized server. I’m working on a RHEL 7.0.

If I disabled SELinux everything work smooth.

Anyone have already got into this? Can be a 7.0 issue?

[root@server1 ~]# systemctl restart nfs-secure-server.service 
Job for nfs-secure-server.service failed. See 'systemctl status nfs-secure-server.service' and 'journalctl -xn' for details.

[root@server1 ~]# journalctl -xn
Jun 02 11:30:51 server1.teo.lab python[2948]: SELinux is preventing /usr/sbin/rpc.svcgssd from read access on the file                                             
                                              *****  Plugin catchall (100. confidence) suggests   **************************
                                              
                                              If you believe that rpc.svcgssd should be allowed read access on the  file by default.
                                              Then you should report this as a bug.
                                              You can generate a local policy module to allow this access.
                                              Do
                                              allow this access for now by executing:
                                              # grep rpc.svcgssd /var/log/audit/audit.log | audit2allow -M mypol
                                              # semodule -i mypol.pp

Lisenet says:

02/06/2018 at 12:40 pm

What’s the actual file that SELinux is preventing read access to?
Teo says:

02/06/2018 at 12:43 pm

This was the line:
Jun 02 11:30:51 server1.teo.lab python[2948]: SELinux is preventing /usr/sbin/rpc.svcgssd from read access on the file

I solved it with this command on both server:
# restorecon -Rv /etc/krb5.keytab

Thanks
Lisenet says:

02/06/2018 at 12:47 pm

Thanks, so it looks like a SELinux label on the keytab file was not correct.

Dan says:

13/07/2018 at 4:28 pm

Hello,

I have been going over this for some time and I am stuck because I get an error from the nfs/client that says mount.nfs: access denied by serer while mounting srv1.rhce.local:/srv/nfs_secure. I get this when running the following command from the nfs/client as root.
mount -t nfs4 -o vers=4.2,sec=krb5 srv1.rhce.local:/srv/nfs_secure /mnt/nfs_secure

RHEL 7.4 server for IPA and another server for NFS
RHEL 7.4 workstation for nfs/client

All other steps are completed successfully.

Thanks

Reply
- Lisenet says:
  
  13/07/2018 at 5:33 pm
  
  Try enabling verbose logging as described in the article and see what happens.

I’ve followed your lab installation (at least I believe I did), and I’m facing an issue with Kerberized NFS that I can’t solve.

As suggested already, I added -vvv to both client and servers. This is the client behaviour :

[root@srv2 log]#  mount -t nfs4 -v -o sec=krb5p srv1.rhce.local:/srv/secureshare /srv/securenfs
mount.nfs4: timeout set for Sat Jul 28 18:10:10 2018
mount.nfs4: trying text-based options 'sec=krb5p,vers=4.1,addr=192.168.122.71,clientaddr=192.168.122.72'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting srv1.rhce.local:/srv/secureshare

[root@srv2 log]# systemctl status nfs-secure
 rpc-gssd.service - RPC security service for NFS client and server
   Loaded: loaded (/usr/lib/systemd/system/rpc-gssd.service; static; vendor preset: disabled)
   Active: active (running) since Sat 2018-07-28 17:59:45 CEST; 1min 35s ago
  Process: 691 ExecStart=/usr/sbin/rpc.gssd $GSSDARGS (code=exited, status=0/SUCCESS)
 Main PID: 698 (rpc.gssd)
   CGroup: /system.slice/rpc-gssd.service
           └─698 /usr/sbin/rpc.gssd -vvv

Jul 28 18:00:22 srv2.RHCE.LOCAL rpc.gssd[698]: No key table entry found for root/[email protected] while getting keytab entry for 'root/[email protected]'
Jul 28 18:00:22 srv2.RHCE.LOCAL rpc.gssd[698]: Success getting keytab entry for 'nfs/[email protected]'
Jul 28 18:00:22 srv2.RHCE.LOCAL rpc.gssd[698]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_RHCE.LOCAL' are good until 1532879989
Jul 28 18:00:22 srv2.RHCE.LOCAL rpc.gssd[698]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_RHCE.LOCAL' are good until 1532879989
Jul 28 18:00:22 srv2.RHCE.LOCAL rpc.gssd[698]: creating tcp client for server srv1
Jul 28 18:00:22 srv2.RHCE.LOCAL rpc.gssd[698]: creating context with server nfs@srv1
Jul 28 18:00:27 srv2.RHCE.LOCAL rpc.gssd[698]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@srv1
Jul 28 18:00:27 srv2.RHCE.LOCAL rpc.gssd[698]: WARNING: Failed to create machine krb5context with cred cache FILE:/tmp/krb5ccmachine_RHCE.LOCAL for server srv1
Jul 28 18:00:27 srv2.RHCE.LOCAL rpc.gssd[698]: WARNING: Failed to create machinekrb5 context with any credentialscache for server srv1
Jul 28 18:00:27 srv2.RHCE.LOCAL rpc.gssd[698]: doing error downcall

/var/log/messages extract :

Jul 28 18:10:15 srv2 rpc.gssd[698]: #012handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 ' (nfs/clnt0)
Jul 28 18:10:15 srv2 rpc.gssd[698]: krb5_use_machine_creds: uid 0 tgtname (null)
Jul 28 18:10:15 srv2 rpc.gssd[698]: Full hostname for 'srv1' is 'srv1'
Jul 28 18:10:15 srv2 rpc.gssd[698]: Full hostname for 'srv2.rhce.local' is 'srv2.rhce.local'
Jul 28 18:10:15 srv2 rpc.gssd[698]: No key table entry found for [email protected] while getting keytab entry for '[email protected]'
Jul 28 18:10:15 srv2 rpc.gssd[698]: No key table entry found for [email protected] while getting keytab entry for '[email protected]'
Jul 28 18:10:15 srv2 rpc.gssd[698]: No key table entry found for root/[email protected] while getting keytab entry for 'root/[email protected]'
Jul 28 18:10:15 srv2 rpc.gssd[698]: Success getting keytab entry for 'nfs/[email protected]'
Jul 28 18:10:15 srv2 rpc.gssd[698]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_RHCE.LOCAL' are good until 1532879989
Jul 28 18:10:15 srv2 rpc.gssd[698]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_RHCE.LOCAL' are good until 1532879989
Jul 28 18:10:15 srv2 rpc.gssd[698]: creating tcp client for server srv1
Jul 28 18:10:15 srv2 rpc.gssd[698]: creating context with server nfs@srv1
Jul 28 18:10:20 srv2 rpc.gssd[698]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@srv1
Jul 28 18:10:20 srv2 rpc.gssd[698]: WARNING: Failed to create machine krb5context with cred cache FILE:/tmp/krb5ccmachine_RHCE.LOCAL for server srv1
Jul 28 18:10:20 srv2 rpc.gssd[698]: WARNING: Machine cache prematurelyexpired or corrupted trying torecreate cache for server srv1
Jul 28 18:10:20 srv2 rpc.gssd[698]: Full hostname for 'srv1' is 'srv1'
Jul 28 18:10:20 srv2 rpc.gssd[698]: Full hostname for 'srv2.rhce.local' is 'srv2.rhce.local'
Jul 28 18:10:20 srv2 rpc.gssd[698]: No key table entry found for [email protected] while getting keytab entry for '[email protected]'
Jul 28 18:10:20 srv2 rpc.gssd[698]: No key table entry found for [email protected] while getting keytab entry for '[email protected]'
Jul 28 18:10:20 srv2 rpc.gssd[698]: No key table entry found for root/[email protected] while getting keytab entry for 'root/[email protected]'
Jul 28 18:10:20 srv2 rpc.gssd[698]: Success getting keytab entry for 'nfs/[email protected]'
Jul 28 18:10:20 srv2 rpc.gssd[698]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_RHCE.LOCAL' are good until 1532879989
Jul 28 18:10:20 srv2 rpc.gssd[698]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_RHCE.LOCAL' are good until 1532879989
Jul 28 18:10:20 srv2 rpc.gssd[698]: creating tcp client for server srv1
Jul 28 18:10:20 srv2 rpc.gssd[698]: creating context with server nfs@srv1
Jul 28 18:10:20 srv2 rpc.gssd[698]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@srv1
Jul 28 18:10:20 srv2 rpc.gssd[698]: WARNING: Failed to create machine krb5context with cred cache FILE:/tmp/krb5ccmachine_RHCE.LOCAL for server srv1
Jul 28 18:10:20 srv2 rpc.gssd[698]: WARNING: Failed to create machinekrb5 context with any credentialscache for server srv1
Jul 28 18:10:20 srv2 rpc.gssd[698]: doing error downcall

Any help appreciated, thanks.

Lisenet says:

28/07/2018 at 6:23 pm

I think that srv1 should use FQDN (looking at /var/log/messages).

When you set up FreeIPA, did you add nfs/srv1.rhce.local or nfs/srv1?
Dédé says:

30/07/2018 at 7:24 pm

I think so too but I couldn’t find the root cause, and I believe that I followed carefully your instructions. At the end I removed this VM srv1 and installed another one srv3, followed some steps on the IPA server, and I could make it work. Thanks for helping and for your website.
Lisenet says:

30/07/2018 at 8:28 pm

No worries, you’re welcome.

Ami says:

31/07/2018 at 4:20 pm

Hello,
I’m also impressed from your guidance and I really thank you for all of that.
maybe I’m confused, but what I’ve been learned from Sander (And in my lab), is that the keytab for NFS client, should be copied from the IPA server with SERVER1 nfs principal and not srv2, as you’ve written…
since ti is the server1 with the nfs krb5 shares, we need the keytab contains nfs/server1.example.com service..

correct me if I’m wrong,
and thanks again
Ami

Reply
- Lisenet says:
  
  31/07/2018 at 6:26 pm
  
  Thanks for your feedback.
  
  In our case srv1.rhce.local acts as an NFS server, and srv2.rhce.local is an NFS client. It’s mentioned in “The Lab” section of the article.
Adamiks says:

01/09/2018 at 8:27 am

Hi Tomas Thank you for the blog and all the material you have put together l have been following your tutorials for some time now and l find them very detailed and well clarified.
l am currently preparing for the RHCE exam will be seating in a month’s time.
I set up a kerberized NFS virtual lab and at some point l got the error:
mount.nfs: remote share not in ‘host:dir’ format

My configurations are as follows
Server side
in the /etc/exports file l have
/public 192.168.8.0(ro)
/protected 192.168.8.0(rw,sec=krb5p)

Client Side l have created mount points
/mnt/public for the /public share
/mnt/secure for the /protected share

l was able to export and mount the non-kerberized public share successfully, it was only the protected kerberized share that
gave me this error.
Can you please assist l am not quite sure where l am making the mistake.If you need more details l am ready to send them through.

Thank You

Reply
- Lisenet says:
  
  01/09/2018 at 10:11 am
  
  Hi, thanks for your feedback!
  
  The error message says that your remote share format is not correct. Looking at your export statement, you’re trying to export to a network address 192.168.8.0 (I take it that you use 192.168.8.0/24) rather that a host. Try something like 192.168.8.5 (or whatever IP is set on your server).

I am still training a lot.. I created three NFS shares (one normal, two kerberized), and after mounting them on the client I get an inconsistent output with the df command. As you see below, mount and df listings of NFS mounts are different. How is it possible ? It’s on RH 7.0 for both server and client.

[root@srv3 mnt]# mount -lt nfs4
srv2:/srv/public-ro on /mnt/public-ro type nfs4 (rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.73,local_lock=none,addr=192.168.122.72,_netdev)
srv2:/srv/protected on /mnt/protected type nfs4 (rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=192.168.122.73,local_lock=none,addr=192.168.122.72,_netdev)
srv2:/srv/project on /mnt/project type nfs4 (rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=192.168.122.73,local_lock=none,addr=192.168.122.72,_netdev)

[root@srv3 mnt]# df -hTt nfs4
Filesystem          Type  Size  Used Avail Use% Mounted on
srv2:/srv/public-ro nfs4  4.0G  852M  3.2G  21% /mnt/public-ro
srv2:/srv/project   nfs4  4.0G  852M  3.2G  21% /mnt/project

Gerardo says:

09/09/2018 at 7:34 am

I have noticed that when I reboot the nfs server the nfs-server and nfs-secure-server take bit to start (like 3 minutes or so) is this normal?

Reply
- Lisenet says:
  
  10/09/2018 at 5:29 pm
  
  I don’t know, but it doesn’t take that long in my test environment.
omipenguin says:

28/10/2018 at 11:30 am

Hi Tomas

Im having issues here configuring kerberised nfs .

IPA IP 10.8.8.2
Serv1 = 10.8.8.50 (NFS Server)
Serv2 = 10.8.8.51 (NFS Client)

on NFS server 10.8.8.50 exports file contains

/srv/nfs_pub 10.8.8.0/24(sec=sys,rw,sync,root_squash,all_squash)
/srv/nfs_group 10.8.8.0/24(sec=sys,rw,sync,root_squash)
/srv/nfs_secure serv2.rhce.local(sec=krb5p,rw,sync)

while mounting on client i get this error

mount.nfs4: timeout set for Sun Oct 28 14:27:25 2018
mount.nfs4: trying text-based options ‘sec=krb5p,addr=10.8.8.50,clientaddr=10.8.8.51’
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting serv1.rhce.local:/srv/nfs_secure

and on NFS Server i see this error message

Oct 25 11:12:16 serv1 rpc.svcgssd[17636]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) – Ticket not yet valid
Oct 25 11:12:16 serv1 rpc.svcgssd[17636]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) – Ticket not yet valid
Oct 25 11:12:16 serv1 rpc.svcgssd[17636]: ERROR: GSS-API: error in handle_nullreq: gss_accept_sec_context(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) – Ticket not yet valid

Note i didnt add Version 4.2 on NFS server i kept it like original RPCNFSDARGS=””

Reply
- Lisenet says:
  
  28/10/2018 at 3:00 pm
  
  Ticket not yet valid suggests that there is a time synchronisation problem. Make sure that all servers are in sync.
- omipenguin says:
  
  29/10/2018 at 8:36 am
  
  Ok Tomas
  
  i fixed the time sync issue, but this time when im trying to mount kerberised nfs on Client , im getting this error
  
  [root@serv2 (~)]$ > mount -v -t nfs4 -o sec=krb5p serv1.rhce.local:/srv/nfs_secure /mnt/nfs_secure
  mount.nfs4: timeout set for Mon Oct 29 11:36:08 2018
  mount.nfs4: trying text-based options ‘sec=krb5p,addr=10.8.8.50,clientaddr=10.8.8.51’
  mount.nfs4: mount(2): Operation not permitted
  mount.nfs4: Operation not permitted
  
  On NFS Server messages file has these errors
  
  Oct 29 11:34:00 serv1 nslcd[2333]: [2342ec] request denied by validnames option
  Oct 29 11:34:00 serv1 nslcd[2333]: [487cb0] request denied by validnames option
- Lisenet says:
  
  29/10/2018 at 5:04 pm
  
  Check Red Hat link to see if it applies in your case: https://access.redhat.com/solutions/2892691
  
  You should be able to ignore this nslcd message, as the problem is likely somewhere else.
- omipenguin says:
  
  30/10/2018 at 6:50 am
  
  From client if i try to mount just using “mount -t nfs -o sec=krb5p ” it will mount successfully but this is mounting using NFSv3 but if i try to mount using “mount -t nfs4 ” im receiving Access denied or Operation not Permitted messages.
Andrew says:

28/10/2018 at 11:40 pm

mount.nfs: an incorrect mount option was specified
I can’t get past this error when mounting a kerberized share.
I have kerberos authentication setup and nfs-secure-server running on both servers.
Any ideas?

Reply
- Lisenet says:
  
  29/10/2018 at 4:58 pm
  
  Make sure that time is in sync, enable verbose logging and check the log file for clues.
Kamrul says:

10/11/2018 at 3:09 pm

Hey Tomas, I am getting this error from server2

[root@srv2 ~]# mount -t nfs4 -o vers=4.2,sec=krb5p srv1.rhce.local:/srv/nfs_secure /mnt/nfs_secure
mount.nfs4: access denied by server while mounting srv1.rhce.local:/srv/nfs_secure

[root@srv2 ~]# hostnamectl
Static hostname: srv2.rhce.local

[root@srv2 ~]# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
—- ————————————————————————–
2 nfs/[email protected]
2 nfs/[email protected]
2 nfs/[email protected]
2 nfs/[email protected]
2 nfs/[email protected]
2 nfs/[email protected]

[root@srv2 ~]# systemctl status nfs-secure -l
nfs-secure.service – Secure NFS
Loaded: loaded (/usr/lib/systemd/system/nfs-secure.service; enabled)
Active: active (running) since Sat 2018-11-10 20:24:55 ALMT; 24min ago
Process: 3310 ExecStart=/usr/sbin/rpc.gssd $RPCGSSDARGS (code=exited, status=0/SUCCESS)
Main PID: 3311 (rpc.gssd)
CGroup: /system.slice/nfs-secure.service
└─3311 /usr/sbin/rpc.gssd -vvv

Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3465]: No key table entry found for root/[email protected] while getting keytab entry for ‘root/[email protected]’
Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3465]: No key table entry found for nfs/[email protected] while getting keytab entry for ‘nfs/[email protected]’
Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3465]: No key table entry found for host/[email protected] while getting keytab entry for ‘host/[email protected]’
Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3465]: No key table entry found for [email protected] while getting keytab entry for ‘SRV2$@’
Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3465]: No key table entry found for root/[email protected] while getting keytab entry for ‘root/srv2.rhce.local@’
Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3465]: No key table entry found for nfs/[email protected] while getting keytab entry for ‘nfs/srv2.rhce.local@’
Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3311]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt17
Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3311]: Closing ‘gssd’ pipe for /var/lib/nfs/rpc_pipefs/nfs/clnt15
Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3311]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt16
Nov 10 20:47:13 srv2.rhce.local rpc.gssd[3311]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt15

Any idea? Thanks

Reply
- Lisenet says:
  
  11/11/2018 at 12:12 pm
  
  It looks like you have a domain mismatch. Your keytab contains entries for @RHCE.LOCAL, but your client is configured to use @EXAMPLE.COM.
- Kamrul says:
  
  11/11/2018 at 6:00 pm
  
  How to configure the client to use @RHCE.LOCAL?
- Lisenet says:
  
  11/11/2018 at 6:11 pm
  
  The same way that you did for @EXAMPLE.COM, but use the correct domain that you have on kerberos.
- Kamrul says:
  
  14/11/2018 at 3:24 pm
  
  Hello again, In client machine, I edited /etc/krb5.conf file and edited EXAMPLE.COM & example.com to RHCE.LOCAL & rhce.local. Then I restarted the nfs-secure service & tried to mount again (mount -t nfs4 -o vers=4.2,sec=krb5p srv1.rhce.local:/srv/nfs_secure /mnt/nfs_secure) and still get access denied error. In systemctl status nfs-secure, I get this error:
  
  Nov 14 20:45:39 srv2.rhce.local rpc.gssd[3607]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt3)
  Nov 14 20:45:39 srv2.rhce.local rpc.gssd[3607]: process_krb5_upcall: service is ”
  Nov 14 20:45:39 srv2.rhce.local rpc.gssd[3607]: Full hostname for ‘srv1.rhce.local’ is ‘srv1.rhce.local’
  Nov 14 20:45:39 srv2.rhce.local rpc.gssd[3607]: Full hostname for ‘srv2.rhce.local’ is ‘srv2.rhce.local’
  Nov 14 20:45:39 srv2.rhce.local rpc.gssd[3607]: No key table entry found for [email protected] while getting keytab entry for ‘[email protected]’
  Nov 14 20:45:39 srv2.rhce.local rpc.gssd[3607]: No key table entry found for root/[email protected] while getting keytab entry for ‘root/[email protected]’
  Nov 14 20:45:39 srv2.rhce.local rpc.gssd[3607]: Success getting keytab entry for ‘nfs/[email protected]’
  Nov 14 20:45:44 srv2.rhce.local rpc.gssd[3598]: Closing ‘gssd’ pipe for /var/lib/nfs/rpc_pipefs/nfs/clnt3
  Nov 14 20:45:44 srv2.rhce.local rpc.gssd[3598]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt4
  Nov 14 20:45:44 srv2.rhce.local rpc.gssd[3598]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt3
  
  Any Idea. I completed all the RHCE objectives other than this objective :( Need help please
- Lisenet says:
  
  14/11/2018 at 7:28 pm
  
  Keytab entry for srv1 was not found, are you sure that the entry exists?
  
  I’d suggest you to start from scratch and use the same domain across all systems when configuring services.
Ovitus says:

03/12/2018 at 8:27 pm

Again, I’m trying to practice with just CentOS 7.0, but when installing ‘nfs-utils’ via yum, it automatically tries to update to 7.5:

Installing:
nfs-utils x86_64 1:1.3.0-0.54.el7 Packages 407 k
Updating:
centos-release x86_64 7-5.1804.el7.centos Packages 24 k

I setup a local repository from packages in the CentOS 7.0 Everything iso. Any way of preventing this? It happens when I try and install ‘sssd’ also.

Reply
- Lisenet says:
  
  08/12/2018 at 11:06 am
  
  Make sure that all repositories are disabled by setting enabled=0 in the .repo files. Then enable the repository which uses the DVD image. Take a look here if you need help setting it up:
  
  https://www.lisenet.com/2016/create-local-yum-repository-from-systemd-mounted-dvd-on-rhel-7/
Remi says:

03/01/2019 at 10:11 pm

Hi!
When I export NFS V4.2 shares with mixed security (2 folders with sys and 1 with krb5p), it doesn’t work for ip ranges or wildcards (192.168.3.0/24 nor *.rhce.local). I can only make it work if I specify a single IP address or hostname for all the shares.

For example, it works with:
/srv/nfspub 192.168.3.0/24(sec=sys,rw,sync,root_squash,all_squash)
/srv/nfsgroup 192.168.3.0/24(sec=sys,rw,sync,root_squash)

It also works with:
/srv/nfssec srv2.rhce.local(sec=krb5p,rw,sync)

Everything works with:
/srv/nfspub srv2.rhce.local(sec=sys,rw,sync,root_squash,all_squash)
/srv/nfsgroup srv2.rhce.local(sec=sys,rw,sync,root_squash)
/srv/nfssec srv2.rhce.local(sec=krb5p,rw,sync)

But nothing works with:
/srv/nfspub 192.168.3.0/24(sec=sys,rw,sync,root_squash,all_squash)
/srv/nfsgroup 192.168.3.0/24(sec=sys,rw,sync,root_squash)
/srv/nfssec srv2.rhce.local(sec=krb5p,rw,sync)

Any idea ?
Thanks!

Reply
- jsiljeg says:
  
  13/12/2019 at 11:59 pm
  
  Mi Remi. I had a similar issue at some point (when mixed sec used).
  I believe it resolved for me when – while mounting on client side – specified sec=sys. When I raised verbosity on client, I noticed on nfs-secure status there are some log messages containing krb stuff (and we want to share /srv/nfsgroup with sec=sys on srv side).
  Hope this may help :)
Vinicius says:

11/04/2019 at 10:18 am

For kerberos, the user alice have an entrance on your kerberos like this ‘[email protected]’?

Reply
- Lisenet says:
  
  12/04/2019 at 8:57 pm
  
  Yes, it does. Kerberos principal: [email protected]
Martan says:

14/04/2019 at 7:31 am

Hello,

I have a question regarding krb5p nfs share. I have exported the share and mounted it on srv2.

Srv1 — nfs-server

drwxr-xr-x. alice root unconfined_u:object_r:nfs_t:s0 secur

/srv/secure srv2.rhce.local(sec=krb5p,rw,sync)

Srv2 — nfs-client
drwxr-xr-x. alice root system_u:object_r:nfs_t:s0 secure

When i switch to user Alice I’m unable to access the share at all. After acquiring the ticket (kinit) I can enter the share and write to it, but after destroying the the kerberos ticket (klist: Credentials cache keyring ‘persistent:1761600001:1761600001’ not found) I still can write to that share with to issues. Is this behaviour correct?

Reply
- Gjorgi says:
  
  26/05/2019 at 7:08 pm
  
  Only Kerberos authorized users can access and write to Kerberos-protected shares, you switched to user alice- which user alice, the local one? Local users can’t write to Kerberos protected shares when those shares are mounted, we may expect to be able to write as some user or even as root since it is a locally mounted share- but that does not work. So you switched to the Kerberos authorized user alice with kinit, it’s not the local user alice anymore, THAT user will have full read/write privileges provided via ownership.
Martan says:

14/04/2019 at 9:10 am

Hello,

I’m having a trouble with making the exported setting persistent. After nfs server reboot I’m unable to access the previously exported share (/var/lib/nfs/etab is empty) and I need to issue exportfs again. NFs is configured with -V 4.2 in /et/sysconfig/nfs . nfs-server is set to autostart. (redhat 7.5)

Reply
Gjorgi says:

21/05/2019 at 4:40 pm

I see you, along with Sander and CertDepot, rely a lot on kinit on both server and client in the Kerberized NFS scenario. Using kinit requires that an account is opened for the client on the IPA server and that you are given that account’s password. I am really not sure any of that will be available on the RHCE exam. Is there a solution that does the role of the IPA and that would require only possesing keytab files for granting the client access to and mount of, the Kerberized NFS share?

Reply
- Lisenet says:
  
  21/05/2019 at 4:56 pm
  
  Yes, there is (good question BTW). You can alternatively download the keytab file if don’t have Kerberos admin credentials. It’s explained in the blog post.
- Gjorgi says:
  
  23/05/2019 at 8:31 am
  
  Tomas, I followed your recommendation to practice with RHEL 7.0 and not CentOS 7.0 given as a response to my comment in your FreeIPA setup guide and have practiced steps given in this guide, the part about mounting Kerberized NFS shares when only keytabs are given when both NFS server and client are RHEL 7.0 and:
  – once you install nfs-utils, even if you insist on installing version 1.3.0, you will no longer be working with RHEL 7.0 but with 7.6 as verified by checking /etc/redhat-release. Succesfull installation of nfs-utils will move you to different version.
  – when given only keytab files and not kinit credentials for authentication, you’ll have to manually redact /etc/krb5.conf and replace every mention of “example.com” with your own custom domain. Keep track of case and punctuation in that file but replace the “example.com” otherwise access will be denied and keytabs invalidated and reported as bad in /var/log/messages.
- Lisenet says:
  
  23/05/2019 at 8:48 am
  
  You should be using RHEL 7.0 DVD as a local repository to avoid this problem. At least that’s what I always do.
  
  See here for more info:
  
  https://www.lisenet.com/2016/create-local-yum-repository-from-systemd-mounted-dvd-on-rhel-7/
  
  I didn’t do any manual changes to /etc/krb5.conf. Instead, I removed all content, and used authconfig-tui to configure Kerberos. See here for more info: https://www.lisenet.com/2016/ldap-and-kerberos-client-authentication-on-rhel-7-using-sssd/
- Gjorgi says:
  
  23/05/2019 at 10:23 pm
  
  I just wrapped up setting up the FreeIPA server as a repo server as well, with the copy of the DVD as the source for the repo. Brilliant solution for a home made, RHEL- compliant lab, have the DVD as a repo and avoid versioning disparities between CentOS and RHEL and whatnot, just for sake of labbing!
  In your guide on using authconfig-tui for properly setting up NFS client to mount a secure NFS share you rely on downloading cacert file- am not sure at all it would be available on the exam.
- Lisenet says:
  
  24/05/2019 at 7:10 pm
  
  Yea, you can also upgrade between RHEL versions easily. Simply mount a RHEL 7.1 DVD, and your repo becomes a RHEL 7.1 repo on a RHEL 7.0 server. Similar way you can go from RHEL 7.0 to RHEL 7.6 if you want to test them all.
  
  With regards to NFS, you have to have a certificate in order to configure the service. Therefore the certificate must provided in one way or another.
Gjorgi says:

25/05/2019 at 6:26 am

With Kerberos-protected NFS shares, only Kerberos-authenticated users can write on those shares when they are mounted on client. Local users on the client can’t write on those shares, not even root. And here lies the catch: with only the keytab files you can mount the secured NFS share- but you can’t write on it. In order to effectively enable Kerberos-authenticated users to write on your mounted share, your client has to join the IPA/Kerberos domain/realm. In order to do that, you need to run ipa-client-install and to succeed at that you’ll need the password of the Kerberos-authenticated user authorized to enroll hosts in the domain. Without authentication detail of that level there can be no writing to protected share.

Reply
Gjorgi says:

04/06/2019 at 3:42 am

… or at exam time, their IdM server is specially set up so that downloading keytab files is sufficient to join client machines to the domain and allow domain-only, Kerberos- authenticated users to log in…
In lieu of that, Kerberos admin credentials HAVE to be listed (and with that run ipa-client-install, by far the easiest way of joining client machines to a domain) otherwise I can’t see how would a client join just by downloading keytab files.

Reply
Andrew says:

09/07/2019 at 10:15 am

Very very useful topic!
Thank you!

Reply
Luud says:

26/12/2019 at 12:53 pm

Followed your tutorial exactly except the v4.2 stuff.
Used the Centos 7.0 download for server1 and server2.
Having trouble with writing to the kerborized share.
After doing a kinit for alice , I can read the directory.
Creating a file results in Permission denied.
After chmod 777 on /srv/nfs_secure, alice can create files , but the are owned by nfsnobody.

/etc/exports on srv1:
/srv/nfs_secure *(rw,sec=krb5p,no_root_squash)

Any help is greatly appreciated.

Reply

Requirements

The Lab

Keep Time in Sync

Base NFS Server Setup

Packages, Services and Firewall

Configure NFS Exports

Kerberos Keytab File

NFS Client Setup

Packages and Services

Mount a Kerberised NFS Share

Mount Non-Kerberised NFS Shares

Problems With Key Version Numbers

References

159 thoughts on “Setting up a Kerberised NFS Server on RHEL 7”

Leave a Reply Cancel reply