RHEL7: Use Kerberos to control access to NFS network shares.

Share this link

Note: This is an RHCE 7 exam objective.

Prerequisites

First, you will have to configure a KDC (Kerberos Distribution Center) called here kbserver.example.com.

Then, you will need two additional servers: a NFS server (here nfsserver.example.com) and a NFS client (here nfsclient.example.com). If you’ve got only two servers/VM, regroup the KDC and the NFS servers on the same machine.
Also, to get Kerberos running, NTP synchronization and hostname resolution must be working.
It is advisable to set up a master DNS server but if none is working, add the following lines in the /etc/hosts file of each server (replace the specified ip addresses with yours):

192.168.1.11 kbserver.example.com
192.168.1.12 nfsserver.example.com
192.168.1.13 nfsclient.example.com

Caution: When adding a new line in the /etc/hosts file, you have to write the fully qualified domain name just after the ip address. If you use one or several aliases and add them before the fully qualified domain name or if you don’t specify the fully qualified domain name, Kerberos will not work.

Kerberos NFS Server Configuration

Before adding the Kerberos configuration, set up the NFS server (use the nfsserver.example.com hostname in this tutorial).

Then, you will have to add the Kerberos client configuration (replace kbclient.example.com with nfsserver.example.com in this tutorial).

Finally, add the specific NFS part to the principals:

# kadmin
Authenticating as principal root/admin@EXAMPLE.COM with password.
Password for root/admin@EXAMPLE.COM: kerberos
kadmin:  addprinc -randkey nfs/nfsserver.example.com
WARNING: no policy specified for host/kbclient.example.com@EXAMPLE.COM; defaulting to no policy
Principal "host/nfsserver.example.com@EXAMPLE.COM" created.

Create a local copy stored by default in the /etc/krb5.keytab file:

kadmin:  ktadd nfs/nfsserver.example.com
Entry for principal host/nfsserver.example.com with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/nfsserver.example.com with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/nfsserver.example.com with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/nfsserver.example.com with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/nfsserver.example.com with kvno 2, encryption type camellia256-cts-cmac added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/nfsserver.example.com with kvno 2, encryption type camellia128-cts-cmac added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/nfsserver.example.com with kvno 2, encryption type des-hmac-sha1 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/nfsserver.example.com with kvno 2, encryption type des-cbc-md5 added to keytab WRFILE:/etc/krb5.keytab.
kadmin:  quit

Edit the /etc/exports file and add the option sec=krb5 (or the option that you want, see note):

/home/tools nfsclient.example.com(rw,no_root_squash,sec=krb5)
/home/guests nfsclient.example.com(rw,no_root_squash,sec=krb5)

Note1: The sec option accepts four different values: sec=sys (no Kerberos use), sec=krb5 (Kerberos user authentication only), sec=krb5i (Kerberos user authentication and integrity checking), sec=krb5p (Kerberos user authentication, integrity checking and NFS traffic encryption). The higher the level, the more you consume resources.
Note2: If you want to use sec=sys (no Kerberos use), you also need to run setsebool -P nfsd_anon_write 1

Export the new configuration:

# exportfs -avr
exporting nfsclient.example.com:/home/guests
exporting nfsclient.example.com:/home/tools

Check your configuration:

# showmount -e localhost
Export list for localhost:
/home/guests nfsclient.example.com
/home/tools  nfsclient.example.com

Activate at boot and start the nfs-secure-server service (RHEL 7.0 only):

# systemctl enable nfs-secure-server && systemctl start nfs-secure-server

Note: If you want to get more information in the /var/log/messages file, edit the /etc/sysconfig/nfs file, assign the “-vvv” string to the RPCIDMAPDARGS/RPCSVCGSSDARGS variables and restart the nfs-idmap/nfs-secure-server daemons.

Kerberos NFS Client Configuration

Before adding the Kerberos configuration, set up the NFS client (use the nfsclient.example.com hostname in this tutorial).

Then, you will have to add the Kerberos client configuration (replace kbclient.example.com with nfsclient.example.com in this tutorial).

Finally, add the specific NFS part to the principals:

# kadmin
Authenticating as principal root/admin@EXAMPLE.COM with password.
Password for root/admin@EXAMPLE.COM: kerberos
kadmin:  addprinc -randkey nfs/nfsclient.example.com
WARNING: no policy specified for host/kbclient.example.com@EXAMPLE.COM; defaulting to no policy
Principal "host/nfsclient.example.com@EXAMPLE.COM" created.

Create a local copy stored by default in the /etc/krb5.keytab file:

kadmin:  ktadd nfs/nfsclient.example.com
Entry for principal host/nfsclient.example.com with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/nfsclient.example.com with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/nfsclient.example.com with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/nfsclient.example.com with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/nfsclient.example.com with kvno 2, encryption type camellia256-cts-cmac added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/nfsclient.example.com with kvno 2, encryption type camellia128-cts-cmac added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/nfsclient.example.com with kvno 2, encryption type des-hmac-sha1 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/nfsclient.example.com with kvno 2, encryption type des-cbc-md5 added to keytab WRFILE:/etc/krb5.keytab.
kadmin:  quit

Activate at boot and start the nfs-secure service (RHEL 7.0 only):

# systemctl enable nfs-secure && systemctl start nfs-secure

Activate at boot and start the nfs-client target (RHEL 7.1 and after):

# systemctl enable nfs-client.target && systemctl start nfs-client.target

Note1: Since RHEL 7.1, the nfs-secure service automatically starts if there is a /etc/krb5.keytab file.
Note2: If you want to get more information in the /var/log/messages file, edit the /etc/sysconfig/nfs file, assign the “-vvv” string to the RPCIDMAPDARGS/RPCGSSDARGS variables and restart the nfs-idmap/nfs-secure daemons.
Note3: With the RHEL 7.3 release, the Systemd init system is able to use aliases. For example, the nfs.service is a symbolic link/alias to the nfs-server.service service file. This enables, for example, using the systemctl status nfs.service command instead of systemctl status nfs-server.service.
Previously, running the systemctl enable command using an alias instead of the real service name failed with an error.

Mount the remote directory:

# mount -t nfs4 -o sec=krb5 nfsserver.example.com:/home/tools /mnt

Note1: If you get the error message “mount.nfs4: an incorrect mount option was specified”, check that you started the correct daemons.
Note2: It is not necessary to specify the rw option, it is done by default.
Note3: You can test what shares are exported by the NFS server with the command showmount -e nfsserver.example.com but you first need to stop Firewalld on the NFS server (or open the 111 udp and 20048 tcp ports on the NFS server).
note4: If you don’t specify the sec option, the security mechanism will be negotiated transparently with the remote server (see details here).

To permanently set up the mount, paste the following line in the /etc/fstab file:

nfsserver.example.com:/home/tools /mnt nfs4 sec=krb5

Switch to the user01 user:

# su - user01

Create a Kerberos ticket:

$ kinit
Password for user01@EXAMPLE.COM: user01

Create a file called testFile:

$ cd /mnt
$ echo "This is a test." >testFile

Check the result:

$ ls -l
total 8
-rw-rw-r--. 1 user01 user01 16 Sep  7 16:42 testFile

Additional Resources

You can also watch Sander van Vugt‘s video about Mounting Kerberized NFS (17 min/2016).

(6 votes, average: 4.33 out of 5)
Loading...
160 comments on “RHEL7: Use Kerberos to control access to NFS network shares.
  1. SkoolofManoovah says:

    When I do that last bit-

    # mount -t nfs4 -o sec=krb5 nfsserver.example.com:/home/guests /mnt

    I get the following error message, can’t find on web what the problem is-

    “mount.nfs4: an incorrect mount option was specified”

    I have checked and rechecked that I followed your instructions precisely except ips and fqdns. Any suggestions?

    • CertDepot says:

      When testing, I got exactly the same error until I found all the steps.
      I think the error message is completely misleading, nothing linked to an incorrect mount option.
      It’s something connected to the nfs-secure/nfs-secure-server daemons but it shouldn’t happen.

    • mrmarcus says:

      For some reason I can not get the nfs-secure-server service to start following these instructions. I’ve followed the directions step by step on several VMs to no avail. Does this service have any pre-requisites?

    • Gjorgi says:

      You will get this message if you haven’t previously started nfs-secure. This service has to be started first, before attempting secure NFS mount.
      On the other hand, I can’t enable nfs-secure on my CentOS 7.2 virtual install. I am only allowed to manually start it. Any workarounds?

      • CertDepot says:

        Until now, the RHCE exam is still in RHEL 7.0. As the NFS configuration changed a lot between RHEL 7.0 and RHEL 7.1, I advice you to prepare the exam using the RHEL 7.0 version (or CentOS 7.0).

  2. YDE says:

    Hi
    I’ve followed it and it works except one problem.
    When I want to write on mounted nfs storage I’ve “Permission denied” error (in Root or other users).
    No problem before Kerberos configuration.
    Any idea ?

    • CertDepot says:

      Hi,
      I did all the tutorial again and I got the same error.
      In fact, I forgot to give full access rights on the nfsserver’s exported directory!
      => [nfsserver]# chmod 777 /home/guests

      • YDE says:

        Thanks, I tried and it works.

        I also search a solution and i’ve found something :

        if you edit /etc/idmapd.conf on nfsserver and nfsclients
        and, in General section, changed :
        #Domain = local.domain.edu
        to
        Domain=example.com

        You haven’t to set full access on exported directory, and only owner with Kerberos authentication can write on it.
        for example :
        I have a user Alice authenticate by Kerberos
        I have exported /nfs/alice/ directory owned by alice, with 700 access on my nfsserver
        When I mount it on my nfsclient, Only Alice can write on it.

        I don’t like to set full access on a directory, i think it’s not secure even for directory that are not important.

        • CertDepot says:

          By default, the Domain directive in the /etc/idmapd.conf file receives the domain name as value.
          Therefore, it is useless to set Domain=example.com because it’s already the default value.

          • Don’t even go there if you’re preparing for RHCE exam. I’ve played with this a lot lately. My findings:
            * on plain RHEL 7.0 – no updates installed, all you have to do it import the key tabs on both servers, mount the share with sec=krb5p and run nfs-secure-server and nfs-secure, and it will work. plain and easy.
            * on centos 7.0 – no way to get the same approach working
            * on centos 7.1 – works as above, without a need to run nfs-secure-server and nfs-secure.

            My $ 0.02: it’s weird stuff. On the RHCE exam: the minute that you need to go deep, like setting idmapd.conf stuff, you’re definitely on the wrong track. It shouldn’t be that hard.

          • tron says:

            Agree on the “it shouldn’t be that hard” but it helps to know what’s going on to better checkpoint your work, at least in my case.

            My .02: the idmap domain is not the same as the kerberos realm. And it’s not by default example.com, it’s your host domain (i.e. FQDN minus first component). It’s nice to know that you can fix the domain (it has to be the same or in the local mapping between client and server) and that you can up the verbosity to peek /var/log/messages for hints.

            Go for the low hanging fruit first though. Time is a premium.

          • student says:

            I am always having problems with Kerberos even I use your VMs. I can not change to user via ssh ldapuser1@server2 and list key with klist. Even getent passwd ldapuser1 shows user information. I think something is wrong with VMs.

          • Sam says:

            Student,What version of OS are you using?
            Did you configure the SSH service to use Kerberos?

            http://www.pmsas.pr.gov.br/wp-content/?id=certdepot-EX200&exam=rhel7-configure-system-authenticate-using-kerberos/

  3. YDE says:

    Other question, In this documentation
    https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/installing-ipa.html
    RedHat uses ipa-server and ipa-client to configure Kerberos authentication and Kerberized NFS / SMB.
    Is it enough for RHCE ?
    Can we use it during exam ? Or manual Kerberos configuration is required ?

  4. jerky_rs says:

    Great article.

    I think Redhat would list IPA somewhere in the Objectives. It is also part of the RH413 expertise course (hidden..) = http://www.redhat.com/en/services/training/rh413-red-hat-server-hardening (Course outline tab).

    So my guess is that you will need to setup NFS server and Kerberos client (as configuring Kerberos server is not an objective). Setting up the same in IPA requires specific knowledge of something that is not listed in objectives or is part of the RH300 course outline.

  5. Shikaz says:

    Do you have an idea how to add the NFS + Kerberos in fstab?
    Is it?: nfsserver.example.com:/home/guests /mnt nfs4 _netdev,krb5 0 0?

    • CertDepot says:

      I have updated the tutorial.
      It is: nfsserver.example.com:/home/guests /mnt nfs4 sec=krb5
      The _netdev option is not necessary. “0 0” is only for local filesystems.

      • Shikaz says:

        Thanks and by the way i really like this site, that’s why whenever I found something that we can add or missing question I post it here, so i wish you don’t get annoyed 🙂 or tell me so.

        • CertDepot says:

          I can’t always answer quickly because I don’t how everything, sometimes I need time to build the configuration or I’m busy doing something else.
          But in any case, you don’t annoy me!

  6. erik says:

    If you want SElinux contexts to be consistent between the server / client, you will need to make the following tweak to the above configs:

    Server:
    In /etc/sysconfig/nfs set RPCNFSDARGS=”-V 4.2″

    Client:
    When you mount the share add the v4.2 option:
    mount -t nfs4 -o sec=krb5p,v4.2 ehansen01:/home/tools /mnt/tools/
    –or fstab–
    server01:/home/tools /mnt/tools nfs sec=krb5p,v4.2

    This is straight out of one of Redhat’s Sysadmin courses

    • BadBrains says:

      The course material only works on Centos 7.0 not Centos 7.1.
      nfs-secure-server unit file is static in the latter so cannot be enabled. The SELINUX context does not appear to work either on version 7.1.
      I am going to try with RHEL instead on Centos. Really hoping the exam version does not change with new releases.

    • tom says:

      Does it work for you? I tested it on both RHEL 7.1 and CentOS 7.1 and it did not work:

      [root@client3 ~]# mount -vvv -t nfs4 -o sec=krb5p,vers=4.2 server3.example.com:/srv/nfsusers /mnt
      mount.nfs4: timeout set for Sun Oct 18 12:31:48 2015
      mount.nfs4: trying text-based options ‘sec=krb5p,vers=4.2,addr=192.168.122.225,clientaddr=192.168.122.16’
      mount.nfs4: mount(2): Protocol not supported
      mount.nfs4: Protocol not supported

      [root@client3 ~]# mount -vvv -t nfs4 -o sec=krb5p,v4.2 server3.example.com:/srv/nfsusers /mnt
      mount.nfs4: timeout set for Sun Oct 18 12:32:48 2015
      mount.nfs4: trying text-based options ‘sec=krb5p,v4.2,addr=192.168.122.225,clientaddr=192.168.122.16’
      mount.nfs4: mount(2): Protocol not supported
      mount.nfs4: Protocol not supported

      It works with vers=4.1 but then SELinux contexts are not exported to client.

      Any ideas?

  7. michaels says:

    Hello,

    CLIENT Side:

    There is no enable option for

    # systemctl enable nfs-secure

    You can start the service independently (# systemctl start nfs-secure) but to enable it through the reboots we should enable the nfs-client service:

    # systemctl enable nfs-client.target

    Server Side:

    There is no enable option for

    # systemctl enable nfs-secure-server

    Again, you can start it independently but to enable it through reboots is enough to enable the nfs-server.service:

    # systemctl enable nfs-server.service

  8. fiend138 says:

    There are a few issues with the steps above.

    On the server side you can’t enable nfs-secure-server.service because it is a static service started by nfs-server.service. However on both RHEL 7.1 and CEntOS 7.1 I noticed a problem. In order for this to work I had to remove two lines from the service file

    ConditionPathExists=|!/run/gssproxy.pid
    ConditionPathExists=|!/proc/net/rpc/use-gss-proxy

    After that just run ‘systemctl enable nfs-server.service’

    On the client side there is a similar issue. nfs-secure.service is also a static service started by the nfs-client.target. However in the version of nfs-utils used this is not enabled by default. There is supposedly a fix in version 1.3.1 but I haven’t had a chance to test it. So instead of ‘systemctl enable nfs-secure.service’ the command should be ‘systemctl enable nfs-client.target’. Found at https://bugzilla.redhat.com/show_bug.cgi?id=1173564

  9. Lee Yang Jae says:

    I found Sander’s Centos 7 server VM nfs version and it can support nfs-secure-server and nfs-secure.
    The version is nfs-utils-1.3.0-0.el7.x86_64

    [root@server1 ~]# rpm -qa | grep nfs-utils
    nfs-utils-1.3.0-0.el7.x86_64
    [root@server1 ~]# systemctl enable nfs
    nfs-blkmap.service nfs-secure-server.service
    nfs-idmap.service nfs-secure.service
    nfslock.service nfs-server.service
    nfs-mountd.service nfs.service
    nfs-rquotad.service

    But after update as nfs verstion by yum update,
    [root@server1 ~]# rpm -qa | grep nfs-utils
    nfs-utils-1.3.0-0.21.el7.x86_64
    [root@server1 ~]# systemctl enable nfs
    nfs-blkmap.service nfs-server.service nfs.service

    there is only nfs-server.

    Could you help to compare “nfs-utils-1.3.0-0.el7.x86_64” and “nfs-utils-1.3.0-0.21.el7.x86_64”.

    Is there a way to use nfs-secure-server and nfs-secure in latest nfs version?

    • CertDepot says:

      I recently updated all the NFS tutorials. Check them and you will see that the NFS configuration has seriously changed between RHEL 7.0 and RHEL 7.1. Unfortunately, changes are still regularly happening: new nfs-client.target, nfs-secure not needed anymore due to nfs-util evolution, …
      Even worse, the official Red Hat NFS documentation is not up-to-date (January 2016)!

  10. circuscowboy says:

    To make things more complicated the virtual machines for the official Red Hat labs are 7.0 and I believe the exam is the same. (February 2016)

    • CertDepot says:

      I think you are right but things should change quickly for two reasons: the situation is unsustainable (NFS interface is very different between RHEL 7.0 and RHEL 7.1) and the RHEL 7.0 release is now pretty old.
      I suspect the change to take time because training books and labs need to be rewritten. Due to the changes brought by the RHEL 7.2 release (Systemd rebase), I shouldn’t be surprised if exams use the RHEL 7.2 version in the coming weeks.

  11. rilindo says:

    As a reminder, when practicing make secure that the clocks are synced (either using ntp or chronyd). Kerberos, after all, is sensitive to time.

    (spent 19 minutes what could have been 5 trying to get kerberos working with NFS)

  12. sandervanvugt says:

    I think that finally I found what goes wrong over and over again with kerberized NFS. The problem is in software updates. The symptom: you start with an installation of CentOS or RHEL 7.2. The procedure works as described as mentioned in my video (see “Additional Resources” above). Then you upgrade, and you get an “access denied” message from the Kerberized NFS server. The fun thing is that this has happened over and over again in my tests, on 7.0, on 7.1 as well as 7.2. Fortunately, it’s just an upgrade issue and not a configuration issue. The solution? DO NOT use the online repositories, DO NOT update your server. Instead, just build your own internal repository server that is stabilised on 7.2, and use that. You’ll see, the procedure just works 🙂

  13. danw says:

    Hello, I need some info for this month of April 2016…

    Did anyone take the RHCE exam last Friday? I would love to know what version of Red Hat is the exam on?

    Hopefully the mess between nfs-client and nfs-secure is not going to compromise my exam…. I will let you know tomorrow when I finish my exam.

  14. asifshabir says:

    I have followed this method and I’m getting errors:

    when I mount it takes a lot of time and then displays this error message

    mount.nfs4: access denied by server while mounting nfsserver1.example.com:/home/tools

    logs from kbserver.
    http://www.heypasteit.com/clip/2MI2

    I am using centos 7

    cat /etc/redhat-release
    CentOS Linux release 7.1.1503 (Core)

    Normal mount is working fine.

    Can you please suggest a solution.

    • CertDepot says:

      Check Time Synchronization with NTP, check Name Resolution (/etc/hosts with FQDN just after the ip address or DNS name server), check SELinux configuration.

  15. Jaz says:

    This Kerberized NFS has become a real pain.
    I am not sure how I am going to fulfil this requirement if I have to configure kerberized nfs during the exam.
    I have tried every possible way and it didn’t work out on RHEL 7.0
    “access denied while mounting” messages appear.
    When I remove “sec=krb5” option from exports file then it mounts normally without any problem.

    Seriously, what is wrong with it?

    • CertDepot says:

      Did you update your RHEL 7.0 distribution? I’m asking this question because it doesn’t seem to be a good idea!

    • Gjorgi says:

      You don’t have nfs-secure running. You probably got the generic keytab.conf downloaded to your client which doesn’t correctly identify Kerberos realm.
      Address these two issues and attempt again.

    • Jaz says:

      Actually, I realized this mistake right after my first comment.
      There is a ‘mistake’ in Micheal Jang’s 7th edition (apparently too many in there). He made me create nfs server and client keytabs both with the same nfs server address which caused the issue.
      # ktadd -k /tmp/server1.keytab server1.example.com
      # ktadd -k /tmp/client.keytab server1.example.com

      So I deleted the old keytab from /etc/ and also from Kerberos server and generated a new one with the client address and it worked.

    • mmhaque4 says:

      I wrote the RHCE 300 exam last 13th May, 2016. I was unable to get the kerberised NFS share part working.

      At the server:
      I added services: mountd, rpc-bind, nfs
      started services: nfs-server, nfs-secure, nfs-secure-servre, rpcbind
      Downloaded the keytab from the given server to : /etc/krb5.keytab.

      At the client:

      started services nfs-server, nfs-secure, rpcbind.

      In the /etc/fstab of the client machine;

      I had
      [ip of nfs server]/nfssecure nfs /mnt/nfs defaults, sec=krbp5 0 0

      At first the error message was:

      mount.nfs: an incorrect mount option was specified

      Then I restarted all the services in both the server and client, changed the sec=krb5p to sec=krb5 and ran the mount -a command again. This time \

      I got the “mount.nfs: access denied by server while mounting ..”

      I did not start either ntpd or chronyd service in any machine though.

      Can anyone please help?

      • CertDepot says:

        Concerning ntpd and chronyd, you didn’t know that a delay bigger than 5 minutes between client and server makes Kerberos out of order?

        • mmhaque4 says:

          Thanks for the reply.

          I forgot to mention – In the client machine I downloaded the keytab to /etc/krb5.keytab from the given link.

          I was actually not aware of ntp/chrony.
          ===============================================
          /etc/chrony.conf. Assuming that your ntp servers are called ‘a.b.c’ and ‘d.e.f’, your chrony.conf

          file could contain as a minimum

          server a.b.c
          server d.e.f
          server g.h.i
          ================================================

          If I have to set up the chrony then which server can I use during the exam? Questions give no indication about ntp/chrony servers.

          • CertDepot says:

            All the servers involved in the Kerberos configuration should be synchronized.
            It seems logical to use the Kerberos KDC as time server.

      • brucemzn says:

        @Jaz You had errors because you started invalid services. Please note that nfs-secure-server.service cannot start without a keytab file (/etc/krb5.keytab). The following services should be enable and started if using Redhat/Centos 7.0 :

        NFS Server:
        systemctl enable nfs-secure-server
        systemctl start nfs-secure-server
        systemctl status nfs-secure-server

        Add the services to the firewall:
        firewall-cmd –permanent –add-service=nfs
        firewall-cmd –permanent –add-service=mountd
        firewall-cmd –permanent –add-service=rpc-bind
        firewall-cmd –reload

        NFS Client:
        systemctl enable nfs-secure
        systemctl start nfs-secure
        systemctl status nfs-secure
        showmount -e nfs-server-hostname (or ip-nfs-server)

        You will obviously get an error “access denied by server while mounting…”
        The solution to this to reboot both nfs server and client machines. The worst part is having the nfs client taking too long to boot. The issue is caused by network.service and NetworkManger.service running simultaneously. The command # “systemd-analyze blame” will confirm. The solution to this is to disable one of the services, on both machines.
        #systemctl disable network.services (for example)
        Reboot and you will see a huge difference in boot time.
        Run the “systemd-analyze blame” command to confirm

        In my experience, the following had no impact on my Redhat 7.0/7.1 lab with regards to NFS Kerberos:
        1. NTP and Chronyd. These two makes no difference on my lab.
        2. SELinux (such as semanage fcontext)
        3. /etc/sysconfig/nfs
        4. Specifying nfs4 instead of nfs in the /etc/fstab file does not make a difference.
        BUT editing line #5 in the /etc/idmapd.conf on both machines had positive impact on assigning write permissions. The user can write into their directories as assigned at the nfs server side. The user can only write after obtaining a valid kerberos ticket.
        Hope this is informative…

      • brucemzn says:

        @ mmhaque4
        This error is caused by either:
        1. The nfs-secure-server is not running on the server side
        2. The nfs-secure service is not running on the client side
        3. The keytab files are stale, meaning that they are not valid.
        4. You did not reboot the nfs server and client
        Try those tricks and let us know how it goes. Check the validity of keytab files by running the “# klist -k” command on both nfs server and client. The results must correspond the hostname, respectively

        • mmhaque4 says:

          @brucemzn,

          I have tested at my home in centos 7.2 machines and was able to mount Kerberos nfs without rebooting server and client.

          Today I wrote the exam again and got the same score (196) as before. The nfs-secure-server was not starting at all in the server machine. I checked the keytab files in both server/client with cat /etc/krb5.keytab and each had their individual name.

          I started nfs-server, nfs-secure both in the server and client.

          I had a new problem today that I did not have on my first attempt. The ‘nmcli con up team0’ was giving error. The team0(link aggregation) was not being up.

          • brucemzn says:

            @mmhaque4,
            Sorry for late response. Please note that the exam is based on Redhat 7.0 unless otherwise stated by Redhat on their official website. For you to pass the exam at this moment, you must practice using Redhat/Centos 7.0

            To confirm keytab files, use # klist -k command

            Looks like you enabled and started all the service, but some were unnecessary, for example, you dont have to start the nfs-secure.service on the server side.

            The following will work, provided you have valid keytab files:
            Server Side:
            enable nfs-secure-server.service
            enable nfs-server.service
            start nfs-secure-server.service
            start nfs-server.service

            Client Side:
            enable nfs-server.service
            enable nfs-secure.service
            start nfs-server.service
            start nfs-secure.service

            Will you be kind to provide the error messages please, then I will assist you to troubleshoot the problem. Awaiting for your response.

            As for link aggregation, use nmtui command. To get right JSON configs:
            cat /usr/share/doc/teamd-1.9/example_configs/activebackup_ethtool_1.conf

          • mmhaque4 says:

            Thanks brucemzn,

            During the exam, the ‘nfs-secure-server.service’ was not starting in the server even though I had the correct krb5.keytab file in the /etc/ directory.

            As for the team configuration, I do not remember what was the error message but it was something ‘nmcli’ and NetworkManager versions mismatch. As a result the new connection ‘team0’ was not turning up. It was created but was not shown under the device column.

            It appears the nmcli and NetworkManager should have exactly the same version numbers.

            I have just emailed RedHat regarding the versions mismatch and requested them to clarify whether it was my task to fix the version conflict issue or they should have provided me with the matched versions.

            Thanks for your help.

          • brucemzn says:

            @mmhaque4
            Thanks for the response. As for the network interface teaming, i would kindly request you to start a thread under Networking teaming, and i will post the solution there. Right now, let talk about Kerberised NFS.

            Its a pity that we cant discuss the exam questions here or anywhere. But we can recreate the scenario under lab environment. My request to you is to recreate the scenario, and let us know what the error messages are. These error messages are important to help troubleshoot. My questions to you are:
            1. Do you have full functional Kerberos Server in your lab environment?
            2. Do you have two client systems running Redhat/Centos 7.0 or 7.1 Either version is not an issue as we can troubleshoot.

            If you don’t have a lab environment, its difficult for us to help pinpoint the errors you are facing

            Finally, don’t forget to start the thread on Network Teaming. The solution will follow.

            Regards
            Bruce Malaudzi

          • mmhaque4 says:

            Hi brucemzn,

            I am happy to let you know that I finally passed the RHCE yesterday! That was my third attempt.

            This time both Network Teaming and Kerborized NFS worked just fine.

            As for teaming nmcli command worked. And for the NFS, I did not have to reboot both VMs before mounting. Also, I did not have to start ntp/chronyd service and selinux fcontext on this.

            Thanks again for your help.

          • CertDepot says:

            Congratulations!

          • brucemzn says:

            Hi mmhaque4.

            Congratulations and thanks for sharing. As you said that you did not start start ntp/chronyd this time. That’s an impressive theory. Like I said before, time does not affect NFS Kerberos. Thanks you for providing us with such information. That’s right, ntp/chronyd does not affect NFS Kerberos in RHCE exam lab environment.

          • CertDepot says:

            How would you explain time doesn’t affect NFS Kerberos when a delay of 5 minutes is said to be a real problem?

          • brucemzn says:

            Hi CertDepot,

            Sorry for the late response. I been have away for quite some time.

            I have two labs. One is based on RHEL-7.0 and the other RHEL-7.1

            When i do the Kerberised NFS task, clients can mount Kerberised NFS shares successfully, on both labs.

            Neither ntpd nor chronyd are running when i do the Kerberised NFS task.

            # systemctl status ntpd (shows dead)
            # systemctl status chronyd (shows dead)

            Therefore my theory that NTP does not affect Kerberos NFS in a lab environment is proven to be true. Please feel free to ask further questions.

            Any input will be appreciated.

          • CertDepot says:

            I believe you. The fact is your servers are required to be synchronized within a 5-minute window. Normally, you use NTP to get this result. If you can get this result in your lab without NTP, it’s fine. You get my point?
            I would be more impressed if you’ve got servers not synchronized at all and get Kerberos working.

          • brucemzn says:

            Hi CertDepot,

            Thanks for the response. I agree and get your point. Its a learning curve.

          • Lisenet says:

            That’s not a proof I’m afraid, you are simply lucky to have your servers in time sync. Try shifting time one hour ahead so that it differs among servers, and let us know if you still get it working.

          • brucemzn says:

            Hi Lisenet,

            Thanks for the response.

            Well, i simulated the environment per your request. Three machines are involved in this scenario, a KDC, an NFS Server and NFS Client. I shifted the time, at least two hours apart from each machine.

            I then deleted existing keytab files. I then removed nfs-utils packages. I then installed nfs-utils from scratch. Then i created new keytab files on both NFS server and NFS client machine.

            The NFS client managed to mount the Kerberised NFS share with no issues at all.

            Please see attached screenshots on the following link. There are four images. Please view all four and analyze them closely. Questions are welcome and i will be grateful to answer them.

            Link: https://drive.google.com/drive/folders/0B3dXAvMr1PE7T2xGenNEdkZmcUk?usp=sharing

            This conclude that NTP does not affect Kerberos NFS. I will be posting a video on youtube in two weeks time, where i demonstrate this live on camera. I will share the video with everyone.

            Thanks for reading and i hope to hear from you soon.

          • Lisenet says:

            I’m sorry, but I cannot replicate it, NFS mount fails when I shift time.

            Try the following:

            1. Sync time with all 3 servers, IPA, NFS server and NFS client, ensure the time is in sync.
            2. On the NFS server, stop chronyd, disable NTP sync, change time so that it’s 1 hour behind compared to IPA/NFS client.
            3. On the NFS client, try to mount the kerberised NFS share.

          • brucemzn says:

            Hi Lisenet

            Thanks for the response. Your failure to replicate the scenario, made me wonder. And it led to one theory. The theory is that “One of us is doing something wrong.” So I went back to the drawing board, to simulate the scenario. After several tests, I came to realise that I’m the one who is wrong.

            I have been using the command # date set-time to shift the times on all three machines, but not realising that time was reverted after a reboot.

            Using the command timedatectl makes permanent changes. NFS client failed to mount the Kerberos share. Therefore you are right, Kerberos needs time synchronization for smooth operation.

            I do apologise to everyone. I did not mean to mislead anyone, but everyday I learn new things by recreating the scenario and testing it over and over. Today I have learned three things:

            1. Using # date command to manipulate time is not persistent. Changes are not saved after a reboot

            2. Using # timedatectl command to manipulate time makes permanent changes.

            3. Time synchronisation is essential for Kerberos to work as expected.

          • CertDepot says:

            You are learning and it’s great!

          • Lisenet says:

            I second that.

  16. David_V says:

    Does the krb5.keytab go on the NFS server machine or the NFS client machine? Sander and Asghar say the exact opposite.. Sander says NFS server, and Asghar says on the NFS client.

  17. raj says:

    I contacted Redhat Training to ask about the exact version of RHEL7 used for the RHCE EX300 exam. I got a response today confirming it as RHEL7.1.
    I have created my own repo based on http://archive.kernel.org/centos-vault/7.1.1503/isos/x86_64/CentOS-7-x86_64-Minimal-1503-01.iso. This ensures I’m not upgraded to RHEL7.2.

    • CertDepot says:

      This is a stunning news.
      Although I believe you, I would like to get an official confirmation from Red Hat.
      There are some serious consequences concerning people’s preparation for the RHCE exam.
      Thanks a lot for this information.

  18. raj says:

    I was surprised as well but that’s the response I got from training-uk@redhat.com. I’m on the course/exam in a few weeks so I’ll confirm and post an update.

  19. raj says:

    Thanks James for the update as I’ve been preparing for an exam based on RHEL7.1 . This uncertainty is very frustrating. I’ve just called Redhat training UK and they said that EX300 is based on RHEL7.0 but refused to comment on the email I have from them stating it was RHEL7.1 .

    Their only statement was email us again and we’ll tell you the current version . I’ve asked for a URL which has the updated version of the exam as it is not on the objectives. Has anyone else contacted Redhat ?

    • Lisenet says:

      There are not that many differences between RHEL 7 versions. Walk the extra mile, practice all tasks on RHEL 7.0, 7.1 and 7.2, and you’ll be fine no matter what version you get on the exam. That’s what I ended up doing.

  20. raj says:

    As I understand it, the exact version (including point release ) is not listed anywhere on their site and can only be obtained by directly contacting Redhat. I’m loosing patience with Redhat now. Here is my latest response from Redhat UK training about 5 mins ago.

    “The EX300 is on Linux Rhel 7 version which you would be able to see clearly on our website as well, for details you can refer the link : https://www.redhat.com/en/services/training/ex300-red-hat-certified-engineer-rhce-exam

    However, if you want to still confirm on which the version is based on you can contact our certification team. We did check on which version it is running and they confirmed that it is on RHEL 7.0.”

    https://www.redhat.com/training/certification/comments.html

    • Lisenet says:

      I know it’s not gonna help you much, but I tend to agree with Randy Russell from Red Hat. His comment about different minor releases made sense:

      “What’s required is knowing underlying principles rather than rote memorization. Dot releases are part of the job.”

      In prod, I need to work with all versions of RHEL.

      • CertDepot says:

        I don’t fully agree with you. This triggers an additional difficulty that is not really necessary.

        • Lisenet says:

          You’re right from that perspective, it does add some difficulty (but whether it’s necessary or not I don’t know). My point is, RHCE is an advanced, I’d say real-life, level certificate, you aren’t expected to memorise tasks, you need to know a bit more than that.

        • Lisenet says:

          On second thought, I actually agree with you. I’ve started studying for RHCA EX436, and that’s where it got tricky. If the exam was on RHEL 6, it would be a real nightmare to prepare for it without knowing the dot release.

          For example, up to RHEL 6.5 pcs and pacemaker weren’t fully supported, now they are, but the original cluster shell (crmsh) is no longer available. Clustering on RHEL 6 was a real pain (it’s a better now with pcs), therefore to prepare properly, one needs to know the dot release to practice on.

  21. raj says:

    Hi Lisenet,
    Your email is interesting, below is the email I got today from the same training administrator. It’s ridiculous that this is not clearly documented on the site as there is no easy way to know if/when the version changes. When I enquired in June the same training administrator stated the version was 7.1 and I rebuilt my home lab to 7.1 . Three months later, I’m now being told it’s 7.0. Is Redhat moving backwards or was I initially given incorrect info?

    Hi ,
    Sorry for the confusion and inconvenience caused to you.
    The EX300 is on Rhel 7.0 not Rhel 7.1.

    We request you to check the below link for more information https://www.redhat.com/en/resources/red-hat-training-and-certification-classroom-requirements

    Regards,
    Shim
    Training Administrator

    Red Hat Training UK
    Tel: 0800 1456153
    Fax: +44 (0) 1252 601 214
    E-mail: training-uk@redhat.com
    Web: http://www.redhat.com/training
    Timings: Mon to Fri (0830 – 1700 Hrs)

  22. sab says:

    Hi,
    Could you please clarify for me how NFS Selinux booleans works?
    Manpage nfsd_selinux(8) says:
    nfs_export_all_ro: If you want to allow any files/directories to be exported read/only via NFS, you must turn on the nfs_export_all_ro boolean. Enabled by default.
    nfs_export_all_rw: If you want to allow any files/directories to be exported read/write via NFS, you must turn on the nfs_export_all_rw boolean. Enabled by default.
    However, no matter if these booleans are enabled or disabled on server, I am able to export shares and client is able to mount with RW permissions.
    And also I have a question regarding context on files on server – server is exporting folders with any context on it and client is able to write to this folder.
    To me it looks like that nfsd is working as a kernel module with kernel_t context and don’t care about booleans or file contexts.

    • Lisenet says:

      These booleans are not required when files to be shared via NFS are labeled with the public_content_t or public_content_rw_t types.

      NFS can share files labeled with these types even if the nfs_export_all_ro and nfs_export_all_rw booleans are off. This might explain why you are able to mount with RW permissions.

      • sab says:

        Thanks for your comments,
        This should work as you describe, however I’m able to export files labeled with any context with disabled booleans:
        [root@nfsserver ~]# getsebool -a | grep ^nfs
        nfs_export_all_ro –> off
        nfs_export_all_rw –> off
        nfsd_anon_write –> off
        [root@nfsserver ~]# ls -ldZ /share/secured/
        drwxr-xr-x. harry root unconfined_u:object_r:default_t:s0 /share/secured/
        [root@nfsserver ~]# cat /etc/exports
        /share/secured *.example.com(rw,sec=krb5p)
        [root@nfsserver ~]# exportfs -rav
        exporting *.example.com:/share/secured
        On the client:
        [root@nfsclient ~]# tail -1 /etc/fstab
        nfsserver.example.com:/share/secured /nfs/secured nfs _netdev,rw,sync,sec=krb5p 0 0
        [root@nfsclient ~]# ls -ldZ /nfs/secured/
        drwxr-xr-x. nobody nobody system_u:object_r:nfs_t:s0 /nfs/secured/
        [harry@nfsclient ~]$ klist
        Ticket cache: KEYRING:persistent:1003:1003
        Default principal: harry@EXAMPLE.COM

        Valid starting Expires Service principal
        10/25/2016 04:21:51 10/26/2016 04:21:50 krbtgt/EXAMPLE.COM@EXAMPLE.COM
        [harry@nfsclient ~]$ touch /nfs/secured/harryfile1
        [harry@nfsclient ~]$

  23. ganastasiou says:

    Hello,
    First of all i would like to thank you for the material and how-to education you have shared. I would like to ask you if you are aware, if during the exam keytabs are shared or either admin password for Kerberos to “get” them locally.

    • CertDepot says:

      Sorry, I don’t know.

    • brucemzn says:

      Keytabs files are premade/pre-generated for you, waiting for download from a remote location/server. Keytab file for server1 cannot be used on server2. During the exam you are provided with a user’s Kerberos password to get a Kerberos key (aka Ticket Granting Ticket (TGT)). You are not provided with Kerberos admin password, because you do not need it.

      • Sam says:

        Exams are subject to change. The keytab may or may not work on server1 and server2. It depends on the way the keytab is generated.
        You may or may not get an admin password. I would suggest you prepare for different scenarios.

  24. waleed8000 says:

    I have a problem after mount nfs and access folder with kerberos user and create file but file permission is nobody:nobody

    • Sam says:

      A quick search, and check your dns settings. Reference http://whacked.net/2006/07/26/nfsv4nfs-mapid-nobody-domain/

    • brucemzn says:

      Hi Waleed
      Let me rephrase your question. Are you saying that nfs share is successfully mounted with Kerberos authentication, but as user cannot access the mount point or create files inside the mount point, although permissions are set to nobody:nobody ?

      • waleed8000 says:

        yes I mean that file permissions nobody:nobody mount with kerberos authentication success

        This is server option ” /home *(rw,no_root_squash,sec=krb5)”

        • Sam says:

          waleed8000, Did you check your DNS settings, what are they? or has this been resolved ?

        • brucemzn says:

          Hi Waleed8000
          Its great that you can mount the NFS share through Kerberos. The user needs a Kerberos tick (aka TGT) to be able to write

          1. Change to the respective user
          # su – user01
          2. Check if this user has a TGT already
          # klist
          3. Get a new TGT
          # kinit
          4. Confirm if the user now have a TGT
          # klist
          5. If this user has a TGT, the user can read and write on allocated NFS shares.

          Important: The user must already been defined on the kerberos server side, because kerberos will prompt you for a password to authenticate.

          The following video might help. Go to 48:00 minutes of the video.
          https://youtu.be/876AWmYQFOE

  25. hunter86_bg says:

    Ok, I found the issue – it was me and my lack of knowledge and experience with NFSv4.
    Everything is ok if I use “/etc/exports” like this:
    # cat /etc/exports
    /public *(rw,all_squash)
    /group *(rw,no_root_squash)
    /private *(rw,sec=krb5p)

    But if I change it to this:
    /nfs/public *(rw,all_squash)
    /nfs/group *(rw,no_root_squash)
    /nfs/private *(rw,sec=krb5p)

    It is successfully mounted as NFSv3. I just realized that Sander van Vugt has mentioned that NFSv4 can be mounted by selecting only the top mount, aka “root” but I am quite perplexed with this idea.

    Could you share your knowledge on this ?

  26. hunter86_bg says:

    Hello friends,
    here is my latest “discovery” about kerberos.

    1.When you are using RHEL7.0 to prepare and nfs-secure.service on the client refuses to mount your mount point until you restart it –> check your client’s Static Hostname. It should be FQDN, no matter that the dns resolves it as simple “server2”.
    2.Kerberos is time aware, but is essential only for the applications. For example if your Kerberized NFS Server is out of sync -> the client will get “access denied” message, but if the client is out of sync – it will connect without issues.

    RedHat’s doc quote:
    Applications that accept Kerberos authentication require time synchronization.
    Kerberos clients running Red Hat Enterprise Linux 7 support automatic time adjustment with the KDC and have no strict timing requirements. This enables better tolerance to clocking differences when deploying IdM clients with Red Hat Enterprise Linux 7.

  27. DanS says:

    Took my Exam today in Texas August 2017 and still on RHEL 7.0
    Also understand my mistakes now on nfs-secure and nfs-secure-server from this site and video. Thanks

  28. maikeu says:

    Hi Certdepot

    I’ve kind of got this going. Going to keep going back to my snapshot to see that I can reconfigure it without referring to instructions.

    The “why” is still confusing me a bit though. I’m not sure if previous failures were due to not having ntp set up properly, or some other issue.

    I’ve been testing with the keytab file on the server, using the ktutil program to awkwardly remove individual keys, but I haven’t fully tracked down exactly which keytabs are required on the client, to be able to mount the kerberised nfs share.

    Between your instructions, Jang’s, and Tecmint https://www.tecmint.com/setting-up-nfs-server-with-kerberos-based-authentication/, I haven’t really found something that breaks it down so that I can understand in a way that is better than “follow this sequence of instructions,” Jang I think might have a mistake in his sequence, while yours and Tecmint’s don’t seem to mention anything about copying keytabs over to the client? Correct me if I’ve gotten that wrong.

    • Sam says:

      Failure of NTP to sync or improper jitter will cause Kerberos NFS to fail strangely.

      I am not to sure where you are going with removing of individual keys in the key tab. I will guess it is the encryption type. ie aes256-cts-hmac-sha1-96
      aes256 – encryption algorithm, advanced encryption standard at 256 bit
      cts – an mode of operation in aes.
      hmac – Hash-based message authentication code
      sha1 – sha 1 hashing algorithm
      96 (not sure on this one)
      Different algorithms have different strengths and weakness.

      You are on the correct track on the last point. It is possible to generate the keytab file on the server or one of several clients. This can be done if the Kerberos admin password is to be kept to one server.

    • Lisenet says:

      Tecmint instructions won’t work for RHEL 7.1 and above so be careful.

      If you want comprehensive instructions, you may want to take a look here: https://www.lisenet.com/2016/kerberised-nfs-server-on-rhel-7/

  29. student says:

    On exam, just copying keytab files will help us? Or should we troubleshoot some issues related to Kerberos? On training everything goes smooth but on exam Kerberos topics make problem. Who passed exam, could you tell us what we need to know exactly?

  30. Oxygen says:

    Thanks for the article. Not sure if that helps someone but I was getting nobody:nobody on the client mount point until I uncommented and specified domain in /etc/idmapd.conf on a server side (restart required). You can increase logs verbosity there as well, by default it’s 0.
    Cheers!

  31. Oxygen says:

    Just a quick remark to my comment, my OS hostname was “name.mydomain.com” but domainname was “(none)”. My kerberos users apparently were bound to “mydomain.com”, that’s why I had to change that.

  32. Oxygen says:

    Sorry, one more thing, /etc/idmapd.conf states:
    # The default is the host’s DNS domain name.
    so that would be myhost.mydomain.com but it only worked when I stated mydomain.com explicitly.

  33. itninja says:

    The problem is that you have to add nfs service for the server side as well, since it is not found in the kerberos database – you can clearly see it in krb5 logs.

  34. amedix says:

    Hi,

    I’ve been told that the file “krb5.keytab” will be provided in the exam?
    But do I still need to install and configure the kerberos server and/or client (krb5.conf)?

    Thanks

    • CertDepot says:

      You won’t have to configure the Kerberos server but you will need to set up the client configuration.

      • amedix says:

        Hi,

        Thanks for your quick reply.

        By client configuration do you mean it’s just updating the file /etc/krb5.conf ?

        • Sam says:

          When you go through the example on this page, you will see the step involved. There is a good bit to the installation/configuration, including getting the correct services to run. In short I would suggest that you go through the example, in full. It is the only realistic answer.

    • Sam says:

      I would suggest that you should know the steps, on how to generate the keytab file(s). There are times that the steps may help with troubleshooting!

  35. Zorbeg says:

    Hi there

    I managed to set up everything, kerberized nfs is exported from server:
    [root@server1 srv]# cat /etc/exports
    /srv/secret *(rw,sec=krb5p)

    [root@server1 srv]# ls -l
    total 0
    drwxr-xr-x. 2 ldapuser1 ldapuser1 41 Jan 25 16:25 secret

    and mounted on client:
    [root@server2 ~]# ls -ld /secret/
    drwxr-xr-x. 2 ldapuser1 ldapuser1 41 Jan 25 16:25 /secret/

    Share is accessed by ldapuser1, it is rw for this user, so everything works as I intended.

    There is however one question that is bugging me.
    ldapuser1 has to execute kinit in order to obtain ticket from IPA server. otherwise user can’t do anything with shared directory.

    Is this behavior expected on exam? Should user somehow automatically get ticket upon logon?

    • Zorbeg says:

      If someone faces same problem in the future:

      Need to ssh to server2 (nfs client) as ldap user, then this user will have ticket for some time and does not have to run kinit every time

      # ssh ldapuser1@localhost
      $ klist # shows ticket for a day or so
      $ touch /secret/whoami # should now work without using “kinit”

  36. SpencerArnold888 says:

    I can confirm this process does not work in Redhat 7.6 either. The more I learn about the “standards” redhat operates their exams under, the less i want to waste my free time and money with them.

Leave a Reply

Upcoming Events (Local Time)

There are no events.

Follow me on Twitter

Archives

vceplus-200-125    | boson-200-125    | training-cissp    | actualtests-cissp    | techexams-cissp    | gratisexams-300-075    | pearsonitcertification-210-260    | examsboost-210-260    | examsforall-210-260    | dumps4free-210-260    | reddit-210-260    | cisexams-352-001    | itexamfox-352-001    | passguaranteed-352-001    | passeasily-352-001    | freeccnastudyguide-200-120    | gocertify-200-120    | passcerty-200-120    | certifyguide-70-980    | dumpscollection-70-980    | examcollection-70-534    | cbtnuggets-210-065    | examfiles-400-051    | passitdump-400-051    | pearsonitcertification-70-462    | anderseide-70-347    | thomas-70-533    | research-1V0-605    | topix-102-400    | certdepot-EX200    | pearsonit-640-916    | itproguru-70-533    | reddit-100-105    | channel9-70-346    | anderseide-70-346    | theiia-IIA-CIA-PART3    | certificationHP-hp0-s41    | pearsonitcertification-640-916    | anderMicrosoft-70-534    | cathMicrosoft-70-462    | examcollection-cca-500    | techexams-gcih    | mslearn-70-346    | measureup-70-486    | pass4sure-hp0-s41    | iiba-640-916    | itsecurity-sscp    | cbtnuggets-300-320    | blogged-70-486    | pass4sure-IIA-CIA-PART1    | cbtnuggets-100-101    | developerhandbook-70-486    | lpicisco-101    | mylearn-1V0-605    | tomsitpro-cism    | gnosis-101    | channel9Mic-70-534    | ipass-IIA-CIA-PART1    | forcerts-70-417    | tests-sy0-401    | ipasstheciaexam-IIA-CIA-PART3    | mostcisco-300-135    | buildazure-70-533    | cloudera-cca-500    | pdf4cert-2v0-621    | f5cisco-101    | gocertify-1z0-062    | quora-640-916    | micrcosoft-70-480    | brain2pass-70-417    | examcompass-sy0-401    | global-EX200    | iassc-ICGB    | vceplus-300-115    | quizlet-810-403    | cbtnuggets-70-697    | educationOracle-1Z0-434    | channel9-70-534    | officialcerts-400-051    | examsboost-IIA-CIA-PART1    | networktut-300-135    | teststarter-300-206    | pluralsight-70-486    | coding-70-486    | freeccna-100-101    | digitaltut-300-101    | iiba-CBAP    | virtuallymikebrown-640-916    | isaca-cism    | whizlabs-pmp    | techexams-70-980    | ciscopress-300-115    | techtarget-cism    | pearsonitcertification-300-070    | testking-2v0-621    | isacaNew-cism    | simplilearn-pmi-rmp    | simplilearn-pmp    | educationOracle-1z0-809    | education-1z0-809    | teachertube-1Z0-434    | villanovau-CBAP    | quora-300-206    | certifyguide-300-208    | cbtnuggets-100-105    | flydumps-70-417    | gratisexams-1V0-605    | ituonline-1z0-062    | techexams-cas-002    | simplilearn-70-534    | pluralsight-70-697    | theiia-IIA-CIA-PART1    | itexamtips-400-051    | pearsonitcertification-EX200    | pluralsight-70-480    | learn-hp0-s42    | giac-gpen    | mindhub-102-400    | coursesmsu-CBAP    | examsforall-2v0-621    | developerhandbook-70-487    | root-EX200    | coderanch-1z0-809    | getfreedumps-1z0-062    | comptia-cas-002    | quora-1z0-809    | boson-300-135    | killtest-2v0-621    | learncia-IIA-CIA-PART3    | computer-gcih    | universitycloudera-cca-500    | itexamrun-70-410    | certificationHPv2-hp0-s41    | certskills-100-105    | skipitnow-70-417    | gocertify-sy0-401    | prep4sure-70-417    | simplilearn-cisa    |
http://www.pmsas.pr.gov.br/wp-content/    | http://www.pmsas.pr.gov.br/wp-content/    |