How to enable Kerberos for Hadoop

A Hadoop cluster in simple mode trusts the user names supplied by its clients. On Linux, Kerberos gives HDFS and YARN a way to authenticate users and daemons before accepting data access or application requests.

Each daemon authenticates with a host-specific service principal and a private keytab. Matching forward and reverse hostnames, synchronized clocks, and a working Kerberos realm are essential because a ticket for one service hostname cannot authenticate a differently named service.

Switching an existing cluster to secure mode interrupts access until its daemons and clients share the new configuration. A maintenance window, recoverable configuration copies, and working administrative access are necessary. The starting cluster has separate HDFS and YARN service accounts, HTTPS keystores, and user accounts available on its worker nodes.

Steps to enable Kerberos for Hadoop:

Back up the active configuration directory on each host before changing authentication.
```
$ sudo cp -a /opt/hadoop/etc/hadoop /opt/hadoop/etc/hadoop.before-kerberos
```
A change on only part of the cluster can prevent daemons and clients from communicating. An existing backup with this name needs a different destination. If startup fails, the previous configuration can be restored on all affected hosts during the same outage before their services are started again. No NameNode formatting is involved.

The paths use an Apache Hadoop 3 installation under /opt/hadoop; a different installation requires its actual active configuration directory. The non-HA cluster has its NameNode, Secondary NameNode, and ResourceManager on master01.example.net and its DataNode and NodeManager on worker01.example.net. The realm is EXAMPLE.NET. Other worker hosts need their own principals and keytabs. Existing storage paths, replication, scheduler settings, and client addresses stay in place.

HDFS and YARN HTTPS must already have usable ssl-server.xml settings and host certificates. The SASL DataNode method requires HTTPS_ONLY; enabling that policy without a keystore prevents startup. Ordinary application users must exist on worker nodes and must not belong to the privileged hadoop group. The Hadoop installation and its parent directories must be root-owned and protected from ordinary users.
Related: Apache Hadoop secure-mode requirements
Stop YARN from the master during the maintenance window.
```
$ sudo -u yarn /opt/hadoop/sbin/stop-yarn.sh
```
The scripts require the intended hosts in workers and working SSH access for hdfs and yarn, including any local host selected by a script. Every daemon they manage needs its service identity before restart. The existing service environment must supply writable log and PID directories. A distribution-managed cluster uses its service manager for the same stop/start operations.
Related: Apache Hadoop cluster operation

Stop HDFS from the master.

$ sudo -u hdfs /opt/hadoop/sbin/stop-dfs.sh

Create a private keytab staging directory on the MIT Kerberos KDC.
```
$ sudo install -d -m 700 /root/hadoop-keytabs
```
Keytabs provide passwordless access as their principals. Only realm administrators should be able to read the staging directory, and its contents must stay out of shared folders, logs, and backups without secret protection.

Create the daemon and HTTP principals on the KDC with random keys.

$ sudo kadmin.local
kadmin.local:  addprinc -randkey nn/master01.example.net@EXAMPLE.NET
kadmin.local:  addprinc -randkey rm/master01.example.net@EXAMPLE.NET
kadmin.local:  addprinc -randkey HTTP/master01.example.net@EXAMPLE.NET
kadmin.local:  addprinc -randkey dn/worker01.example.net@EXAMPLE.NET
kadmin.local:  addprinc -randkey nm/worker01.example.net@EXAMPLE.NET
kadmin.local:  addprinc -randkey HTTP/worker01.example.net@EXAMPLE.NET
kadmin.local:  quit

kadmin.local runs on the primary MIT KDC with realm-administrator privileges. Existing principals need their existing keys rather than recreation. The uppercase HTTP service name is intentional. JournalNodes, a separately hosted Secondary NameNode, or a MapReduce JobHistory Server need their own service identities and configuration.

Export each service keytab into the private staging directory without changing its keys.

$ sudo kadmin.local
kadmin.local:  ktadd -norandkey -k /root/hadoop-keytabs/nn-master01.keytab nn/master01.example.net@EXAMPLE.NET
kadmin.local:  ktadd -norandkey -k /root/hadoop-keytabs/rm-master01.keytab rm/master01.example.net@EXAMPLE.NET
kadmin.local:  ktadd -norandkey -k /root/hadoop-keytabs/HTTP-master01.keytab HTTP/master01.example.net@EXAMPLE.NET
kadmin.local:  ktadd -norandkey -k /root/hadoop-keytabs/dn-worker01.keytab dn/worker01.example.net@EXAMPLE.NET
kadmin.local:  ktadd -norandkey -k /root/hadoop-keytabs/nm-worker01.keytab nm/worker01.example.net@EXAMPLE.NET
kadmin.local:  ktadd -norandkey -k /root/hadoop-keytabs/HTTP-worker01.keytab HTTP/worker01.example.net@EXAMPLE.NET
kadmin.local:  quit

MIT ktadd normally randomizes a principal's keys, invalidating older copies. The local -norandkey option preserves them.
Related: MIT Kerberos principal and keytab administration

Deliver each host's keytabs to its root-only /root/hadoop-keytabs staging directory through the existing secret-management channel.

The master receives nn-master01.keytab, rm-master01.keytab, and HTTP-master01.keytab. The worker receives dn-worker01.keytab, nm-worker01.keytab, and HTTP-worker01.keytab. A host receives only its own keys.
Create the restricted keytab directory on each daemon host.
```
$ sudo install -d -o root -g hadoop -m 750 /etc/security/keytabs
```
HDFS daemons run as hdfs and YARN daemons as yarn. The HTTP keytab is group-readable because both service accounts use the web authentication identity on a shared host.

Install the master's three service keytabs with their restricted owners and permissions.

$ sudo install -o hdfs -g hadoop -m 400 /root/hadoop-keytabs/nn-master01.keytab /etc/security/keytabs/nn.service.keytab

$ sudo install -o yarn -g hadoop -m 400 /root/hadoop-keytabs/rm-master01.keytab /etc/security/keytabs/rm.service.keytab

$ sudo install -o root -g hadoop -m 640 /root/hadoop-keytabs/HTTP-master01.keytab /etc/security/keytabs/http.service.keytab

Install the worker's three service keytabs with their restricted owners and permissions.

$ sudo install -o hdfs -g hadoop -m 400 /root/hadoop-keytabs/dn-worker01.keytab /etc/security/keytabs/dn.service.keytab

$ sudo install -o yarn -g hadoop -m 400 /root/hadoop-keytabs/nm-worker01.keytab /etc/security/keytabs/nm.service.keytab

$ sudo install -o root -g hadoop -m 640 /root/hadoop-keytabs/HTTP-worker01.keytab /etc/security/keytabs/http.service.keytab

Set Kerberos authentication and service-to-account mappings in core-site.xml.

The XML edits apply on both the master and worker under /opt/hadoop/etc/hadoop, using the usual text editor and preserving unrelated settings. The container-executor.cfg edit applies on each worker. Each XML property belongs inside its existing <configuration> element. A property already present needs its value replaced rather than a duplicate entry. Client machines need the same core-site.xml and HDFS connection/security settings.

core-site.xml

  <property>
    <name>hadoop.security.authentication</name>
    <value>kerberos</value>
  </property>
  <property>
    <name>hadoop.security.authorization</name>
    <value>true</value>
  </property>
  <property>
    <name>hadoop.rpc.protection</name>
    <value>privacy</value>
  </property>
  <property>
    <name>hadoop.security.auth_to_local</name>
    <value>RULE:[2:$1/$2@$0]([nd]n/.*@EXAMPLE\.NET)s/.*/hdfs/
RULE:[2:$1/$2@$0]([rn]m/.*@EXAMPLE\.NET)s/.*/yarn/
DEFAULT</value>
  </property>
  <property>
    <name>hadoop.http.authentication.type</name>
    <value>kerberos</value>
  </property>
  <property>
    <name>hadoop.http.authentication.kerberos.principal</name>
    <value>HTTP/_HOST@EXAMPLE.NET</value>
  </property>
  <property>
    <name>hadoop.http.authentication.kerberos.keytab</name>
    <value>/etc/security/keytabs/http.service.keytab</value>
  </property>

The mappings turn nn and dn identities from this realm into hdfs, and rm and nm into yarn. Ordinary users in the default realm retain their user names. _HOST is expanded to each daemon's fully qualified hostname; it is never a literal KDC principal. RPC privacy adds encryption as well as authentication.

Add the HDFS service identities and SASL transfer settings to hdfs-site.xml.

hdfs-site.xml

  <property>
    <name>dfs.namenode.kerberos.principal</name>
    <value>nn/_HOST@EXAMPLE.NET</value>
  </property>
  <property>
    <name>dfs.namenode.keytab.file</name>
    <value>/etc/security/keytabs/nn.service.keytab</value>
  </property>
  <property>
    <name>dfs.secondary.namenode.kerberos.principal</name>
    <value>nn/_HOST@EXAMPLE.NET</value>
  </property>
  <property>
    <name>dfs.secondary.namenode.keytab.file</name>
    <value>/etc/security/keytabs/nn.service.keytab</value>
  </property>
  <property>
    <name>dfs.namenode.secondary.https-address</name>
    <value>master01.example.net:9869</value>
  </property>
  <property>
    <name>dfs.datanode.kerberos.principal</name>
    <value>dn/_HOST@EXAMPLE.NET</value>
  </property>
  <property>
    <name>dfs.datanode.keytab.file</name>
    <value>/etc/security/keytabs/dn.service.keytab</value>
  </property>
  <property>
    <name>dfs.web.authentication.kerberos.principal</name>
    <value>HTTP/_HOST@EXAMPLE.NET</value>
  </property>
  <property>
    <name>dfs.web.authentication.kerberos.keytab</name>
    <value>/etc/security/keytabs/http.service.keytab</value>
  </property>
  <property>
    <name>dfs.block.access.token.enable</name>
    <value>true</value>
  </property>
  <property>
    <name>dfs.data.transfer.protection</name>
    <value>privacy</value>
  </property>
  <property>
    <name>dfs.http.policy</name>
    <value>HTTPS_ONLY</value>
  </property>
  <property>
    <name>dfs.datanode.address</name>
    <value>0.0.0.0:9866</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir.perm</name>
    <value>700</value>
  </property>

The Secondary NameNode runs alongside the NameNode as hdfs on master01.example.net. Both use that host's nn principal and keytab; the separate Secondary NameNode properties configure its login. Its HTTPS address names the checkpoint host used by start-dfs.sh.

SASL with privacy authenticates and encrypts DataNode transfers on the nonprivileged port 9866. The DataNode runs as hdfs without jsvc or root startup. HDFS_DATANODE_SECURE_USER must remain unset for this method. Existing network policy must allow the selected data-transfer and HTTPS ports between the intended hosts.

Add the YARN identities and LinuxContainerExecutor settings to yarn-site.xml.

yarn-site.xml

  <property>
    <name>yarn.resourcemanager.principal</name>
    <value>rm/_HOST@EXAMPLE.NET</value>
  </property>
  <property>
    <name>yarn.resourcemanager.keytab</name>
    <value>/etc/security/keytabs/rm.service.keytab</value>
  </property>
  <property>
    <name>yarn.nodemanager.principal</name>
    <value>nm/_HOST@EXAMPLE.NET</value>
  </property>
  <property>
    <name>yarn.nodemanager.keytab</name>
    <value>/etc/security/keytabs/nm.service.keytab</value>
  </property>
  <property>
    <name>yarn.nodemanager.container-executor.class</name>
    <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
  </property>
  <property>
    <name>yarn.nodemanager.linux-container-executor.group</name>
    <value>hadoop</value>
  </property>
  <property>
    <name>yarn.nodemanager.linux-container-executor.path</name>
    <value>/opt/hadoop/bin/container-executor</value>
  </property>
  <property>
    <name>yarn.http.policy</name>
    <value>HTTPS_ONLY</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.spnego-principal</name>
    <value>HTTP/_HOST@EXAMPLE.NET</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.spnego-keytab-file</name>
    <value>/etc/security/keytabs/http.service.keytab</value>
  </property>
  <property>
    <name>yarn.nodemanager.webapp.spnego-principal</name>
    <value>HTTP/_HOST@EXAMPLE.NET</value>
  </property>
  <property>
    <name>yarn.nodemanager.webapp.spnego-keytab-file</name>
    <value>/etc/security/keytabs/http.service.keytab</value>
  </property>

LinuxContainerExecutor launches applications as the submitting Unix user. DefaultContainerExecutor runs them as the NodeManager account and is unsuitable for this secure Linux setup. Existing yarn.resourcemanager.hostname, local directories, and resource settings remain unchanged.

Set the privileged executor group and denied service accounts in container-executor.cfg.
container-executor.cfg
```
yarn.nodemanager.linux-container-executor.group=hadoop
banned.users=hdfs,yarn,mapred,bin
min.user.id=1000
```
The native container-executor binary can switch Unix users. Its group must contain trusted service accounts only. The minimum user ID must fit the site's normal application accounts; service accounts remain banned regardless of their IDs.

The native executor reads the configuration directory selected when it was built. The Apache distribution uses its Hadoop configuration directory; a custom build may use another path.
Assign the native executor binary to root and the trusted service group on each worker.
```
$ sudo chown root:hadoop /opt/hadoop/bin/container-executor
```
The binary is setuid/setgid, so a writable parent directory or an untrusted member of hadoop can compromise the host. The trusted, root-owned installation is a prerequisite.
Set the executor binary to the required setuid and setgid permissions.
```
$ sudo chmod 6050 /opt/hadoop/bin/container-executor
```

Assign the executor configuration file to root on each worker.

$ sudo chown root:root /opt/hadoop/etc/hadoop/container-executor.cfg

Restrict the executor configuration file to root-only reading.

$ sudo chmod 400 /opt/hadoop/etc/hadoop/container-executor.cfg

Check the native executor setup as the NodeManager account on each worker.
```
$ sudo -u yarn /opt/hadoop/bin/container-executor --checksetup
```
A successful check returns without output. A permission or configuration error needs correction before startup.
Start HDFS from the master with the secure configuration.
```
$ sudo -u hdfs /opt/hadoop/sbin/start-dfs.sh
```
Startup errors such as Failed to login, missing keys, or invalid secure DataNode settings appear in the daemon logs. The log and PID locations remain those of the existing installation.
Related: How to restart Hadoop services
Related: How to view Hadoop daemon logs
Start YARN from the master with the secure configuration.
```
$ sudo -u yarn /opt/hadoop/sbin/start-yarn.sh
```

Confirm that an unticketed user cannot access HDFS.

$ hdfs dfs -ls /
##### snipped #####
ls: DestHost:destPort master01.example.net:9000 , LocalHost:localPort master01.example.net/192.0.2.10:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

This check runs as an ordinary user with no ticket or delegation token in the session. A successful listing here means that authentication has not been enforced on the contacted cluster.

Obtain a Kerberos ticket as the ordinary application user.
```
$ kinit alice@EXAMPLE.NET
Password for alice@EXAMPLE.NET:
```
alice is an existing realm user with a matching Unix account on each worker and write access to /user/alice in HDFS. HDFS ownership and ACLs still control file access after authentication.
Related: How to set HDFS file permissions

Check that the ticket belongs to the intended user.

$ klist
Ticket cache: FILE:/tmp/krb5cc_1100
Default principal: alice@EXAMPLE.NET

Valid starting     Expires            Service principal
11/08/26 22:40:10  11/09/26 22:40:10  krbtgt/EXAMPLE.NET@EXAMPLE.NET

List the HDFS root directory as the authenticated user.

$ hdfs dfs -ls /
Found 1 items
drwxr-xr-x   - hdfs supergroup          0 2026-11-08 22:39 /user

Submit a one-container YARN application as the same authenticated user.

$ yarn jar /opt/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.5.0.jar org.apache.hadoop.yarn.applications.distributedshell.Client -jar /opt/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.5.0.jar -shell_command id -num_containers 1 -master_memory 256 -container_memory 256
##### snipped #####
2026-11-08 22:40:13,789 INFO impl.YarnClientImpl: Submitted application application_1794177604866_0001
##### snipped #####
2026-11-08 22:40:19,852 INFO distributedshell.Client: Application completed successfully

The bundled DistributedShell JAR name follows the installed Hadoop version. Application completed successfully reports completion of the submitted job. Both the authenticated HDFS listing and the successful YARN application must work before the cluster is returned to users.

Written by

Mohd Shakir Zakaria

Mohd Shakir Zakaria is a cloud architect with deep roots in software development and open-source advocacy. Certified in AWS, Red Hat, VMware, ITIL, and Linux, he specializes in designing and managing robust cloud and on-premises infrastructures.