The Problem:

My project triggered an error when pushing messages to a Kafka cluster with Kerberos authentication enabled.

javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]) occurred when evaluating SASL token received from the Kafka Broker. This may be caused by Java's being unable to resolve the Kafka Broker's hostname correctly. You may want to try to adding '-Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS environment. Users must configure FQDN of kafka brokers when authenticating using SASL and socketChannel.socket().getInetAddress().getHostName() must match the hostname in principal/hostname@realm Kafka Client will go to AUTH_FAILED state.

at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:293)

at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslToken(SaslClientAuthenticator.java:210)

at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:178)

at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:64)

at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:338)

at org.apache.kafka.common.network.Selector.poll(Selector.java:291)

at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)

at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:236)

at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:135)

at java.lang.Thread.run(Thread.java:745)

Caused by: javax.security.sasl.SaslException: GSS initiate failed

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)

at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$2.run(SaslClientAuthenticator.java:275)

at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$2.run(SaslClientAuthenticator.java:273)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:273)

... 9 common frames omitted

Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)

at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770)

at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)

at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)

... 14 common frames omitted

Caused by: sun.security.krb5.KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER

at sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:73)

at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:259)

at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:270)

at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:302)

at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:120)

at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)

at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)

... 17 common frames omitted

Caused by: sun.security.krb5.Asn1Exception: Identifier doesn't match expected value (906)

at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)

at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)

at sun.security.krb5.internal.TGSRep.(TGSRep.java:60)

at sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:55)

Solution:

To resolve the issue, I updated the Kafka client configuration by adding the following:

kerberos.domain.name = hadoop.xxxx.com

The value of the kerberos.domain.name property varies depending on your environment.

I identified the correct value by checking the KDC server logs from a working client.