kafka使用kerberos协议踩坑实录

编程语言 2021-11-16 19:56:17 阅读次数: 0

  • 1，安装kdc docker run -d -p 88:88/udp -p 749:749/udp beargiles/kdc-sandbox

       生成keytab
   参考：https://registry.hub.docker.com/r/beargiles/kdc-sandbox
复制代码

  • 2，配置kafka服务启用kerberos认证

    config/server.properties配置：

      listeners=SASL_PLAINTEXT://10.156.4.129:39192
  security.inter.broker.protocol=SASL_PLAINTEXT
  sasl.mechanism.inter.broker.protocol=GSSAPI
  sasl.enabled.mechanisms=GSSAPI
  
  sasl.kerberos.service.name=kafka
复制代码

    jaas_server.conf配置：

     KafkaServer {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      storeKey=true
      keyTab="D:\develop\krt\server.keytab"
      principal="[email protected]";
  };
复制代码

    配置/bin/kafka-run-class.sh：

      -Djava.security.krb5.conf=D:\develop\krt\krb5.conf
  -Djava.security.auth.login.config=D:\develop\krt\jaas_server.conf
复制代码

  • 3，kafka客户端链接配置

jaas_client.conf配置：

  KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    keyTab="D:\develop\krt\client.keytab"
    principal="[email protected]"
    useTicketCache=true;
  };
复制代码

krb5.conf配置：

[libdefaults]
    default_realm = EXAMPLE.COM
    dns_lookup_realm = false
    dns_lookup_kdc = false
    kdc_timesync = 1
    ccache_type = 4
    forwardable = true
    proxiable = true
[realms]
    EXAMPLE.COM = {
            kdc = example.com
            admin_server = example.com
    }

[domain_realm]
    example.com = EXAMPLE.COM
    .example.com = EXAMPLE.COM
复制代码

KafkaProducer配置：

public static void main(String[] args) throws Exception{
    System.setProperty("java.security.auth.login.config", "D:\develop\krt\jaas_client.conf");
    System.setProperty("java.security.krb5.conf", "D:\develop\krt\krb5.conf");

    Properties props = new Properties();
    props.put("bootstrap.servers", "10.156.4.129:39192");
    props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

    props.put("security.protocol", "SASL_PLAINTEXT");
    props.put("sasl.mechanism", "GSSAPI");
    props.put("sasl.kerberos.service.name", "kafka");

    Random random=new Random();
    KafkaProducer kafkaProducer = new KafkaProducer<>(props);
    while (true) {
      ProducerRecord record = new ProducerRecord("test", random.nextInt());
      kafkaProducer.send(record);
      Thread.sleep(1000);
    }

}
复制代码

KafkaConsumer配置：

 public static void main(String[] args) {
 
    System.setProperty("java.security.auth.login.config", "D:\develop\krt\jaas_client.conf");
    System.setProperty("java.security.krb5.conf", "D:\develop\krt\krb5.conf");

    Properties props = new Properties();
    props.put("bootstrap.servers", "10.156.4.129:39192");
    props.put("group.id", "group-1");
    props.put("enable.auto.commit", "false");
    props.put("auto.commit.interval.ms", "1000");
    props.put("auto.offset.reset", "earliest");
    props.put("session.timeout.ms", "30000");
    props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
    props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

    props.put("security.protocol", "SASL_PLAINTEXT");
    props.put("sasl.mechanism", "GSSAPI");
    props.put("sasl.kerberos.service.name", "kafka");

    KafkaConsumer kafkaConsumer = new KafkaConsumer<>(props);
    kafkaConsumer.subscribe(Arrays.asList("test"));
    while (true) {
        ConsumerRecords<String, String> records = kafkaConsumer.poll(1);
        for (ConsumerRecord<String, String> record : records)
            System.out.println("Partition: " + record.partition() + " Offset: " + record.offset() + " Value: " + record.value() + " ThreadID: " + Thread.currentThread().getId());

    }
}
复制代码

踩坑记录：

  1. 安装kdc时，由于kdc和admin_server的端口使用的是udp协议，docker启动port映射需指定udp协议。

  2. 容器内不找不到EXAMPLE.COM的hosts，而kerberos使用DNS获取hosts地址，如果环境支持DNS则可以配置域名DNS，或者使用配置 * dns_lookup_kdc = false* 取消DNS配置，直接使用本地hosts：

       Caused by: sun.security.krb5.KrbException: Cannot locate KDC
        at sun.security.krb5.Config.getKDCList(Unknown Source)
        at sun.security.krb5.KdcComm.send(Unknown Source)
        at sun.security.krb5.KdcComm.send(Unknown Source)
        at sun.security.krb5.KrbAsReqBuilder.send(Unknown Source)
        at sun.security.krb5.KrbAsReqBuilder.action(Unknown Source)
复制代码

3.客户端找不到krb5.conf配置，或者配置文件格式错误,检查改配置文件是否配置正确：

Caused by: java.io.IOException: Configuration Error:
        Line 1: expected [{], found [libdefaults]
        at sun.security.provider.ConfigFile$Spi.ioException(Unknown Source)
        at sun.security.provider.ConfigFile$Spi.match(Unknown Source)
        at sun.security.provider.ConfigFile$Spi.parseLoginEntry(Unknown Source)
        at sun.security.provider.ConfigFile$Spi.readConfig(Unknown Source)
        at sun.security.provider.ConfigFile$Spi.init(Unknown Source)
        at sun.security.provider.ConfigFile$Spi.init(Unknown Source)
        
 
 
复制代码

  1. keytab文件配置错误，检查keytab配置文件：

     Caused by: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner  authentication information from the user
     at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Unknown Source)
     at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Unknown Source)
     at com.sun.security.auth.module.Krb5LoginModule.login(Unknown Source)
     at sun.reflect.GeneratedMethodAccessor215.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
     at java.lang.reflect.Method.invoke(Unknown Source)
     at javax.security.auth.login.LoginContext.invoke(Unknown Source)
     at javax.security.auth.login.LoginContext.access$000(Unknown Source)
     at javax.security.auth.login.LoginContext$4.run(Unknown Source)
     at javax.security.auth.login.LoginContext$4.run(Unknown Source)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.login.LoginContext.invokePriv(Unknown Source)
     at javax.security.auth.login.LoginContext.login(Unknown Source)
     at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:60)
     at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:103)
     at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:64)
     at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:114)
     at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:142)
     
复制代码

  2. sasl.kerberos.service.name客户端配置应与服务端配置相同，这里为kafka，不相同会产生如下错误：

     javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]) occurred when evaluating SASL token received from the Kafka Broker. This may be caused by Java's being unable to resolve the Kafka Broker's hostname correctly. You may want to try to adding '-Dsun.net.spi.nameservice.provider.1=dns,sun' to your client's JVMFLAGS environment. Users must configure FQDN of kafka brokers when authenticating using SASL and `socketChannel.socket().getInetAddress().getHostName()` must match the hostname in `principal/hostname@realm` Kafka Client will go to AUTH_FAILED state.
     at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:293)
     at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslToken(SaslClientAuthenticator.java:210)
     at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:178)
     at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:64)
     at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:338)
     at org.apache.kafka.common.network.Selector.poll(Selector.java:291)
     at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
     at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:236)
     at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:135)
     at java.lang.Thread.run(Thread.java:745)
复制代码

猜你喜欢

转载自juejin.im/post/7031117859539550245
今日推荐
富文本编辑器 Quill 2.0 重磅发布，特性、可靠性与开发者体验大幅提升
“开源信徒”周鸿祎开源360智脑大模型
周排行
Ubuntu 14.04 下Fuel6.0安装部署
香港一小巴侧翻致1死16伤 警方：未见机件故障
pikachu--XSS盲打
阅读 深入理解JVM虚拟机笔记一
java.sql.SQLException: ORA-00932: 数据类型不一致: 应为 -, 但却获得 CLOB
oracle delete all object under an user
[LeetCode]20 Valid Parentheses 有效的括号
树形DP求树的直径 【模板】
Context propagation over HTTP in Go
【PAT】（B）1053 住房空置率 (20)*
每日归档
更多
2024-04-18(0)
2024-04-17(5)
2024-04-16(70)
2024-04-15(42)
2024-04-14(0)
2024-04-13(119)
2024-04-12(38)
2024-04-11(14)
2024-04-10(68)
2024-04-09(5)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022