kubernetes API Server permission management practice

Introduction to API Server Permission Control

API Server permission control is divided into three types: Authentication (authentication), Authorization (authorization), and AdmissionControl (admission control).

Authentication:

When a client initiates an API request to a Kubernetes non-read-only port, Kubernetes authenticates the user's legitimacy in three ways. In kubernetes, there are three ways to verify whether a user has permission to operate an API: certificate authentication, token authentication, and basic information authentication.

Certificate authentication

Set the startup parameters of apiserver: --client_ca_file=SOMEFILE, the certificate of the verification client contained in the referenced file, if it is verified, the principal object in the verification record will be used as the username of the request.

Token authentication

Set the startup parameters of apiserver: --token_auth_file=SOMEFILE. The format of the token file contains three columns: token, username, userid. When using token as the authentication method, in the http request to the apiserver, add a Header field: Authorization, and set its value to: Bearer SOMETOKEN.

Basic information authentication

Set the startup parameters of apiserver: --basic_auth_file=SOMEFILE, if the password in the file is changed, only restart the apiserver to make it take effect again. The basic format of its file contains three columns: passwork, username, userid. When using this as the authentication method, in the http request to the apiserver, add a Header field: Authorization, and set its value to: Basic BASE64ENCODEDUSER:PASSWORD.

Authorization:

In Kubernetes, authentication and authorization are separated, and authorization occurs after authentication is completed. The authentication process is to verify that the user who initiated the API request is who he claims to be. The authorization process determines whether the user has the permission to execute the API request, so the authorization is based on the result of the authentication. The Kubernetes authorization module is applied to all HTTP access requests to APIServer (except read-only ports), and access to read-only ports does not require authentication and authorization processes. When APIServer starts, the authorization_mode is set to AlwaysAllow mode by default, that is, it is always allowed.

The Kubernetes authorization module inspects each HTTP request and extracts the required attributes in the request context (eg: user, resource kind, namespace) to compare with the access control rules. Any API request needs to be validated by one or more access control rules before being processed.

Currently, Kubernetes supports and implements the following authorization modes (authorization_mode). These authorization modes can be selected by passing in parameters when the apiserver starts.

--authorization_mode=AlwaysDeny

--authorization_mode=AlwaysAllow

--authorization_mode=ABAC

AlwaysDeny mode blocks all requests (generally used for testing). AlwaysAllow mode allows all requests, and the default apiserver starts with AlwaysAllow mode). ABAC (Attribute-Based Access Control, that is, attribute-based access control) mode allows users to customize authorization access control rules.

ABAC mode:

There are 4 attributes in an API request that are used in the user authorization process:

UserName: String type, used to identify the user who initiated the request. If no authentication or authorization operation is performed, the string is empty.

ReadOnly: bool type, indicating whether the request is only a read-only operation (GET is a read-only operation).

Kind: String type, used to identify the type of Kubernetes resource object to be accessed. When accessing API endpoints such as /api/v1beta1/pods, the Kind property is not empty, but when accessing other endpoints, such as /version, /healthz, etc., the Kind property is empty.

Namespace: String type, used to identify the namespace where the Kubernetes resource object to be accessed is located.

For ABAC mode, in addition to the --authorization_mode=ABAC option, you also need to specify --authorization_policy_file=SOME_FILENAME when the apiserver starts. Each line of the authorization_policy_file file is a JSON object, which is a map data structure without nesting, representing an access control rule object. An access control rule object is a map with the following fields:

user: The user string specified by --token_auth_file.

readonly: true or false, if true, the rule only applies to GET requests.

kind: Kubernetes built-in resource object type, such as pods, events, etc.

namespace: can also be abbreviated as ns.

A simple access control rules file is shown below, each line defines a rule.

{"user":"admin"}

{"user":"alice", "ns": "projectCaribou"}

{"user":"kubelet", "readonly": true, "kind": "pods"}

{"user":"kubelet", "kind": "events"}

{"user":"bob", "kind": "pods", "readonly": true, "ns": "projectCaribou"}

Note: The default field is equivalent to the zero value of the field type (empty string, 0, false, etc.).

The rules are explained line by line as follows.

The first line shows that admin can do anything, not limited by namespace, resource type, and request type.

The second line shows that alice can do anything in the namespace "projectCaribou", regardless of resource type and request type.

The third line indicates that the kubelet has permission to read information about any pod.

The fourth line shows that the kubelet has permission to read and write any event.

The fifth line shows that Bob has permission to read information about all pods in the namespace "projectCaribou".

An authorization process is a process of comparing whether each attribute in the API request matches the corresponding field in the access control rule file. When the apiserver receives an API request, the attributes of the request have been determined. If there is an attribute that is not set, the apiserver will set it to an empty value of this type (empty string, 0, false, etc.). The matching rules are simple as shown below.

If an attribute in the API request is empty, it is specified that the attribute matches the corresponding field in the access control rule file.
If a field of the access control rule is empty, it is specified that the field matches the corresponding attribute of the API request.
If the attribute value in the API request is not null and the value of a field of the access control rule is not null, the two values are compared, if they are the same, they match, otherwise they do not match.
The attribute tuple (tuple) requested by the API will match all the rules in the access control rule file one by one. As long as there is one match, the match is successful, otherwise, the authorization fails.

Admission Control:

The admission control admission controller is essentially a piece of admission code. In the process of requesting the kubernetes api, the sequence is to go through authentication and authorization first, then perform the admission operation, and then operate the target object. This admission code is in the apiserver and must be compiled into the binary to be executed.

When a request is made to the cluster, each admission control code is executed in a certain order. If an admission control rejects the request, the result of the entire request will be returned immediately, and the user will be prompted with the corresponding error message.

In some cases, the admission logic may change the target object in order to adapt to the application's configuration. In addition, the admission logic also changes some of the relevant resources for the requested operation.

effect

In kubernetes, the prerequisite for the normal operation of some advanced features is that some admission modules are in the enable state. To sum up, for kubernetes apiserver, if the admission control module is not properly configured, it cannot be called a complete server, and some functions will not take effect normally.

open method

There is a parameter in kubernetes apiserver: admission_control, whose value is an ordered list of admission modules connected by commas. After setting, a certain sequence of admission module calls can be executed before the object is operated.

Module function

AlwaysAdmit: allow all requests

AlwaysDeny: All requests are prohibited, mostly used in test environments.

DenyExecOnPrivileged: It will intercept all requests to execute commands on the privileged container. If your cluster supports privileged containers and you want to restrict users from executing commands on these privileged containers, it is strongly recommended to use it.

ServiceAccount: This plug-in automates serviceAccounts and is highly recommended if you want to use ServiceAccount objects.

The description of serviceAccount is as follows: A serviceAccount adds the corresponding authentication information for the process running in the pod. When this plugin is enabled in the admission module (enabled by default), then when a pod is created or modified, it will do the following:

If the pod does not have a serviceAccount attribute, set the pod's serviceAccount attribute to "default";
Make sure pods always exist using de serviceAccount;
If LimitSecretReferences is set to true, when the pod references the Secret object but not the ServiceAccount object, discard the pod;
If the pod does not contain any ImagePullSecrets, the ImagePullSecrets of the serviceAccount are added to the pod;
If MountServiceAccountToken is true, add a VolumeMount to the container in the pod.

SecurityContextDeny: This plugin will disable all options defined in pods that use SecurityContext. SecurityContext defines OS-level security settings (uid, gid, capabilities, SELinux, etc.) in the container.

ResourceQuota: It observes all requests to ensure that the container listed at the ResourceQuota object in the namespace does not have any exceptions. If the ResourceQuota object is used in kubernetes, this plugin must be used to constrain the container. It is recommended that this plugin is the last one in the admission control parameter list.

LimitRanger: He will observe all requests to ensure that the defined constraints are not violated. These conditions are defined in the LimitRange object in the namespace. If you use LimitRange objects in kubernetes, you must use this plugin.

NamespaceExists: It observes all requests, if a request tries to create a namespace that does not exist, the request is rejected.

Recommended plugin order

--admission_control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount, ResourceQuota

verification process

1. Some default attributes related to authorization authentication are set in the initialization parameters of API Server:

Secure listening port: SecurePort: 8443 read/write permission, support x509 security certificate and x509 private key authentication
Insecure listening port: InsecurePort: 8080 No user authentication and authorization, read/write permission
Authorization Mode: AuthorizationMode: "AlwaysAllow",
Admission Control Plugin: AdmissionControl: "AlwaysAdmit"

2. When the API Server starts, you can set parameters related to authorization authentication:

--insecure_port custom insecure listening port

--secure_port Custom secure listening port

--tls_cert_file set security certificate file

--tls_private_key_file set the private key file

--cert_dir This property is ignored when the security certificate file and private key file are set. When the security certificate file and private key file are not set, the apiserver will automatically

The port-bound public IP address generates a self-registered certificate file and key respectively and stores them /var/run/kubernetesunder

--service_account_key_file service account file, including x509 public and private keys

--client_ca_file client certificate file

--token_auth_file token statement

--basic_auth_file basic information authentication file

--authorization_mode authorization mode

--ahtuorization_policy_file authorization file

--admission_control admission control module list

--admission_control_config_file admission control configuration file

3. Parse the input parameters and extract the authentication information:

Public and private key file settings: Check whether ServerAccountKeyFile has been specified. If it is not specified and TLSPrivateKeyFile is specified, then judge whether TLSPrivateKeyFile contains a valid RSA key. If so, use TLSPrivateKeyFile as ServerAccountKeyFile.
Identity authentication information extraction: Take out the username, userID, password (or token) from the CSV file set by the parameter and encapsulate it into a map structure, the key is username, and the value is a struct with three attributes. Get user information from basicAuthFile, clientCAFile, tokenFile, serviceAccountKeyFile (serviceAccountLookup), and get a map array of authentication information.
Authorization information extraction: Read the set authorization file, parse the string, and return an array of authorization information. (including username, group, resource, read only, namespace) Make the "cluster" Kinds be one API group (minions, bindings, events, endpoints). The "user" Kinds are another (pods, services, replicationControllers, operations).
Admission control plug-in: get all plug-in names, return the admission control interface (execute all plug-ins)

4. Use the identity authentication information, authorization information, and admission control plug-in as the configuration of the Master, New Master.

5. Request authentication:

Call the NewRequestAttributeGetter method of the apiserver, extract the authorization information from the request, and call the WithAuthorizationCheck method (authorization verification).
Call the handler's NewRequestAuthenticator method, extract the authencate information from the Request, and call the AuthenticateRequest method (there are different authentication methods for client certificates, token, and basic auth).

Replenish

Authentication:

Token authentication, when requesting, add the Authorization:bearer token string to the request header. In the CSV file, the three columns are token, username, and userid. Authentication succeeds when there are rows in the CSV that match the requested Authorization.

For basic auth authentication, when requesting, add the user:password string of Authorization:basic base64 encoding to the request header. In the CSV file, the three columns are password, username, and userid. Authentication succeeds when there are rows in the CSV file that match the requested Ahtuorization.

Certificate verification:

When the API Server starts, specify the server digital certificate and key (if not specified, it will be automatically generated when the server starts), and specify the client ca file. When the server starts, it will parse the ca file, traverse the cert in it, and join the certpool. Specify the authentication mode in the TLSConfig of the Server: Currently, RequestClientCert is used (no mandatory authentication, no connection is refused without authentication, other authentication is allowed), and there are other authentication modes requireAndVerifyClientCert (mandatory verification). Use ListenAndServeTLS (with the server digital certificate and key as parameters) to listen on a secure port.

API Server permission control operation (temporarily not added to namespace ) test:

Start the server: specify the token verification file, authorization method, and authorization file

./_output/local/bin/linux/amd64/canary-apiserver --logtostderr=true --log-dir=/tmp --v=4 --etcd_servers=http://127.0.0.1:4001 --insecure_bind_address=127.0.0.1 --insecure_port=8088 --secure_port=8442 --kubelet_port=10250 --service-cluster-ip-range=10.1.1.0/24 --allow_privileged=true --runtime-config="api/v1beta3=false" --redis-addr=localhost:6379 --profiling=true --token_auth_file=token.csv --authorization_mode=ABAC --authorization_policy_file=abac.csv

Token file content:

abcdef,hankai,123456

abcdefg,hk,123457

abcd,admin,1234

abc,hhh,111

Authorization file content:

{“user”:”admin”}

{“user”:”hankai”,”readonly”:true}

{“user”:”hhh”,”resource”:”apps”}

{“user”:”hk”,”readonly”:true,”resource”:”namespaces”}

Authentication: admin (with permission to read and write all resources)

curl -X GET -H "Content-Type: application/json" -H "Authorization: bearer abcd" -khttps://10.57.104.59:8442/api/v1/apps

curl -X GET -H "Content-Type: application/json" -H "Authorization: bearer abcd" -khttps://10.57.104.59:8442/api/v1/namespaces

curl -X POST -H "Content-Type: application/json" -H "Authorization: bearer abcd" -d@'n1.json' -khttps://10.57.104.59:8442/api/v1/namespaces

curl -X POST -H "Content-Type: application/json" -H "Authorization: bearer abcd" -d@'app_demo1.json' -khttps://10.57.104.59:8442/api/v1/apps

Authenticate hankai (only read permission GET)

curl -X POST -H "Content-Type: application/json" -H "Authorization: bearer abcdef" -d@'app_demo1.json' -khttps://10.57.104.59:8442/api/v1/apps forbidden

curl -X POST -H "Content-Type: application/json" -H "Authorization: bearer abcdef" -d@'n1.json' -khttps://10.57.104.59:8442/api/v1/namespaces forbidden

curl -X GET -H "Content-Type: application/json" -H "Authorization: bearer abcdef" -khttps://10.57.104.59:8442/api/v1/namespaces

curl -X GET -H "Content-Type: application/json" -H "Authorization: bearer abcdef" -khttps://10.57.104.59:8442/api/v1/apps

Authenticate hk (only GET rights to namespaces)

curl -X GET -H "Content-Type: application/json" -H "Authorization: bearer abcdefg" -khttps://10.57.104.59:8442/api/v1/apps

forbidden

curl -X GET -H "Content-Type: application/json" -H "Authorization: bearer abcdefg" -khttps://10.57.104.59:8442/api/v1/namespaces

curl -X POST -H "Content-Type: application/json" -H "Authorization: bearer abcdefg" -d@'n1.json' -khttps://10.57.104.59:8442/api/v1/namespaces forbidden

curl -X POST -H "Content-Type: application/json" -H "Authorization: bearer abcdefg" -d@'app_demo1.json' -khttps://10.57.104.59:8442/api/v1/apps forbidden

Verify hhh (has read and write rights to apps)

curl -X POST -H "Content-Type: application/json" -H "Authorization: bearer abc" -d@'app_demo1.json' -khttps://10.57.104.59:8442/api/v1/apps

curl -X GET -H "Content-Type: application/json" -H "Authorization: bearer abc" -khttps://10.57.104.59:8442/api/v1/apps

curl -X GET -H "Content-Type: application/json" -H "Authorization: bearer abc" -khttps://10.57.104.59:8442/api/v1/namespaces

forbidden

curl -X POST -H "Content-Type: application/json" -H "Authorization: bearer abc" -d@'n1.json' -khttps://10.57.104.59:8442/api/v1/namespaces

forbidden

Note: In the follow-up, you only need to specify the namespace in each column of the abac.csv file to realize the user's operation permission for the specified namespace.

New: TSL client certificate authentication

Use self-generated certificate test: use openssl to generate server.crt, server.key, ca.key, ca.crt . When the server starts, pass in --tls_cert_file=server.crt --tls_private_key_file=server.key --client_ca_file=ca.crt

./_output/local/bin/linux/amd64/canary-apiserver --logtostderr=true --log-dir=/tmp --v=4 --etcd_servers=http://127.0.0.1:4001 --insecure_bind_address=7.0.0.1 --insecure_port=8088 --secure_port=8442 --kubelet_port=10250 --service-cluster-ip-range=10.1.1.0/24 --allow_privileged=true --runtime-config="api/v1beta3=false" --redis-addr=localhost:6379 --profiling=true --tls_cert_file=server.crt --tls_private_key_file=server.key --client_ca_file=ca.crt --token_auth_file=token.csv --authorization_mode=ABAC --authorization_policy_file=abac.csv

When requesting, specify the client certificate through -cacert (you can specify the path of the client certificate by modifying the configuration file of opnessl, or import the client certificate in the browser) curl -X GET --cacert ca.crt -H "Content-Type : application/json" -H "Authorization: bearer abcd" -k https://10.57.104.59:8442/api/v1/apps to achieve authentication.

https://my.oschina.net/wayhk/blog/491973