Try Kubernetes

Reference Documentation

Design docs, concept definitions, and references for APIs and CLIs.

Edit This Page

kubeadm init

Run this command in order to set up the Kubernetes master.

Synopsis

Run this command in order to set up the Kubernetes master.

kubeadm init

Options

      --apiserver-advertise-address string      The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
      --apiserver-bind-port int32               Port for the API Server to bind to. (default 6443)
      --apiserver-cert-extra-sans stringSlice   Optional extra Subject Alternative Names (SANs) to use for the API Server serving certificate. Can be both IP addresses and DNS names.
      --cert-dir string                         The path where to save and store the certificates. (default "/etc/kubernetes/pki")
      --config string                           Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
      --cri-socket string                       Specify the CRI socket to connect to. (default "/var/run/dockershim.sock")
      --dry-run                                 Don't apply any changes; just output what would be done.
      --feature-gates string                    A set of key=value pairs that describe feature gates for various features. Options are:
CoreDNS=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (ALPHA - default=false)
HighAvailability=true|false (ALPHA - default=false)
SelfHosting=true|false (BETA - default=false)
StoreCertsInSecrets=true|false (ALPHA - default=false)
SupportIPVSProxyMode=true|false (ALPHA - default=false)
      --ignore-checks-errors stringSlice        A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.
      --kubernetes-version string               Choose a specific Kubernetes version for the control plane. (default "stable-1.8")
      --node-name string                        Specify the node name.
      --pod-network-cidr string                 Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
      --service-cidr string                     Use alternative range of IP address for service VIPs. (default "10.96.0.0/12")
      --service-dns-domain string               Use alternative domain for services, e.g. "myorg.internal". (default "cluster.local")
      --skip-token-print                        Skip printing of the default bootstrap token generated by 'kubeadm init'.
      --token string                            The token to use for establishing bidirectional trust between nodes and masters.
      --token-ttl duration                      The duration before the bootstrap token is automatically deleted. If set to '0', the token will never expire. (default 24h0m0s)

Init workflow

kubeadm init bootstraps a Kubernetes master node by executing the following steps:

  1. Runs a series of pre-flight checks to validate the system state before making changes. Some checks only trigger warnings, others are considered errors and will exit kubeadm until the problem is corrected or the user specifies --skip-preflight-checks.

  2. Generates a self-signed CA (or using an existing one if provided) to set up identities for each component in the cluster. If the user has provided their own CA cert and/or key by dropping it in the cert directory configured via --cert-dir (/etc/kubernetes/pki by default) this step is skipped as described in the Using custom certificates document.

  3. Writes kubeconfig files in /etc/kubernetes/ for the kubelet, the controller-manager and the scheduler to use to connect to the API server, each with its own identity, as well as an additional kubeconfig file for administration named admin.conf.

  4. If kubeadm is invoked with --feature-gates=DynamicKubeletConfig enabled, it writes the kubelet init configuration into the /var/lib/kubelet/config/init/kubelet file. See Set Kubelet parameters via a config file and Reconfigure a Node’s Kubelet in a Live Cluster for more information about Dynamic Kubelet Configuration. This functionality is now by default disabled as it is behind a feature gate, but is expected to be a default in future versions.

  5. Generates static Pod manifests for the API server, controller manager and scheduler. In case an external etcd is not provided, an additional static Pod manifest are generated for etcd.

    Static Pod manifests are written to /etc/kubernetes/manifests; the kubelet watches this directory for Pods to create on startup.

    Once control plane Pods are up and running, the kubeadm init sequence can continue.

  6. If kubeadm is invoked with --feature-gates=DynamicKubeletConfig enabled, it completes the kubelet dynamic configuration by creating a ConfigMap and some RBAC rules that enable kubelets to access to it, and updates the node by pointing Node.spec.configSource to the newly-created ConfigMap. This functionality is now by default disabled as it is behind a feature gate, but is expected to be a default in future versions.

  7. Apply labels and taints to the master node so that no additional workloads will run there.

  8. Generates the token that additional nodes can use to register themselves with the master in the future. Optionally, the user can provide a token via --token, as described in the kubeadm token docs.

  9. Makes all the necessary configurations for allowing node joining with the Bootstrap Tokens and TLS Bootstrap mechanism:

    • Write a ConfigMap for making available all the information required for joining, and set up related RBAC access rules.

    • Let Bootstrap Tokens access the CSR signing API.

    • Configure auto-approval for new CSR requests.

    See kubeadm join for additional info.

  10. Installs the internal DNS server (kube-dns) and the kube-proxy addon components via the API server. If kubeadm is invoked with –feature-gates=CoreDNS=true, then CoreDNS will be installed as the default internal DNS server instead of kube-dns.
    Please note that although the DNS server is deployed, it will not be scheduled until CNI is installed.

  11. If kubeadm init is invoked with the alpha self-hosting feature enabled, (--feature-gates=SelfHosting=true), the static Pod based control plane is transformed into a self-hosted control plane.

Using kubeadm init with a configuration file

Caution: The config file is still considered alpha and may change in future versions.

It’s possible to configure kubeadm init with a configuration file instead of command line flags, and some more advanced features may only be available as configuration file options. This file is passed in the --config option.

apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
api:
  advertiseAddress: <address|string>
  bindPort: <int>
etcd:
  endpoints:
  - <endpoint1|string>
  - <endpoint2|string>
  caFile: <path|string>
  certFile: <path|string>
  keyFile: <path|string>
  dataDir: <path|string>
  extraArgs:
    <argument>: <value|string>
    <argument>: <value|string>
  image: <string>
kubeProxy:
  config:
    mode: <value|string>
networking:
  dnsDomain: <string>
  serviceSubnet: <cidr>
  podSubnet: <cidr>
kubernetesVersion: <string>
cloudProvider: <string>
nodeName: <string>
authorizationModes:
- <authorizationMode1|string>
- <authorizationMode2|string>
token: <string>
tokenTTL: <time duration>
selfHosted: <bool>
apiServerExtraArgs:
  <argument>: <value|string>
  <argument>: <value|string>
controllerManagerExtraArgs:
  <argument>: <value|string>
  <argument>: <value|string>
schedulerExtraArgs:
  <argument>: <value|string>
  <argument>: <value|string>
apiServerExtraVolumes:
- name: <value|string>
  hostPath: <value|string>
  mountPath: <value|string>
controllerManagerExtraVolumes:
- name: <value|string>
  hostPath: <value|string>
  mountPath: <value|string>
schedulerExtraVolumes:
- name: <value|string>
  hostPath: <value|string>
  mountPath: <value|string>
apiServerCertSANs:
- <name1|string>
- <name2|string>
certificatesDir: <string>
imageRepository: <string>
unifiedControlPlaneImage: <string>
featureGates:
  <feature>: <bool>
  <feature>: <bool>

Passing custom arguments to control plane components

If you would like to override or extend the behaviour of a control plane component, you can provide extra arguments to kubeadm. When the component is deployed, these additional arguments are added to the Pod command itself.

For example, to add additional feature-gate arguments to the API server, your configuration file will need to look like this:

apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
apiServerExtraArgs:
  feature-gates: APIResponseCompression=true

To customize the scheduler or controller-manager, use schedulerExtraArgs and controllerManagerExtraArgs respectively.

More information on custom arguments can be found here:

Using custom images

By default, kubeadm pulls images from k8s.gcr.io, unless the requested Kubernetes version is a CI version. In this case, gcr.io/kubernetes-ci-images is used.

You can override this behavior by using kubeadm with a configuration file. Allowed customization are:

Using custom certificates

By default, kubeadm generates all the certificates needed for a cluster to run. You can override this behavior by providing your own certificates.

To do so, you must place them in whatever directory is specified by the --cert-dir flag or CertificatesDir configuration file key. By default this is /etc/kubernetes/pki.

If a given certificate and private key pair exists, kubeadm skips the generation step and existing files are used for the prescribed use case. This means you can, for example, copy an existing CA into /etc/kubernetes/pki/ca.crt and /etc/kubernetes/pki/ca.key, and kubeadm will use this CA for signing the rest of the certs.

External CA mode

It is also possible to provide just the ca.crt file and not the ca.key file (this is only available for the root CA file, not other cert pairs). If all other certificates and kubeconfig files are in place, kubeadm recognizes this condition and activates the “External CA” mode. kubeadm will proceed without the CA key on disk.

Instead, run the controller-manager standalone with --controllers=csrsigner and point to the CA certificate and key.

Managing the kubeadm drop-in file for the kubelet

The kubeadm package ships with configuration for how the kubelet should be run. Note that the kubeadm CLI command never touches this drop-in file. This drop-in file belongs to the kubeadm deb/rpm package.

This is what it looks like:

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS

Here’s a breakdown of what/why:

Use kubeadm with other CRI runtimes

Since v1.6.0, Kubernetes has enabled the use of CRI, Container Runtime Interface, by default. The container runtime used by default is Docker, which is enabled through the built-in dockershim CRI implementation inside of the kubelet.

Other CRI-based runtimes include:

After you have successfully installed kubeadm and kubelet, execute these two additional steps:

  1. Install the runtime shim on every node, following the installation document in the runtime shim project listing above.

  2. Configure kubelet to use the remote CRI runtime. Please remember to change RUNTIME_ENDPOINT to your own value like /var/run/{your_runtime}.sock:

cat > /etc/systemd/system/kubelet.service.d/20-cri.conf <<EOF
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=$RUNTIME_ENDPOINT"
EOF
systemctl daemon-reload

Now kubelet is ready to use the specified CRI runtime, and you can continue with the kubeadm init and kubeadm join workflow to deploy Kubernetes cluster.

You may also want to set --cri-socket to kubeadm init and kubeadm reset when using an external CRI implementation.

Using internal IPs in your cluster

In order to set up a cluster where the master and worker nodes communicate with internal IP addresses (instead of public ones), execute following steps.

  1. When running init, you must make sure you specify an internal IP for the API server’s bind address, like so:

    kubeadm init --apiserver-advertise-address=<private-master-ip>

  2. When a worker node has been provisioned, add a flag to /etc/systemd/system/kubelet.service.d/10-kubeadm.conf that specifies the private IP of the worker node:

    --node-ip=<private-node-ip>

  3. Finally, when you run kubeadm join, make sure you provide the private IP of the API server addressed as defined in step 1.

Self-hosting the Kubernetes control plane

As of 1.8, you can experimentally create a self-hosted Kubernetes control plane. This means that key components such as the API server, controller manager, and scheduler run as DaemonSet pods configured via the Kubernetes API instead of static pods configured in the kubelet via static files.

Caution: Self-hosting is alpha, but is expected to become the default in a future version. To create a self-hosted cluster, pass the --feature-gates=SelfHosting=true flag to kubeadm init.

Warning: see self-hosted caveats and limitations.

Caveats

Self-hosting in 1.8 has some important limitations. In particular, a self-hosted cluster cannot recover from a reboot of the master node without manual intervention. This and other limitations are expected to be resolved before self-hosting graduates from alpha.

By default, self-hosted control plane Pods rely on credentials loaded from hostPath volumes. Except for initial creation, these credentials are not managed by kubeadm. You can use --feature-gates=StoreCertsInSecrets=true to enable an experimental mode where control plane credentials are loaded from Secrets instead. This requires very careful control over the authentication and authorization configuration for your cluster, and may not be appropriate for your environment.

In kubeadm 1.8, the self-hosted portion of the control plane does not include etcd, which still runs as a static Pod.

Process

The self-hosting bootstrap process is documented in the kubeadm design document.

In summary, kubeadm init --feature-gates=SelfHosting=true works as follows:

  1. Waits for this bootstrap static control plane to be running and healthy. This is identical to the kubeadm init process without self-hosting.

  2. Uses the static control plane Pod manifests to construct a set of DaemonSet manifests that will run the self-hosted control plane. It also modifies these manifests where necessary, for example adding new volumes for secrets.

  3. Creates DaemonSets in the kube-system namespace and waits for the resulting Pods to be running.

  4. Once self-hosted Pods are operational, their associated static Pods are deleted and kubeadm moves on to install the next component. This triggers kubelet to stop those static Pods.

  5. When the original static control plane stops, the new self-hosted control plane is able to bind to listening ports and become active.

This process (steps 3-6) can also be triggered with kubeadm phase selfhosting convert-from-staticpods.

Running kubeadm without an internet connection

For running kubeadm without an internet connection you have to pre-pull the required master images for the version of choice:

Image Name v1.8 release branch version v1.9 release branch version
k8s.gcr.io/kube-apiserver-${ARCH} v1.8.x v1.9.x
k8s.gcr.io/kube-controller-manager-${ARCH} v1.8.x v1.9.x
k8s.gcr.io/kube-scheduler-${ARCH} v1.8.x v1.9.x
k8s.gcr.io/kube-proxy-${ARCH} v1.8.x v1.9.x
k8s.gcr.io/etcd-${ARCH} 3.0.17 3.1.10
k8s.gcr.io/pause-${ARCH} 3.0 3.0
k8s.gcr.io/k8s-dns-sidecar-${ARCH} 1.14.5 1.14.7
k8s.gcr.io/k8s-dns-kube-dns-${ARCH} 1.14.5 1.14.7
k8s.gcr.io/k8s-dns-dnsmasq-nanny-${ARCH} 1.14.5 1.14.7

Here v1.8.x means the “latest patch release of the v1.8 branch”.

${ARCH} can be one of: amd64, arm, arm64, ppc64le or s390x.

If using --feature-gates=CoreDNS=true image coredns/coredns:1.0.2 is required (instead of the three k8s-dns-* images).

Automating kubeadm

Rather than copying the token you obtained from kubeadm init to each node, as in the basic kubeadm tutorial, you can parallelize the token distribution for easier automation. To implement this automation, you must know the IP address that the master will have after it is started.

  1. Generate a token. This token must have the form <6 character string>.<16 character string>. More formally, it must match the regex: [a-z0-9]{6}\.[a-z0-9]{16}.

    kubeadm can generate a token for you:

    kubeadm token generate

  2. Start both the master node and the worker nodes concurrently with this token. As they come up they should find each other and form the cluster. The same --token argument can be used on both kubeadm init and kubeadm join.

Once the cluster is up, you can grab the admin credentials from the master node at /etc/kubernetes/admin.conf and use that to talk to the cluster.

Note that this style of bootstrap has some relaxed security guarantees because it does not allow the root CA hash to be validated with --discovery-token-ca-cert-hash (since it’s not generated when the nodes are provisioned). For details, see the kubeadm join.

What’s next

Analytics

Create an Issue Edit this Page