Concepts

Detailed explanations of Kubernetes system concepts and abstractions.

Edit This Page

Federation

This page explains why and how to manage multiple Kubernetes clusters using federation.

Why federation

Federation makes it easy to manage multiple clusters. It does so by providing 2 major building blocks:

Some other use cases that federation enables are:

Federation is not helpful unless you have multiple clusters. Some of the reasons why you might want multiple clusters are:

Caveats

While there are a lot of attractive use cases for federation, there are also some caveats:

Hybrid cloud capabilities

Federations of Kubernetes Clusters can include clusters running in different cloud providers (e.g. Google Cloud, AWS), and on-premises (e.g. on OpenStack). Simply create all of the clusters that you require, in the appropriate cloud providers and/or locations, and register each cluster’s API endpoint and credentials with your Federation API Server (See the federation admin guide for details).

Thereafter, your API resources can span different clusters and cloud providers.

Setting up federation

To be able to federate multiple clusters, you first need to set up a federation control plane. Follow the setup guide to set up the federation control plane.

API resources

Once you have the control plane set up, you can start creating federation API resources. The following guides explain some of the resources in detail:

The API reference docs list all the resources supported by federation apiserver.

Cascading deletion

Kubernetes version 1.6 includes support for cascading deletion of federated resources. With cascading deletion, when you delete a resource from the federation control plane, you also delete the corresponding resources in all underlying clusters.

Cascading deletion is not enabled by default when using the REST API. To enable it, set the option DeleteOptions.orphanDependents=false when you delete a resource from the federation control plane using the REST API. Using kubectl delete enables cascading deletion by default. You can disable it by running kubectl delete --cascade=false

Note: Kubernetes version 1.5 included cascading deletion support for a subset of federation resources.

Scope of a single cluster

On IaaS providers such as Google Compute Engine or Amazon Web Services, a VM exists in a zone or availability zone. We suggest that all the VMs in a Kubernetes cluster should be in the same availability zone, because:

It is okay to have multiple clusters per availability zone, though on balance we think fewer is better. Reasons to prefer fewer clusters are:

Reasons to have multiple clusters include:

Selecting the right number of clusters

The selection of the number of Kubernetes clusters may be a relatively static choice, only revisited occasionally. By contrast, the number of nodes in a cluster and the number of pods in a service may change frequently according to load and growth.

To pick the number of clusters, first, decide which regions you need to be in to have adequate latency to all your end users, for services that will run on Kubernetes (if you use a Content Distribution Network, the latency requirements for the CDN-hosted content need not be considered). Legal issues might influence this as well. For example, a company with a global customer base might decide to have clusters in US, EU, AP, and SA regions. Call the number of regions to be in R.

Second, decide how many clusters should be able to be unavailable at the same time, while still being available. Call the number that can be unavailable U. If you are not sure, then 1 is a fine choice.

If it is allowable for load-balancing to direct traffic to any region in the event of a cluster failure, then you need at least the larger of R or U + 1 clusters. If it is not (e.g. you want to ensure low latency for all users in the event of a cluster failure), then you need to have R * (U + 1) clusters (U + 1 in each of R regions). In any case, try to put each cluster in a different zone.

Finally, if any of your clusters would need more than the maximum recommended number of nodes for a Kubernetes cluster, then you may need even more clusters. Kubernetes v1.3 supports clusters up to 1000 nodes in size. Kubernetes v1.8 supports clusters up to 5000 nodes. See Building Large Clusters for more guidance.

What’s next

Analytics

Create an Issue Edit this Page