1 / 28

Matthias Sohn Adel Zaalouk SAP

Gardener is a solution that provides and establishes Kubernetes clusters as a service, homogeneously on hyper-scale providers and for private clouds. It aims to achieve minimal TCO and full day-2 operations support, including automation, backup & recovery, resilience, self-healing, and more.

barbarapeterson
Download Presentation

Matthias Sohn Adel Zaalouk SAP

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Matthias Sohn Adel Zaalouk SAP

  2. From Containers to Kubernetes Container Scheduler Benefits Isolation Immutable infrastructure Portability Faster deployments Versioning Ease of sharing Container • Kubernetes • Orchestration ofclusterofcontainersacross multiple hosts • Automaticplacements, networking, deployments, scaling, roll-out/-back, A/B testing Container Runtime Challenges Networking Deployments Service Discovery Auto Scaling Persisting Data Logging, Monitoring Access Control Host OS • Declarative – not procedural • Declaretargetstate, reconciletodesiredstate • Self-healing VM • WorkloadPortability • Abstract fromcloudproviderspecifics • Multiple containerruntimes Docker

  3. What does Kubernetes not cover ? • Install and manage many clusters • Across Multi-Cloud • Public Cloud Providers • Private Cloud • Zero Ops • Minimal TCO • Manage Nodes • Manage Control Planes • Day 2 Operations Gardener

  4. WHAT do we want to achieve with the Gardener? Provide and establish solution for Kubernetes Clusters as a Service Central Provisioning Engage with Open Source community,foster adoption, become CNCF project Large scale organisations need hundreds or thousands of clusters

  5. WHAT do we want to achieve with the Gardener? Homogenously on Hyper-Scale Providers and for the Private Cloud Full Control of Kubernetes, Homogeneous Across All Installations AWS, Azure, GCP, Alibaba and Others Private DCs for Data Privacy: OpenStack and eventually Bare Metal

  6. WHAT do we want to achieve with the Gardener? with Minimal TCO and Full Day-2 Operations Support Full Automation, Backup & Recovery, High Resilience and Robustness, Self-Healing, Auto-Scaling, … Rollout Bug Fixes, Security Patches, Updates of Kubernetes, OS, Infrastructure,Certificate Management, …

  7. Gardener Mission Provide and establish solution for Kubernetes Clusters as a Service Homogenously on Hyper-Scale Providers and for the Private Cloud with Minimal TCO and Full Day-2 Operations Support

  8. Primary Gardener Architecture Principle Following the definition of Kubernetes… Kubernetes is a system for automatingdeployment, scaling, and management of containerized software …we do the following: We use Kubernetes to deploy, host and operate Kubernetes Control planes are “seeded” into already existing clusters

  9. Common Kubernetes Cluster Setup Master Master Master Worker Worker Worker Master Master Master Worker Worker Worker Master Worker Master Worker Master HA HA HA Worker Master Master Worker Worker Master Worker Worker Worker Master Master Worker Master Worker HA Worker Worker Master Master HA Worker Worker The host the control plane, often in HA and on separated hardware (usually underutilized or, worse, overutilized) green machines The host the actual workload and are managed by Kubernetes (usually pretty well utilized) blue machines

  10. Gardener Kubernetes Cluster Setup Shoot Clusters Gardener Cluster Seed Cluster Master Master Worker Worker Worker Worker Master Worker Worker Worker Master Worker Master Worker Master Worker Worker HA HA Worker manages Worker Worker Worker Worker Zooming into the Seed Cluster reveals… API Server ETCD Worker Worker Controller Mgr Scheduler Multiple Shoot Cluster Control Planes Worker ETCD API Server Controller Mgr Scheduler Gardener Machine Controller Manager Machine ProvisioningSelf-Healing Auto-UpdateAuto-Scaling … ETCD API Server Inside a Seed Cluster Worker

  11. Primary Gardener Design Principle “Let Kubernetes drive the design of the Gardener.” Do not reinvent the wheel …

  12. Lingua Franca – Gardener Cluster Resource apiVersion: garden.sapcloud.io/v1 kind: Shoot metadata: name: my-cluster namespace: garden-project spec: dns: provider: aws-route53 domain: cluster.ondemand.com cloud: aws: networks: vpc: cidr: 10.250.0.0/16 workers: - name: cpu-worker machineType: m4.xlarge autoScalerMin: 5 autoScalerMax: 20 kubernetes: version: 1.11.2 kubeAPIServer: featureGates: ... runtimeConfig: ... admissionPlugins: ... kubeControllerManager: featureGates: ... kubeScheduler: featureGates: ... kubelet: featureGates: ... maintenance: timeWindow: begin: 220000+0000 end: 230000+0000 autoUpdate: kubernetesVersion: true status: ... Native Kubernetes Resource Gardener or Self-Managed DNS Define Your Infrastructure Needs Specify Worker Pools Set Kubernetes Version Tweak Kubernetes Control Plane Avoid Vendor Lock-In Define When and What to Update Gardener Reported Status

  13. Garden Cluster Seed Cluster Shoot Cluster R R R R R R R R R R R R R R R R R R R R R R R R R R R R R Administrator End-User R R R R R R R R R Kubify gardenctl Gardener Dashboard kubectl Kubernetes Dashboard kubectl Kubernetes Dashboard HTTPS Garden Cluster Seed Cluster Shoot Cluster Garden Cluster API LB Ingress LB Seed Cluster API LB Shoot Cluster API LB Shoot Cluster VPN LB Storage [K8s] DS, RS, SS, J, ... [CRD] Shoot, Seed, ... Storage [K8s] DS, RS, SS, J, ...[CRD] MachineDeployment Garden Cluster Control Plane Seed Cluster Control Plane R Worker Worker W Worker Kubelet + Container Runtime Kubelet + Container Runtime Kubelet + Container Runtime etcd Events SS etcd Main Backup SS Events PV VPN D Gardener Dashboard D Main PV Calico DS Gardener API Server D API Server VPN D Kube Proxy DS Gardener Controller Manager D ... Core DNS D Scheduler D New Shoot Clusters canbecreated via the Gardener dashboardorbyuploading a newShootresourcetothe Garden Cluster. The Gardener picksitupandstarts a Terraform jobtocreatethenecessaryIaaScomponents. ThenitdeploystheShoot Cluster Control Plane intotheSeed Cluster andrequiredadd-onsintotheShoot Cluster. Update ordeleteoperationsarehandledbythe Gardener fullyautomaticallyaswell. Optional Addons Controller Manager D Addon Manager D Machine Controller D Actual Workload ... Terraformer J Shoot Cluster Control Plane R Monitoring VPN ... Logging ... ... SCP ... IaaS IaaS

  14. Following the Design Principle Gardener uses… Workload Additional Tooling Network policies Cert Manager Add-On Manager Cluster Autoscaler Prometheus Calico Helm Cert Broker EFK Stack K8S buildingblocks Stateful Sets Reconciliation Jobs Deployments Secrets Admission Control PVs PVCs Controllers CRDs Replicasets Load-Balancer ConfigMaps RBAC Driver API Server Extension Pods Kubernetes as deployment underlay

  15. Where are all these clusters coming from? • Garden clusters are installed on a bootstrap cluster • in GKE, EKS, AKS • set up using Gardener’s Kubify • DR setup with the Gardener Ring (planned) Seed clusters are created as shoot clusters by the Gardener Shoot clusters are created by their seed cluster which is managed by the Gardener

  16. Gardener Demo

  17. GardenerCommunity Installer Setting up a Gardener landscape is not trivial,so we have a community installer: https://github.com/gardener/landscape-setup • Many shortcuts to make it simple (Gardener and Seed in a single cluster) • Do not use productively! • You can use it as a starter for a productive setup • Different cluster and different cloud provider accounts recommended

  18. Gardener is Open Source Gardener BlogCNCF PresentationKubernetes PodcastHacker NewsReddit Long-Term Goal Become CNCF Project

  19. Thank You! GitHub https://github.com/gardener Home Page https://gardener.cloud Wiki https://github.com/gardener/documentation/wiki Mailing List https://groups.google.com/forum/?fromgroups#!forum/gardener Slack Channel https://kubernetes.slack.com/messages/gardener Community Installer https://github.com/gardener/landscape-setup

  20. Kubernetes Machine Controller Manager Problem • Node provisioning and de-provisioning is out of scope of current Kubernetes • In the beginning we used terraform scripts ➦ unmanageable • No mechanism • to smoothly scale clusters • upgrade cluster nodes for all providers Machine Controller Manager • Node custom resources to manage nodes via k8s API • Plugins enable support for different cloudproviders • Enables cluster auto-scaling and upgrade of cluster nodes

  21. MCM Model Model for Kubernetes deployments works great So why not use it for machines? Pod Machine ReplicaSet MachineSet Deployment MachineDeployment

  22. MCM Custom Resources Secret Name: test-secret Cloudconfig: abc….xzy AccessKeyId: abc123 SecretAccessKey: xyz789 AWS-Machine-Class (Template) Name: v1 Machine Type: t2.large Disk Size: 50GB Secret: test-secret …… Machine Name: test-machine MachineClass: v1 Machine-Set Name: test-ms Replicas: 3 MachineClass: v1 Machine-Deployment Name: test-md Replicas: 3 UpdateStrategy: Rolling MachineClass: v1

  23. Working of MCM Kubernetes Controller Manager 3 VMs Cloud Provider API Node objects help in monitoring the machine status – Health Machine Controller Manager Machine Deployment Class: V1 Replicas: 3 Machine Node Machine Controller Machine Node Machine Class + Secret V1 Machine Class + Secret Machine Node Machine Set Controller Kubernetes API Server Machine Set Replicas: 3 kubectl Machine Deployment Controller ETCD (Key-value store) Machine Class + Secret V1 Machine Deployment Class: V1 Replicas: 3

  24. Autoscaling Kubernetes Controller Manager Forked Cluster Autoscaler Cloud Provider API Now assume that all the nodes resources are nearly consumed and a new pod is created Machine Node 4 Machine Controller Manager Machine Node 3 Machine Controller Pod Image: Nginx Node: - Machine Node 2 Machine Node 1 Machine Set Controller Kubernetes API Server kubectl Machine Deployment Controller ETCD (Key-value store) Machine Deployment Class: V1 Replicas: 3 Machine Deployment Class: V1 Replicas: 4 Pod Image: Nginx Node: Node4 Pod Image: Nginx Node: Unschedulable

  25. Machine Controller Manager - Components Create/update Machine-sets to perform updates Create/delete Machines to maintain required replicas Update no. of replicas based on load Machine Controller Manager Machine Controller Machine-Set Controller Machine- Deployment Controller Cluster Auto-scaler Responsible for Maintaining set of healthy Machine replicas Responsible for Managing Machine-sets (used for updates) Scales the number of replicas based on load in the cluster Responsible for Managing Machines Parent-child relationship: Adoption of orphaned children Controllers cooperate, rather than racing with each other !

More Related