ECS or EKS: Native cloud containers or kubernetes ?

There’s a lot of articles comparing the two and telling you about their differences and when you should go with one or another. I’ll make it slightly different.

I’ll tell you what I want to accomplish considering technology, culture, operations, costs, support and compliance and what is my choice to do so.

I want to provide developer teams a platform so they can easily build, deploy and run their apps in compliance with the Cloud Engineering Team’s best practices and Security Team’s policies.

Speedup cloud usage, cloud services details abstraction, best practices and security policies enforced, costs and permissions controlled with no bureaucracy and 100% cloud provider support.

What I want to do

I don’t want to add any man/hour effort and cost creating and operating a platform that would end up limiting the features of native cloud services. I want developers to use all the native cloud services potential so they can benefit from all of their latest features, integrations and be able to contact service support directly if they want. But it does not mean uncontrolled usage.

I want it as simple as possible in a way I wouldn’t limit any service or people potential and still avoid tools lock-in, because tech people should not be limited by tools.

More detailed list of things of what must be covered:

It’s a platform to run 12-factors aware (or not) containerized apps.
Developers should not care about the platform running their containers.
Developers should not care about any network stuff (connectivity, DNS, URLs, load balancers, target groups, rules, WAFs, security groups etc).
Developers should have easy visibility of their applications costs.
Don’t add unnecessary costs with the platform itself (spare headroom, operational team etc)
Don’t have costs when no app is running (for when non-production environments are turned off).
Enforce Cloud Engineering team best practices.
Enforce Security team policies.
Apps should have their monitoring automatically configured (logging, alerting, notifications etc).
Apps must have full access to all AWS services features and integrations (like trigger a task on a S3 event).
If the team has people with AWS knowledge, let them use it.
Don’t add an abstraction layer that limit the native cloud features or requires any development to support what is natively supported by the cloud.
Cloud Engineering team can always help with cloud services and troubleshooting as they have this knowledge, but the developers with cloud knowledge should be able to do it by themselves as they are using cloud directly.
Cloud Engineering team should not be needed to intermediate anything because that can slow developers down.
Developers should be able to open support tickets to AWS directly if they feel comfortable.
Not a centralized platform to need additional tools for cost management, permissions complexity etc.
Run the apps on each team account, following the multi-account best practices keeping apps and their cloud services together and where the developers already have access (maybe as read-only, not a problem) and their costs are known.
Decentralized but not uncontrolled regarding costs, security and best practices.

My Choice

I chose ECS to provide this platform. In fact, EKS (as it’s usually used) wouldn’t meet many of the requirements.

And to provide all these things on top of the cloud, I only need two other tools: one for CI/CD and another for IaC. In my case: Bitbucket Pipelines and Terraform. With k8s, many other tools and people to operate them would be needed.

Terraform modules takes care of the abstraction layer for things such as load balancers, security groups, IAM roles while including best practices and Bitbucket Pipelines still enforce security policies.

Why not EKS ?

I think EKS is designed for those who are already used to k8s ecosystem and depends on kubernetes for many services and don’t want to switch to cloud native services.

I think kubernetes itself is useful in virtualized environments in legacy datacenters as it’s much more efficient to run containers than virtual machines, and using certificates, service discovery and other embeded services are usually the best option.

K8s provides services like ssl certificates, load balancers, service discovery, monitoring etc. It’s a “cloud in a box”, it’s really a must have on traditional datacenters where you only have access to virtualization platform.

You deploy k8s on top of the virtualization platform (or on top of metal if you can) to provide developers a cloud-like platform then.

If you use k8s on one cloud and want to migrate to another cloud, it is potentially very easy if you’re using everything as k8s services. But if you are using k8s just to run containers and not using these features provided by k8s, then you have no reason to use k8s.

But if you are already in the cloud, and only in the cloud, k8s is like a cloud in a box, in a cloud. And that box needs to be supported and managed and it will not be by the cloud provider. Doesn’t make sense to me.

Let’s take a look at the most common k8s implementation in the cloud: a centralized EKS cluster based on EC2 instances (no Fargate) on the Cloud Engineering team account running all of the company’s applications.

I don’t like this approach because of the following reasons:

Culture: you’ll need a team to operate the k8s cluster. It reassembles me that old days of traditional centralized virtualization clusters run by an infrastructure team whom developers always blame for their applications problems. And they say back “But there are many other apps on our cluster without problems!”. And all that kind of problems the DevOps culture came to solve.
Costs: along with the people cost to operate the cluster (that does not scales in proportion of apps), it’s hard to separate costs for each app, not only the apps container costs, but some apps will use more k8s services like certificates, ingress features etc, and some teams will require more tshoot help because teams maturity are different.
Same network for all apps: the network have to be a very big network and you can’t do security group rules for anything that must be accessed through the network because all apps from all teams share the same network address space.
Complexity for cloud permissions. For AWS, you are just running EC2 instances and there’s no native solution to give each containers permissions to access other cloud resources. Although there are k8s solutions to deal with it, it could be much easier to use IAM roles as ECS does.
Manage EC2 instances. There are cpu, memory and network limits that affects containers allocations that you need to think of when choosing EC2 instance types because each type have different limits, and you have to review it frequently and you don’t want to have large spare capacity distributed on the cluster. That’s man/hour cost. Hopefully there are tools that will help you save money with these costs by automatically changing your cluster configuration for you.
You can save money using Spot Instances, but it adds complexity and failure prone situations. You’ll have to deal with multiple launch configurations to deal with spot instances and on demand instances and make sure that the total cluster instances can deal with the actual capacity requirements and that containers of the same apps won’t run on spot instances that can be claimed at the same time (ECS+Fargate+Spot don’t have this problem).
The centralized cluster is a Single-Point-Of-Failure. Although EKS don’t usually have any problems by itself (not caused by human errors), many teams sharing the same resources can always lead to some kind of problems. If a team need a feature/option/configuration that is common to the cluster, it could impact other team. If a team need some kind of log parsing and you are doing this as a deamon set container, you add complexity to that log parser to serve different needs. And potential problems will grow as it’s the nature of adding complexity to a central thing that needs to serve different needs.

So, when run this way, k8s is just the new VMware in my opinion – I’m not talking about the technology itself, but it brings back the same old problems the traditional centralized virtualization did that the devops culture aimed to solve and the cloud solved by design.

When cloud providers didn’t have a managed container platform, k8s on top of virtualized instances was the choice for sure. But nowadays all the cloud providers have better ways to run your containerized apps that have better integrations with their other services.

There are really no reasons to not use the managed plataform to run containers that integrates natively within every services on the cloud provider. Really. In my opinion.

Why ECS ?

ECS Clusters don’t have any costs by themselves, they are just a “logical unit for organization”.

So you can have one ECS Cluster inside each team account or even one cluster for each application, it makes no difference at all as it won’t add any costs and no operational effort, as ECS+Fargate is not based on EC2 instances. So you do not have any effort managing EC2 scaling policies, cluster headroom, EC2 bring-up time gap problems, EC2 network limitations etc.

So, no cost with the Cluster itself and no man/hour cost to manage it.

Costs visibility problems and permissions complexity are solved because the application is running on the team account. The multi-account model is the answer for these problems.

With ECS and terraform you just say: “Hey AWS, get my container image from this registry and scale it as needed with these limits X and Y.”

You’ll be able to use the latest features and integrations from AWS like triggering a task on an s3 event, scaling tasks according to SQS queue size and anything AWS provides you and will support you tu use it.

And it doesn’t apply just for AWS ECS. Every cloud has its managed platform to run containers. In Google it’s Cloud Run (I think).

You can save money using “Spot Instances” at ECS even using Fargate ! And as you are not using EC2, you don’t even have to deal with the cluster capacity. You just say a percentage of your tasks that can’t be reclaimed by AWS and the other tasks will just be relaunched if reclaimed. You don’t have to deal with any capacity questions and your tasks will be automatically relaunched always keeping the number of tasks needed to meet your scaling policy.

In other words: your app won’t have any performance impact even using “spot instances” on ECS without instances.

My opinion

When you’re building something new, there’s no reason to go with k8s.

If you don’t know why you NEED kubernetes, don’t go for it. It comes with complexity and cost. Of course you can use tools to create abstraction layers, but this is unneeded problem solving.

If you need k8s because your deployment already uses it, you don’t need it, you’re just lazy to get rid of it. And it could be a good reason as it would have man/hour cost to adapt your deployment to ECS to save money later on.

If you need k8s because you really need it for a reason you know why and you know ECS and AWS would not meet it, that’s ok, keep it ! It works very well if you can solve the culture problems that it brings with it. I would rather try to make your app simple enough to be technology agnostic and portable to use any cloud native services.

How about k8s for multicloud ?

I don’t think it worth enough. Why ?

All the problems I mapped above was just on AWS cloud. Imagine repeating it across multiple clouds ??

One can argue that using k8s and all services within it would provide an easier multi-cloud environment and/or migration. I totally agree with that. But who wants it ?

If you want an app to run multicloud, the same app, running containers of the same app simultaneously in more than one cloud (to make it clear) you are very wrong. One day I’ll write about this, but there’s no reason to want this. This would require so many network and compatibility layers that you would not be using cloud anymore, you would end up transforming the cloud into a legacy virtualization layer.

My idea of multicloud

If you want to be able to run apps in more than one cloud (and maybe the same app on one cloud or another), this is a fair wish, I agree.

I would do it as an abstraction in the IaC layer (in my case, terraform). Write terraform modules that looks the same for who uses them, and they use each cloud services to acomplish the task.

Hybrid cloud environments

Yes, that’s the unique situation I think k8s is still desirable.

When cloud providers didn’t have a managed container platform, k8s on top of virtualized instances was the choice for sure. But nowadays all the cloud providers have better ways to run your apps and better integrations with their other services.

But if you need to run your apps in a hybrid environment, simultaneously, go for k8s.

You can run k8s on top of virtualization or directly on metal on the traditional datacenter and have a managed k8s cluster on any cloud provider. Of course you are keeping the traditional centralized virtualization model and it’s problems, but now with containers that are much better than complete virtual machines.

Good articles about ECS vs EKS

Most of them does not take a side, they usually just compare both and try to tell you why you should consider on or another for what they are best.

Nops.io – EKS vs ECS: The ultimate guide

Highlights:

“EKS has higher operational overhead due to Kubernetes’ complexity, with a higher learning curve and deeper understanding of Kubernetes required. However, it provides deeper control. In addition, the Kubernetes ecosystem is vast, with a wealth of tools, extensions, and a large community for support and development with solutions for logging, monitoring, CI/CD, security, and more. EKS is a powerful platform for developers looking to leverage the flexibility and extensibility of Kubernetes, supported by robust AWS infrastructure.”
“AWS ECS is a proprietary AWS container management service that provides an efficient and secure way to run and scale containerized applications on AWS. ECS is deeply integrated with AWS services. It provides a seamless experience for AWS users and is designed for simplicity, with the inherent trade-off of offering less fine-tuned control and flexibility. With ECS, there’s no need to manage a control plane, nodes, or add-ons, making it easier to get started. It is advantageous for quick deployments or when a more straightforward approach suffices. In addition, with ECS you don’t have to pay for a control plane, meaning that it can potentially be cheaper.”
“AWS EKS brings Kubernetes, an open-source container orchestration platform, into the AWS cloud. Kubernetes offers high flexibility, a robust ecosystem and community, and consistent open-source API which offers extensibility and portability. As a result, it is often more suited to complex applications, multi-cloud environments, and other situations which require more fine-grained control. EKS abstracts away some of the complexity of managing Kubernetes, allowing users to leverage the power of Kubernetes without the operational overhead of setting up and maintaining the control plane. It also automates numerous aspects of running a Kubernetes cluster, including patching, node provisioning, and updates.”

Reddit – EKS vs ECS

Highlights:

“Kubernetes is like a cloud in a box. But if you’re already in the cloud, then there’s no need to have a cloud in a box, in the cloud. I used Kubernetes exclusively for about 8 months after a bunch of ECS time, and just kept thinking “this would be so much easier on ECS”. Kubernetes is like the new WordPress.. everyone just uses it because there are so many plugins/helmcharts. The whole vendor lock-in is overstated in my opinion because so much of it now makes use of cloud specific APIs anyways.”
“ECS is the old clunky model. You need ECS, Parameter Store, Secrets Manager, ACM, SSM, Cloudwatch, etc to do what you can do in EKS. My personal pet peeve about ECS is how long it took them to give you reasonable access to your containers, especially for Fargate. It is a must for debugging. With Kubernetes it is a kubectl exec away. Look at just how much software you can install via Helm, Artifact Hub.“
“ECS is best if you are OK with the AWS ecosystem. Fargate == ECS except you don’t have to manage the underlying hosts yourself and can fit into their performance tiers. EKS is best if you already rely on Kubernetes features, or are pursuing a scenario where you run cloud+onprem, or multi-cloud, such that you want a consistent application experience across all of those environments. IMO use ECS unless you have a compelling reason otherwise. It is just less work for you.”

ARGLabs

IaC made simple.