André Ramoni

How should repository variables and secrets get onto containers ?

Every application needs configuration, and some of it is secret. At ARGLabs the rule for getting both onto a container is one line long: a Bitbucket repository variable named ENV_<NAME> reaches the container as the plain environment variable <NAME>; a variable named SECRET_<NAME> is stored in AWS Secrets Manager and injected by the ECS agent as <NAME>. Nothing else reaches the container.

Adding or removing one is just a change in the repository settings and a pipeline run. No Terraform edit, no pipeline edit, no code change. This post shows the mechanism end to end on the arglabs-workload application, with real output from the dev environment, and then explains why it is a good practice.

What the repository has today

These are the seven repository variables of arglabs-workload on Bitbucket, read from the Pipelines API. Secured ones have no readable value, by design.

One detail matters: ENV_CAMUFLADO is marked secured in Bitbucket, but it still starts with ENV_, so it becomes a plain variable. Inside a pipeline a secured variable is indistinguishable from a plain one (secured only masks the log), so the prefix is what says “this one is a secret”, not the Bitbucket flag.

What the container sees

The example app has two endpoints for exactly this check. GET /env dumps the container’s environment and should have both the secrets and env vars values, GET /secret answers only the names that were injected as secrets. Both show values in the clear on purpose, since this is a lab application and the point is to see what arrived. Here is /env on dev, trimmed to the names that matter (the OpenTelemetry, ECS metadata and PATH entries were cut):

$ curl -s https://workload.dev.arglabs.com/env
{
  "env": {
    "APP_ENV_KEYS": "CAMUFLADO,DB_USER,NOVA1,TESTEVAR",
    "APP_SECRET_KEYS": "DB_PASS,NOVA2,TESTSECRET",
    "CAMUFLADO": "testepraveroqueacontece",
    "DB_PASS": "argarglabs",
    "DB_USER": "Arglabs",
    "HOSTNAMES": "workload-use1.dev.arglabs.com,workload.dev.arglabs.com",
    "NOVA1": "wedwedwedw",
    "NOVA2": "nova2secret",
    "REGION": "use1",
    "REGION_FULL": "us-east-1",
    "TESTEVAR": "testado",
    "TESTSECRET": "testado"
  },
  "timestamp": "2026-10-13T19:10:35.389175+00:00"
}

Every ENV_ and every SECRET_ name from the repository is there, without its prefix. Two extra names are added by Terraform: APP_ENV_KEYS lists what came as plain variables and APP_SECRET_KEYS what came as secrets, because inside the container a secret is an environment variable like any other and the process has no other way to tell them apart. That is what /secret reads:

$ curl -s -w 'HTTP %{http_code}\n' https://workload.dev.arglabs.com/secret
{
  "secrets": {
    "DB_PASS": "argarglabs",
    "NOVA2": "nova2secret",
    "TESTSECRET": "testado"
  },
  "timestamp": "2026-10-13T19:10:35.776568+00:00"
}
HTTP 200

DB_USER is not listed: it came from ENV_DB_USER, so it is a plain variable, not a secret. The endpoint answers 200 only when every listed secret is present and non-empty.

The task definition

This is where the two kinds part ways. The relevant parts of the app container in task definition aio-workload-use1-dev:71 (the same trimming as above):

"environment": [
  { "name": "APP_ENV_KEYS",    "value": "CAMUFLADO,DB_USER,NOVA1,TESTEVAR" },
  { "name": "APP_SECRET_KEYS", "value": "DB_PASS,NOVA2,TESTSECRET" },
  { "name": "CAMUFLADO",       "value": "testepraveroqueacontece" },
  { "name": "DB_USER",         "value": "Arglabs" },
  { "name": "NOVA1",           "value": "wedwedwedw" },
  { "name": "TESTEVAR",        "value": "testado" }
],
"secrets": [
  { "name": "DB_PASS",    "valueFrom": "arn:aws:secretsmanager:us-east-1:293174400224:secret:aio/workload/dev/app:DB_PASS::" },
  { "name": "NOVA2",      "valueFrom": "arn:aws:secretsmanager:us-east-1:293174400224:secret:aio/workload/dev/app:NOVA2::" },
  { "name": "TESTSECRET", "valueFrom": "arn:aws:secretsmanager:us-east-1:293174400224:secret:aio/workload/dev/app:TESTSECRET::" }
]

The plain variables carry their value. The secrets carry an ARN and a JSON key, nothing else: one Secrets Manager secret per region, aio/workload/dev/app, holds all of them as a JSON object, and each entry points at one key of it. No secret value is in the task definition, so it can be described, diffed and logged freely.

How it gets there

Four hops, each driven by data, none of them naming a variable.

1. The pipeline turns the prefixes into two JSON maps

Every Terraform step runs scripts/vars-appvars.sh (shared by every ARGLabs app through arglabs-cloudteam-sharedpipelines). It reads the names in the environment and keeps the ones carrying a prefix:

__appvars_names() {
  awk -v p="$1" 'BEGIN {
    for (k in ENVIRON)
      if (index(k, p) == 1 && length(k) > length(p)) print k
  }' | sort
}

export TF_VAR_app_env="$(__appvars_map ENV_)"

if [ -f terraform/tf-secrets.tf ]; then
  export TF_VAR_app_secrets="$(__appvars_map SECRET_)"
else
  export TF_VAR_app_secrets="{}"
  echo "terraform/tf-secrets.tf is parked: SECRET_* variables are ignored."
fi

The prefix is the allowlist. Everything Bitbucket injects on its own (BITBUCKET_*, AWS_*, the OIDC credentials) is left behind by construction, and AWS_SECRET_ACCESS_KEY does not start with SECRET_ either. The step log of the last deploy shows the result; Bitbucket masks the secured values, which is why CAMUFLADO prints as its variable name:

app_env keys:     CAMUFLADO DB_USER NOVA1 TESTEVAR
app_secret keys:  DB_PASS NOVA2 TESTSECRET
TF_VAR_app_env={"CAMUFLADO":"$ENV_CAMUFLADO","DB_USER":"Arglabs","NOVA1":"wedwedwedw","TESTEVAR":"$SECRET_TESTSECRET"}
TF_VAR_app_secrets=<redacted>

2. Terraform receives them as two maps

In terraform/variables.tf the two maps land in two variables, the second one sensitive. Only the names are ever taken out of the sensitive map, and the secret’s name is fixed by convention so the file that creates it and the file that reads it never need to reference each other:

variable "app_env"     { type = map(string), default = {} }
variable "app_secrets" { type = map(string), default = {}, sensitive = true }

locals {
  app_env          = merge(var.app_env, { APP_ENV_KEYS = join(",", sort(keys(var.app_env))) })
  app_secrets      = var.app_secrets
  app_secret_names = nonsensitive(sort(keys(var.app_secrets)))
  app_secret_name  = "${local.scope_parsed}/${local.app_name}/${local.env}/app"
}

3. The secrets go to Secrets Manager, write-only

terraform/tf-secrets.tf creates one secret per region and writes the whole map as its JSON value. The version uses a write-only argument: the value is sent to the API and dropped, it never lands in the Terraform state. The version number is derived from a hash of the content, so a changed value rewrites the secret with no manual bump, and an empty value fails the plan instead of starting a container with an empty credential:

resource "aws_secretsmanager_secret" "app" {
  for_each = toset(local.secret_regions)
  region   = each.key
  name     = local.app_secret_name
}

resource "aws_secretsmanager_secret_version" "app" {
  for_each  = toset(local.secret_regions)
  region    = each.key
  secret_id = aws_secretsmanager_secret.app[each.key].id

  secret_string_wo         = jsonencode(local.app_secrets)
  secret_string_wo_version = parseint(substr(sha256(jsonencode(local.app_secrets)), 0, 8), 16)

  lifecycle {
    precondition {
      condition     = alltrue([for name in local.app_secret_names : local.app_secrets[name] != ""])
      error_message = "At least one SECRET_* Bitbucket variable is empty. ..."
    }
  }
}

The secret also gets a resource policy. The execution role already carries secretsmanager:GetSecretValue from its identity policy, so an Allow would restrict nothing; what narrows the access to this one app is an explicit Deny for everyone else. This is the live policy on aio/workload/dev/app in us-east-1:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DenyEveryoneElse",
    "Effect": "Deny",
    "Principal": { "AWS": "*" },
    "Action": "secretsmanager:GetSecretValue",
    "Resource": "*",
    "Condition": { "StringNotEquals": { "aws:PrincipalArn": [
      "arn:aws:iam::293174400224:role/aio-workload-use1-execution-role-dev",
      "arn:aws:iam::293174400224:role/deployer"
    ] } }
  }]
}

4. The task definition is built from the names

In terraform/tfmod-ecs-app.tf the plain variables become the container’s environment and the secret names become its secrets, each pointing at its key in the regional secret by ARN. ECS rejects a task definition where a name is both a variable and a secret, so any ENV_X that collides with a SECRET_X is dropped from the environment: the secret wins.

task_definition_environments = { for r in local.regions : r => merge(
  { for k, v in local.app_env : k => v if !contains(local.task_secret_names, k) },
  { APP_ENV_KEYS    = join(",", [for n in local.app_env_names : n if !contains(local.task_secret_names, n)]) },
  { for k, v in { APP_SECRET_KEYS = join(",", sort(local.task_secret_names)) } : k => v if length(local.task_secret_names) > 0 },
) }

task_definition_secrets = { for r in local.regions : r => [
  for name in local.app_secret_names : {
    name               = name
    secret_manager_arn = "arn:aws:secretsmanager:${r}:${local.aws_account_id}:secret:${local.app_secret_name}"
    secret_key         = name
  }
] }

At task start the ECS agent, using the execution role, calls GetSecretValue once per region secret, picks each key, and hands the values to the process as environment variables. The application makes no AWS call and needs no SDK for this. The whole /secret handler of the example app is:

names = [k for k in os.environ.get('APP_SECRET_KEYS', '').split(',') if k]
secrets = {name: os.environ.get(name) for name in names}
self.send_response(200 if names and all(secrets.values()) else 503)

Why nothing changes to add or remove one

Every hop above is data-driven. The names come from the environment, the Terraform variables are map(string), the resources iterate the maps with for_each and jsonencode, and the secret version is keyed by a hash of its content. A new SECRET_X on the next push is one more key in the JSON secret, one more entry in secrets, one more name in APP_SECRET_KEYS, and a new task. A removed one disappears from all three. With no SECRET_* left, the secret itself is destroyed.

The proof is in the repository: none of NOVA1, NOVA2, TESTEVAR, TESTSECRET or CAMUFLADO appears in any file of arglabs-workload. They exist only in the Bitbucket settings, and they reached the container anyway.

Why this is a good practice

  • One source of truth per environment. The repository’s variables are the configuration. There is no .env file, no value in Terraform code, no value in the Terraform state (write-only), no secret baked into the image.
  • The convention is the allowlist. Nothing reaches the container unless somebody named it on purpose with the prefix. Pipeline credentials and Bitbucket’s own variables are excluded by construction, not by a list that has to be maintained.
  • Secrets are resolved by the platform, not by the app. The ECS agent fetches them with the execution role, under a resource policy that names exactly two principals. The application reads environment variables and nothing else, so it stays portable and has no secret-fetching code to get wrong.
  • A name is a variable or a secret, never both. Promoting ENV_X to SECRET_X is a rename in Bitbucket; the secret wins on collision, so nothing has to be cleaned up first.
  • Auditable. Every read is a GetSecretValue event in CloudTrail; tf-secrets.tf turns them into metrics, an alarm on failed reads and a dashboard per region. The task definition holds no value and can be diffed between revisions.
  • Same mechanism everywhere. The example Lambda of the same repository gets the same ENV_* as variables and reads the same secret at cold start, from APP_SECRET_ID and APP_SECRET_KEYS, with its own role in the same policy.

Two honest caveats. The Terraform plan artifact the pipeline publishes carries every root variable in clear text, sensitive ones included, so anyone who can download the pipeline artifacts can read the secrets; anything that must stay private cannot travel as a TF_VAR. And the /env and /secret endpoints exist to show what arrived on this lab application; a real application must not answer its configuration, let alone its secrets, over HTTP.

Code: arglabs-workload (the application, terraform/tf-secrets.tf and terraform/tfmod-ecs-app.tf) and arglabs-cloudteam-sharedpipelines (scripts/vars-appvars.sh).

Posted in Best Practices