Provision AWS Secret Manager Retrieval of Secrets

Every modern web app has secrets that need to be shared securely with it. You web application may need to have a environment variable like an API key. That key should be stored in AWS secrets manager, it is a great use case for us to learn about how to give access to instance to the secret.

Overview

We are going to create infra same as before but with the secrets manager integrated

alt text

Github Repository

This guide full code is available in https://github.com/Ship-Infra/ship-infra-project/tree/main/v7-secrets. Feel free to clone it and follow along!

Creating a Secret in AWS Secret Manager

Now lets add the actual secrets. We need to add secrets only once, and then reference them. This is the process that we will repeat everytime the secret has to be updated. Let's do this from a command line using aws cli:

aws secretsmanager create-secret \
    --name "my_app/v1/credentials" \
    --description "Application credentials" \
    --secret-string '{
        "my_api_key": "your-api-key",
    }'

This command will create the secret values. If everything goes well, you should see an output like below:

{
    "ARN": "arn:aws:secretsmanager:us-east-1:088656249151:secret:<your-service>/<some-path>/credentials-MhS9Um",
    "Name": "<your-service>/<some-path>/credentials",
    "VersionId": "61c33df3-6e38-4f7f-8b9d-fda8bad3c880"
}

That output indicates your newly created secrets container with the secret.

Adding a secret

Now, we can add a dummy secret value. This can be done via UI, or CLI. Here is an example using CLI:

## Update secrets

#!/bin/bash

# Check if .env file exists
if [ ! -f .env ]; then
    echo "Error: .env file not found!"
    exit 1
fi

# Read .env file and create JSON string
JSON_STRING="{"
FIRST=true

while IFS='=' read -r key value; do
    # Skip empty lines and comments
    [[ -z "$key" || "$key" =~ ^# ]] && continue

    # Remove quotes from value
    value=$(echo "$value" | sed 's/^"//;s/"$//')

    # Add comma if not first item
    if [ "$FIRST" = true ]; then
        FIRST=false
    else
        JSON_STRING="$JSON_STRING,"
    fi

    # Add key-value pair to JSON
    JSON_STRING="$JSON_STRING\"$key\": \"$value\""
done < .env

JSON_STRING="$JSON_STRING}"

# Check if secret exists
if aws secretsmanager describe-secret --secret-id "my_app/v1/credentials" >/dev/null 2>&1; then
    # Update existing secret
    aws secretsmanager update-secret \
        --secret-id "my_app/v1/credentials" \
        --secret-string "$JSON_STRING"
    echo "Secret updated successfully!"
else
    # Create new secret
    aws secretsmanager create-secret \
        --name "portfolio/app/credentials" \
        --description "Application credentials" \
        --secret-string "$JSON_STRING"
    echo "Secret created successfully!"
fi

The above script will read the .env file from a local folder, convert it to JSON format, and then either create a new secret or update an existing one in AWS Secrets Manager.

Now let's create an .env file in the same folder as the script:

my_api_key="your-api-key"
# Note, the new line is important at the end of the file

Let's run the script:

bash update-secrets.sh

# Output below
{
    "ARN": "arn:aws:secretsmanager:us-east-1:088656249151:secret:my_app/v1/credentials-5x50LZ",
    "Name": "my_app/v1/credentials",
    "VersionId": "8202b548-5236-45d2-94f4-4d87d146ad00"
}
Secret updated successfully!

Provisioning AWS Secret Manager and using Secrets

Let's start using the secrets in our infrastructure:

Retrieving Secret Container

# modules/secrets/main.tf
data "aws_caller_identity" "current" {}

data "aws_secretsmanager_secret" "app_secrets" {
  name = var.credentials_name
}

The code above defines a new secret in AWS Secrets Manager. It doesn't contain the actual secret value, just metadata about the secret. The name parameter should match the secret name you created using AWS CLI.

data "aws_caller_identity" "current" {} is a special AWS data source that provides information about the AWS account and IAM principal (user or role) currently being used to make API calls. It requires no arguments and returns three key pieces of information:

account_id - Your AWS account number
arn - The ARN (Amazon Resource Name) of the IAM user or role
user_id - The unique identifier of the IAM user or role

Getting Secret Value

Next, we will define a data source to retrieve the secret value:

# modules/secrets/main.tf

data "aws_secretsmanager_secret_version" "current" {
  secret_id = data.aws_secretsmanager_secret.app_secrets.id
}

We are using id from the previous data source to find the correct secret. By default it gets the latest version of the secret.

Since the secret is a JSON string, we use jsondecode to convert this into Terraform map. We have to provide the variables.tf and outputs.tf` files as well:

# modules/secrets/variables.tf
variable "credentials_name" { type = string }

# modules/secrets/outputs.tf
output "secrets" {
  value = jsondecode(data.aws_secretsmanager_secret_version.current.secret_string)
}

Now, to use it in main module and in our ec-2 instance, we can do the following:

# main.tf
module "secrets" {
  source = "./modules/secrets"
  credentials_name = "my_app/v1/credentials"
}

module "ec2_first_instance" {
  source            = "./modules/ec2"
  instance_ami      = var.instance_ami
  instance_type     = var.instance_type
  security_group_id = module.security_group.security_group_id
  subnet_id         = module.network.public_subnet_ids[0]
  ssh_public_key    = var.ssh_public_key
  ssh_key_name    = "ec2-key-first-instance"
  user_data = <<-EOF
              #!/bin/bash
              sudo apt-get update -y
              sudo apt-get install -y docker.io
              sudo systemctl start docker
              sudo systemctl enable docker

              # Add user to docker group
              sudo usermod -aG docker $USERNAME

              sudo docker run -d -p 80:80 \
              -e MY_API_KEY=${module.secrets.secrets.my_api_key} \
              nginx
              EOF
}

module "ec2_second_instance" {
  source            = "./modules/ec2"
  instance_ami      = var.instance_ami
  instance_type     = var.instance_type
  security_group_id = module.security_group.security_group_id
  subnet_id         = module.network.public_subnet_ids[1]
  ssh_public_key    = var.ssh_public_key # Note, you can use the same public key for both instances, but not recommended for production
  ssh_key_name    = "ec2-key-second-instance"
  user_data = <<-EOF
              #!/bin/bash
              sudo apt-get update -y
              sudo apt-get install -y docker.io
              sudo systemctl start docker
              sudo systemctl enable docker

              # Add user to docker group
              sudo usermod -aG docker $USERNAME

              sudo docker run -d -p 80:80 \
              -e MY_API_KEY=${module.secrets.secrets.my_api_key} \
              nginx
              EOF
}

Note that we are passing the secret value as an environment variable to the docker container using -e (environment variable) flag. Let's apply the changes here to ensure nothing breaks so far. Run:

terraform init
terraform apply --auto-approve

Testing the setup

To test if our setup is working, we can SSH into our EC-2 instance and check if the environment variable is set correctly.

ssh-add path/to/your/private-key.pem # Adds your private key to the ssh-agent
ssh ubuntu@<EC2_PUBLIC_IP> # Replace with your EC2 public IP

## Inside the EC2 instance, run:
ubuntu@ip-10-0-61-134:~$ sudo docker ps

# Outputs:
CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS        PORTS                               NAMES
e4dba1c33ac9   nginx     "/docker-entrypoint.…"   3 seconds ago   Up 1 second   0.0.0.0:80->80/tcp, :::80->80/tcp   recursing_haslett

The above command shows that our nginx container is running. Now, let's check if the environment variable is set correctly inside the container:

sudo docker exec -it e4dba1c33ac9 /bin/bash # Replace with your container ID, you can get it from `docker ps` command

## Inside the container

root@e4dba1c33ac9:/# echo $MY_API_KEY
## Outputs
your-api-key

Destroying Infra

Remember, infra has costs. When you are done experimenting, you can destroy the infra like follows:

terraform destroy --auto-approve

Conclusion

So far this might have worked, but the secrets now get stored in plain text on the EC-2 instance. This is not ideal from security perspective. For instance, anyone with SSH access to the EC-2 instance can see the user data as follows:

curl http://169.254.169.254/latest/user-data

# Reveals user data in plain text
#!/bin/bash
sudo apt-get update -y
sudo apt-get install -y docker.io
sudo systemctl start docker
sudo systemctl enable docker

# Add user to docker group
sudo usermod -aG docker $USERNAME

sudo docker run -d -p 80:80 \
-e MY_API_KEY=your-api-key \
nginx

So while this tutorial shows how to setup the secrets store with AWS (Which you should do), the retrieval of secrets approach is not secure. Although it works. Providing a secure way is our of scope for now, since we are focusing on setting up a running infra rather than secure. Baby steps.

Next, we will explore the next very important resource in AWS - The RDS (Relational Database Service) and see how to provision a managed database in AWS.

Provision AWS Secret Manager Retrieval of Secrets

On this page