config/deploy.production.yml should be default behavior, rather than config/deploy.yml

[motdotla]

Background: The .env file is used for development secrets and config for many developers but Kamal treats it like 'production' data.

The developer experience of kamal initializing a config/deploy.yml file and associating that with a .env file feels not explicit enough.

What if instead kamal init:

1. generated a config/deploy.production.yml file
2. generated a .env.production file it not already existing

This would be less surprising to me. By default I'm using Kamal to deploy to production and associating a .env.production file with it.

The current setup of config/deploy.yml and a .env file made me confused how I could modify it to support production secrets. Would I have to use some different mechanism, switch to Rails credentials completely, do some manual switching out of .env* files just before running kamal env push and kamal env deploy, etc. I think it Kamal was more explicit by generating config/deploy.production.yml out the gate this confusion would be minimized.

[igor-alexandrov]

It would be better if kamal init would generate both deploy.yml and config/deploy.production.yml. I believe the destination param should become mandatory and have production as a default value. This will allow to solve the problem that was described by @motdotla and will make clearer configurations with multiple destinations.

[nickhammond]

Another idea is passing the destination with the kamal init command so you could easily do kamal init -d staging or kamal init -d production. This isn't currently supported but something I was thinking about today since I ended up doing this to configure the staging environment:

kamal init
mv .env .env.staging
echo "{}" > config/deploy.staging.yml

This also has the benefit of not breaking or modifying the current workflow, it's more of an opt-in process. Some people really will have just one remote destination and that's perfectly fine.

[marckohlbrugge]

+1 on changing .env to something else.

⚠️ The current setup is dangerous and can lead to potential loss of PRODUCTION data.

Depending on how your Rails app is configured, it may automatically pick up ENV vars from .env – So when you kamal envify from your local machine, it generates a .env file, and your local codebase picks that up without you being aware. This could lead to your production DATABASE_URL being loaded in your development environment. Now imagine running some truncation or sanitization rake job, and poof your production data is gone. 💨

A few things have to go wrong for this to happen, but it's not that far fetched. I came pretty far into this process myself, but caught it just on time.

[motdotla]

yeah, with you. i ended up doing this to avoid the risk:

1. creating .env.kamal
2. using dotenvx to explicitly select the env i want from .env.kamal
3. write those on deploy/build as a variable to env.clear in deploy.yml

an example:

service: myservice
image: myimage

# custom registry
registry:
  server: ghcr.io
  username: myusername
  password: <%= `dotenvx get KAMAL_REGISTRY_PASSWORD --env-file=.env.kamal`.strip %>

# customize builder
builder:
  args:
    GIT_COMMIT_HASH: <%= `git rev-parse --short HEAD`.strip %>
    GIT_TAG: <%= `git tag -l | sort -V | tail -n1`.strip %>

servers:
  - <%= `dotenvx get SERVER_PRODUCTION --env-file=.env.kamal`.strip %>

env:
  clear:
    DOTENV_KEY: <%= `dotenvx get DOTENV_KEY_PRODUCTION --env-file=.env.kamal`.strip %>
    RAILS_MASTER_KEY: <%= `dotenvx get RAILS_MASTER_KEY_PRODUCTION --env-file=.env.kamal`.strip %>

volumes:
  - "storage:/rails/storage"
  

this way i can also keep this in my code and only share the .env.kamal file with those who have permission to deploy (or my ci/ci)

[pch]

+1

Just ran into this: .env was loaded with the production DATABASE_URL 🤦🏻‍♂️ Almost migrated my production DB thanks to the automated db:prepare hook.

(partly my fault because my IP is allowed to connect to the production db, but that's still a big footgun here)

This seems to be caused by foreman (default for Procfile.dev) loading .env file.

[bhtabor]

+1

Foreman picked up the production DATABASE_URL I've setup for Kamal from .env and Rails used it in development even though I've hard coded database configuration for development and testing. For now, I've changed the DATABASE_URL environment to PRODUCTION_DATABASE_URL. Perhaps using .env.production instead of .env is preferable.

You can connect to the database by setting an environment variable ENV['DATABASE_URL'] or by using a configuration file called config/database.yml.

https://guides.rubyonrails.org/configuring.html#configuring-a-database

[pch]

Kamal already generates a .kamal directory. What if both the .env.erb and .env lived inside this directory instead?

The current default seems to be intended for CI where the risk of overriding vars is much lower.

[cinemast]

Did I understood this conversation correctly, kamal does not enable for multiple environments, e.g. dev, stage, prod?

[strzibny]

It does but by default the split is not done.

[abratashov]

I've found a good solution to support several hosts (production, demo, etc.) with the same config but different env values.
I just fill the deploy.yml with escaped values like:

service: <%= ENV['DOCKER_CONTAINER'] %>

image: <%= ENV['DOCKER_IMAGE'] %>

registry:
  server: <%= ENV['DOCKER_SERVER'] %>
  username: <%= ENV['DOCKER_USERNAME'] %>
  password:
    - DOCKER_REGISTRY_PASSWORD

ssh:
  user: <%= ENV['SERVER_USER'] %>
...

and deploy.demo.yml:

# Just copied one line to avoid an error of the nil Hash keys
service: <%= ENV['DOCKER_CONTAINER'] %>

and added .env.demo

################################################# Docker
DOCKER_SERVER=registry.gitlab.com
DOCKER_USERNAME=user
DOCKER_REGISTRY_PASSWORD=glpat-token
DOCKER_IMAGE=user/myapp/myapp-demo
DOCKER_CONTAINER=myapp-demo

################################################# Server
SERVER_USER=deployer
SERVER_IP=1.2.3.4
SERVER_HOSTNAME=myapp.host.org

################################################# DB Postgres
DB_HOST=1.2.3.4
DB_NAME=DB_NAME
DB_USER=deployer
POSTGRES_PASSWORD=pass
# ...

Deploy works smoothly:

kamal setup -d demo

If you need more servers, just create yet one-line file deploy.prod.yml and .env.prod

Also, there is the full process of Rails project deployment with Kamal https://github.com/abratashov/kamal-blog/blob/main/docs/install_prod.md

[Flixt]

Here to cross reference rails/rails#51829. There really seem to be some problems related to Kamal/Foreman/DotEnv confusions.

[project-orcon]

This is really bad, I just connected to my production database running rails console in development - had no idea DATABASE_URL ENV var would override my development db settings

[trinitytakei]

+1 - not as severe as DATABASE_URL and such, bust I still lost an hour after running kamal init that added RAILS_MASTER_KEY=DIESEL_POWERED_FOOTGUN_LOL to my .env file, which overrode config/master.key and ofc encryption/decryption went to hell. I managed to blow away my config/master.key during the investigation (yeah yeah, I'm an idiot, we all know 'no one, not even ChatGPT-3000-quantum will be able to decrypt your credentials if you lose master.key' etc). Thankfully it was simple/fast to rebuild my credentials, but still, not my idea of fun.

In hindsight this was trivial to realize/fix, I should have known better, lesson learned, etc.

That said... the suggested fix would be awesome.