New profile is making structure rigid than configurable

[perilbrain]

The Use case

This problem is more clear with an example use case:

Say I have following environments

1. 1 Production server visible to world
2. 1 Staging server visible to clients
3. 1 Dev server visible to developers
4. N Local servers specific to individual developers say around 30 developers.

The project is using Git version control. To deploy or to load a project the developer had to do following things

git clone <repository_url>/project
cd project
gradle bootstrap <env_name>
gradle build -x test
gradle bootRun

It was easier to do because application.properties was git ignored and running gradle bootstrap <env_name> generated inside it spring.profiles.active=<env_name> . If provided <env_name> didn't exist a new profile with name application-local_<env_name>.properties was created from the template application-local.properties. The developer was free to choose from a set of properties for their specific needs. Thus building a configuration was as easy as writing code using spring.profiles.include.

What changed

The new profile processing has put a new set of restrictions which is unable to answer a very simple problem i.e. given a constraint, a feature and a requirement like

• Constraint: No developer should commit their local configurations in repository.
• Feature: Every developer should have their own configurations.
• Requirement: All configurations for production, staging and dev should be tracked lets say them major configurations.

Lets try to address this problem

Trial 1:

Use profile groups by putting all major configurations in application.yml. Let user specify active profile and its configuration.

• Local configuration starts being tracked

Trial 2:

Add default spring.profiles.active=this_env in application.yml and make it mandatory for user to create this file and add their customization there.

• Now user can't reference any of the configuration in application.yml and has to copy whole thing again in their configuration. Because The spring.profile.group property cannot be used in profile-specific documents.
• If we are using activate.on-profile then developers will be forced to use a set of rules.

Trial 3:

Make application.yml.example with all contents and let developer copy and override it. Which is quit rough work.

I am unable to find a proper solution to this problem, but for more understanding I am explaining my current set up so someone can guide how to address the problem if I am missing something .

Project files

└── resources
    ├── config
    │   ├── .gitignore
    │   ├── application-common_pg.properties
    │   ├── application-common.properties
    │   ├── application-local_monkeycoder.properties
    │   ├── application-local_pg_customized.properties
    │   ├── application-local.properties
    │   ├── application-prod.properties
    │   ├── application.properties
    │   ├── application.properties.example
    │   ├── application-security.properties
    │   ├── application-security.properties.example
    │   └── application-stage.properties
    ├── db
    │   └── migration
    ├── META-INF
    │   └── spring-configuration-metadata.json
    ├── static
    └── templates

.gitignore has

application-security.properties
application.properties
application-local_*.properties

application.properties.example has

spring.profiles.active=stage

application-common.properties has things like

spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.data.rest.RepositoryRestMvcAutoConfiguration

application-common_pg.properties has

spring.jpa.properties.hibernate.dialect=com.hello.hibernate.dialect.PostgreSQL13Dialect

application-security.properties.example has passwords and others have their own data source urls. For example

application-prod.properties has at top

spring.profiles.include=common,security,common-pg
logging.level.web=info
# ....

This is very difficult to migrate to the new structure I can't think of an idea without breaking one or two things.

[philwebb]

I'm not sure I've totally understood your setup, if you have an example project that I can clone it might help me. I think you're saying that you have no application.properties file in your git repo and gradle bootstrap generates it somehow?

One possible idea that you could look into is using the new spring.config.import property. This might allow you to import your properties files without needing to use profiles. For example, you might have an application.properties:

spring.config.import=optional:classpath:applicationthisenv.properties

This will import applicationthisenv.properties if it exists.

What do you do in your dev/staging/production environments? Do those get run with a specific profile active when they're started? Also, do you have @Profile elements in your code, or are you only really using profiles to get application-<profile>.properties files to import?

[perilbrain]

if you have an example project that I can clone it might help me.

I am attaching a sample project with README which will help you understand.
demo.zip
Edit: Sorry, there is a little error in README , to switch profile use gradle bootstrap -Pprofile=<profile_name> ex. gradle bootstrap -Pprofile=dev

saying that you have no application.properties file in your git repo and gradle bootstrap generates it somehow?

Exactly... Tracking something that changes with each environment is not feasible and will only create problems. This .example pattern has been imported from php world.

One possible idea that you could look into is using the new spring.config.import property

I guess I had seen that somewhere, but still can't figure out how to include properties from different configurations. This situation is exactly like as you had seen in #24172 except for no tracking for application.properties.

Do those get run with a specific profile active when they're started?

Yes you are right. We have lot of hardwares :D [legacy style] and no cloud like setup to start from env.

Also, do you have @Profile elements in your code, or are you only really using profiles to get application-.properties files to import?

No profile engineering with code, its all plain properties loading their required properties. We never felt any need to use code for this purpose.

[philwebb]

Thanks for the example, it helped a lot to clarify things.

I think it should be possible to use spring.config.import to do what you need. Assuming that you still want to keep the basic profiles, you can change things as follows:

You're main application.properties would active the profile that you're using. For example:

spring.profiles.active=local_john

You'd then have a application-local_john.properties file but rather than using spring.profiles.include you'd use spring.config.import:

spring.config.import=./common.properties,./security.properties
application.conf.env_type=application-local.properties

This will import common.properties and security.properties (I'd rename them from application-common.properties and application-security.properties to make it clear that they're no longer profile specific files).

This should give you the same result as your previous solution. If you wanted to (since you're not using @Profile in your code), you could go one step further and replace all spring.profiles.active properties with spring.config.import.

Please let me know if that solution works for you.

[perilbrain]

This looks promising I was not able to conclude that this syntax spring.config.import=./common.properties,./security.properties could work. I was thinking in terms of yaml having all main configuration profiles and how to import that all without polluting the space.
Thanks for figuring this out, I guess I can close the issue.