S3 reference inconsistency

[sbernauer]

Affected version

nightly

Current and expected behavior

Currently we have the following CRD

    s3bucket:
      inline:
        bucketName: some-dummy-value
        connection:
          reference: spark

As we only need to tell s3a the endpoint, credentials etc. but not he bucket we should use a S3ConnectionDef as follows:

    s3conncetion:
      reference: spark

Second inconsistency is in

spark-k8s-operator/rust/crd/src/constants.rs

Lines 20 to 21 in c47e149

	pub const ACCESS_KEY_ID: &str = "accessKeyId";
	pub const SECRET_ACCESS_KEY: &str = "secretAccessKey";

All the operators (and the official concepts docs) use accessKey and secretKey in the credentials secret.
We need to adopt the spark-k8s operator to adhere to the common convention so that S3Connections can be reused across multiple products.

Possible solution

• Use S3ConnectionDef rather then S3BucketDef
• Use accessKey and secretKey rather than accessKeyId and secretAccessKey

Additional context

No response

Environment

No response

Would you like to work on fixing this bug?

No response

[razvan]

Implemented as part of #187