Skip to content

Commit 0900249

Browse files
abhinavdahiyaabhinavdahiya
committed
docs: add details on install time customization for machine configuration
The docs include an example for removing `NoSchedule` taint on masters to allow for single master cluster. Although the changes described in the example are not completely sufficient to achieve the single master cluster, as few operators specifcally set workloads to worker nodes. I chose this example because the Code Ready Containers team will be using somehting similar to achieve single node libvirt cluster.
1 parent 9596c7f commit 0900249

File tree

1 file changed

+169
-0
lines changed

1 file changed

+169
-0
lines changed

‎docs/user/customization.md

Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,175 @@ The `manifests` target will render the manifest templates and output the result
2727

2828
The `manifest-templates` target will output the unrendered manifest templates into the asset directory. This allows modification to the templates before they have been rendered, which may be useful to users who wish to reuse the templates between cluster deployments.
2929

30+
### Install Time Customization for Machine Configuration
31+
32+
**IMPORTANT**:
33+
34+
- These customizations require using the `manifests` target that does not provide compatibility guarantees, for more information [check here](versioning.md#versioning).
35+
- This can affect upgradability of your cluster as the `machine-config-operator` can mark clusters tainted when user defined [MachineConfig][machine-config] objects are present in the cluster.
36+
37+
In most cases, user applications should be run on the cluster via Kubernetes workload objects (e.g. DaemonSet, Deployment, etc). For example, DaemonSets are the most stable way to run a logging agent on all hosts. However, there may be some cases where these workloads need to be executed prior to the node joining the Kubernetes cluster. For example, a compliance mandate like "the user must run auditing tools as soon as the operating system comes up" might require a custom systemd unit for an auditing container in the Ignition config for some or all nodes.
38+
39+
The configuration of machines in OpenShift is controlled using `MachineConfig` objects and what configuration is applied to a machine in the OpenShift cluster is based on the [MachineConfigPool][machine-config-pool] objects. To allow customization of machine configuration which is not possible as Day 2 operation, the installer allows users to bring their own custom `MachineConfig` objects.
40+
41+
1. `openshift-install --dir $INSTALL_DIR create manifests`
42+
43+
2. Copy files with `MachineConfig` objects to `$INSTALL_DIR/openshift/` directory.
44+
45+
These custom `MachineConfig` objects are black boxes to the installer and the installer only plays the role of `oc create -f <custom-machine-config-object>` early enough into cluster bootstrap to make sure the configuration is used by the [MachineConfigOperator][machine-config-operator].
46+
47+
3. `openshift-install --dir $INSTALL_DIR create cluster`
48+
49+
#### Control plane with no Taints
50+
51+
All control plane nodes by default register with a taint `node-role.kubernetes.io/master=:NoSchedule` making them unschedulable by most normal workloads. An installation that requires the control plane to boot without that taint can push a custom `MachineConfig` object with a `kubelet.service` that doesn't include the taint.
52+
53+
For example:
54+
55+
1. Run `manifests` target to create all the manifests.
56+
57+
```console
58+
$ mkdir no-taint-cluster
59+
60+
$ cp aws-install-config.yaml no-taint-cluster/install-config.yaml
61+
62+
$ openshift-install --dir no-taint-cluster create manifests
63+
INFO Consuming "Install Config" from target directory
64+
65+
$ ls -l no-taint-cluster/**
66+
no-taint-cluster/manifests:
67+
total 68
68+
-rw-r--r--. 1 xxxxx xxxxx 169 Feb 28 10:54 04-openshift-machine-config-operator.yaml
69+
-rw-r--r--. 1 xxxxx xxxxx 1589 Feb 28 10:54 cluster-config.yaml
70+
-rw-r--r--. 1 xxxxx xxxxx 149 Feb 28 10:54 cluster-dns-02-config.yml
71+
-rw-r--r--. 1 xxxxx xxxxx 243 Feb 28 10:54 cluster-infrastructure-02-config.yml
72+
-rw-r--r--. 1 xxxxx xxxxx 154 Feb 28 10:54 cluster-ingress-02-config.yml
73+
-rw-r--r--. 1 xxxxx xxxxx 557 Feb 28 10:54 cluster-network-01-crd.yml
74+
-rw-r--r--. 1 xxxxx xxxxx 327 Feb 28 10:54 cluster-network-02-config.yml
75+
-rw-r--r--. 1 xxxxx xxxxx 264 Feb 28 10:54 cvo-overrides.yaml
76+
-rw-r--r--. 1 xxxxx xxxxx 275 Feb 28 10:54 etcd-service.yaml
77+
-rw-r--r--. 1 xxxxx xxxxx 283 Feb 28 10:54 host-etcd-service-endpoints.yaml
78+
-rw-r--r--. 1 xxxxx xxxxx 268 Feb 28 10:54 host-etcd-service.yaml
79+
-rw-r--r--. 1 xxxxx xxxxx 118 Feb 28 10:54 kube-cloud-config.yaml
80+
-rw-r--r--. 1 xxxxx xxxxx 1299 Feb 28 10:54 kube-system-configmap-etcd-serving-ca.yaml
81+
-rw-r--r--. 1 xxxxx xxxxx 1304 Feb 28 10:54 kube-system-configmap-root-ca.yaml
82+
-rw-r--r--. 1 xxxxx xxxxx 3877 Feb 28 10:54 kube-system-secret-etcd-client.yaml
83+
-rw-r--r--. 1 xxxxx xxxxx 4030 Feb 28 10:54 machine-config-server-tls-secret.yaml
84+
-rw-r--r--. 1 xxxxx xxxxx 856 Feb 28 10:54 pull.json
85+
86+
no-taint-cluster/openshift:
87+
total 28
88+
-rw-r--r--. 1 xxxxx xxxxx 293 Feb 28 10:54 99_binding-discovery.yaml
89+
-rw-r--r--. 1 xxxxx xxxxx 181 Feb 28 10:54 99_kubeadmin-password-secret.yaml
90+
-rw-r--r--. 1 xxxxx xxxxx 330 Feb 28 10:54 99_openshift-cluster-api_cluster.yaml
91+
-rw-r--r--. 1 xxxxx xxxxx 1015 Feb 28 10:54 99_openshift-cluster-api_master-machines-0.yaml
92+
-rw-r--r--. 1 xxxxx xxxxx 2655 Feb 28 10:54 99_openshift-cluster-api_master-user-data-secret.yaml
93+
-rw-r--r--. 1 xxxxx xxxxx 1750 Feb 28 10:54 99_openshift-cluster-api_worker-machineset.yaml
94+
-rw-r--r--. 1 xxxxx xxxxx 2655 Feb 28 10:54 99_openshift-cluster-api_worker-user-data-secret.yaml
95+
```
96+
97+
2. Create a `MachineConfig` that includes `kubelet.service` that has no taints.
98+
99+
```sh
100+
cat > no-taint-cluster/openshift/99-master-kubelet-no-taint.yaml <<EOF
101+
apiVersion: machineconfiguration.openshift.io/v1
102+
kind: MachineConfig
103+
metadata:
104+
labels:
105+
machineconfiguration.openshift.io/role: master
106+
name: 02-master-kubelet
107+
spec:
108+
config:
109+
ignition:
110+
version: 2.2.0
111+
systemd:
112+
units:
113+
- contents: |
114+
[Unit]
115+
Description=Kubernetes Kubelet
116+
Wants=rpc-statd.service
117+
118+
[Service]
119+
Type=notify
120+
ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests
121+
ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state
122+
EnvironmentFile=-/etc/kubernetes/kubelet-workaround
123+
EnvironmentFile=-/etc/kubernetes/kubelet-env
124+
125+
ExecStart=/usr/bin/hyperkube \
126+
kubelet \
127+
--config=/etc/kubernetes/kubelet.conf \
128+
--bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
129+
--rotate-certificates \
130+
--kubeconfig=/var/lib/kubelet/kubeconfig \
131+
--container-runtime=remote \
132+
--container-runtime-endpoint=/var/run/crio/crio.sock \
133+
--allow-privileged \
134+
--node-labels=node-role.kubernetes.io/master \
135+
--minimum-container-ttl-duration=6m0s \
136+
--client-ca-file=/etc/kubernetes/ca.crt \
137+
--cloud-provider=aws \
138+
--volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec \
139+
\
140+
--anonymous-auth=false \
141+
142+
Restart=always
143+
RestartSec=10
144+
145+
[Install]
146+
WantedBy=multi-user.target
147+
enabled: true
148+
name: kubelet.service
149+
EOF
150+
```
151+
152+
`machineconfiguration.openshift.io/role: master` label attaches this `MachineConfig` to the [master][master-machine-config-pool] `MachineConfigPool`. The [default][default-kubelet-service] configuration for the `kubelet.service` on libvirt includes the taint.
153+
154+
3. Run `cluster` target to create the cluster using the custom manifests.
155+
156+
```console
157+
$ openshift-install --dir no-taint-cluster create cluster
158+
INFO Consuming "Openshift Manifests" from target directory
159+
INFO Consuming "Master Machines" from target directory
160+
INFO Consuming "Common Manifests" from target directory
161+
INFO Creating cluster...
162+
INFO Waiting up to 30m0s for the Kubernetes API...
163+
...
164+
```
165+
166+
Check that no control plane nodes registered with taints:
167+
168+
```console
169+
$ oc --config no-taint-cluster/auth/kubeconfig get nodes -ojson | jq '.items[] | select(.metadata.labels."node-role.kubernetes.io/master" == "") | .spec.taints'
170+
null
171+
```
172+
173+
Check that the `02-master-kubelet` `MachineConfig` exists in the cluster:
174+
175+
```console
176+
oc --config no-taint-cluster/auth/kubeconfig get machineconfigs
177+
NAME GENERATEDBYCONTROLLER IGNITIONVERSION CREATED
178+
00-master 3.11.0-744-g5b05d9d3-dirty 2.2.0 137m
179+
00-master-ssh 3.11.0-744-g5b05d9d3-dirty 137m
180+
00-worker 3.11.0-744-g5b05d9d3-dirty 2.2.0 137m
181+
00-worker-ssh 3.11.0-744-g5b05d9d3-dirty 137m
182+
01-master-container-runtime 3.11.0-744-g5b05d9d3-dirty 2.2.0 137m
183+
01-master-kubelet 3.11.0-744-g5b05d9d3-dirty 2.2.0 137m
184+
02-master-kubelet 2.2.0 137m
185+
01-worker-container-runtime 3.11.0-744-g5b05d9d3-dirty 2.2.0 137m
186+
01-worker-kubelet 3.11.0-744-g5b05d9d3-dirty 2.2.0 137m
187+
99-master-3c81ffa3-3b8d-11e9-ac1e-52fdfc072182-registries 3.11.0-744-g5b05d9d3-dirty 133m
188+
99-worker-3c83a226-3b8d-11e9-ac1e-52fdfc072182-registries 3.11.0-744-g5b05d9d3-dirty 133m
189+
master-55491738d7cd1ad6c72891e77c35e024 3.11.0-744-g5b05d9d3-dirty 2.2.0 137m
190+
worker-edab0895c59dba7a566f4b955d87d964 3.11.0-744-g5b05d9d3-dirty 2.2.0 137m
191+
```
192+
193+
[default-kubelet-service]: https://github.com/openshift/machine-config-operator/blob/master/templates/master/01-master-kubelet/_base/units/kubelet.yaml
194+
[machine-config-operator]: https://github.com/openshift/machine-config-operator#machine-config-operator
195+
[machine-config-pool]: https://github.com/openshift/machine-config-operator/blob/master/docs/MachineConfigController.md#machinepool
196+
[machine-config]: https://github.com/openshift/machine-config-operator/blob/master/docs/MachineConfiguration.md
197+
[master-machine-config-pool]: https://github.com/openshift/machine-config-operator/blob/master/manifests/master.machineconfigpool.yaml
198+
30199
## OS Customization (unvalidated)
31200

32201
In rare circumstances, certain modifications to the bootstrap and other machines may be necessary. The installer provides the "ignition-configs" target, which allows arbitrary modification to the [Ignition Configs][ignition] used to boot these machines. Note that there is currently no validation on the modifications that are made, so it is possible that the changes will result in a non-functioning cluster.

0 commit comments

Comments
 (0)