(DK) OPENSHIFT 4.7 STEP by STEP 구성

OPENSHIFT는 설치 후 정리하는 것도 더 힘든거 같다. 워낙 여러가지를 만져야 되고.. 간편하게 설정 하는 

openshift-install create cluster --dir ocp 라는 명령어를 치고 나서 바로 되면 좋겠지만.. 원인 모를 이유로 설치가 잘 되지 않아서 아래와 같은 방법으로 결국엔 할 수 밖에 없었다.

 

참고는 블로그 및 유투브를 참고를 많이 했다... 참 어렵네 오픈시프트..

 

1. DNS 구성

		Cluster Name	Base Domain	IP
	api	openshift	vcf.local	10.253.107.152
	api-int	openshift	vcf.local	10.253.107.152
*	apps	openshift	vcf.local	10.253.107.152
	bootstrap	openshift	vcf.local	10.253.107.10
	master0	openshift	vcf.local	10.253.107.11
	master1	openshift	vcf.local	10.253.107.12
	master2	openshift	vcf.local	10.253.107.13
	worker0	openshift	vcf.local	10.253.107.14
	worker1	openshift	vcf.local	10.253.107.15
	worker2	openshift	vcf.local	10.253.107.16

2. DHCP 구성을 위해 NSXT 설정

DHCP 구성 시 정적 바인딩 할 IP 대역은 DHCP범위에서 제거 후 DHCP 정적 바인딩 설정

10.253.107.20-10.253.107.199

DHCP 정적 바인등 클릭

정적 바인딩 MAC 등록 및 IP 구성

3. 도커 설치

sudo yum install -y yum-utils

sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

sudo yum install docker-ce docker-ce-cli containerd.io -y

systemctl start docker

systemctl enable docker

4. 인증관련

ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_rsa

eval "$(ssh-agent -s)"

ssh-add ~/.ssh/id_rsa

## vCenter에서 인증서 다운로드 후 신뢰 인증서 등록

mv ca.crt /usr/local/share/ca-certificates/

update-ca-certificates

신뢰할 수 있는 루트 CA인증서 다운로드 받으면, LINUX, MAC, WINDOWS 3가지 나옴 OS맞는걸 선택 하면 됨

다운로드 받는곳

4. 오픈시프트 구성 파일 다운로드

### wget으로 받아도 되지만 어디서 받는지 알기 위해서 캡처 첨부 함

mkdir -p /var/tmp/openshift
cd /var/tmp/openshift
 
wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-linux.tar.gz

wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-install-linux.tar.gz

tar -xzvf openshift-client-linux.tar.gz

tar -xzvf openshift-install-linux.tar.gz

mv oc kubectl openshift-install /usr/local/bin/

웹 페이지 접속 

Cluster > Datacenter > vSphere

Recommanded

OpenShift Installer , Command line interface를 각 각 다운로드 받는다

Download pull secret 은 복사를 해서 잘 저장해 둔다.

5. NGINX LB 구성

필자는 귀찮아서 기 만들었던 NGINX를 사용 했지만, NSX를 사용해도 됨.

(DK) NGINX LB (TCP/UDP) 구성

6. 오픈시프트 설치

## SELINUX & FIREWALLD DISABLED

vi /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

reboot

systemctl stop firewalld
systemctl disable firewalld


openshift-install create install-config --dir ocp

vi ocp/install-config
apiVersion: v1
baseDomain: vcf.local
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform: {}
  replicas: 0 ## 0으로 변경
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 3
metadata:
  creationTimestamp: null
  name: openshift
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:       ## 삭제
  - cidr: 10.0.0.0/16   ## 삭제
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  vsphere:
    folder: "/OBDC/vm/openshift" ### 추가
    apiVIP: 10.253.107.253   ## 삭제
    cluster: OBCLUSTER
    datacenter: OBDC
    defaultDatastore: vsanDatastore
    ingressVIP: 10.253.107.254    ## 삭제
    network: LS-OPENSHIFT-MGMT-10.253.107.x
    password: Openbase!234
    username: administrator@vsphere.local
    vCenter: vcsa01.vcf.local
  fips: false   ## 추가
{ PULL SECRET }
{ SSH KEY }


## Manifests 실행
openshift-isntall create manifests --dir ocp

cd ~/ocp/openshift
rm -rf 99_openshift-cluster-api_master-*
rm -rf 99_openshift-cluster-api_worker-machineset-0.yaml

cd ~/ocp/manifests/

vi cluster-scheduler-02-config.yml

apiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
  creationTimestamp: null
  name: cluster
spec:
  mastersSchedulable: false    ### true > false change
  policy:
    name: ""
status: {}


openshift-install create ignition-configs --dir ocp

cd ocp

base64 -w0 master.ign > master.64
base64 -w0 worker.ign > worker.64


cat << EOF | tee append-bootstrap.ign
{
  "ignition": {
    "config": {
      "merge": [
        {
          "source": "http://10.253.107.152:8080/bootstrap.ign"
        }
      ] 
    },
    "version": "3.1.0"
  }
}
EOF

base64 -w0 append-bootstrap.ign > append-bootstrap.64


cp *.ign /usr/share/nginx/html/files/

chmod 644 /usr/share/nginx/html/files/*.ign

7. VM 생성

https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/latest/

Index of /pub/openshift-v4/dependencies/rhcos/latest/latest

다운로드 후 OVA 배포

8. TEMP로 MASTER, WORKER, BOOTSTRAP 복제

MASTER 생성 (모든 VM 생성은 openshift 폴더로 지정)

 

cd ~/ocp/

cat master.64

cat worker.64

Master는 Master의 맞는 master.64를 , worker.64 , bootstrap은 append-bootstrap.64 값을 넣으면 된다.


### 각 VM에 고급 설정 ## 예시;
guestinfo.ignition.config.data : eyJpZ25pdGlvbiI6eyJjb25maWciOnsibWVyZ2UiOlt7InNvdXJjZSI6Imh0dHBzOi8vYXBpLWludC5vcGVuc2hpZnQudmNmLmxvY2FsOjIyNjIzL2NvbmZpZy9tYXN0ZXIifV19LCJzZWN1cml0eSI6eyJ0bHMiOnsiY2VydGlmaWNhdGVBdXRob3JpdGllcyI6W3sic291cmNlIjoiZGF0YTp0ZXh0L3BsYWluO2NoYXJzZXQ9dXRmLTg7YmFzZTY0LExTMHRMUzFDUlVkSlRpQkRSVkpVU1VaSlEwRlVSUzB0TFMwdENrMUpTVVJGUkVORFFXWnBaMEYzU1VKQlowbEpVa2hLTTFVeE9IcEJlVUYzUkZGWlNrdHZXa2xvZG1OT1FWRkZURUpSUVhkS2FrVlRUVUpCUjBFeFZVVUtRM2hOU21JelFteGliazV2WVZkYU1FMVNRWGRFWjFsRVZsRlJSRVYzWkhsaU1qa3dURmRPYUUxQ05GaEVWRWw0VFVSamQwNXFSVE5OUkVGNlRURnZXQXBFVkUxNFRVUmpkMDVFUlROTlJFRjZUVEZ2ZDBwcVJWTk5Ra0ZIUVRGVlJVTjRUVXBpTTBKc1ltNU9iMkZYV2pCTlVrRjNSR2RaUkZaUlVVUkZkMlI1Q21JeU9UQk1WMDVvVFVsSlFrbHFRVTVDWjJ0eGFHdHBSemwzTUVKQlVVVkdRVUZQUTBGUk9FRk5TVWxDUTJkTFEwRlJSVUYzU1VWaVJFZHBUbmtyU0RRS05GRmpTRGRyTjFwSVVqVmpNRUZzUWpadlMzazRRM0UxZW1NM05VSkRMekZIV0VSQ05rZHVOMU13U1ZCaGQxQnZMemxyU0d4SlVYWm1TM0JyYlU1MlNRcElkbUpOWlhSNk5YUndlakpPYzNrNVYwdHFURmRXYXpFNFZYY3hObUpxZUZaUGRFSTFjVkV3VGpFNVMzRkJXRUp5WVVoelRXbHdlakpuTmxCTFMySjRDbnBWWXpJdldFTXhXVGx0YXpCUFJqQTNVblV2UjIxQ2FrazNSWFZPYkdFNFZXbEJOa1ZCZVVJMFVWUnhORUZTVGxWNE9GbFlZbEpDTDJ0TldHUkNlV1VLYVU5blJWaHZSazVKZFhwd1NVMUdibTlvWmxoalpuTnFiRUptTnpaSmFEUlhNSGR4YjNwQ1pXaHFiMWRFTW5velEzcEtUVVpqV25OWlJHMUROSG94TndvNGIxWTJRbUp5TVhVMU1GQmlUVmhCVG5obVUyZHhSUzlMWXpKbmFHbG9OVTgyVW10TU1YVmxlVmh6T0ZsbU9XcEJaRWt2TTNOeVFqTjRWSEpvVEdaekNtRlFaV012VG5jMlZGRkpSRUZSUVVKdk1FbDNVVVJCVDBKblRsWklVVGhDUVdZNFJVSkJUVU5CY1ZGM1JIZFpSRlpTTUZSQlVVZ3ZRa0ZWZDBGM1JVSUtMM3BCWkVKblRsWklVVFJGUm1kUlZYWkRRU3RuTURaa01FVlpVemhrVnpCRlpEbGtXR3BOYVVFdlJYZEVVVmxLUzI5YVNXaDJZMDVCVVVWTVFsRkJSQXBuWjBWQ1FVeG5TbVpEWW5VMVNFMXdWMmh4Y0hOVlIyMWxhbk5FYWpCRWVERmpkRE5UTVhZMVptUnRWRVZuUW5SMVYyNDJSR3R2VmxkaVUzUXZOVE15Q2psMWFsVkhielZoT1hKUVlXdzBZVVJJVVd4WE1GVXdielkxY0VsbVNTOHhVa3BEZDFSblYxb3ZaakpyVEV4dldVSXhRbmhCYzNkTGRqZHFRWFYzVlRnS1FscEZZV1oxVEc5WFpWZzJZMWRxY0ZFM2N6aFFkMWxUYlhBMlJXOVdNa3RyT1hKWFlraHhZMGhuVUZScVJ6Qlhaa1J0YVRKTU9FeGllVTlyTkVodmRBcHhWMGRhU0M5clRuQXdWMHcyTjFkNmRIVkhkamhPY1d4Qk5qVjZhbnBCWlZsRUwwZDRabTE1TVdkelUzYzVSblZuTkZjMllscFFVSGswVUVkVVVIRnJDblpUVDJkVFkxRjJNSFV2TWtsMVQxaHpWR0p1ZVROeE5ERlNNbEY1YkZOWFZuZFNNVVkwYXpONFZtUkpXRkZrU1ZGMGRIQXhUVzFFY25wMmJEWktUallLYUVWcFYxWkZNVGhxZVZKeWVuVTJVWGhZUzBGVVpqaG1SbEpGUFFvdExTMHRMVVZPUkNCRFJWSlVTVVpKUTBGVVJTMHRMUzB0Q2c9PSJ9XX19LCJ2ZXJzaW9uIjoiMy4yLjAifX0=
guestinfo.ignition.config.data.encoding : base64
disk.EnableUUID : TRUE

위에 처럼 모두 배포 하면 아래와 같이 나옴

BOOTSTRAP이 정상적으로 되면 IP 및 HOSTNAME을 가지고 온다.

JUMPHOST에서

ssh core@10.253.107.10  접속 한 후 아래처럼 완료가 되며

 

export KUBECONFIG=~/ocp/auth/kubeconfig


## 확인
oc get nodes

 

아래가 모두 True로 나와야 한다.

oc get co

이제 나머지 Worker NODE를 POWER ON 한다. 

정상 부팅 화면

그리고 워커노드 부팅 후 연동 하는 것은 좀 오래 걸린다.

watch -n 5 "oc get csr | grep Pending"

oc adm certificate approve csr-ddq25 csr-k4xst csr-mlp4t

oc adm certificate approve csr-77hvg csr-878fh csr-jjxzc

watch -n 5 oc get nodes

오래 기다리면 NotReady에서 Ready로 변경됨 (위에 등록 하고 5분정도 걸렸음)

oc get --all-namespaces -o jsonpath='{range.items[*]}{range .status.ingress[*]}{.host}{"\n"}{end}{end}' routes

oc get route -n openshift-console

 

https://console-openshift-console.apps.openshift.vcf.local

cat ~/ocp/auth/kubeadmin-password

로그인 화면

인터넷이 되어 있으면 레드햇 접속 하면 아래처럼 확인이 가능하다.