hazelcast.xml - Kubernetes İçin Network Join Ayarları

Giriş

Açıklaması şöyle

For production environments, the usage of UDP is not the best choice. So, among others, Hazelcast supports clusters which are deployed in Kubernetes environment without multicasting. For the detailed information, look into the documentation on Configuring Kubernetes. To activate the discovery on Kubernetes, set the properties as follows:

1. Bir kubernetes servisi yaratılır

2. Bu servise kubernetes tag içinde atıfta bulunulur

Örnek

Şöyle yaparız

<kubernetes enabled="true">
  <namespace>bjy</namespace>
  <service-name>hazelcast-service-lb</service-name>
</kubernetes>

Örnek

Şöyle yaparız. Burada ismi hazelcast olan bir servise atıfta bulunuluyor

hazelcast:
  instance-name: users-app
  cluster-name: users-app
  network:
    join:
      multicast:
        enabled: false
      kubernetes:
        enabled: true
        service-name: hazelcast

Açıklaması şöyle

This disables the multicast, enables Kubernetes discovery, and defines the name of the service hazelcast, which will be used to search for cluster members via Kubernetes API. Then, we need to create this service, which can be declared as the next

Şöyle yaparız

apiVersion: v1
kind: Service
metadata:
  name: hazelcast
  labels:
    app: hazelcast
spec:
  ports:
    - port: 5701
      protocol: TCP
  selector:
    app: app-users
  type: ClusterIP

Açıklaması şöyle

To allow Hazelcast to use the service inside Kubernetes for the discovery, we also need to grant certain permissions. An example of RBAC configuration for default namespace you can find in Hazelcast documentation.

Şöyle yaparız. Burada 'default' namespace içindeki 'default' service account'a ClusterRole bağlanıyor. ClusterRole ile get ve list verb'lerini çalıştırabiliyor.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: hazelcast-cluster-role
rules:
  - apiGroups:
      - ""
    resources:
      - endpoints
      - pods
      - nodes
      - services
    verbs:
      - get
      - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: hazelcast-cluster-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: hazelcast-cluster-role
subjects:
  - kind: ServiceAccount
    name: default
    namespace: default

hazelcast.stale.join.prevention.duration.seconds Alanı

Örnek

5 yapılması öneriliyor. Şöyle yaparız

hazelcast.stale.join.prevention.duration.seconds=5'

Açıklaması şöyle

A member that’s starting up looks for candidate members according to its network join config. Once it has established an address for the cluster’s master, it issues join requests. It can be the case that the currently starting member sends out more than 1 request until master processes its join request and it becomes a member of the cluster.

A “stale join request” is one that is processed by master member after the sender is already in master’s cluster members list, so it is ignored. Master member maintains a list of “recently joined members” for this purpose.

How does this relate to kubernetes? With k8s managing pods and persistence enabled (so member has a persistent identity), it can be the case that this happens:

• a pod is started, hazelcast member joins but fails startup shortly after joining.
• k8s reschedules the failed pod quickly (shorter than “stale join prevention duration”)
• new pod starts but master member ignores the join requests -> probably the member decides to start as standalone or it fails after cluster join timeout. In both cases, bad things happen.

To avoid such a situation, it is recommended to lower the stale join prevention duration.