Kubernetes service discovery in SonataFlow
The Kubernetes service discovery allows you to describe the Kubernetes resource you want to perform HTTP requests on using a custom URI. Under the hood, it will discover the network endpoint (URL) to where to make the request.
The Kubernetes service discovery feature works during the workflow application startup, in which this feature scans all the Quarkus configurations in search of the URI pattern. Therefore, you must remember that if the application startup time matters, consider using a known static URL instead.
Following is the custom URI pattern in Kubernetes service discovery:
kubernetes:<kind>.<version>.<group>/<namespace>/<resourceName>?<attributeName>=<attributeValue>
\________/ \____/ \_______/ \_____/ \_________/ \____________/ \______________________________/
scheme kind version group namespace resourceName additional resource attributes
\____________________/ \__________________________/
GVK Supported values:
- port-name={PORT_NAME}
- labels=label-name=label-value;other-label=other-value
The following scheme values are supported in the URI pattern:
-
kubernetes
-
openshift
-
knative
The following resources are supported for the Kubernetes GVK (Group, Version, and Kind):
-
services.v1
-
services.v1.serving.knative.dev
-
pods.v1
-
deployments.v1.apps
-
deploymentconfigs.v1.apps.openshift.io
-
statefulsets.v1.apps
-
routes.v1.route.openshift.io
-
ingresses.v1.networking.k8s.io
When using
The above URI looks directly for services.v1.serving.knative.dev resource. |
The |
- Query parameters in URI
-
Also known as query string. The query parameters are defined the similar way with URLs to assign value to specific attributes.
The following query parameters help the engine to be more precise when querying for a given Kubernetes resource:
-
Custom labels: The custom labels are used to filter services in case there are more than one service with the same label selector but exposing different ports. In this case, you can instruct the engine that if more than one service is found, then the engine must use the service containing the provided label.
The label is defined with the following expression and in case of multiple labels, you can use semicolon (;):
labels=label-name=namevalue;another-label=another-value
Example label definition in URIkubernetes:pods.v1/<namespace>/<pod-name>?labels=label-name=test-label
Using the previous URI example, if there are more than one service exposing the given pod, the
label-name=test-label
label is used to filter the service. If the label does not exist, the first found service is used. -
Custom port name: The custom port name is used to determine which port to use when multiple ports are configured in the target service or container. You can configure the port name to be queried using the following pattern:
port-name=<PORT_NAME>
-
Required Kubernetes roles
The service discovery engine requires that the Kubernetes Service Account running the application has read permissions for the discovered objects.
The following Kubernetes Role resource has all the required permissions for the service discovery to work:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: service-discovery-role
rules:
- apiGroups:
- ""
resources:
- pods
- services
verbs:
- get
- list
- apiGroups:
- apps
resources:
- deployments
- statefulsets
verbs:
- get
- list
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
# Knative
- apiGroups:
- serving.knative.dev
resources:
- services
verbs:
- get
- list
If running on OpenShift, you must also add the following rules:
rules:
- apiGroups:
- route.openshift.io
resources:
- routes
verbs:
- get
- list
- apiGroups:
- apps.openshift.io
resources:
- deploymentconfigs
verbs:
- get
- list
You must then bind the Service Account with the Role
via a RoleBinding
:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: service-discovery-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: service-discovery-role
subjects:
- kind: ServiceAccount
name: default
Usually, pods run using the |
You must create these two objects in every namespace that you wish to deploy Quarkus Workflow applications with Service Discovery enabled.
As an alternative, you can create a ClusterRole
instead of a Role
in every namespace. In this case, the role will reflect globally in your cluster. For security reasons, you should avoid this approach.
Configuration in Kubernetes service discovery
There is no specific configuration required for the Kubernetes service discovery except by using the expected URI pattern. However, the okhttp
communication interceptor, which logs the communication between the application and the Kubernetes API is disabled by default.
You can enable the okhttp
communication interceptor if there is a need to debug the communication between the client and the Kubernetes API by setting the following application property:
okhttp
communication interceptorquarkus.log.category."okhttp3.OkHttpClient".level=INFO
Precedence in Kubernetes service discovery
Based on the resource to be discovered, the Kubernetes service discovery follows specific paths as shown in the following figure:
Example of Kubernetes service discovery in SonataFlow
The Kubernetes service discovery is performed at the STATIC_INIT time of Quarkus during the workflow application startup. First, the service discovery scans the Quarkus configuration values and searches for the Kubernetes URI pattern. If the URI pattern is found, the engine parses the URI, queries the Kubernetes API searching for the given resource, and overrides the given application property.
For example, consider an application that consumes a resource running on Kubernetes. This resource is a Knative service that exposes a function, which can be discovered using the following URI:
org.kie.kogito.sw.knative.service=knative:services.v1/serverless-workflow-greeting-quarkus/greeting-quarkus-cli
The service discovery engine does not read the application property |
Once the workflow application is started, you can see the Kubernetes service discovery into action in the logs:
$ java -jar target/quarkus-app/quarkus-run.jar
__ ____ __ _____ ___ __ ____ ______
--/ __ \/ / / / _ | / _ \/ //_/ / / / __/
-/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/
2022-08-23 12:21:04,360 DEBUG [org.kie.kog.add.qua.k8s.SWDiscoveryConfigSourceInterceptor] (main) Configuring k8s client...
2022-08-23 12:21:05,245 INFO [io.qua.sma.ope.run.OpenApiRecorder] (main) Default CORS properties will be used, please use 'quarkus.http.cors' properties instead
2022-08-23 12:21:05,420 DEBUG [org.kie.kog.add.qua.k8s.par.KubeURI] (main) KubeURI successfully parsed: KubeURI{protocol='knative', gvk=GVK{group='', version='v1', kind='service'}, namespace='serverless-workflow-greeting-quarkus', resourceName='greeting-quarkus-cli'}
2022-08-23 12:21:06,043 INFO [org.kie.kog.add.qua.k8s.KubeResourceDiscovery] (main) Connected to kubernetes cluster v1.23.4, current namespace is serverless-workflow-greeting-quarkus. Resource name for discovery is greeting-quarkus-cli
2022-08-23 12:21:06,045 DEBUG [org.kie.kog.add.qua.k8s.KnativeResourceDiscovery] (main) Trying to adapt kubernetes client to knative
2022-08-23 12:21:06,316 DEBUG [org.kie.kog.add.qua.k8s.KnativeResourceDiscovery] (main) Found Knative endpoint at http://greeting-quarkus-cli.serverless-workflow-greeting-quarkus.10.99.154.147.sslip.io
2022-08-23 12:21:06,375 INFO [org.kie.kog.sw.AppLifecycle] (main) The application is starting
2022-08-23 12:21:06,382 INFO [org.kie.kog.add.qua.mes.com.QuarkusKogitoExtensionInitializer] (main) Registered Kogito CloudEvent extension
2022-08-23 12:21:06,460 INFO [io.quarkus] (main) sw-service-discovery 1.0.0-SNAPSHOT on JVM (powered by Quarkus 999-SNAPSHOT) started in 2.360s. Listening on: http://0.0.0.0:8080
2022-08-23 12:21:06,462 INFO [io.quarkus] (main) Profile prod activated.
2022-08-23 12:21:06,462 INFO [io.quarkus] (main) Installed features: [cache, cdi, jackson-jq, kogito-addon-kubernetes-extension, kogito-addon-messaging-extension, kogito-processes, kogito-serverless-workflow, kubernetes, kubernetes-client, openshift-client, qute, reactive-routes, rest-client, rest-client-jackson, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-openapi, smallrye-reactive-messaging, smallrye-reactive-messaging-http, vertx]
In the previous example, the URI is translated to |
The Kubernetes service discovery scans the Quarkus configuration during the startup, which can also cause small delay as follows:
(main) sw-service-discovery 1.0.0-SNAPSHOT on JVM (powered by Quarkus 999-SNAPSHOT) started in 2.360s. Listening on: http://0.0.0.0:8080
(main) sw-service-discovery 1.0.0-SNAPSHOT on JVM (powered by Quarkus 999-SNAPSHOT) started in 1.507s. Listening on: http://0.0.0.0:8080
Enabling Kubernetes service discovery
When using the Kubernetes service discovery feature, you need to balance if your application can afford the delayed startup time.
If the URI pattern is not found in the application properties, then discovery is not triggered. However, the scanning is performed anyway. Therefore, a short time is required to go through the startup scan.
You can enable the Kubernetes service discovery by adding a service discovery implementation to the application’s dependencies as shown in the following Maven dependencies:
<dependency>
<!-- Service Discovery -->
<dependency>
<groupId>org.kie.kogito</groupId>
<artifactId>kogito-addons-quarkus-kubernetes</artifactId>
</dependency>
<groupId>org.kie.kogito</groupId>
<artifactId>kogito-addons-quarkus-fabric8-kubernetes-service-catalog</artifactId>
</dependency>
Found an issue?
If you find an issue or any misleading information, please feel free to report it here. We really appreciate it!