Kubernetes service discovery in Serverless Workflow

The Kubernetes service discovery allows you to have a static URI, defining a Kubernetes resource, which is used to perform HTTP requests. The Kubernetes resource defined in the URI is queried in the current Kubernetes cluster and translated in a valid URL.

The Kubernetes service discovery feature works during the workflow application startup, in which this feature scans all the Quarkus configuration in search of the URI pattern. Therefore, you must keep in mind that if application startup time matters, then consider to use a static URL instead.

Following is the URI pattern in Kubernetes service discovery:

URI pattern in Kubernetes service discovery

kubernetes:<group>/<version>/<kind>/<namespace>/<resourceName>?<attributeName>=<attributeValue>
\________/ \_____/ \_______/ \____/ \_________/ \____________/ \______________________________/
  scheme    group   version   kind   namespace   resourceName   additional resource attributes
            \____________________/                               \__________________________/
                     GVK                                           Supported values:
                                                                   - port-name={PORT_NAME}
                                                                   - labels=label-name=label-value;other-label=other-value

The following scheme values are supported in the URI pattern:

kubernetes
openshift
knative

The following resources are supported for the Kubernetes GVK (Group, Version, and Kind):

v1/service
serving.knative.dev/v1/service
v1/pod
apps/v1/deployment
apps.openshift.io/v1/deploymentconfig
apps/v1/statefulset
route.openshift.io/v1/route
networking.k8s.io/v1/ingress

The Kubernetes GVK is mandatory and cannot be empty. In case the Kubernetes GVK is empty, the engine throws a validation issue and discovery is not performed. The last value in the URI is always considered as resource name.

The namespace in the URI is optional, however, if namespace contains an empty value, the current namespace or context is used.

Query parameters in URI

Also known as query string. The query parameters are defined the similar way with URLs to assign value to specific attributes.

The following query parameters help the engine to be more precise when querying for a given Kubernetes resource:

Custom labels: The custom labels are used to filter services in case there are more than one service with the same label selector but exposing different ports. In this case, you can instruct the engine that if more than one service is found, then the engine must use the service containing the provided label.

The label is defined with the following expression and in case of multiple labels, you can use semicolon (;):

labels=label-name=namevalue;another-label=another-value
Example label definition in URI
```
kubernetes:v1/pod/<namespace>/<pod-name>?labels=label-name=test-label
```
Using the previous URI example, if there are more than one service exposing the given pod, the label-name=test-label label is used to filter the service. If the label does not exist, the first found service is used.
Custom port name: The custom port name is used to determine which port to use when multiple ports are configured in the target service or container. You can configure the port name to be queried using the following pattern:

port-name=<PORT_NAME>

Configuration in Kubernetes service discovery

There is no specific configuration required for the Kubernetes service discovery except by using the expected URI pattern. However, the okhttp communication interceptor, which logs the communication between the application and the Kubernetes API is disabled by default.

You can enable the okhttp communication interceptor if there is a need to debug the communication between the client and the Kubernetes API by setting the following application property:

Application property to enable okhttp communication interceptor

quarkus.log.category."okhttp3.OkHttpClient".level=INFO

Precedence in Kubernetes service discovery

Based on the resource to be discovered, the Kubernetes service discovery follows specific paths as shown in the following figure:

Example of Kubernetes service discovery in Serverless Workflow

The Kubernetes service discovery is performed at the STATIC_INIT time of Quarkus during the workflow application startup. First, the service discovery scans the Quarkus configuration values and searches for the Kubernetes URI pattern. If the URI pattern is found, the engine parses the URI, queries the Kubernetes API searching for the given resource, and overrides the given application property.

For example, consider an application that consumes a resource running on Kubernetes. This resource is a Knative service that exposes a function, which can be discovered using the following URI:

Example URI

org.kie.kogito.sw.knative.service=knative:v1/Service/serverless-workflow-greeting-quarkus/greeting-quarkus-cli

The service discovery engine does not read the application property name, but only value.

Once the workflow application is started, you can see the Kubernetes service discovery into action in the logs:

Example logs

$ java -jar target/quarkus-app/quarkus-run.jar
__  ____  __  _____   ___  __ ____  ______
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/
2022-08-23 12:21:04,360 DEBUG [org.kie.kog.add.qua.k8s.SWDiscoveryConfigSourceInterceptor] (main) Configuring k8s client...
2022-08-23 12:21:05,245 INFO  [io.qua.sma.ope.run.OpenApiRecorder] (main) Default CORS properties will be used, please use 'quarkus.http.cors' properties instead
2022-08-23 12:21:05,420 DEBUG [org.kie.kog.add.qua.k8s.par.KubeURI] (main)  KubeURI successfully parsed: KubeURI{protocol='knative', gvk=GVK{group='', version='v1', kind='service'}, namespace='serverless-workflow-greeting-quarkus', resourceName='greeting-quarkus-cli'}
2022-08-23 12:21:06,043 INFO  [org.kie.kog.add.qua.k8s.KubeResourceDiscovery] (main) Connected to kubernetes cluster  v1.23.4, current namespace is serverless-workflow-greeting-quarkus. Resource name for discovery is greeting-quarkus-cli
2022-08-23 12:21:06,045 DEBUG [org.kie.kog.add.qua.k8s.KnativeResourceDiscovery] (main) Trying to adapt kubernetes client to knative
2022-08-23 12:21:06,316 DEBUG [org.kie.kog.add.qua.k8s.KnativeResourceDiscovery] (main) Found Knative endpoint at http://greeting-quarkus-cli.serverless-workflow-greeting-quarkus.10.99.154.147.sslip.io
2022-08-23 12:21:06,375 INFO  [org.kie.kog.sw.AppLifecycle] (main) The application is starting
2022-08-23 12:21:06,382 INFO  [org.kie.kog.add.qua.mes.com.QuarkusKogitoExtensionInitializer] (main) Registered Kogito CloudEvent extension
2022-08-23 12:21:06,460 INFO  [io.quarkus] (main) sw-service-discovery 1.0.0-SNAPSHOT on JVM (powered by Quarkus 999-SNAPSHOT) started in 2.360s. Listening on: http://0.0.0.0:8080
2022-08-23 12:21:06,462 INFO  [io.quarkus] (main) Profile prod activated.
2022-08-23 12:21:06,462 INFO  [io.quarkus] (main) Installed features: [cache, cdi, jackson-jq, kogito-addon-kubernetes-extension, kogito-addon-messaging-extension, kogito-processes, kogito-serverless-workflow, kubernetes, kubernetes-client, openshift-client, qute, reactive-routes, rest-client, rest-client-jackson, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-openapi, smallrye-reactive-messaging, smallrye-reactive-messaging-http, vertx]

In the previous example, the URI is translated to http://greeting-quarkus-cli.serverless-workflow-greeting-quarkus.10.99.154.147.sslip.io when the application started. The translated URI is used at runtime, when needed.

The Kubernetes service discovery scans the Quarkus configuration during the startup, which can also cause small delay as follows:

Kubernetes service discovery is enabled
Kubernetes service discovery is disabled

(main) sw-service-discovery 1.0.0-SNAPSHOT on JVM (powered by Quarkus 999-SNAPSHOT) started in 2.360s. Listening on: http://0.0.0.0:8080

(main) sw-service-discovery 1.0.0-SNAPSHOT on JVM (powered by Quarkus 999-SNAPSHOT) started in 1.507s. Listening on: http://0.0.0.0:8080

When using the Kubernetes service discovery feature, you need to balance if your application can afford the delayed startup time.

If the URI pattern is not found in the application properties, then discovery is not triggered. However, the scanning is performed anyway. Therefore, a short time is required to go through the startup scan.

You can disable the Kubernetes service discovery by excluding the kogito-addons-quarkus-kubernetes library from the classpath of the application as shown in the following Maven exclusions:

Example Maven exclusions

<dependency>
  <groupId>org.kie.kogito</groupId>
  <artifactId>kogito-quarkus-serverless-workflow</artifactId>
  <exclusions>
    <exclusion>
      <groupId>org.kie.kogito</groupId>
      <artifactId>kogito-addons-quarkus-kubernetes</artifactId>
    </exclusion>
    <exclusion>
      <groupId>org.kie.kogito</groupId>
      <artifactId>kogito-addons-quarkus-kubernetes-deployment</artifactId>
    </exclusion>
  </exclusions>
</dependency>

Additional resources

Configuring the OpenAPI services endpoints

Found an issue?

If you find an issue or any misleading information, please feel free to report it here. We really appreciate it!