Getting Started with Azure Logic Apps Hybrid on RKE2

With Azure Logic Apps Hybrid, you can run the Logic Apps runtime directly on your Kubernetes cluster while still enjoying the benefits of Azure for authoring, deploying, and monitoring. In our last post, we explored how to set this up using Red Hat OpenShift. Now, we’ll focus on RKE2. We created a single-node RKE2 cluster, assigned an ingress IP using MetalLB, connected it to Azure Arc, and successfully deployed an end-to-end Hybrid Logic App. However, there were several key steps that needed attention, which we will cover here.

If you’ve previously implemented Hybrid setups on AKS or K3s, the process will feel quite familiar. The official requirements document outlines the general process. Since RKE2 is quite similar to standard Kubernetes, most of the integration works seamlessly. However, four aspects require special attention. Three of these involve DNS configuration, which we’ll highlight as we go.

az extension add --name connectedk8s
az extension add --name k8s-extension
az extension add --name customlocation
az extension add --name containerapp

This should only take one command and a few minutes:

curl -sfL https://get.rke2.io | sudo sh -
sudo systemctl enable --now rke2-server.service

# kubectl and kubeconfig will be placed in their location
export PATH=$PATH:/var/lib/rancher/rke2/bin
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
kubectl get nodes          # Should be ready in about 2-5 minutes after the first image pull

By default, RKE2 servers are schedulable, meaning this single node comprises the entire cluster without taints to remove. If you’re planning to use kubectl from your laptop, be sure to include the node’s public IP or hostname in the tls-san of /etc/rancher/rke2/config.yaml before the initial start. Afterward, you can copy the rke2.yaml file and replace 127.0.0.1 with that address.

Quick Sanity Check: Deploy a temporary nginx service as a LoadBalancer: kubectl create deploy nginx --image=nginx && kubectl expose deploy nginx --port=80 --type=LoadBalancer. Initially, the EXTERNAL-IP will display as . This is where MetalLB comes into play in the next step. Don’t forget to clean up the deployment and service after your check.

In this setup, all Logic Apps use a single Envoy ingress, which requires an IP address. While cloud infrastructures handle this automatically, on-premises setups leave you with two choices:

• In-cluster Load Balancer: Use MetalLB (or options like Cilium, kube-vip, OpenELB) to assign an IP from a pool within your node network. Keep envoy.serviceType=LoadBalancer in Step 4, which is what we opted for.
• External L4 Load Balancer in front of NodePort: If you already have an F5, HAProxy, NetScaler, or Azure Standard Load Balancer in place, you can continue using it. Set envoy.serviceType=NodePort (along with envoy.nodeHttpsPort and envoy.nodeHttpPort), and direct your LB to those NodePorts. You’ll need to pass the LB’s virtual IP as --static-ip in Step 6.

We chose to go with MetalLB in L2 mode:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.9/config/manifests/metallb-native.yaml
kubectl wait --for=condition=Ready pods --all -n metallb-system --timeout=180s

Next, you will need to specify a pool from your node subnet and advertise it over L2:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: hybrid-pool
  namespace: metallb-system
spec:
  addresses:
    - 192.168.1.240-192.168.1.250   # Ensure this range is free on your node network
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: hybrid-l2
  namespace: metallb-system
spec:
  ipAddressPools:
    - hybrid-pool

Be cautious with two things: the IP range must be free on your subnet and should not conflict with RKE2’s cluster CIDR (10.42.0.0/16) or service (10.43.0.0/16) ranges. Run the test for the temporary LoadBalancer from Step 1 again; the EXTERNAL-IP should now match one of the addresses from your pool.

Running MetalLB on a single cloud VM? L2 mode broadcasts ARP for the pool IPs on the node’s network, but cloud software-defined networks won’t route an unassigned IP to your NIC from outside. Inside the cluster, it works fine, but to access Envoy externally from the VM, you’ll have to forward the VM’s public IP to the LB IP using socat, or add the pool IP as a secondary interface on your NIC. On true on-prem L2 networks, you won’t need to worry about this.

az connectedk8s connect --name  --resource-group  --location 
kubectl get pods -n azure-arc      # All agents should report Running shortly

If your host has strict egress rules (like in our case with an internal MS VM governed by a denied inbound policy), remember that Arc onboarding requires only outbound connectivity and a local kubeconfig. We executed the connect command on the VM directly using a system-assigned managed identity (az login --identity). There’s no need to open inbound ports or deal with browser prompts.

This step involves activating the extension that will make the cluster a host for Logic Apps. For RKE2, you can use the straightforward command without any Azure.Cluster.Distribution or coreDNSVersion overrides. If you attempt to copy these from an OpenShift guide, the installation may succeed but lead to confusion during operation.

az k8s-extension create \
      --resource-group  --cluster-name  \
      --cluster-type connectedClusters --name logicapps-aca-extension \
      --extension-type Microsoft.App.Environment \
      --release-train stable --auto-upgrade-minor-version true --scope cluster \
      --release-namespace logicapps-aca-ns \
      --configuration-settings "Microsoft.CustomLocation.ServiceAccount=default" \
      --configuration-settings "appsNamespace=logicapps-aca-ns" \
      --configuration-settings "clusterName=" \
      --configuration-settings "keda.enabled=true" \
      --configuration-settings "keda.logicAppsScaler.enabled=true" \
      --configuration-settings "keda.logicAppsScaler.replicaCount=1" \
      --configuration-settings "containerAppController.api.functionsServerEnabled=true" \
      --configuration-settings "functionsProxyApiConfig.enabled=true" \
      --configuration-settings "envoy.serviceType=LoadBalancer"

Initially, our instantiation returned “Succeeded,” but nearly half the pods encountered CrashLoopBackOff. The logs from components like billing and containerapp-controller reported:

The configured user limit (128) on the number of inotify instances has been reached

Since these services are built on .NET, and make use of FileSystemWatcher, the standard setting of fs.inotify.max_user_instances=128 on Ubuntu is insufficient. You must increase it on the node, delete the pods, and then you’re set:

sudo sysctl -w fs.inotify.max_user_instances=8192
sudo sysctl -w fs.inotify.max_user_watches=1048576
echo -e "fs.inotify.max_user_instances=8192\nfs.inotify.max_user_watches=1048576" | sudo tee /etc/sysctl.d/99-inotify.conf
kubectl delete pods --all -n logicapps-aca-ns

After making those adjustments, everything came up cleanly, and Envoy obtained a MetalLB address automatically. To confirm:

az k8s-extension show -g  --cluster-name  \
      --cluster-type connectedClusters --name logicapps-aca-extension \
      --query provisioningState -o tsv          # -> Succeeded
kubectl get svc microsoft-app-environment-k8se-envoy -n logicapps-aca-ns   # Check for EXTERNAL-IP from your pool

Next, you need to enable the custom-locations feature and create a location that links the cluster, extension, and namespace:

az connectedk8s enable-features -n  -g  \
      --features cluster-connect custom-locations \
      --custom-locations-oid $(az ad sp show --id bc313c14-388c-4e7d-a58e-70017303ee3b --query id -o tsv)

EXT_ID=$(az k8s-extension show -g  --cluster-name  \
      --cluster-type connectedClusters --name logicapps-aca-extension --query id -o tsv)
CC_ID=$(az connectedk8s show -g  -n  --query id -o tsv)

az customlocation create \
      --resource-group  --name  \
      --location  --host-resource-id $CC_ID \
      --namespace logicapps-aca-ns --cluster-extension-ids $EXT_ID

Just one note: the extension type may not be available in every Azure region. At the time of writing, it can be found in eastus, westus, westeurope, among others, but not in eastus2. Remember, the region for the connectedCluster resource is independent of your nodes’ location, so select a supported region here.

CL_ID=$(az customlocation show -g  -n  --query id -o tsv)

az containerapp connected-env create \
      --resource-group  --name  \
      --location  --custom-location $CL_ID

Because MetalLB has already assigned an EXTERNAL-IP to the Envoy LoadBalancer service, the platform will automatically detect it. The --static-ip parameter will only be necessary if you opted for the NodePort with external Load Balancer setup in Step 2.

Application hostnames like *...k4apps.io need to resolve to the Envoy ingress from within the cluster. On AKS or OpenShift, az containerapp arc setup-core-dns takes care of that. Unfortunately, RKE2 is not a supported distribution for this command yet, so a manual setup is required, but it’s manageable.

The extension handles the necessary rewrite and forwarding rules within a coredns-custom ConfigMap. However, CoreDNS still needs to mount this ConfigMap and import it from the Corefile.

We initially attempted to patch the Corefile ConfigMap directly, make necessary updates to the Deployment to include the volume, and then restart it. This worked temporarily, but RKE2’s Helm controller would eventually overwrite those changes during restarts. The correct method involves creating a HelmChartConfig. The rke2-coredns chart includes extraConfig (for settings outside the default zone block) as well as extraVolumes/extraVolumeMounts, which is exactly what we need:

# /var/lib/rancher/rke2/server/manifests/rke2-coredns-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-coredns
  namespace: kube-system
spec:
  valuesContent: |-
    extraConfig:
      import:
        parameters: /etc/coredns/custom/*.server
    extraVolumes:
      - name: custom-config-volume
        configMap:
          name: coredns-custom
          optional: true
    extraVolumeMounts:
      - name: custom-config-volume
        mountPath: /etc/coredns/custom

Save this file in /var/lib/rancher/rke2/server/manifests/, or you can also apply it directly using kubectl apply -f (the Helm controller observes both methods). Within a minute, CoreDNS will redeploy, now with the new mount and import settings, ensuring that your configurations persist through restarts and upgrades.

Run the following command to validate the setup by checking DNS resolution:

nslookup test.

If everything is configured correctly, you should receive the microsoft-app-environment-k8se-envoy-internal ClusterIP.

Final DNS Adjustment: You may need to create a kube-dns alias. The Logic Apps platform tries to locate cluster DNS by looking for a service named kube-dns in the kube-system namespace. Since RKE2 uses a different name (rke2-coredns-rke2-coredns) and does not create the alias automatically, the platform fails to identify the DNS, leading to 502 errors on revision calls. To correct this, simply create the alias as follows:

apiVersion: v1
kind: Service
metadata: { name: kube-dns, namespace: kube-system, labels: { k8s-app: kube-dns } }
spec:
  selector: { k8s-app: kube-dns }
  ports:
    - { name: dns, port: 53, protocol: UDP, targetPort: 53 }
    - { name: dns-tcp, port: 53, protocol: TCP, targetPort: 53 }

This is simply a straightforward service; nothing about it will get templated or changed, ensuring it remains intact through reconciles or upgrades.

After ensuring the plumbing is sorted, go to the Azure portal to create a new Logic App (Standard) with the Hybrid hosting option. Connect it to your Azure-managed environment, link it to your SMB share for storage, and you’re ready to start. You can author your workflows as you usually would, and they will execute on your RKE2 cluster instead of Azure directly.