Troubleshooting a service network
Typically, you can create a service network without referencing this troubleshooting guide. However, this guide provides some tips for situations when the service network does not perform as expected.
See Resolving common problems if you have encountered a specific issue using the skupper
CLI.
A typical troubleshooting workflow is to check all the sites and create debug tar files.
Checking sites
Using the skupper
command-line interface (CLI) provides a simple method to get started with troubleshooting Skupper.
-
Check the site status:
$ skupper status --namespace west Skupper is enabled for namespace "west" in interior mode. It is connected to 2 other sites. It has 1 exposed services.
The output shows:
-
A site exists in the specified namespace.
-
A link exists to two other sites.
-
A service is exposed on the service network and is accessible from this namespace.
-
-
Check the service network:
$ skupper network status --namespace west Sites: ├─ [local] 05f8c38 - west │ URL: 10.110.15.54 │ mode: interior │ name: west │ namespace: west │ version: 1.0.2 │ ╰─ Services: │ ╰─ name: backend │ address: backend: 8080 │ protocol: tcp ╰─ [remote] 1537b82 - east URL: 10.97.26.100 name: east namespace: east sites linked to: 05f8c38-west version: 1.0.2 ╰─ Services: ╰─ name: backend address: backend: 8080 protocol: tcp ╰─ Targets: ├─ name: backend-77f8f45fc8-smckp ├─ name: backend-77f8f45fc8-gh6tp ╰─ name: backend-77f8f45fc8-m58tg
If the output is not what you expected, you might want to check links before proceeding. The output shows:
-
There are 2 sites on the service network,
east
andwest
. -
Details for each site, for example the namespace names.
-
The original services that are exposed (Targets), in this case the three backend services exposed using the
tcp
protocol. -
The services available on the service network, including the port number. For example,
backend:8080
.
-
-
Check the status of services exposed on the service network:
$ skupper service status Services exposed through Skupper: ╰─ backend (tcp port 8080) ╰─ Targets: ╰─ app=backend name=backend
The output shows the
backend
service and the related target of that service.The related targets for services are only displayed when the target is available on the current cluster. -
List the events for a site:
$ skupper debug events NAME COUNT AGE GatewayQueryRequest 3 9m12s 3 gateway request 9m12s SiteQueryRequest 3 9m12s 3 site data request 9m12s ServiceControllerEvent 9 10m24s 2 service event for west/frontend 10m24s 1 service event for west/backend 10m26s 1 Checking service for: backend 10m26s 2 Service definitions have changed 10m26s 1 service event for west/skupper-router 11m4s DefinitionMonitorEvent 15 10m24s 2 service event for west/frontend 10m24s 1 service event for west/backend 10m26s 1 Service definitions have changed 10m26s 5 deployment event for west/frontend 10m34s 1 deployment event for west/skupper-service-controller 11m4s ServiceControllerUpdateEvent 1 10m26s 1 Updating skupper-internal 10m26s ServiceSyncEvent 3 10m26s 1 Service interface(s) added backend 10m26s 1 Service sync sender connection to 11m4s amqps://skupper-router-local.west.svc.cluster.local:5671 established 1 Service sync receiver connection to 11m4s amqps://skupper-router-local.west.svc.cluster.local:5671 established IpMappingEvent 5 10m34s 1 172.17.0.7 mapped to frontend-6b4688bf56-rp9hc 10m34s 2 mapped to frontend-6b4688bf56-rp9hc 10m54s 1 172.17.0.4 mapped to 11m4s skupper-service-controller-6c97c5cf5d-6nzph 1 172.17.0.3 mapped to skupper-router-547dffdcbf-l8pdc 11m4s TokenClaimVerification 1 10m59s 1 Claim for efe3a241-3e4f-11ed-95d0-482ae336eb38 succeeded 10m59s
The output shows sites being linked and a service being exposed on a service network. However, this output is most useful when reporting an issue and is included in the Skupper debug tar file.
Checking links
You must link sites before you can expose services on the service network. This section outlines some advanced options for checking links.
-
Check the link status:
$ skupper link status --namespace east Links created from this site: ------------------------------- Link link1 is active
A link exists from the specified site to another site, meaning a token from another site was applied to the specified site.
Running skupper link status
on a connected site produces output only if a token was used to create a link.If you use this command on a site where you did not create the link, but there is an incoming link to the site:
$ skupper link status --namespace west Links created from this site: ------------------------------- There are no links configured or active Currently active links from other sites: ---------------------------------------- A link from the namespace east on site east(536695a9-26dc-4448-b207-519f56e99b71) is active
-
Check the verbose link status:
$ skupper link status link1 --verbose --namespace east Cost: 1 Created: 2022-10-24 12:50:33 +0100 IST Name: link1 Namespace: east Site: east-536695a9-26dc-4448-b207-519f56e99b71 Status: Active
The output shows detail about the link, including a timestamp of when the link was created and the associated relative cost of using the link.
The status of the link must be
Active
to allow service traffic.
Creating a Skupper debug tar file
The debug tar file contains all the logs from the Skupper components for a site and provides detailed information to help debug issues.
-
Create the debug tar file:
$ skupper debug dump my-site Skupper dump details written to compressed archive: `my-site.tar.gz`
-
You can expand the file using the following command:
$ tar -xvf kind-site.tar.gz k8s-versions.txt skupper-versions.txt skupper-router-deployment.yaml skupper-router-867f5ddcd8-plrcg-skstat-g.txt skupper-router-867f5ddcd8-plrcg-skstat-c.txt skupper-router-867f5ddcd8-plrcg-skstat-l.txt skupper-router-867f5ddcd8-plrcg-skstat-n.txt skupper-router-867f5ddcd8-plrcg-skstat-e.txt skupper-router-867f5ddcd8-plrcg-skstat-a.txt skupper-router-867f5ddcd8-plrcg-skstat-m.txt skupper-router-867f5ddcd8-plrcg-skstat-p.txt skupper-router-867f5ddcd8-plrcg-router-logs.txt skupper-router-867f5ddcd8-plrcg-config-sync-logs.txt skupper-service-controller-deployment.yaml skupper-service-controller-7485756984-gvrf6-events.txt skupper-service-controller-7485756984-gvrf6-service-controller-logs.txt skupper-site-configmap.yaml skupper-services-configmap.yaml skupper-internal-configmap.yaml skupper-sasl-config-configmap.yaml
These files can be used to provide support for Skupper, however some items you can check:
- versions
-
See
*versions.txt
for the versions of various components. - ingress
-
See
skupper-site-configmap.yaml
to determine theingress
type for the site. - linking and services
-
See the
skupper-service-controller-*-events.txt
file to view details of token usage and service exposure.
Resolving common problems
The following issues and workarounds might help you debug simple scenarios when evaluating Skupper.
Cannot initialize skupper
If the skupper init
command fails, consider the following options:
-
Check the load balancer.
If you are evaluating Skupper on minikube, use the following command to create a load balancer:
$ minikube tunnel
For other Kubernetes flavors, see the documentation from your provider.
-
Initialize without ingress.
This option prevents other sites from linking to this site, but linking outwards is supported. Once a link is established, traffic can flow in either direction. Enter the following command:
$ skupper init --ingress none
See skupper init for more options.
Cannot link sites
To link two sites, one site must be accessible from the other site. For example, if one site is behind a firewall and the other site is on an AWS cluster, you must:
-
Create a token on the AWS cluster site.
-
Create the link on the site inside the firewall.
By default, a token is only valid for a single link. If you need to link many sites, create separate tokens for each link. |
Cannot access Skupper console
Use skupper status
to find the console URL.
Use the following command to display the password for the admin
user:doctype: article
$ kubectl get secret/skupper-console-users -o jsonpath={.data.admin} | base64 -d
Cannot create a token for linking clusters
You might see the following note after using the skupper token create
command:
Token written to <path> (Note: token will only be valid for local cluster)
This output indicates that the site was deployed without an ingress option. For example skupper init --ingress none
.
You must specify an ingress to allow sites on other clusters to link to your site.
You can also use the skupper token create
command to check if an ingress was specified when the site was created.