ERR invalid character '}' looking for beginning of object key string: goroutine 20932 in kiali
Question
I am trying to install Prometheus and Kiali using Istio sample addons. I modifies the file to add nodePort for Service and changed imagePullPolicy to IfNotPresent for Kiali container. The pod comes up, but I am unable to view the graph. It gives the following error:
ERR invalid character '}' looking for beginning of object key string: goroutine 20932
Kiali pod gives the following stack trace:
invalid character '}' looking for beginning of object key string
Additional Detail:
goroutine 2066 [running]:
runtime/debug.Stack(0xc00173aa80, 0x40, 0xc0020d0cc0)
/usr/local/go/src/runtime/debug/stack.go:24 +0x9f
github.com/kiali/kiali/handlers.handlePanic(0x1d5bad0, 0xc001819dc0)
/home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/handlers/graph.go:86 +0x185
panic(0x1841540, 0xc0020d0cc0)
/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/kiali/kiali/graph.CheckError(...)
/home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/graph/util.go:38
github.com/kiali/kiali/graph/telemetry/istio/appender.ServiceEntryAppender.AppendGraph(0xc000734f90, 0xc0008ae8f7, 0xc, 0xc000735380, 0xc0007352c0, 0xc0020d0a80)
/home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/graph/telemetry/istio/appender/service_entry.go:59 +0x1c6
github.com/kiali/kiali/graph/telemetry/istio.BuildNamespacesTrafficMap(0xc000734f90, 0x0, 0xc0020ed6d0, 0x5, 0x5, 0x100, 0xc000734ea0, 0xdf8475800, 0xc0008ae8f7, 0xc, ...)
/home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/graph/telemetry/istio/istio.go:54 +0x282
github.com/kiali/kiali/graph/api.graphNamespacesIstio(0xc001e625a0, 0xc000d95c00, 0x1af6a6c, 0x9, 0x1af1831, 0x5, 0xc0008ae939, 0x3, 0xdf8475800, 0xc0008ae8f7, ...)
/home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/graph/api/api.go:44 +0xb8
github.com/kiali/kiali/graph/api.GraphNamespaces(0xc001e625a0, 0x1af6a6c, 0x9, 0x1af1831, 0x5, 0xc0008ae939, 0x3, 0xdf8475800, 0xc0008ae8f7, 0xc, ...)
/home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/graph/api/api.go:26 +0x178
github.com/kiali/kiali/handlers.GraphNamespaces(0x1d5bad0, 0xc001819dc0, 0xc000eecc00)
/home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/handlers/graph.go:51 +0x13c
net/http.HandlerFunc.ServeHTTP(0x1bc10a0, 0x1d5bad0, 0xc001819dc0, 0xc000eecc00)
/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/kiali/kiali/routing.metricHandler.func1(0x1d5bad0, 0xc001819dc0, 0xc000eecc00)
/home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/routing/router.go:92 +0xa3
net/http.HandlerFunc.ServeHTTP(0xc000132fc0, 0x1d5bad0, 0xc001819dc0, 0xc000eecc00)
/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/kiali/kiali/handlers.AuthenticationHandler.Handle.func1(0x1d5bad0, 0xc001819dc0, 0xc000eecb00)
/home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/handlers/authentication.go:649 +0x1f7
net/http.HandlerFunc.ServeHTTP(0xc000689230, 0x1d5bad0, 0xc001819dc0, 0xc000eecb00)
/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/kiali/kiali/server.plainHttpMiddleware.func1(0x1d5bad0, 0xc001819dc0, 0xc000eecb00)
/home/fedora/workspace/kiali-workflow/kiali-release/src/github.com/kiali/kiali/server/server.go:127 +0x6c
net/http.HandlerFunc.ServeHTTP(0xc00061d740, 0x1d5bad0, 0xc001819dc0, 0xc000eecb00)
/usr/local/go/src/net/http/server.go:2069 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00031c0c0, 0x1d5bad0, 0xc001819dc0, 0xc000eec500)
/home/fedora/workspace/kiali-workflow/kiali-release/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0xd3
github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1(0x1d5b9e0, 0xc00075c460, 0xc000eec500)
/home/fedora/workspace/kiali-workflow/kiali-release/pkg/mod/github.com/!n!y!times/gziphandler@v1.1.1/gzip.go:336 +0x22d
net/http.HandlerFunc.ServeHTTP(0xc00025e510, 0x1d5b9e0, 0xc00075c460, 0xc000eec500)
/usr/local/go/src/net/http/server.go:2069 +0x44
net/http.(*ServeMux).ServeHTTP(0xc00014f800, 0x1d5b9e0, 0xc00075c460, 0xc000eec500)
/usr/local/go/src/net/http/server.go:2448 +0x1ad
net/http.serverHandler.ServeHTTP(0xc00021c000, 0x1d5b9e0, 0xc00075c460, 0xc000eec500)
/usr/local/go/src/net/http/server.go:2887 +0xa3
net/http.(*conn).serve(0xc0002c65a0, 0x1d5f6b8, 0xc00131fa40)
/usr/local/go/src/net/http/server.go:1952 +0x8cd
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:3013 +0x39b
Version details:
- Kubernetes: 1.21.1
- Istio: 1.10.0
- Prometheus: 2.24.0
- Kiali: 1.34
Edit: Adding manifest files used for deployment.
Kiali Configmap:
# Source: kiali-server/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: kiali
namespace: istio-system
labels:
helm.sh/chart: kiali-server-1.34.0
app: kiali
app.kubernetes.io/name: kiali
app.kubernetes.io/instance: kiali-server
version: "v1.34.0"
app.kubernetes.io/version: "v1.34.0"
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/part-of: "kiali"
data:
config.yaml: |
auth:
openid: {}
openshift:
client_id_prefix: kiali
strategy: anonymous
deployment:
accessible_namespaces:
- '**'
additional_service_yaml: {}
affinity:
node: {}
pod: {}
pod_anti: {}
custom_dashboards:
excludes:
- ""
includes:
- '*'
hpa:
api_version: autoscaling/v2beta2
spec: {}
image_name: quay.io/kiali/kiali
image_pull_policy: IfNotPresent
image_pull_secrets: []
image_version: v1.34
ingress_enabled: false
logger:
log_format: text
log_level: info
sampler_rate: "1"
time_field_format: 2006-01-02T15:04:05Z07:00
namespace: istio-system
node_selector: {}
override_ingress_yaml:
metadata: {}
pod_annotations:
sidecar.istio.io/inject: "false"
pod_labels: {}
priority_class_name: ""
replicas: 1
resources: {}
secret_name: kiali
service_annotations: {}
service_type: ""
tolerations: []
version_label: v1.34.0
view_only_mode: false
external_services:
custom_dashboards:
enabled: true
identity:
cert_file: ""
private_key_file: ""
istio_namespace: istio-system
login_token:
signing_key: CHANGEME
server:
metrics_enabled: true
metrics_port: 9090
port: 20001
web_root: /kiali
---
# Source: kiali-server/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: kiali
namespace: istio-system
labels:
helm.sh/chart: kiali-server-1.34.0
app: kiali
app.kubernetes.io/name: kiali
app.kubernetes.io/instance: kiali-server
version: "v1.34.0"
app.kubernetes.io/version: "v1.34.0"
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/part-of: "kiali"
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: kiali
app.kubernetes.io/instance: kiali-server
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
name: kiali
labels:
helm.sh/chart: kiali-server-1.34.0
app: kiali
app.kubernetes.io/name: kiali
app.kubernetes.io/instance: kiali-server
version: "v1.34.0"
app.kubernetes.io/version: "v1.34.0"
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/part-of: "kiali"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
kiali.io/runtimes: go,kiali
sidecar.istio.io/inject: "false"
spec:
serviceAccountName: kiali
containers:
- image: "quay.io/kiali/kiali:v1.34"
imagePullPolicy: IfNotPresent
name: kiali
command:
- "/opt/kiali/kiali"
- "-config"
- "/kiali-configuration/config.yaml"
ports:
- name: api-port
containerPort: 20001
- name: http-metrics
containerPort: 9090
readinessProbe:
httpGet:
path: /kiali/healthz
port: api-port
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 30
livenessProbe:
httpGet:
path: /kiali/healthz
port: api-port
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 30
env:
- name: ACTIVE_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: LOG_LEVEL
value: "info"
- name: LOG_FORMAT
value: "text"
- name: LOG_TIME_FIELD_FORMAT
value: "2006-01-02T15:04:05Z07:00"
- name: LOG_SAMPLER_RATE
value: "1"
volumeMounts:
- name: kiali-configuration
mountPath: "/kiali-configuration"
- name: kiali-cert
mountPath: "/kiali-cert"
- name: kiali-secret
mountPath: "/kiali-secret"
volumes:
- name: kiali-configuration
configMap:
name: kiali
- name: kiali-cert
secret:
secretName: istio.kiali-service-account
optional: true
- name: kiali-secret
secret:
secretName: kiali
optional: true
---
# Source: kiali-server/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: kiali
namespace: istio-system
labels:
helm.sh/chart: kiali-server-1.34.0
app: kiali
app.kubernetes.io/name: kiali
app.kubernetes.io/instance: kiali-server
version: "v1.34.0"
app.kubernetes.io/version: "v1.34.0"
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/part-of: "kiali"
annotations:
spec:
ports:
- name: http
protocol: TCP
port: 20001
nodePort: 31334
- name: http-metrics
protocol: TCP
port: 9090
nodePort: 31335
selector:
app.kubernetes.io/name: kiali
app.kubernetes.io/instance: kiali-server
type: LoadBalancer
Prometheus:
# Source: prometheus/templates/server/cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
labels:
component: "server"
app: prometheus
release: prometheus
chart: prometheus-13.6.0
heritage: Helm
name: prometheus
namespace: istio-system
data:
alerting_rules.yml: |
groups:
- name: Alerts
rules:
- alert: High Pod Memory
expr: sum(container_memory_usage_bytes) > 1
for: 5m
labels:
severity: slack
annotations:
summary: High Memory Usage
alerts: |
{}
prometheus.yml: |
global:
evaluation_interval: 1m
scrape_interval: 15s
scrape_timeout: 10s
rule_files:
- /etc/config/recording_rules.yml
- /etc/config/alerting_rules.yml
- /etc/config/rules
- /etc/config/alerts
alerting:
alertmanagers:
- scheme: http
static_configs:
- targets:
- "alertmanager:9093"
scrape_configs:
- job_name: prometheus
static_configs:
- targets:
- localhost:9090
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: kubernetes-apiservers
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- action: keep
regex: default;kubernetes;https
source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_service_name
- __meta_kubernetes_endpoint_port_name
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: kubernetes-nodes
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- replacement: kubernetes.default.svc:443
target_label: __address__
- regex: (.+)
replacement: /api/v1/nodes/$1/proxy/metrics
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: kubernetes-nodes-cadvisor
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- replacement: kubernetes.default.svc:443
target_label: __address__
- regex: (.+)
replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
- job_name: kubernetes-service-endpoints
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scrape
- action: replace
regex: (https?)
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scheme
target_label: __scheme__
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_path
target_label: __metrics_path__
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_service_annotation_prometheus_io_port
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: kubernetes_namespace
- action: replace
source_labels:
- __meta_kubernetes_service_name
target_label: kubernetes_name
- action: replace
source_labels:
- __meta_kubernetes_pod_node_name
target_label: kubernetes_node
- job_name: kubernetes-service-endpoints-slow
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scrape_slow
- action: replace
regex: (https?)
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scheme
target_label: __scheme__
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_path
target_label: __metrics_path__
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_service_annotation_prometheus_io_port
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: kubernetes_namespace
- action: replace
source_labels:
- __meta_kubernetes_service_name
target_label: kubernetes_name
- action: replace
source_labels:
- __meta_kubernetes_pod_node_name
target_label: kubernetes_node
scrape_interval: 5m
scrape_timeout: 30s
- honor_labels: true
job_name: prometheus-pushgateway
kubernetes_sd_configs:
- role: service
relabel_configs:
- action: keep
regex: pushgateway
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_probe
- job_name: kubernetes-services
kubernetes_sd_configs:
- role: service
metrics_path: /probe
params:
module:
- http_2xx
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_probe
- source_labels:
- __address__
target_label: __param_target
- replacement: blackbox
target_label: __address__
- source_labels:
- __param_target
target_label: instance
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels:
- __meta_kubernetes_namespace
target_label: kubernetes_namespace
- source_labels:
- __meta_kubernetes_service_name
target_label: kubernetes_name
- job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
- action: replace
regex: (https?)
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scheme
target_label: __scheme__
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
target_label: __metrics_path__
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_pod_annotation_prometheus_io_port
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: kubernetes_namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: kubernetes_pod_name
- action: drop
regex: Pending|Succeeded|Failed
source_labels:
- __meta_kubernetes_pod_phase
- job_name: kubernetes-pods-slow
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow
- action: replace
regex: (https?)
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scheme
target_label: __scheme__
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
target_label: __metrics_path__
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_pod_annotation_prometheus_io_port
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: kubernetes_namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: kubernetes_pod_name
- action: drop
regex: Pending|Succeeded|Failed
source_labels:
- __meta_kubernetes_pod_phase
scrape_interval: 5m
scrape_timeout: 30s
recording_rules.yml: |
{}
rules: |
{}
---
# Source: prometheus/templates/server/deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
component: "server"
app: prometheus
release: prometheus
chart: prometheus-13.6.0
heritage: Helm
name: prometheus
namespace: istio-system
spec:
selector:
matchLabels:
component: "server"
app: prometheus
release: prometheus
replicas: 1
template:
metadata:
annotations:
sidecar.istio.io/inject: "false"
labels:
component: "server"
app: prometheus
release: prometheus
chart: prometheus-13.6.0
heritage: Helm
spec:
serviceAccountName: prometheus
containers:
- name: prometheus-server-configmap-reload
image: "jimmidyson/configmap-reload:v0.5.0"
imagePullPolicy: "IfNotPresent"
args:
- --volume-dir=/etc/config
- --webhook-url=http://127.0.0.1:9090/-/reload
resources:
{}
volumeMounts:
- name: config-volume
mountPath: /etc/config
readOnly: true
- name: prometheus-server
image: "prom/prometheus:v2.24.0"
imagePullPolicy: "IfNotPresent"
args:
- --storage.tsdb.retention.time=15d
- --config.file=/etc/config/prometheus.yml
- --storage.tsdb.path=/data
- --web.console.libraries=/etc/prometheus/console_libraries
- --web.console.templates=/etc/prometheus/consoles
- --web.enable-lifecycle
ports:
- containerPort: 9090
#readinessProbe:
#httpGet:
#path: /-/ready
#port: 9090
#initialDelaySeconds: 0
#periodSeconds: 5
#timeoutSeconds: 4
#failureThreshold: 3
#successThreshold: 1
#livenessProbe:
#httpGet:
#path: /-/healthy
#port: 9090
#initialDelaySeconds: 30
#periodSeconds: 15
#timeoutSeconds: 10
#failureThreshold: 3
#successThreshold: 1
volumeMounts:
- name: config-volume
mountPath: /etc/config
- name: storage-volume
mountPath: /data
subPath: ""
hostNetwork: false
dnsPolicy: ClusterFirst
securityContext:
fsGroup: 65534
runAsGroup: 65534
runAsNonRoot: true
runAsUser: 65534
terminationGracePeriodSeconds: 300
volumes:
- name: config-volume
configMap:
name: prometheus
- name: storage-volume
emptyDir:
{}
---
# Source: prometheus/templates/server/service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
component: "server"
app: prometheus
release: prometheus
chart: prometheus-13.6.0
heritage: Helm
name: prometheus
namespace: istio-system
spec:
ports:
- name: http
port: 9090
protocol: TCP
targetPort: 9090
nodePort: 31333
selector:
component: "server"
app: prometheus
release: prometheus
sessionAffinity: None
type: LoadBalancer
1 answers
solution1
0 2021-12-15 11:50:31
If there is no particular reason for LoadBalancer I would recommend reverting those manifests back to the original, provided with Istio.
If you want to access dashboard you can do so by issuing
istioctl dashboard kiali
for Kiali, or
istioctl dashboard promtheus
for Prometheus.
Optionally you can expose those dashboards using Istio Gateway and VirtualService . More details are available here .
However, I would not recommend exposing Kiali and Prometheus to outside, as it creates a huge security risk.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Unable to view kiali dashboard
Detect kafka events in kiali
How to install Kiali on GKE?
Kiali is not working with VictoriaMetricsHello
Istio ServiceEntry ignored in kiali
what's the unknow service in Istio Kiali?
Secure Kiali (in Istio) by source ip address
Istio 1.4.7 - Kiali pod fails to start
create VirtualService for kiali, tracing, grafana
Changing secrets of kiali in istio is not working
Related Tags