在 kubernetes 集群中部署 sentry helm

Question

I have deployed sentry on kubernetes last week, using this helm chart stable/sentry .

Pods work fine but i cannot access to the website, it crash every time I access to the endpoint.

I checked logs of the worker, sentry-web and postgres pods and see this.

This is logs of the website pod:

    self._result_cache = list(self.iterator())
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 220, in iterator
    for row in compiler.results_iter():
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 713, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/django/__init__.py", line 396, in execute
    return real_execute(self, sql, params)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 80, in inner
    raise_the_exception(self.db, e)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 78, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 22, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 101, in inner
    six.reraise(exc_info[0], exc_info[0](msg), exc_info[2])
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 94, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 74, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
django.db.utils.ProgrammingError: ProgrammingError('relation "sentry_projectkey" does not exist\nLINE 1: ...te_limit_window", "sentry_projectkey"."data" FROM "sentry_pr...\n                                                             ^\n',)
SQL: SELECT "sentry_projectkey"."id", "sentry_projectkey"."project_id", "sentry_projectkey"."label", "sentry_projectkey"."public_key", "sentry_projectkey"."secret_key", "sentry_projectkey"."roles", "sentry_projectkey"."status", "sentry_projectkey"."date_added", "sentry_projectkey"."rate_limit_count", "sentry_projectkey"."rate_limit_window", "sentry_projectkey"."data" FROM "sentry_projectkey" WHERE ("sentry_projectkey"."project_id" = %s  AND "sentry_projectkey"."roles" =  "sentry_projectkey"."roles" | 1) LIMIT 1

And this is logs of the postgres pod:

ERROR:  relation "sentry_option" does not exist at character 114
STATEMENT:  SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."last_updated" >= '2019-09-13 14:02:49.793126+00:00' 
ERROR:  relation "sentry_monitor" does not exist at character 328
STATEMENT:  SELECT "sentry_monitor"."id", "sentry_monitor"."guid", "sentry_monitor"."organization_id", "sentry_monitor"."project_id", "sentry_monitor"."name", "sentry_monitor"."status", "sentry_monitor"."type", "sentry_monitor"."config", "sentry_monitor"."next_checkin", "sentry_monitor"."last_checkin", "sentry_monitor"."date_added" FROM "sentry_monitor" WHERE ("sentry_monitor"."next_checkin" < '2019-09-13 15:02:50.996171+00:00'  AND "sentry_monitor"."type" IN (2, 3) AND NOT ("sentry_monitor"."status" IN (1, 2, 3))) LIMIT 10000
ERROR:  relation "sentry_scheduledjob" does not exist at character 179
STATEMENT:  SELECT "sentry_scheduledjob"."id", "sentry_scheduledjob"."name", "sentry_scheduledjob"."payload", "sentry_scheduledjob"."date_added", "sentry_scheduledjob"."date_scheduled" FROM "sentry_scheduledjob" WHERE "sentry_scheduledjob"."date_scheduled" <= '2019-09-13 15:02:51.984402+00:00'  LIMIT 101
ERROR:  relation "sentry_authidentity" does not exist at character 288
STATEMENT:  SELECT "sentry_authidentity"."id", "sentry_authidentity"."user_id", "sentry_authidentity"."auth_provider_id", "sentry_authidentity"."ident", "sentry_authidentity"."data", "sentry_authidentity"."last_verified", "sentry_authidentity"."last_synced", "sentry_authidentity"."date_added" FROM "sentry_authidentity" WHERE "sentry_authidentity"."last_synced" <= '2019-09-13 14:02:56.685172+00:00' 

And the worker pods:

    result = func(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/tasks/options.py", line 36, in sync_options
    for option in Option.objects.filter(last_updated__gte=cutoff_dt).iterator():
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 220, in iterator
    for row in compiler.results_iter():
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 713, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/django/__init__.py", line 396, in execute
    return real_execute(self, sql, params)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 80, in inner
    raise_the_exception(self.db, e)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 78, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 22, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 101, in inner
    six.reraise(exc_info[0], exc_info[0](msg), exc_info[2])
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 94, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 74, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
ProgrammingError: ProgrammingError('relation "sentry_option" does not exist\nLINE 1: ...ion"."value", "sentry_option"."last_updated" FROM "sentry_op...\n                                                             ^\n',)
SQL: SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."last_updated" >= %s 
15:09:24 [ERROR] celery.worker.job: Task sentry.tasks.options.sync_options[e1684c22-6593-48d9-94c7-2c47ee7fc397] raised unexpected: ProgrammingError(u'ProgrammingError(\'relation "sentry_option" does not exist\\nLINE 1: ...ion"."value", "sentry_option"."last_updated" FROM "sentry_op...\\n                                                             ^\\n\',)\nSQL: SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."last_updated" >= %s ',) (data={u'hostname': 'celery@teq-sentry-worker-6ddfb9658c-gbkth', u'name': 'sentry.tasks.options.sync_options', u'args': '[]', u'internal': False, u'kwargs': '{}', u'id': 'e1684c22-6593-48d9-94c7-2c47ee7fc397'})

Answer 1

您必须手动迁移架构，运行以下命令来迁移架构：

kubectl exec -it <your-pod-name> sentry upgrade

Answer 2

Solve. Thank you everyone for your help. The problem is I'm not have enough RAM in order to run the migration script, so I upgrade the VPS to 8GB RAM and then it's work.

Because the migration script have some memory leak so pods need at least 4GB free memory to run. See this comment on Github

Answer 3

Faced a similar issue. In our case, we have enabled ingress (Nginx). And this setting conflict with service.type: LoadBalancer , just change it to ClusterIP and do the clear setup again.

TLDR - DB init starts on "post-install" event (all deployed components is in a ready state) but in case of LoadBallancer, ingress and --wait combination the web Service must have assigned IP ( https://github.com/helm/helm/blob/master/docs/using_helm.md#helpful-options-for-installupgraderollback ). For some reason it wasn't true and web service always returns false . After change the service setting from LoadBalancer to ClusterIP the issue is gone.