Operations guides for Sourcegraph on Kubernetes

The Kustomize deployment type is planned for deprecation and will be sunset in a future release. We recommend using the Helm deployment for all new Kubernetes installations.

Operations guides specific to managing Sourcegraph on Kubernetes installations.

Featured guides

Trying to deploy Sourcegraph on Kubernetes? Refer to our installation guide.

Configure

We strongly recommend referring to our Configuration guide to learn about how to configure your Sourcegraph with Kubernetes instance.

Deploy

Refer to our installation guide for details on how to deploy Sourcegraph.

Migrating from another deployment type? Refer to our migration guides.

Deploy with Kustomize

In order to deploy Sourcegraph that is configured for your cluster:

Building manifests

Build a new set of manifests using an overlay you've created following our configuration guide for Kustomize:

BASH
$ kubectl kustomize $PATH_TO_OVERLAY -o cluster.yaml

Reviewing manifests

Review the manifests generated in the previous step:

BASH
$ less cluster.yaml

Applying manifests

Run the command below to apply the manifests from the output file cluster.yaml to the connected cluster:

BASH
$ kubectl apply --prune -l deploy=sourcegraph -f cluster.yaml

Once you have applied your changes:

Watch - verify your deployment has started:

BASH
$ kubectl get pods -A -o wide --watch

Port-forward - verify Sourcegraph is running by temporarily making the frontend port accessible:
```
SH
$ kubectl port-forward svc/sourcegraph-frontend 3080:30080
```
Log in - browse to your Sourcegraph deployment, login, and verify the instance is working as expected.

Below are the commands that will output the differences between the two overlays, allowing you to review and compare the changes and ensure that the new overlay produces similar resources as the ones currently being used by the active cluster or another overlay you want to compare with, before applying the new overlay.

Between two overlays

To compare resources between two different Kustomize overlays:

BASH
$ diff \
    <(kubectl kustomize $PATH_TO_OVERLAY_1) \
    <(kubectl kustomize $PATH_TO_OVERLAY_2) |\
    more

Example 1: compare diff between resources generated by the k3s overlay for size xs instance and the k3s overlay for size xl instance:

BASH
$ diff \
    <(kubectl kustomize examples/k3s/xs) \
    <(kubectl kustomize examples/k3s/xl) |\
    more

Example 2: compare diff between the new base cluster and the old cluster:

BASH
$ diff \
    <(kubectl kustomize examples/base) \
    <(kubectl kustomize examples/old-cluster) |\
    more

Example 3: compare diff between the output files from two different overlay builds:

BASH
$ kubectl kustomize examples/old-cluster -o old-cluster.yaml
$ kubectl kustomize examples/base -o new-cluster.yaml
 
$ diff old-cluster.yaml new-cluster.yaml

Between an overlay and a running cluster

To compare the difference between the manifests generated by an overlay and the resources that are being used by the running cluster connected to the kubectl tool:

BASH
$ kubectl kustomize $PATH_TO_OVERLAY | kubectl diff -f  -

The command will output the differences between the customizations specified in the overlay and the resources currently running in the cluster, allowing you to review the changes and ensure that the overlay produces similar resources to the ones currently being used by the active cluster before applying the new overlay.

Example: compare diff between the k3s overlay for size xl instance and the instance that is connected with kubectl:

BASH
$ kubectl kustomize examples/k3s/xl | kubectl diff -f  -

List pods in cluster

List all pods in your cluster and the corresponding health status of each pod:

BASH
$ kubectl get pods -o=wide

Tail logs for specific pod

Tail the logs for the specified pod:

BASH
$ kubectl logs -f $POD_NAME

If Sourcegraph is unavailable and the sourcegraph-frontend-* pod(s) are not in status Running, then view their logs with $ kubectl logs -f sourcegraph-frontend-$POD_ID (filling in $POD_ID from the $ kubectl get pods output). Inspect both the log messages printed at startup (at the beginning of the log output) and recent log messages.

Retrieving resource information

Display detailed information about the status of a single pod:

BASH
$ kubectl describe $POD_NAME

List all Persistent Volume Claims (PVCs) and their statuses:

BASH
$ kubectl get pvc

List all Persistent Volumes (PVs) that have been provisioned. In a healthy cluster, there should be a one-to-one mapping between PVs and PVCs:

BASH
$ kubectl get pv

List all events in the cluster's history:

BASH
$ kubectl get events

Delete failing pod so it gets recreated, possibly on a different node:

BASH
$ kubectl delete pod $POD_NAME

Remove all pods from a node and mark it as unschedulable to prevent new pods from arriving

BASH
$ kubectl drain --force --ignore-daemonsets --delete-local-data $NODE

Restarting Sourcegraph Instance:

BASH
$ kubectl rollout restart deployment sourcegraph-frontend

Access the database

Check that the pgsql pod is running:

BASH
$ kubectl get pods -l app=pgsql
NAME      READY   STATUS    RESTARTS   AGE
pgsql-0   2/2     Running   0          19m

Make sure you are operating under the correct namespace (i.e. add -n prod if your pod is under the prod namespace).

Open a PostgreSQL interactive terminal:

BASH
$ kubectl exec -it pgsql-0 -- psql -U sg

Run your SQL query:

SQL
SELECT * FROM users;

NOTE: To execute an SQL query against the database without first creating an interactive session (as below), append --command "SELECT * FROM users;" to the docker container exec command.

Backup and restore

The following instructions are specific to backing up and restoring the sourcegraph databases in a Kubernetes deployment. These do not apply to other deployment types.

WARNING: Only core data will be backed up.

These instructions will only back up core data including user accounts, configuration, repository-metadata, etc. Other data will be regenerated automatically:

Repositories will be re-cloned

Search indexes will be rebuilt from scratch

The above may take a while if you have a lot of repositories. In the meantime, searches may be slow or return incomplete results. This process rarely takes longer than 6 hours and is usually much faster.

NOTE: Add -n $NAMESPACE to commands if you are not using the default namespace. The database pods are pgsql-0, codeintel-db-0, and codeinsights-db-0. More kubectl configuration options can be found here: kubectl Cheat Sheet

Back up Sourcegraph databases

These instructions will back up the primary sourcegraph database, the codeintel database, and the codeinsights database.

Verify your Sourcegraph pods are running / healthy, especially the 3 database pods

BASH
$ kubectl get pods

Check for corrupt database indexes. The check blocks writes to each table while it checks that table's indexes. If amcheck returns errors, please reach out to support@sourcegraph.com

Save this query in a file named amcheck.sql:

SQL
create extension if not exists amcheck;
 
select bt_index_parent_check(c.oid, true), c.relname, c.relpages
from pg_index i
join pg_opclass op ON i.indclass[0] = op.oid
join pg_am am ON op.opcmethod = am.oid
join pg_class c ON i.indexrelid = c.oid
join pg_namespace n ON c.relnamespace = n.oid
where am.amname = 'btree'
-- Don't check temp tables, which may be from another session:
and c.relpersistence != 't'
-- Skip partitioned indexes; the indexes of their partitions are checked:
and c.relkind = 'i'
-- Function may throw an error when this is omitted:
and i.indisready AND i.indisvalid;

Run it on each database. Each command prints any errors, and exits 0 when every index is intact:

BASH
$ kubectl exec -i pgsql-0 -- \
    psql -v ON_ERROR_STOP=1 --username sg sg \
    < amcheck.sql > /dev/null
 
$ kubectl exec -i codeintel-db-0 -- \
    psql -v ON_ERROR_STOP=1 --username sg sg \
    < amcheck.sql > /dev/null
 
$ kubectl exec -i codeinsights-db-0 -- \
    psql -v ON_ERROR_STOP=1 --username postgres postgres \
    < amcheck.sql > /dev/null

Generate the database dumps

BASH
$ kubectl exec pgsql-0 -- \
    pg_dump -C --clean --if-exists --username sg sg \
    > sourcegraph_db.out
 
$ kubectl exec codeintel-db-0 -- \
    pg_dump -C --clean --if-exists --username sg sg \
    > codeintel_db.out
 
$ kubectl exec codeinsights-db-0 -- \
    pg_dump -C --clean --if-exists --username postgres postgres \
    > codeinsights_db.out

Run these commands without -it: with a terminal attached, every line of the dump ends with a carriage return.

Ensure the sourcegraph_db.out, codeintel_db.out and codeinsights_db.out files are moved to a safe and secure location.

Restore Sourcegraph databases

These instructions apply to both a new deployment and an existing one. You don't need to delete the database volumes: the restore drops and recreates each database.

On the machine where you run kubectl, cd to the directory with the database dump files: sourcegraph_db.out, codeintel_db.out and codeinsights_db.out
Stop every Sourcegraph service except the databases, so nothing writes to the databases during the restore

BASH
$ kubectl scale deployment,statefulset --replicas=0 -l '
    deploy=sourcegraph,
    app.kubernetes.io/component notin
        (pgsql,codeintel-db,codeinsights-db)'

Wait until kubectl get pods shows no Sourcegraph pods other than pgsql-0, codeintel-db-0, and codeinsights-db-0.

Copy the database files into the pods

BASH
$ kubectl cp sourcegraph_db.out pgsql-0:/data/sourcegraph_db.out
$ kubectl cp codeintel_db.out codeintel-db-0:/data/codeintel_db.out
$ kubectl cp codeinsights_db.out \
    codeinsights-db-0:/var/lib/postgresql/data/codeinsights_db.out

The database containers have a read-only root filesystem, so copy the files into the database volumes. Step 5 deletes them after the restore.

Restore the databases

DROP DATABASE ... WITH (FORCE) disconnects the other sessions on each database, such as the pgsql-exporter sidecar, and drops it, so the dump can recreate it. ON_ERROR_STOP=1 stops the restore at the first error; if a restore fails, fix the cause and run it again.

BASH
$ kubectl exec -it pgsql-0 -- \
    psql -v ON_ERROR_STOP=1 --username sg \
    -c "DROP DATABASE IF EXISTS sg WITH (FORCE)" \
    -f /data/sourcegraph_db.out postgres
 
$ kubectl exec -it codeintel-db-0 -- \
    psql -v ON_ERROR_STOP=1 --username sg \
    -c "DROP DATABASE IF EXISTS sg WITH (FORCE)" \
    -f /data/codeintel_db.out postgres
 
$ kubectl exec -it codeinsights-db-0 -- \
    psql -v ON_ERROR_STOP=1 --username postgres \
    -c "DROP DATABASE IF EXISTS postgres WITH (FORCE)" \
    -f /var/lib/postgresql/data/codeinsights_db.out template1

After all three restores succeed, delete the backup files from the pods

BASH
$ kubectl exec pgsql-0 -- rm /data/sourcegraph_db.out
$ kubectl exec codeintel-db-0 -- rm /data/codeintel_db.out
$ kubectl exec codeinsights-db-0 -- \
    rm /var/lib/postgresql/data/codeinsights_db.out

Check for corrupt database indexes, with the commands in step 2 of Back up Sourcegraph databases. If amcheck returns errors, please reach out to support@sourcegraph.com
Start the remaining Sourcegraph services, by running helm upgrade with the chart version and values file of your deployment (helm list shows the chart version). It sets every service back to the number of replicas in your values.

BASH
$ helm upgrade --install --values ./override.yaml \
    --version <chart_version> \
    sourcegraph sourcegraph/sourcegraph

If you deployed with Kustomize, follow the steps in applying manifests instead.

Browse to your Sourcegraph deployment, log in, and verify your existing configuration has been restored

List of ports

To see a list of ports that are currently being used by your Sourcegraph instance:

BASH
$ kubectl get services

Example output:

BASH
NAME                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
blobstore                       ClusterIP   10.72.3.144    <none>        9000/TCP                     25h
cadvisor                        ClusterIP   10.72.14.130   <none>        48080/TCP                    23h
codeinsights-db                 ClusterIP   10.72.6.240    <none>        5432/TCP,9187/TCP            25h
codeintel-db                    ClusterIP   10.72.5.10     <none>        5432/TCP,9187/TCP            25h
gitserver                       ClusterIP   None           <none>        10811/TCP                    25h
grafana                         ClusterIP   10.72.6.245    <none>        30070/TCP                    25h
indexed-search                  ClusterIP   None           <none>        6070/TCP                     25h
indexed-search-indexer          ClusterIP   None           <none>        6072/TCP                     25h
kubernetes                      ClusterIP   10.72.0.1      <none>        443/TCP                      25h
node-exporter                   ClusterIP   10.72.5.60     <none>        9100/TCP                     25h
otel-collector                  ClusterIP   10.72.9.221    <none>        4317/TCP,4318/TCP,8888/TCP   25h
pgsql                           ClusterIP   10.72.6.23     <none>        5432/TCP,9187/TCP            25h
precise-code-intel-worker       ClusterIP   10.72.11.102   <none>        3188/TCP,6060/TCP            25h
prometheus                      ClusterIP   10.72.12.201   <none>        30090/TCP                    25h
redis-cache                     ClusterIP   10.72.15.138   <none>        6379/TCP,9121/TCP            25h
redis-store                     ClusterIP   10.72.4.162    <none>        6379/TCP,9121/TCP            25h
searcher                        ClusterIP   None           <none>        3181/TCP,6060/TCP            23h
sourcegraph-frontend            ClusterIP   10.72.12.103   <none>        30080/TCP,6060/TCP           25h
sourcegraph-frontend-internal   ClusterIP   10.72.9.155    <none>        80/TCP                       25h
syntect-server                  ClusterIP   10.72.14.49    <none>        9238/TCP,6060/TCP            25h
worker                          ClusterIP   10.72.7.72     <none>        3189/TCP,6060/TCP            25h

Troubleshoot

See the Troubleshooting docs.