# Operations guides for Sourcegraph on Kubernetes

<Callout type="warning">
	The Kustomize deployment type is planned for deprecation and will be sunset in a future release. We recommend using the [Helm deployment](/self-hosted/deploy/kubernetes) for all new Kubernetes installations.
</Callout>

Operations guides specific to managing [Sourcegraph on Kubernetes](/self-hosted/deploy/kubernetes/) installations.

<QuickLinks>

    <QuickLink title="Installation" icon='lightbulb' href="/self-hosted/deploy/kubernetes/" />
    <QuickLink title="Introduction" icon='theming' href="/self-hosted/deploy/kubernetes/kustomize" />
    <QuickLink title="Configuration" icon='installation' href="/self-hosted/deploy/kubernetes/configure" />
    <QuickLink title="Maintenance" icon='presets' href="/self-hosted/deploy/kubernetes/operations" />

</QuickLinks>

## Featured guides

Trying to deploy Sourcegraph on Kubernetes? Refer to our [installation guide](/self-hosted/deploy/kubernetes/#installation).

## Configure

We strongly recommend referring to our [Configuration guide](/self-hosted/deploy/kubernetes/configure) to learn about how to configure your Sourcegraph with Kubernetes instance.

## Deploy

Refer to our [installation guide](/self-hosted/deploy/kubernetes/) for details on how to deploy Sourcegraph.

Migrating from another [deployment type](/self-hosted/deploy/)? Refer to our [migration guides](/self-hosted/deploy/migrate-backup).

## Deploy with Kustomize

In order to deploy Sourcegraph that is configured for your cluster:

#### Building manifests

Build a new set of manifests using an overlay you've created following our [configuration guide for Kustomize](/self-hosted/deploy/kubernetes/kustomize/):

```bash
$ kubectl kustomize $PATH_TO_OVERLAY -o cluster.yaml
```

#### Reviewing manifests

Review the manifests generated in the previous step:

```bash
$ less cluster.yaml
```

#### Applying manifests

Run the command below to apply the manifests from the output file `cluster.yaml` to the connected cluster:

```bash
$ kubectl apply --prune -l deploy=sourcegraph -f cluster.yaml
```

Once you have applied your changes:

-   _Watch_ - verify your deployment has started:

    ```bash
    $ kubectl get pods -A -o wide --watch
    ```

-   _Port-forward_ - verify Sourcegraph is running by temporarily making the frontend port accessible:

    ```sh
    $ kubectl port-forward svc/sourcegraph-frontend 3080:30080
    ```

-   _Log in_ - browse to your Sourcegraph deployment, login, and verify the instance is working as expected.

## Compare overlays

Below are the commands that will output the differences between the two overlays, allowing you to review and compare the changes and ensure that the new overlay produces similar resources as the ones currently being used by the active cluster or another overlay you want to compare with, before applying the new overlay.

### Between two overlays

To compare resources between two different Kustomize overlays:

```bash
$ diff \
    <(kubectl kustomize $PATH_TO_OVERLAY_1) \
    <(kubectl kustomize $PATH_TO_OVERLAY_2) |\
    more
```

Example 1: compare diff between resources generated by the k3s overlay for size xs instance and the k3s overlay for size xl instance:

```bash
$ diff \
    <(kubectl kustomize examples/k3s/xs) \
    <(kubectl kustomize examples/k3s/xl) |\
    more
```

Example 2: compare diff between the new base cluster and the old cluster:

```bash
$ diff \
    <(kubectl kustomize examples/base) \
    <(kubectl kustomize examples/old-cluster) |\
    more
```

Example 3: compare diff between the output files from two different overlay builds:

```bash
$ kubectl kustomize examples/old-cluster -o old-cluster.yaml
$ kubectl kustomize examples/base -o new-cluster.yaml

$ diff old-cluster.yaml new-cluster.yaml
```

### Between an overlay and a running cluster

To compare the difference between the manifests generated by an overlay and the resources that are being used by the running cluster connected to the kubectl tool:

```bash
$ kubectl kustomize $PATH_TO_OVERLAY | kubectl diff -f  -
```

The command will output the differences between the customizations specified in the overlay and the resources currently running in the cluster, allowing you to review the changes and ensure that the overlay produces similar resources to the ones currently being used by the active cluster before applying the new overlay.

Example: compare diff between the k3s overlay for size xl instance and the instance that is connected with `kubectl`:

```bash
$ kubectl kustomize examples/k3s/xl | kubectl diff -f  -
```

## List pods in cluster

List all pods in your cluster and the corresponding health status of each pod:

```bash
$ kubectl get pods -o=wide
```

## Tail logs for specific pod

Tail the logs for the specified pod:

```bash
$ kubectl logs -f $POD_NAME
```

If Sourcegraph is unavailable and the `sourcegraph-frontend-*` pod(s) are not in status `Running`, then view their logs with `$ kubectl logs -f sourcegraph-frontend-$POD_ID` (filling in `$POD_ID` from the `$ kubectl get pods` output). Inspect both the log messages printed at startup (at the beginning of the log output) and recent log messages.

## Retrieving resource information

Display detailed information about the status of a single pod:

```bash
$ kubectl describe $POD_NAME
```

List all Persistent Volume Claims (PVCs) and their statuses:

```bash
$ kubectl get pvc
```

List all Persistent Volumes (PVs) that have been provisioned.
In a healthy cluster, there should be a one-to-one mapping between PVs and PVCs:

```bash
$ kubectl get pv
```

List all events in the cluster's history:

```bash
$ kubectl get events
```

Delete failing pod so it gets recreated, possibly on a different node:

```bash
$ kubectl delete pod $POD_NAME
```

Remove all pods from a node and mark it as unschedulable to prevent new pods from arriving

```bash
$ kubectl drain --force --ignore-daemonsets --delete-local-data $NODE
```

Restarting Sourcegraph Instance:

```bash
$ kubectl rollout restart deployment sourcegraph-frontend
```

## Access the database

Check that the `pgsql` pod is running:

```bash
$ kubectl get pods -l app=pgsql
NAME      READY   STATUS    RESTARTS   AGE
pgsql-0   2/2     Running   0          19m
```

Make sure you are operating under the correct namespace (i.e. add `-n prod` if your pod is under the `prod` namespace).

Open a PostgreSQL interactive terminal:

```bash
$ kubectl exec -it pgsql-0 -- psql -U sg
```

Run your SQL query:

```sql
SELECT * FROM users;
```

> NOTE: To execute an SQL query against the database without first creating an interactive session (as below), append `--command "SELECT * FROM users;"` to the docker container exec command.

## Backup and restore

The following instructions are specific to backing up and restoring the sourcegraph databases in a Kubernetes deployment. These do not apply to other deployment types.

> WARNING: **Only core data will be backed up**.
>
> These instructions will only back up core data including user accounts, configuration, repository-metadata, etc. Other data will be regenerated automatically:
>
> -   Repositories will be re-cloned
> -   Search indexes will be rebuilt from scratch
>
> The above may take a while if you have a lot of repositories. In the meantime, searches may be slow or return incomplete results. This process rarely takes longer than 6 hours and is usually **much** faster.

> NOTE: Add `-n $NAMESPACE` to commands if you are not using the default namespace. The database pods are `pgsql-0`, `codeintel-db-0`, and `codeinsights-db-0`.
> More kubectl configuration options can be found here: [kubectl Cheat Sheet](https://kubernetes.io/docs/reference/kubectl/cheatsheet/)

### Back up Sourcegraph databases

These instructions will back up the primary `sourcegraph` database, the [codeintel](/code-navigation/) database, and the [codeinsights](/code-insights/) database.

1. Verify your Sourcegraph pods are running / healthy, especially the 3 database pods

```bash
$ kubectl get pods
```

2. Check for corrupt database indexes. The check blocks writes to each table while it checks that table's indexes. If amcheck returns errors, please reach out to [support@sourcegraph.com](mailto:support@sourcegraph.com)

Save this query in a file named `amcheck.sql`:

```sql
create extension if not exists amcheck;

select bt_index_parent_check(c.oid, true), c.relname, c.relpages
from pg_index i
join pg_opclass op ON i.indclass[0] = op.oid
join pg_am am ON op.opcmethod = am.oid
join pg_class c ON i.indexrelid = c.oid
join pg_namespace n ON c.relnamespace = n.oid
where am.amname = 'btree'
-- Don't check temp tables, which may be from another session:
and c.relpersistence != 't'
-- Skip partitioned indexes; the indexes of their partitions are checked:
and c.relkind = 'i'
-- Function may throw an error when this is omitted:
and i.indisready AND i.indisvalid;
```

Run it on each database. Each command prints any errors, and exits 0 when every index is intact:

```bash
$ kubectl exec -i pgsql-0 -- \
    psql -v ON_ERROR_STOP=1 --username sg sg \
    < amcheck.sql > /dev/null

$ kubectl exec -i codeintel-db-0 -- \
    psql -v ON_ERROR_STOP=1 --username sg sg \
    < amcheck.sql > /dev/null

$ kubectl exec -i codeinsights-db-0 -- \
    psql -v ON_ERROR_STOP=1 --username postgres postgres \
    < amcheck.sql > /dev/null
```

3. Generate the database dumps

```bash
$ kubectl exec pgsql-0 -- \
    pg_dump -C --clean --if-exists --username sg sg \
    > sourcegraph_db.out

$ kubectl exec codeintel-db-0 -- \
    pg_dump -C --clean --if-exists --username sg sg \
    > codeintel_db.out

$ kubectl exec codeinsights-db-0 -- \
    pg_dump -C --clean --if-exists --username postgres postgres \
    > codeinsights_db.out
```

Run these commands without `-it`: with a terminal attached, every line of the dump ends with a carriage return.

Ensure the `sourcegraph_db.out`, `codeintel_db.out` and `codeinsights_db.out` files are moved to a safe and secure location.

### Restore Sourcegraph databases

These instructions apply to both a new deployment and an existing one. You don't need to delete the database volumes: the restore drops and recreates each database.

1. On the machine where you run `kubectl`, `cd` to the directory with the database dump files: `sourcegraph_db.out`, `codeintel_db.out` and `codeinsights_db.out`

2. Stop every Sourcegraph service except the databases, so nothing writes to the databases during the restore

```bash
$ kubectl scale deployment,statefulset --replicas=0 -l '
    deploy=sourcegraph,
    app.kubernetes.io/component notin
        (pgsql,codeintel-db,codeinsights-db)'
```

Wait until `kubectl get pods` shows no Sourcegraph pods other than `pgsql-0`, `codeintel-db-0`, and `codeinsights-db-0`.

3. Copy the database files into the pods

```bash
$ kubectl cp sourcegraph_db.out pgsql-0:/data/sourcegraph_db.out
$ kubectl cp codeintel_db.out codeintel-db-0:/data/codeintel_db.out
$ kubectl cp codeinsights_db.out \
    codeinsights-db-0:/var/lib/postgresql/data/codeinsights_db.out
```

The database containers have a read-only root filesystem, so copy the files into the database volumes. Step 5 deletes them after the restore.

4. Restore the databases

`DROP DATABASE ... WITH (FORCE)` disconnects the other sessions on each database, such as the `pgsql-exporter` sidecar, and drops it, so the dump can recreate it. `ON_ERROR_STOP=1` stops the restore at the first error; if a restore fails, fix the cause and run it again.

```bash
$ kubectl exec -it pgsql-0 -- \
    psql -v ON_ERROR_STOP=1 --username sg \
    -c "DROP DATABASE IF EXISTS sg WITH (FORCE)" \
    -f /data/sourcegraph_db.out postgres

$ kubectl exec -it codeintel-db-0 -- \
    psql -v ON_ERROR_STOP=1 --username sg \
    -c "DROP DATABASE IF EXISTS sg WITH (FORCE)" \
    -f /data/codeintel_db.out postgres

$ kubectl exec -it codeinsights-db-0 -- \
    psql -v ON_ERROR_STOP=1 --username postgres \
    -c "DROP DATABASE IF EXISTS postgres WITH (FORCE)" \
    -f /var/lib/postgresql/data/codeinsights_db.out template1
```

5. After all three restores succeed, delete the backup files from the pods

```bash
$ kubectl exec pgsql-0 -- rm /data/sourcegraph_db.out
$ kubectl exec codeintel-db-0 -- rm /data/codeintel_db.out
$ kubectl exec codeinsights-db-0 -- \
    rm /var/lib/postgresql/data/codeinsights_db.out
```

6. Check for corrupt database indexes, with the commands in step 2 of [Back up Sourcegraph databases](#back-up-sourcegraph-databases). If amcheck returns errors, please reach out to [support@sourcegraph.com](mailto:support@sourcegraph.com)

7. Start the remaining Sourcegraph services, by running `helm upgrade` with the chart version and values file of your deployment (`helm list` shows the chart version). It sets every service back to the number of replicas in your values.

```bash
$ helm upgrade --install --values ./override.yaml \
    --version <chart_version> \
    sourcegraph sourcegraph/sourcegraph
```

If you deployed with Kustomize, follow the steps in [applying manifests](#applying-manifests) instead.

8. Browse to your Sourcegraph deployment, log in, and verify your existing configuration has been restored

## List of ports

To see a list of ports that are currently being used by your Sourcegraph instance:

```bash
$ kubectl get services
```

Example output:

```bash
NAME                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
blobstore                       ClusterIP   10.72.3.144    <none>        9000/TCP                     25h
cadvisor                        ClusterIP   10.72.14.130   <none>        48080/TCP                    23h
codeinsights-db                 ClusterIP   10.72.6.240    <none>        5432/TCP,9187/TCP            25h
codeintel-db                    ClusterIP   10.72.5.10     <none>        5432/TCP,9187/TCP            25h
gitserver                       ClusterIP   None           <none>        10811/TCP                    25h
grafana                         ClusterIP   10.72.6.245    <none>        30070/TCP                    25h
indexed-search                  ClusterIP   None           <none>        6070/TCP                     25h
indexed-search-indexer          ClusterIP   None           <none>        6072/TCP                     25h
kubernetes                      ClusterIP   10.72.0.1      <none>        443/TCP                      25h
node-exporter                   ClusterIP   10.72.5.60     <none>        9100/TCP                     25h
otel-collector                  ClusterIP   10.72.9.221    <none>        4317/TCP,4318/TCP,8888/TCP   25h
pgsql                           ClusterIP   10.72.6.23     <none>        5432/TCP,9187/TCP            25h
precise-code-intel-worker       ClusterIP   10.72.11.102   <none>        3188/TCP,6060/TCP            25h
prometheus                      ClusterIP   10.72.12.201   <none>        30090/TCP                    25h
redis-cache                     ClusterIP   10.72.15.138   <none>        6379/TCP,9121/TCP            25h
redis-store                     ClusterIP   10.72.4.162    <none>        6379/TCP,9121/TCP            25h
searcher                        ClusterIP   None           <none>        3181/TCP,6060/TCP            23h
sourcegraph-frontend            ClusterIP   10.72.12.103   <none>        30080/TCP,6060/TCP           25h
sourcegraph-frontend-internal   ClusterIP   10.72.9.155    <none>        80/TCP                       25h
syntect-server                  ClusterIP   10.72.14.49    <none>        9238/TCP,6060/TCP            25h
worker                          ClusterIP   10.72.7.72     <none>        3189/TCP,6060/TCP            25h
```

## Troubleshoot

See the [Troubleshooting docs](/self-hosted/deploy/kubernetes/troubleshoot).