Ask questionssourcegraph-frontend local disk usage grows unboundedly
sourcegraph-frontend uses local disk for caching purposes. This appears to appear unboundedly over time (up to 80 GB on sourcegraph.com). In most customer deployments, this seems fine (as occasional restarts are mostly benign), but we're seeing more customers now with tighter restrictions on disk space.
We can restrict this in the k8s deployment by requesting/limiting a certain amount of ephemeral storage (https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#local-ephemeral-storage), but even if we do that, the pod can be evicted, since it doesn't look like the application itself does any garbage collection. Can we implement garbage collection or some other mechanism to limit the size of the frontend cache?
Relevant to https://app.hubspot.com/contacts/2762526/company/557475882
Update (2020.5.12): This is now also affecting https://app.hubspot.com/contacts/2762526/company/464956351. It manifests as a pod eviction due to running out of ephemeral storage. We currently set no ephemeral storage request on the frontend pod, so it actually can be evicted if there's any disk pressure on the host. For this customer, at the time of eviction, the frontend pod was using 800GB of disk.
Answer questions slimsag
This has caused a customer's frontend pods to get evicted and prevents us from adding Kubernetes ephemeral storage limits, see: https://github.com/sourcegraph/sourcegraph/issues/9604
I don't think this is extremely urgent - but could you make sure this is on your roadmap somewhere @sourcegraph/cloud ? (if we can fix in the next 1-3 months I think that would be ideal)