profile
viewpoint
Geoffrey Gilmore ggilmore San Francisco, California

issue openedsourcegraph/sourcegraph

deploy-sourcegraph-dhall: incrementally migrate sourcegraph.com services over to dhall

In order to gather more feedback from @sourcegraph/distribution about the following aspects of deploy-sourcegraph-dhall:

  • what it's like to make larger changes to deploy-sourcegraph-dhall's components (adding new templating logic for more configuration options, adding new fields to the global configuration type)

  • what the pure-yaml -> dhall migration experience is like

I propose the following:

A member of @sourcegraph/distribution (other than @ggilmore or @uwedeportivo) will migrate the following subset of sourcegraph.com services to deploy-sourcegraph-dhall:

  • Frontend (still thinking about this one - this is the riskiest since it sees the most configuration changes/commits by far)
  • Redis
  • Prometheus

In the end state:

  • The above components will have their folders / K8s YAML files removed from 'base/' on deploy-sourcegraph-dot-com
  • deploy-sourcegraph-dot-com will contain a dhall configuration pipeline that outputs a K8s YAML file that's semantically equivalent to what was in base/ before. This YAML file will be checked into the repository.
  • deploy-sourcegraph-dot-com's CI will kubectl apply this new K8s YAML from dhall in addition to the remaining K8s YAML from services that haven't been migrated

This change should be entirely transparent since the K8s YAML that deploy-sourcegraph-dhall generates will be semantically identical to what was there before.


Note: Unlike https://github.com/sourcegraph/sourcegraph/issues/11830, deploy-sourcegraph-dot-com doesn't have a clean diff that clearly shows the transformations that will need to be made to the base configuration. This is a known problem that will need to be solved (see https://github.com/sourcegraph/sourcegraph/issues/12110).

To start off with, the implementer will rely on the git diff output between deploy-sourcegraph-dot-com and deploy-sourcegraph. If this proves to be insufficient, then @ggilmore will use the implementer's feedback to figure which of the solutions from https://github.com/sourcegraph/sourcegraph/issues/12110 we should invest in.

created time in 3 days

issue openedsourcegraph/sourcegraph

deploy-sourcegraph-dhall: implement migration path from pure-yaml -> dhall

We should have a story for capturing and translating the modifications that users built on top of pure-yaml deploy-sourcegraph over to deploy-sourcegraph-dhall.

Here are a few ideas for this:

  • Ask for the git diff between their fork and the upstream deploy-sourcegraph, and then do the translation by hand in the pipeline file.
  • Use kubectl diff against the output of deploy-sourcegraph-dhall and iteratively tweak the pipeline configuration until the diff is a no-op (not sure if the kubectl diff output is very clean - might include runtime information).
  • Write scripts / programs to read the existing cluster configuration (either from the YAML or from what the kubectl api server returns) and translate it into a dhall record that's structured in the same way that https://github.com/sourcegraph/deploy-sourcegraph-dhall/blob/master/src/base/component.dhall specifies. dhall diff can give us a fine-grained diff that can be used to iterate on the pipeline configuration until the diff is empty.
  • Compare the YAML from the existing deployment and the dhall pipeline's output. The YAML will need to be cleaned and normalized (collapsed into single kubernetes list, keys recusively sorted, whitespace trimmed, etc.) so that the diff is clean.

created time in 4 days

issue openedsourcegraph/sourcegraph

deploy-sourcegraph-dhall: grafana: implement new configuration logic for k8s.sgdev.org

Right now the generate function in Grafana https://github.com/sourcegraph/deploy-sourcegraph-dhall/blob/master/src/base/grafana/generate.dhall doesn't actually read the configuration that's passed to it .

https://github.com/sourcegraph/deploy-sourcegraph-dogfood-k8s/blob/ca1a4d1a7e678f6fa65abd2fdab5047fff31000e/generated/params.sh#L38-L44

This issue tracks the work that's needed to implement the k8s.sgdev.org transformations specified for Redis in the above file.

  • new configuration fields will need to be added to https://github.com/sourcegraph/deploy-sourcegraph-dhall/blob/master/src/base/grafana/configuration.dhall
  • new logic will need to be written in https://github.com/sourcegraph/deploy-sourcegraph-dhall/blob/master/src/base/grafana/generate.dhall to support these new transformations.

The PR implementing this functionality will include an example pipeline configuration in the description that declares the same transformations as the ones linked above.

created time in 4 days

issue openedsourcegraph/sourcegraph

deploy-sourcegraph-dhall: redis: implement configuration logic for k8s.sgdev.org

Right now the generate function in Redis https://github.com/sourcegraph/deploy-sourcegraph-dhall/blob/master/src/base/redis/generate.dhall doesn't actually read the configuration that's passed to it .

https://github.com/sourcegraph/deploy-sourcegraph-dogfood-k8s/blob/ca1a4d1a7e678f6fa65abd2fdab5047fff31000e/generated/params.sh#L92-L98

This issue tracks the work that's needed to implement the k8s.sgdev.org transformations specified for Redis in the above file.

  • new configuration fields will need to be added to https://github.com/sourcegraph/deploy-sourcegraph-dhall/blob/master/src/base/redis/configuration.dhall
  • new logic will need to be written in https://github.com/sourcegraph/deploy-sourcegraph-dhall/blob/master/src/base/redis/generate.dhall to support these new transformations.

The PR implementing this functionality will include an example pipeline configuration in the description that declares the same transformations as the ones linked above.

created time in 4 days

issue openedsourcegraph/sourcegraph

deploy-sourcegraph-dhall: frontend: implement new configuration logic for k8s.sgdev.org

Right now the generate function in Frontend https://github.com/sourcegraph/deploy-sourcegraph-dhall/blob/master/src/base/frontend/generate.dhall doesn't support all the transformations that are currently specified in the k8s.sgdev.org transformation pipeline:

https://github.com/sourcegraph/deploy-sourcegraph-dogfood-k8s/blob/ca1a4d1a7e678f6fa65abd2fdab5047fff31000e/generated/params.sh#L13-L25

New configuration fields will need to be added to https://github.com/sourcegraph/deploy-sourcegraph-dhall/blob/master/src/base/frontend/configuration.dhall and new logic will need to be written in https://github.com/sourcegraph/deploy-sourcegraph-dhall/blob/master/src/base/frontend/generate.dhall to support these new transformations.

The PR implementing this functionality will include an example pipeline configuration in the description that declares the same transformations as the ones linked above.

created time in 4 days

issue openedsourcegraph/sourcegraph

RFC 141: prepare k8s.sgdev.org infrastructure for dhall-based pipeline

https://github.com/sourcegraph/deploy-sourcegraph-dogfood-k8s is currently running the old bash-based prototype. The following work needs to be done to transition over to the new deploy-sourcegraph-dhall-based pipeline for dogfooding.

CI

  • [ ] the buildkite agents will need to have the dhall-asdf plugin baked into their docker images
  • Various scripts will need to be written:
  • [ ] a script to render the dhall configuration pipleine into a single kubernetes yaml file
  • [ ] a script apply the kubernetes yaml file from the step above to the cluster
  • [ ] a script that checks that the kubernetes yaml file that's checked into the repository is in sync with what the dhall-pipeline renders

Infrastructure

  • [ ] deploy-sourcegraph-dogfood-k8s's existing CI pipeline seems to be broken as a result of the testing for https://github.com/sourcegraph/sourcegraph/issues/10455. In addition, it's possible that there will be some downtime on k8s.sgdev.org during this bash->dhall transition period. k8s.sgdev.org is used for demos (this will need to be fixed separately), so it's important that it stay up. To "fix" both of these and move as quickly as possible, I propose to create a clone of the existing k8s.sgdev.org cluster under a separate domain (k8s-dhall.sgdev.org). The postgres taable frok k8s.sgdev.org will need to be dumped and restored on the new cluster.

created time in 4 days

create barnchsourcegraph/deploy-sourcegraph-dhall

branch : schemas

created branch time in 5 days

delete branch sourcegraph/deploy-sourcegraph-dhall

delete branch : revert-4-ci-change-package

delete time in 5 days

delete branch sourcegraph/deploy-sourcegraph-dhall

delete branch : simplify.sh

delete time in 5 days

create barnchsourcegraph/deploy-sourcegraph-dhall

branch : simplify.sh

created branch time in 5 days

delete branch sourcegraph/deploy-sourcegraph-dhall

delete branch : simplify.sh

delete time in 5 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 5986f86c64b87160778fb391cf37da1ca131918e

remove boilerplate from raw yaml-to-dhall output (#8)

view details

push time in 5 days

PR merged sourcegraph/deploy-sourcegraph-dhall

remove boilerplate from raw yaml-to-dhall output

Thanks to https://github.com/dhall-lang/dhall-haskell/pull/1902 and @Gabriel439, I was able to use a preview build of dhall schemas to partially automate the use of K8s types instead of the raw yaml-to-dhall output.

+2520 -30952

0 comment

14 changed files

ggilmore

pr closed time in 5 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha f1182559ec5476739bbf572d0d60d25bc7b76738

rm simple.dhall helper files

view details

Geoffrey Gilmore

commit sha dd34acf2848d8368375e999b0d1f18fd04a1c5cd

rm simplify.sh helper script

view details

push time in 5 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 9f220246da113bc4108d555b5070548606fc4c7b

simplify redis

view details

Geoffrey Gilmore

commit sha 359a20ab640b3d03ecafeb9ebdf723c884919e6d

simplify replacer

view details

Geoffrey Gilmore

commit sha 8bc649ba1065ab9393366bb9a810df11fb1fb4ab

simplify repo-updater

view details

Geoffrey Gilmore

commit sha eea41f2f07ebd064e90c2460793870782d7e39dd

simple searcher

view details

Geoffrey Gilmore

commit sha 72bf14290309dafdb7185d8e6db0dc07adb590fa

simple symbols

view details

Geoffrey Gilmore

commit sha 84a081f65051af017f63eb508293f48859eeca16

simple syntax highlighter

view details

push time in 5 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha c80c69464bc3c8c7666679f3b07ebd1a95ce0669

simplify prometheus

view details

Geoffrey Gilmore

commit sha ed57a3414ef07ed343617bde0410cd742c23c4ee

simplify query runner

view details

push time in 5 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 30d74c02351fae398104d74a329d6747aad70482

simplify indexed-search

view details

Geoffrey Gilmore

commit sha 4973e5d8de9fbe396d6431a714814a482b9f267d

simplify jaeger

view details

Geoffrey Gilmore

commit sha 4c9dce9f84fea786502933477a6e38bf612bf3fb

simplify precise-code-intel

view details

push time in 5 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 19740a4a82b1ff9dd02bb98e8c03445e89cd8026

simplify grafana

view details

push time in 5 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 31d1eb60fef344c6ee115f541b33630865f4db88

simplify cadvisor

view details

Geoffrey Gilmore

commit sha 03308add46807f02f728921ffcc089490213227d

simpliify github proxy

view details

push time in 5 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha aef740df0594a18d4adeac8f50497e55eea96f1f

query-runner

view details

Geoffrey Gilmore

commit sha 1d2a4e8568d885209b1d5253102e1ac07850e0b4

redis

view details

Geoffrey Gilmore

commit sha 119e78366106cecf58b413ae7c0b6aae251930c6

redis

view details

Geoffrey Gilmore

commit sha cca4bded66b100847ef69949c63bde614841ad2d

replacer

view details

Geoffrey Gilmore

commit sha 03c009941541029decc48609a2a3ad992b150a1e

repo-updater

view details

Geoffrey Gilmore

commit sha c85a5efff9c4087d1f46c057f6d4450d08190482

searcher

view details

Geoffrey Gilmore

commit sha 7ddd0c3eae6577ee1604b2377fa3b954f67a51e8

symbols

view details

Geoffrey Gilmore

commit sha 9342571fb4641ff59b72bf9a7a7e54dc024da454

syntax highlighter

view details

push time in 6 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha f992af6ce4c042ef27fc962d63f6426ce53bf187

correct import

view details

Geoffrey Gilmore

commit sha 2dd0a2b7289b9355bd0a0e3a97b5b97457124861

jaeger

view details

Geoffrey Gilmore

commit sha ad7f560937547b3f909039b62e13695864a12687

postgres

view details

Geoffrey Gilmore

commit sha ff2e1a069b6f61a9061e093c1d7077eac7c03245

precise-code-intel.dhall

view details

Geoffrey Gilmore

commit sha e240b0cac78340d03c1f873c69c2fca729080417

prometheus

view details

push time in 6 days

create barnchsourcegraph/deploy-sourcegraph-dhall

branch : simplify.sh

created branch time in 6 days

delete branch sourcegraph/deploy-sourcegraph-dhall

delete branch : more-config

delete time in 6 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha cbf603bba06a63f1b295992925faac63cdd34b4e

update services to more recent deploy-sourcegraph configuration (#7)

view details

push time in 6 days

PR merged sourcegraph/deploy-sourcegraph-dhall

update services to more recent deploy-sourcegraph configuration

Based off of https://github.com/sourcegraph/deploy-sourcegraph/commit/9fae2fbcb3d619e4679bef41cc5aae03bcebe69c

Fixes https://github.com/sourcegraph/sourcegraph/issues/11829

+35385 -34726

0 comment

82 changed files

ggilmore

pr closed time in 6 days

issue closedsourcegraph/sourcegraph

deploy-sourcegraph-dhall: bring resources up-to-date with latest deploy-sourcegraph commits

https://github.com/sourcegraph/deploy-sourcegraph-dhall is based off an old commit of deploy-sourcegraph: https://github.com/sourcegraph/deploy-sourcegraph/tree/dhall-rfc-141

The configuration needs to be caught up to date with https://github.com/sourcegraph/deploy-sourcegraph/commit/3437ce590576d46ac275b19849f2fea8ed6f4ac0.


This should be fairly straightforward: it's mostly just a lot of copy pasting.

The following rough script creates a dhall record of all the resources (keyed by the resource's kind when run in the root of a component's folder (example: https://github.com/sourcegraph/deploy-sourcegraph/tree/3437ce590576d46ac275b19849f2fea8ed6f4ac0/base/cadvisor).

#!/usr/bin/env bash

cd "$(dirname "${BASH_SOURCE[0]}")"
set -euo pipefail

record="{=}"

for f in *.yaml; do
  kind=$(yq read "$f" 'kind')
  dhallExpression=$(printf "let kubernetes = https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/master/1.17/package.dhall in kubernetes.%s.Type" "$kind")

  k8sDhall="$(yaml-to-dhall --file "$f" --records-loose "${dhallExpression}")"
  record="$record /\ { $kind = ${k8sDhall} }"
done

echo "$record" | dhall --explain --output upstream.dhall

[Note that won't work off-the-shelf for every compoment (see sourcegraph-frontend's two services) - but it's easy enough to hack the above script for those purposes)].

The upstream.dhall file can be compared against the current component with dhall diff.

Repeat this copy-paste process for each component in https://github.com/sourcegraph/deploy-sourcegraph-dhall

closed time in 6 days

ggilmore

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 9181284fee4526339698ea252e02fb5516791c19

diff.dhall generate-plain.dhall

view details

Geoffrey Gilmore

commit sha 65b90fb3199bf48141c78898f65bd9468e7b55d4

adjust ci check

view details

push time in 6 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 1ddc8bac2d312968bb78caffe30913de83c3dd5d

remove freeze from pipeline

view details

push time in 6 days

push eventsourcegraph/deploy-sourcegraph

Geoffrey Gilmore

commit sha 9fae2fbcb3d619e4679bef41cc5aae03bcebe69c

wip

view details

push time in 6 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 1e767bcd168f4bdc9217a4f0fe8297da1cad491c

ci: add generate to typecheck stop

view details

push time in 6 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha fa5da0bfb799074d8f1db4d04dc77a53bad198ab

remove diff.sh

view details

push time in 6 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 651bd672711a7cebb6481787417ff966fa3fcea2

remove upstream files

view details

push time in 6 days

create barnchsourcegraph/deploy-sourcegraph

branch : dhall-again

created branch time in 6 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 2966ea4f4476e8439d169e774f074dcf88a49f16

update all components

view details

push time in 6 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 566def8a580a23735d4da2f8a1536034278a8a1a

remove generate from package.dhall

view details

push time in 7 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 0f76cde3b6e44f1155eb173e4a753839aec0257e

extend refactor to render.dhall

view details

push time in 8 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha ca1e7c19a7168867ae515609007d5fb15f65401f

add all render functions

view details

Geoffrey Gilmore

commit sha b33ffbd19a7e6bcd239e34495c52776baac033bf

lint

view details

push time in 8 days

delete branch ggilmore/tep-alumni-website

delete branch : ci-each-commit

delete time in 9 days

push eventggilmore/tep-alumni-website

Geoffrey Gilmore

commit sha bf068a85c966452b20f09bd23c87f89a694116cf

ci: run suite on every commit (#8)

view details

push time in 9 days

delete branch ggilmore/tep-alumni-website

delete branch : dhall-1.33.1

delete time in 9 days

push eventggilmore/tep-alumni-website

Geoffrey Gilmore

commit sha 813bbd07a49571933ad0fcfdb73162724a835535

dhall 1.33.1

view details

Geoffrey Gilmore

commit sha 97f6f0851c4debf228f010ee658638807900dc0a

switch back to upstream github-actions

view details

push time in 9 days

create barnchggilmore/tep-alumni-website

branch : ci-each-commit

created branch time in 9 days

delete branch ggilmore/tep-alumni-website

delete branch : shortcode

delete time in 9 days

push eventggilmore/tep-alumni-website

Geoffrey Gilmore

commit sha fdf7d3905abe7abb707892e9c5a2f6043defaa06

add donation buttons back (#7)

view details

push time in 9 days

create barnchggilmore/tep-alumni-website

branch : shortcode

created branch time in 9 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 8eec7fba8f8421c377268ad49922d15c80729a37

lint

view details

push time in 12 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha d4497bbe2e736fed7878d59129caef863ff7027f

ci: run checks on every commit (#5)

view details

Geoffrey Gilmore

commit sha 289839b6efdb7c23a53ca254d19a379b431c51dd

dev: use watchexec instead of entr (#6)

view details

Geoffrey Gilmore

commit sha de1f3cf37378ad2d949eac3b4a6a54e8b1197080

Merge branch 'master' into more-config

view details

Geoffrey Gilmore

commit sha 200c2e515225dc5aebbdc3bc3abe26e5e3ecc7af

Merge remote-tracking branch 'origin/master' into more-config

view details

Geoffrey Gilmore

commit sha 48094aa65bd0ce9d6c5653135e96eacb89391b79

wip

view details

push time in 12 days

delete branch sourcegraph/deploy-sourcegraph-dhall

delete branch : use-watchexec

delete time in 13 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 289839b6efdb7c23a53ca254d19a379b431c51dd

dev: use watchexec instead of entr (#6)

view details

push time in 13 days

create barnchsourcegraph/deploy-sourcegraph-dhall

branch : use-watchexec

created branch time in 13 days

delete branch sourcegraph/deploy-sourcegraph-dhall

delete branch : ci-all-commits

delete time in 13 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha d4497bbe2e736fed7878d59129caef863ff7027f

ci: run checks on every commit (#5)

view details

push time in 13 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha f1e8bcb55fa61644f549e7a601586ed25cc5a010

rename some checks

view details

push time in 13 days

create barnchsourcegraph/deploy-sourcegraph-dhall

branch : ci-all-commits

created branch time in 13 days

create barnchsourcegraph/deploy-sourcegraph-dhall

branch : more-config

created branch time in 13 days

issue openedsourcegraph/sourcegraph

deploy-sourcegraph-dhall: extend configuration layer for subset of services on k8s.sgdev.org

https://github.com/sourcegraph/deploy-sourcegraph-dogfood-k8s/blob/4669f10e139feccd1e8a68490faecd9c5b76437d/generated/params.sh#L11-L44

Right now, k8s.sgdev.org is configured via a bash-based configuration pipeline as a result RFC 141. This will be replaced by https://github.com/sourcegraph/deploy-sourcegraph-dhall as part of the Dhall dogfooding process.

Part of this will be done by @ggilmore to lay the groundwork. The rest will be done by another member of the distribution team to spread knowledge and solicit feedback. This other team member will need to need to go through the dhall tutorial + pair with me in order to get up to speed about the Dhall language itself and https://github.com/sourcegraph/deploy-sourcegraph-dhall's architecture.

@ggilmore proposes that he'll replicate frontend, grafana, and redis from https://github.com/sourcegraph/deploy-sourcegraph-dogfood-k8s/blob/release/generated/params.sh. The other distribution teammate will handle the other services

The work involved here includes:

  • writing scripts to invoke dhall in the k8s.sgdev.org pipeline
  • refactoring k8s.sgdev.org's bash-based configuration scripts to merge the output of the dhall-based-pipeline can be deployed alongside the legacy bash-based one
  • Writing more dhall code in deploy-sourcegraph-dhall to handle new configuration needs (such as increasing the storage capacity of a persistent volume claim - needed for gitserver, etc.)
  • Wiring up the configuration layer for all services in deploy-sourcegraph-dhall, not just the ones listed in https://github.com/sourcegraph/sourcegraph/issues/10936.

created time in 14 days

issue openedsourcegraph/sourcegraph

deploy-sourcegraph-dhall: bring resources up-to-date with latest deploy-sourcegraph commits

https://github.com/sourcegraph/deploy-sourcegraph-dhall is based off an old commit of deploy-sourcegraph: https://github.com/sourcegraph/deploy-sourcegraph/tree/dhall-rfc-141

The configuration needs to be caught up to date with https://github.com/sourcegraph/deploy-sourcegraph/commit/3437ce590576d46ac275b19849f2fea8ed6f4ac0.


This should be fairly straightforward: it's mostly just a lot of copy pasting.

The following rough script creates a dhall record of all the resources (keyed by the resource's kind when run in the root of a component's folder (example: https://github.com/sourcegraph/deploy-sourcegraph/tree/3437ce590576d46ac275b19849f2fea8ed6f4ac0/base/cadvisor).

#!/usr/bin/env bash

cd "$(dirname "${BASH_SOURCE[0]}")"
set -euo pipefail

record="{=}"

for f in *.yaml; do
  kind=$(yq read "$f" 'kind')
  dhallExpression=$(printf "let kubernetes = https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/master/1.17/package.dhall in kubernetes.%s.Type" "$kind")

  k8sDhall="$(yaml-to-dhall --file "$f" --records-loose "${dhallExpression}")"
  record="$record /\ { $kind = ${k8sDhall} }"
done

echo "$record" | dhall --explain --output upstream.dhall

[Note that won't work off-the-shelf for every compoment (see sourcegraph-frontend's two services) - but it's easy enough to hack the above script for those purposes)].

The upstream.dhall file can be compared against the current component with dhall diff.

Repeat this copy-paste process for each component in https://github.com/sourcegraph/deploy-sourcegraph-dhall

created time in 14 days

delete branch sourcegraph/deploy-sourcegraph-dhall

delete branch : ci-change-package

delete time in 14 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha e35a476a0edb9e99ccc60322872bd278fda4b955

ci: switch to upsteam package instead of my fork (#4)

view details

push time in 14 days

create barnchsourcegraph/deploy-sourcegraph-dhall

branch : ci-change-package

created branch time in 14 days

issue commentdhall-lang/dhall-haskell

performance advice for dhall-kubernetes based package

Hey @Gabriel439 thanks for looking into this! A few responses:

  • I am using dhall 1.33.1 for the benchmark https://github.com/sourcegraph/deploy-sourcegraph-dhall/blob/285b1515ade18682d1fac33f8e70bb32100dd67c/.tool-versions#L1.
  • 🤦 I realize now that I was freezing everything besides pipeline.dhall for the benchmark. Here are my benchmark numbers after freezing that file:
    ~/dev/go/src/github.com/sourcegraph/ds-dhall master* 28s
    ❯ bench  'dhall-to-yaml --file pipeline.dhall'
    benchmarking dhall-to-yaml --file pipeline.dhall
    time                 8.996 s    (8.203 s .. NaN s)
                         0.995 R²   (0.991 R² .. 1.000 R²)
    mean                 8.468 s    (8.285 s .. 8.781 s)
    std dev              299.8 ms   (45.39 ms .. 371.0 ms)
    variance introduced by outliers: 19% (moderately inflated)
    
    • I still find it surprising that there is a ~ 14 second speedup after I freeze the pipeline.dhall file alone. Can you explain why this is the case?
  • One of the ways I'm exploring to reduce the size of the cached expression is to see if we can encode things more compactly by specifying the union type only once

    • Ah, I thought it'd make more sense for each component to construct its own list of resources so that it'd be easier to concatenate them later without having to know all the resources in each component.
ggilmore

comment created time in 15 days

issue commentsourcegraph/sourcegraph

Distribution: 3.18 Tracking issue

This week I mainly focused on https://github.com/sourcegraph/deploy-sourcegraph-dhall for RFC 141:

  • I imported the rest of the deploy-sourcegraph services
  • Wrote scripts to set up a proper dev environment (linting, formatting, compiling on save)
  • Added CI (also rendered in Dhall)
  • Added benchmarks and opened https://github.com/dhall-lang/dhall-haskell/issues/1890 to ask for help investigating performance issues

I'm in the process of:

  • incorporating the configuration layer into the newly added services
  • generating a diff between https://github.com/sourcegraph/deploy-sourcegraph-dogfood-k8s and https://github.com/sourcegraph/deploy-sourcegraph-dhall so that we can migrate over and do proper dogfooding
slimsag

comment created time in 17 days

issue openeddhall-lang/dhall-haskell

performance advice with dhall-kubernetes based package

Forked from https://functionalprogramming.slack.com/archives/C82N1L0MB/p1592337474256100

My team is investigating Dhall to see if it'd be a good way to ship + customize our complex Kubernetes application on our internal clusters and for our customers to use.

https://github.com/sourcegraph/deploy-sourcegraph-dhall is my PoC implementation of our Kubernetes configuration. I like what I see so far, but I'm wondering how I can improve the render times that I'm seeing.

I've posted benchmark's in the https://github.com/sourcegraph/deploy-sourcegraph-dhall 's README: https://github.com/sourcegraph/deploy-sourcegraph-dhall#benchmarks


Going off of https://discourse.dhall-lang.org/t/figuring-out-performance-bottlenecks/251/5,one thing that I want to call out is that my normal form is apparently ~300x the size of the Prelude.

I know that dhall-kubernetes has a lot of types, but is this size expected?

  • Are there non-ideal things things that I'm doing in https://github.com/sourcegraph/deploy-sourcegraph-dhall that cause this bloat?
  • Are there some tools that can visualize the normal form and identify common patterns?

created time in 18 days

delete branch sourcegraph/deploy-sourcegraph-dhall

delete branch : editor-condig

delete time in 18 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha c002f47fe8508c95f823e8cc3aa688e002cb8423

ci: add editorconfig for shfmt (#3)

view details

push time in 18 days

create barnchsourcegraph/deploy-sourcegraph-dhall

branch : editor-condig

created branch time in 18 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha a3f32062b10e08435c6ad73085a5df8c4c30c00d

Update README.md

view details

push time in 18 days

delete branch sourcegraph/deploy-sourcegraph-dhall

delete branch : add-pipeline-check

delete time in 18 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 285b1515ade18682d1fac33f8e70bb32100dd67c

ci: add step to see if package compiles correctly (#2)

view details

push time in 18 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha dd0aa9425437d8ed6a9466c3a8e76eb3978d8783

fix case sensitivity issue

view details

push time in 18 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha b69a86f996601e9dad5871c9dfe02c6d6e8051d2

debug

view details

push time in 18 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 3e479effa86e3cea55c17444e3ab30cabd859b65

debug

view details

push time in 18 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha a3537b2b5a562e431aff692fab9df538e243698b

more debug

view details

push time in 18 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha ba603911b00609a29d81a3ca11a3695b1fefbe7a

try removing submodules

view details

push time in 18 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 01dd79267770bc5bf18bb13e92c6b78ff6167b46

remove themes?

view details

push time in 18 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 60e3dd2a14de44c17ae1c090343f88b019682597

debug

view details

push time in 18 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha bd755bd760a9fc9e85a40bdbc411e807823988bc

more debug

view details

push time in 18 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha cf6eb991334fe6e164d7dda625a9fb06e2c80814

debug

view details

push time in 18 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 43e1be10b87ffc48034dac935bb03eabd00b36f2

debug

view details

push time in 18 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 8ef8ebf9989446311d12c0179e5222a9fb99c3eb

try fixiing script

view details

push time in 18 days

create barnchsourcegraph/deploy-sourcegraph-dhall

branch : add-pipeline-check

created branch time in 18 days

push eventsourcegraph/deploy-sourcegraph-dhall

Geoffrey Gilmore

commit sha 6ebfad4852b42f46ed38282795ff2e9b7d75d18c

add rest of the deploy-sourcegraph services (#1)

view details

push time in 18 days

more