profile
viewpoint

beyang/cheerio 13

Some utility commands you wish were in Python's pip

beyang/black 1

Make everything black

beyang/about 0

Sourcegraph blog, feature announcements, and website (about.sourcegraph.com)

beyang/ajenti 0

Ajenti - the web admin panel everyone wants.

beyang/angular-app 0

Reference application for AngularJS

beyang/angular-dataform 0

Reliable, componentized data form controls for AngularJS

beyang/angular-seed 0

Seed project for angular apps.

fork beyang/opencv

Open Source Computer Vision Library

https://opencv.org

fork in 17 hours

push eventsourcegraph/sourcegraph

Beyang Liu

commit sha 7ba275d7126569d832ba50d9dd36a9f01f585e60

doc: codeintel: link to lsif quickstart from how-to page

view details

push time in 2 days

push eventsourcegraph/sourcegraph

Adam Harvey

commit sha 94671fb19895e4b3b6272f511788f65e144aaebf

campaigns: migrate pre-spec campaigns to have specs (#13001) * Basic migrations. * wip * Fix capitalisation on the JSON tag for ExternalID. * Implement the rest of the migration. * Fix test breakage from the database migration. * Apply generate changes. * Update internal/campaigns/types.go Co-authored-by: Erik Seliger <erikseliger@me.com> * Continue for changesets that never had campaigns. * Remove weird duplicate log15. * Handle the partial changeset migration case in the down migration. * Add ON DELETE SET NULL constraints. * More migration updates. * Add delete cascade integration test. * Fix generated schema. Co-authored-by: Erik Seliger <erikseliger@me.com>

view details

Eric Fritz

commit sha 380718a2624da12c6ae898bec2303b061cc2e9bd

doc/dev: Add code intelligence development documentation (#12949)

view details

Eric Fritz

commit sha 49d206bed71386cc11a47a0786c15361d9e43094

codeintel: Add fast-path edge unmarshalling (#12878)

view details

Adam Harvey

commit sha e0ae49d12196e18f345d877601610937dbec6ed8

campaigns: migrate to allow changeset spec user IDs to be NULL (#13020)

view details

Erik Seliger

commit sha 15230795276fc0c3f11763f3fe297d9f5027c470

Don't break campaign resolver when a related user is deleted (#13006)

view details

uwedeportivo

commit sha dcbd0181d398ac2e75335a075f70ed5e68baa526

e2e test: search visibility test refs wrong repo (#13021)

view details

Rijnard van Tonder

commit sha a9946a99a666be13562ed9934a923be047a52439

Revert "Update comby/comby Docker tag to v0.18.1 (#13003)" (#13027) This reverts commit 94dc08eb65896f04824f142092929086675da64a.

view details

Adam Harvey

commit sha 05170f3dc459fa2e9440c3ce5aed5314b846d16a

campaigns: update a migration with an UPDATE (#13030) The boat burning migration that was merged in #13001 didn't account for the possibility that a campaign may have been created in the narrow time window between the campaign_spec_id and last_updated_at fields being added. Since the sum total of people who might be affected is likely three, and they all work here, we'll deal with this by updating the existing migration.

view details

Erik Seliger

commit sha 41ccf166a444adf5928939a47324933b203ddea8

Cleanup campaign connection resolver (#13029) * Split up campaigns resolver in connection and node resolvers This follows the pattern we've been using with the other API entities as well. Also, I've added a separate test for the connection resolver to test pagination and moved the existing test for namespace resolvers from resolvers_test, which is way too large anyways. * Add cursor based pagination for campaigns This uses the exact same pattern we use for the changeset spec connection.

view details

Erik Seliger

commit sha d428d27ed7b4e9c8992f5057a63b4f2ce1cb8170

Add test for reconciler state variables (#13031) Especially important for error, as that path wasn't tested at all.

view details

Erik Seliger

commit sha b4d4ea7767ddccf35854711441f427483530b53d

Remove unused files (#13033)

view details

Erik Seliger

commit sha b36a0e4bbc9fb90468b14b8417ce1a0f0df9e5ee

Add method to access faster pre-computed diffstat (#13034) .. rather than parsing all diffs for the list view.

view details

Juliana Peña

commit sha a625db1e49f4fa6f314a5be8f5210d9050e23d7e

search: Monaco query editor should not syntax highlight links or unknown filters (#12992) Fixes #12961 with two changes to the Monaco query input: - URLs are no longer clickable - Input that looks like a filter (eg. 'https:' or C++ namespaces) are no longer treated as unknown filters or syntax highlighted Also fixes #10056 ![image](https://user-images.githubusercontent.com/206864/90187090-16fc5c80-dd6e-11ea-968d-738717a2bf83.png) ![image](https://user-images.githubusercontent.com/206864/90187145-2aa7c300-dd6e-11ea-9ca8-a1999baf15e5.png)

view details

Rijnard van Tonder

commit sha 068b3ca0004aba97ebd2defcdfd0d2b8d892d3ac

changelog entries for search operators (#13040)

view details

Erik Seliger

commit sha bde4f4a64bce185513c11539371077ac2be0775c

Unclutter the bitbucket server types (#13038)

view details

Quinn Slack

commit sha 6b66c6def8881590975d12366777cc49b013556d

fix names of programming languages (#13042)

view details

Aida DeWitt

commit sha 505a9aadadcdd296a8b91fa2be08d3b0d7277190

Improving code intelligence docs and LSIF indexer installation instructions (#13041) * Making code intelligence documents more use friendly and direct * renamed lsif.md to feature name: precise_code_intelligence * created a features page Co-authored-by: Eric Fritz <eric@sourcegraph.com> * combined javascript/typescript into one line * replaced "built-in" with "basic" for consistency

view details

ᴜɴᴋɴᴡᴏɴ

commit sha d76e96439c98a238216de86eaf08f3d4f1978f01

auth: enforce length requirement when reset password (#12971)

view details

Farhan Attamimi

commit sha 84bf8188a6a1c2efda244be83e90bdb90faf52a7

Changelog: add entries for repogroup pages and search homepage (#11921)

view details

Quinn Slack

commit sha a76ad2c9fe821bd68ac6966859e450d8a4f7f46d

clean up PrivateCodeCta (#13044) - Use <a> not <Link> for external links (if there were no `target="_blank"` I believe this would not have worked; it would have treated the `to` prop value as a relative path) - To style a link as a button, use the Bootstrap classes directly, instead of embedding a button in a link (which is unnecessary and could have unintended consequences if this element is inside a form element) - Use consistent language: self-hosted not private, capitalize/spell Git/Perforce/Subversion in the standard way

view details

push time in 2 days

PR closed sourcegraph/sourcegraph

Reviewers
Don't regenerate if input files are older than output file (take 2)

First commit is what was pushed in #13782, second commit fixes #14039.

Empirically, this improves time-to-user-readiness for enterprise/dev/start.sh from 40-45s to 32-36s.

+94 -37

2 comments

4 changed files

beyang

pr closed time in 2 days

Pull request review commentsourcegraph/sourcegraph

Don't regenerate if input files are older than output file (take 2)

 const build = gulp.series(generate, webWebpack) /**  * Watches everything and rebuilds on file changes.  */-const dev = gulp.parallel(watchGenerate, webWebpackDevServer)+const dev = gulp.series(

We want to wait for the initial generate steps to complete before running the webpack dev server for 2 reasons:

  • The generated files may be missing when the the webpack dev server expects them to exist, causing errors to be printed.
  • The generate steps will immediately update the generated files, causing webpack dev server to immediately recompile before its first compilation is complete, wasting CPU cycles for no observable improvement in time-to-ready
beyang

comment created time in 4 days

PullRequestReviewEvent
PullRequestReviewEvent

pull request commentsourcegraph/sourcegraph

Don't regenerate if input files are older than output file (take 2)

@felixfbecker PTAL

beyang

comment created time in 4 days

push eventsourcegraph/sourcegraph

Beyang Liu

commit sha 18b93a120bf428a35292790a90c882fc7ea45bf5

web:build: don't regenerate files if generated files are newer than source

view details

push time in 4 days

push eventsourcegraph/sourcegraph

Dax McDonald

commit sha ebb9650b1e1cf08feb167d10035181701c5ad46f

Update sha for dev/private (#14077)

view details

Robert Lin

commit sha a2adb9f9ea68f29d41930b259b27aa1aad889262

prometheus: only override Slack username if it is empty (#14079)

view details

Beyang Liu

commit sha 8383a5ddf0e7354baac66b0b3b71ac0f0b4bcee7

add tracing to internal/db/repos/repos.Count (#14020)

view details

Robert Lin

commit sha cc74e8e33f5b61403517899a7c5c9b47a8b38325

codenotify: add bobheadxi subscriptions, fix entry (#14080)

view details

Quinn Slack

commit sha 1976511aa1ae525a4775c0126efb1be84859eca7

update release version on docs.sourcegraph.com (#14084)

view details

Erik Seliger

commit sha 20e2597bfb17f1a62abbd3162d8852a9bd279272

Implement a CodeSnippet component that encapsules the design (#14062)

view details

Asdine El Hrychy

commit sha 2008a8bfb9d6e8c60519af415b17f4040503ba6c

Drop sources column (#14094) * Drop sources column * Regenerate schema

view details

renovate[bot]

commit sha ccafaea0b167a60ee1ff4f3c04859d3d740f2974

Update dependency typescript to ^4.0.3 (#14040) Co-authored-by: Renovate Bot <bot@renovateapp.com>

view details

Eric Fritz

commit sha 180bbec981e36039bfa00f4da9c68bde419e5019

codenotify: Configure efritz's subscriptions (#14060)

view details

Eric Fritz

commit sha 4ac0214132549835b99569d50395c434cc7c8ba9

codeintel: Refactor command runner in indexer (#14102)

view details

Eric Fritz

commit sha d1e0f498d443c9dc3ab9342b0c70d0ebf396c54b

codeintel: Lower indexer output to debug level (#14103)

view details

Quinn Slack

commit sha a9ebb0fbc4136b79471185d623733ac41ff5e476

indent and highlight JSON campaign specs (#14087) A campaign spec is a YAML document. YAML is a superset of JSON. If a campaign spec is pure-JSON, then it's nice to indent it (in case it's all on one line, for example) and highlight it using the JSON highlighter. The button is still "Download YAML" and the filename is still "whatever.yaml", but that's OK because YAML is a superset of JSON, so those are correct.

view details

Eric Fritz

commit sha d233e758e35673446b8aae817c2cf2bc81760003

tracking-issue: Fix strikethrough on closed (unmerged) PRs (#14107)

view details

Eric Fritz

commit sha 9263b31038b03e29bf2c358b92daa4a1cf2709aa

codeintel: Refactor index command construction (#14105)

view details

Eric Fritz

commit sha 947f7da87983d36bdf763621045c114d948ef85f

codeintel: Move away from hardcoding image names (#14114)

view details

Felix Becker

commit sha d2f2b5f079f8f29716298a5a68ab4c525c581fd3

Pin cssnano version that treats :focus-visible correctly (#14086)

view details

Felix Becker

commit sha 1d56631d51f086e09da0f7294b18727fa8d2245f

Properly truncate path breadcrumbs (#14097) * Properly truncate path breadcrumbs Fixes #13535 * Add changelog entry * Prettier * add whitespace nowrap to path part * apply whitespace nowrap to all fp breadcrumbs Co-authored-by: TJ Kandala <kandalatj@gmail.com>

view details

Eric Fritz

commit sha 91d59f56fad75723103c07e94d020fab28420de1

codeintel: Refactor construction of copyfiles flags (#14116)

view details

Eric Fritz

commit sha 91ce92dd9d446011bf3e7f569d16a69bb9910151

codeintel: Add orderedKeys to indexer (#14117)

view details

Quinn Slack

commit sha bc1864ec4dddd6362970902881a1da0fe8a14052

show command palette immediately, don't wait for fade-in (#14119) This improves UI responsiveness and is consistent with other similar menus in the UI (that immediately appear without a fade-in).

view details

push time in 4 days

startedptitSeb/box86

started time in 5 days

startedcode-mancers/invoker

started time in 5 days

PullRequestEvent

pull request commentbeyondgrep/website

Add Sourcegraph to list of indexing tools

I was going through my outstanding PRs and closing ones that hadn't had any activity on them in awhile. Sorry if this was premature!

beyang

comment created time in 5 days

push eventsourcegraph/sourcegraph

Beyang Liu

commit sha 4e59a515ee4f10281e75060d73620f45dafffce2

oauth2: better debugging and validation * debug logging toggled by INSECURE_OAUTH2_LOG_TRACES * github client secret validator * troubleshooting docs Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>

view details

push time in 6 days

delete branch sourcegraph/sourcegraph

delete branch : bl/oauth2-logging

delete time in 6 days

PR merged sourcegraph/sourcegraph

Reviewers
oauth2: debug logging toggled by INSECURE_OAUTH2_LOG_TRACES

Logs will look like this, when INSECURE_OAUTH2_LOG_TRACES=true in the frontend environment:

18:06:37                          frontend | >>>>> HTTP Request: POST https://github.com/login/oauth/access_token                                                                                                                                                                                 
18:06:37                          frontend |       Header: map[Authorization:[Basic xxxxxxxxxxxxxxxxxxxxxx] Content-Type:[application/x-www-form-urlencoded]]                                                                       
18:06:37                          frontend |       Body: code=xxxxxxxxxxx&grant_type=authorization_code                                                                                                                                                                                                          
18:06:37                          frontend | >>>>> HTTP Request: POST https://github.com/login/oauth/access_token                                                                                                                                                                                                                                           
18:06:37                          frontend |       Header: map[Content-Type:[application/x-www-form-urlencoded]]                                                                                                                                                                                                                                            
18:06:37                          frontend |       Body: client_id=xxxxxxxxxxxxx&client_secret=xxxxxxxxxxxxxxxxxxxxxxx&code=xxxxxxxxxxxxxxxxxxxxxxx&grant_type=authorization_code                                                                                                                                                      

Also added validation for the GitHub client ID and secret to ensure no hidden or non-alphanumeric characters are used.

+111 -24

2 comments

7 changed files

beyang

pr closed time in 6 days

PR closed beyondgrep/website

Add Sourcegraph to list of indexing tools

Would it be appropriate to add Sourcegraph (https://github.com/sourcegraph/sourcegraph) to the list of indexing tools on the web page? It looks like the other tools listed are open source, while Sourcegraph is open core (main application is open-source, but there's an enterprise version that contains proprietary code for things like SSO). Unsure if that rules it out of this list, but a lot of our customers were using one of the indexing tools listed here (e.g., OpenGrok, Hound, cscope) before they adopted Sourcegraph, so thought it might be a worthy addition . If not, feel free to close and sorry about the noise!

P.S. thanks for creating this useful resource for people looking for better code search utilities. I've used ack a lot over the years, its speed and usability are fantastic, and from my vantage point (I used ack before anything else), it kicked off the explosion of better grep alternatives listed here!

+15 -0

0 comment

1 changed file

beyang

pr closed time in 6 days

push eventsourcegraph/sourcegraph

Beyang Liu

commit sha 3761e8c3f593b51f2001bf20d5ed424cc5dbbb17

fix test

view details

push time in 6 days

PR closed cool-repositories/sessions

Add GitHub action to generate LSIF for Go repositories

Adds a GitHub action that generates LSIF for all Go repositories (selected by the query file:go.mod$ fork:yes on this Sourcegraph instance.

+17 -0

0 comment

1 changed file

beyang

pr closed time in 6 days

PR closed cool-repositories/csrf

Add GitHub action to generate LSIF for Go repositories

Adds a GitHub action that generates LSIF for all Go repositories (selected by the query file:go.mod$ fork:yes on this Sourcegraph instance.

+17 -0

0 comment

1 changed file

beyang

pr closed time in 6 days

PR closed cool-repositories/schema

Add GitHub action to generate LSIF for Go repositories

Adds a GitHub action that generates LSIF for all Go repositories (selected by the query file:go.mod$ fork:yes on this Sourcegraph instance.

+17 -0

0 comment

1 changed file

beyang

pr closed time in 6 days

PR closed cool-repositories/websocket

Add GitHub action to generate LSIF for Go repositories

Adds a GitHub action that generates LSIF for all Go repositories (selected by the query file:go.mod$ fork:yes on this Sourcegraph instance.

+17 -0

0 comment

1 changed file

beyang

pr closed time in 6 days

PR closed cool-repositories/handlers

Add GitHub action to generate LSIF for Go repositories

Adds a GitHub action that generates LSIF for all Go repositories (selected by the query file:go.mod$ fork:yes on this Sourcegraph instance.

+17 -0

0 comment

1 changed file

beyang

pr closed time in 6 days

push eventsourcegraph/sourcegraph

Ryan Slade

commit sha 40da5f1228d4167d40e1f7b699aba3e4b25afce6

repo-updater: Remove NewChangesetSource (#14585) We now use the repos.Sourcer to create a repos.Source and then cast it to a ChangesetSource.

view details

ᴜɴᴋɴᴡᴏɴ

commit sha af4b27f5c0ffb4822316f6a33d09b5e6e52f0ea6

licensing: add new plans and feature tags (#14236)

view details

Keegan Carruthers-Smith

commit sha 1e9624670fd101f0777e0b800f066b7da971eaaa

gosum: commit gosum values (#14615) Someone didn't run there code after updating go.mod :)

view details

Keegan Carruthers-Smith

commit sha ed134c2c1cd25e18e83cbdb6d9bfad54c78da75e

graphqlbackend: Markdown resolver is string (#14616) Our Markdown resolver can just be `type Markdown string`. This will avoid some allocations and simplify interactions with the code. Code in graphqlbackend pkg updated with comby: comby -in-place '*markdownResolver' 'Markdown' 'go' comby -in-place '&markdownResolver{text: :[a]}' 'Markdown(:[a])' 'go'

view details

Keegan Carruthers-Smith

commit sha 72cb6d7c9116a1df55a5b8ba82fa20528dd1c9c3

graphqlbackend: remove deprecated updateAllMirrorRepositories and noRepositoriesEnabled (#14400)

view details

Felix Becker

commit sha 392fbc71f5b934b305626ebe4662a3ca0a6cf4bd

Auto-retry integration tests as a bandaid (#14620)

view details

Thorsten Ball

commit sha 460c7ec5da4e265d8d2d0b88b3669807dfac1795

Split up campaigns docs into getting started, how-tos, tutorials (#14612) * Split up campaigns docs into getting started, how-tos, tutorials * Add link to campaing spec yaml * Change fonts of campaigns docs * Change getting started * Change wording * Change styling of Getting started links * Split out the explanations * Fix broken links * Rename 'Managing access' to 'Permissions in campaigns' * Rename outdated link * Add a file-icon to links on campaigns docs index * Remove images * Reorder screenshots * Pull out styling to global CSS * Add recommendation for latest version to docs * Update doc/user/campaigns/how-tos/creating_a_campaign.md Co-authored-by: Erik Seliger <erikseliger@me.com> Co-authored-by: Erik Seliger <erikseliger@me.com>

view details

Robert Lin

commit sha 0ab2b83b580f661fdc5269174f5ac6b1b3edb25c

release: unify command naming, remove tracking issue announcement (#14621) Documented in https://github.com/sourcegraph/about/pull/1749

view details

Robert Lin

commit sha 886d3f120b960c25bdc48c882d2a394b15473e83

release: add open issues to status, only post if issue created

view details

Dan Adler

commit sha abd1397705ae4e81a938ab3deedbfaefb8ba50d6

Remove code intel latencies from pings (#14554)

view details

Robert Lin

commit sha e86f5a3854dfac63a96a1307df5c0045b6d9bf86

release: align issues query with tracking issue (#14627)

view details

Dan Adler

commit sha b81b3b4071dffb0c5c75e8ba4a3965b1c1375475

remove stages from pings (#14551) * Remove stages from pings * Update changelog and docs pages. * Fix types

view details

uwedeportivo

commit sha e08eb08cb4c9fe92404a7336dd193d18280a7c7d

admin docs: describes experimental validation command in the src-cli (#12001) * admin docs: describes experimental validation command in the src-cli * link to it * Update doc/admin/validation.md Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update doc/admin/validation.md Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update doc/admin/validation.md Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update doc/admin/validation.md Co-authored-by: Robert Lin <robert@bobheadxi.dev> Co-authored-by: Robert Lin <robert@bobheadxi.dev>

view details

TJ Kandala

commit sha 17dc10bcfa091f46a9ac6a8d7c0088a8939efa95

web: import dialog css (#14631)

view details

uwedeportivo

commit sha 16576ab2d7f86a6252daa4feb85f17407acac62d

admin doc: fix code block in validation page (#14634)

view details

Chris Pine

commit sha b01d3043e5595b6bb6cae4447b9f9c0d4dac5695

handle gzipped graphql requests (#14391) * handle gzipped graphql requests * cleanup * close gzip reader * revert unwanted change

view details

Juliana Peña

commit sha ecfc49a1a2241ea7fbc09ebf2e373ea6625c9e56

search: fix version context dropdown styles (#14542)

view details

renovate[bot]

commit sha 9f34563c12ae27e45c5da154ae92f4915ce040ca

Update dependency @sourcegraph/eslint-config to ^0.20.9 (#14602) Co-authored-by: Renovate Bot <bot@renovateapp.com> Co-authored-by: Felix Becker <felix.b@outlook.com>

view details

Eric Fritz

commit sha 15b8df087682cbbeb3703ef6a2f480aa6d81ccdb

codeintel: Determine nearest upload efficiently in query path (#14581)

view details

TJ Kandala

commit sha 3dabc0a78a8a6eeed48e3ab1bd88c04162107de1

Rewrite browser extension options menu with UX improvements (RFC 221) (#14345) Co-authored-by: Marek <marekz@gmail.com> Co-authored-by: Felix Becker <felix.b@outlook.com>

view details

push time in 6 days

delete branch sourcegraph/sourcegraph

delete branch : bl/oauth2-logging-tmp

delete time in 6 days

push eventsourcegraph/sourcegraph

Beyang Liu

commit sha 8ec977ccc6b4af26859003eb35ac7b573c029f88

use lazyregexp

view details

Beyang Liu

commit sha 62aa053c3c3feae8a285ecdf9d2d027d95bf23c3

refactor

view details

push time in 6 days

push eventsourcegraph/sourcegraph

Beyang Liu

commit sha 5c5db966d377d1d3324842798e00f498b8f3b834

authz: github: GitHub PAT may not have write access (#14700) When the GitHub PAT does not have write access, don't sync an empty user list for repo permissions; otherwise the repository will be rendered inaccessible to non-admin users.

view details

push time in 7 days

delete branch sourcegraph/sourcegraph

delete branch : bl/repo-perm-fix

delete time in 7 days

PR merged sourcegraph/sourcegraph

authz: github: GitHub PAT may not have write access

When the GitHub PAT does not have write access, don't sync an empty user list for repo permissions; otherwise the repository will be rendered inaccessible to non-admin users.

+5 -4

1 comment

1 changed file

beyang

pr closed time in 7 days

push eventsourcegraph/sourcegraph

uwedeportivo

commit sha 1e1abb4c00bf9c370a4a63dd402a7c365ee693f6

e2e: temporarily skip 2 tests to get 3.20 rc build (#13831)

view details

Juliana Peña

commit sha e1873b6eded648d95f87015f9d5bd85123a65a37

Fix interactive search e2e test failures (#13866)

view details

uwedeportivo

commit sha e984c51eb79a6011bd33a6a76daa07d94df657a0

e2e: keep up with search input UI changes (#13867)

view details

uwedeportivo

commit sha 0be50ddf11c47ba47ee59e7be5bab9193628b12b

better selector and skip two tests that influence later tests (#13881)

view details

Rijnard van Tonder

commit sha dfa2d7915a0e166da62fbb768231a69f144727d6

search: fix NOT erasing rest of query (#13925)

view details

uwedeportivo

commit sha 82e9bdfbe694608b08ca8dac367bf9323c82c8f3

Update latest release to 3.20.0 (#13962)

view details

Keegan Carruthers-Smith

commit sha 4024d8feabe3802d11272ad987fb3b4ae90d2a73

gomod: rollback go-diff to v0.5.3 (#13973) Causing a panic in production. We need to fix the issue in v0.6.0 and then we can update again.

view details

Loïc Guychard

commit sha f085e1031a64beed9c2eccd396f59f29f8f1768c

Fix query state getting erased on location changes (#13954) As a result, every time the `location` was updated, for example when clicking on tokens in a code view, any query state that wasn't persisted to the URL (a partially edited query, or the scoped query added automatically when navigating to a repo/file) would be erased. I've simplified the useEffect() block in <GlobalNavbar/> (which could end up calling onNavbarQueryChange() multiple times, had some comments implying that it wouldn't update the query in interactive mode but still did it further down...), fixing the issue as a result, by restricting the cases in which we update the query state.

view details

uwedeportivo

commit sha b673842d93c910a5832c03edbd88ea3affe34ebc

update docker deployment docs and handler to point to 3.20.1 (#14053) * update docker deployment docs and handler to point to 3.20.1 * correct src update version

view details

Beyang Liu

commit sha 71a10292a91e464c397f01ca12d703230d78b920

authz: github: GitHub PAT may not have write access When the GitHub PAT does not have write access, don't sync an empty user list for repo permissions; otherwise the repository will be rendered inaccessible to non-admin users.

view details

push time in 7 days

PR opened sourcegraph/sourcegraph

authz: github: GitHub PAT may not have write access

When the GitHub PAT does not have write access, don't sync an empty user list for repo permissions; otherwise the repository will be rendered inaccessible to non-admin users.

+5 -4

0 comment

1 changed file

pr created time in 7 days

create barnchsourcegraph/sourcegraph

branch : bl/repo-perm-fix

created branch time in 7 days

create barnchsourcegraph/sourcegraph

branch : bl/oauth2-logging-tmp

created branch time in 7 days

pull request commentsourcegraph/about

Updating company values

Great values, thanks for making this change!

christinaforney

comment created time in 8 days

PullRequestReviewEvent

push eventsourcegraph/sourcegraph

Beyang Liu

commit sha 4f5a2c55d4dfb0f3d4f14f1c7d754b5bfcd19b5b

go.mod: update golang.org/x/oauth2 (#14607)

view details

push time in 8 days

delete branch sourcegraph/sourcegraph

delete branch : bl/oauth2

delete time in 8 days

PR merged sourcegraph/sourcegraph

Reviewers
Update oauth2 dep

Update to latest master of our fork (have a pending PR to merge to upstream: https://github.com/golang/oauth2/pull/442).

+3 -1

1 comment

2 changed files

beyang

pr closed time in 8 days

Pull request review commentsourcegraph/sourcegraph

Update oauth2 dep

 replace github.com/russross/blackfriday => github.com/russross/blackfriday v1.5.  replace github.com/dghubble/gologin => github.com/sourcegraph/gologin v1.0.2-0.20181110030308-c6f1b62954d8 -replace golang.org/x/oauth2 => github.com/sourcegraph/oauth2 v1.0.0+replace golang.org/x/oauth2 => github.com/sourcegraph/oauth2 v0.0.0-20201011192344-605770292164

go mod edit -replace golang.org/x/oauth2=github.com/sourcegraph/oauth2@605770292164aba6b415b7b243a20006ac97b90d

beyang

comment created time in 8 days

PullRequestReviewEvent

push eventsourcegraph/about

Beyang Liu

commit sha 8cbfd7cbd934965ec1f7e9006501be72ae39d600

podcast: publishe ep 12

view details

push time in 8 days

push eventsourcegraph/about

Beyang Liu

commit sha 3d6b2f77185ef5b189c24cfd1dca58a3dbe7490a

podcast fix typo

view details

push time in 8 days

push eventsourcegraph/about

Beyang Liu

commit sha 34a18b7aa2863f11092fe576b69649929e23c68e

Revert "podcast: publish ep 12" This reverts commit c8baf27d7a58f68f6149db0a1f92004baaf17ce6.

view details

push time in 9 days

push eventsourcegraph/about

Beyang Liu

commit sha c8baf27d7a58f68f6149db0a1f92004baaf17ce6

podcast: publish ep 12

view details

push time in 9 days

push eventsourcegraph/about

Beyang Liu

commit sha ceb61fdb53613efc332c18cf44093d9dd6730a04

podcast: clean up ep 12 transcript

view details

push time in 9 days

push eventsourcegraph/sourcegraph

push time in 9 days

Pull request review commentsourcegraph/sourcegraph

oauth2: debug logging toggled by INSECURE_OAUTH2_LOG_TRACES

 func newOAuthFlowHandler(serviceType string) http.Handler { // golang.org/x/oauth2 package will use our http client which is configured // with proxy and TLS settings/etc. func withOAuthExternalHTTPClient(r *http.Request) *http.Request {-	ctx := context.WithValue(r.Context(), oauth2.HTTPClient, httpcli.ExternalHTTPClient())+	client := httpcli.ExternalHTTPClient()+	if traceLogEnabled {

This wraps the global external HTTP client in a logging roundtripper that is used by the underlying oauth2 package. Code feels a little icky to me due to the type checks; if reviewer(s) have any suggestions for doing this in a cleaner way, please comment!

beyang

comment created time in 9 days

PullRequestReviewEvent

push eventsourcegraph/sourcegraph

Beyang Liu

commit sha 1ba61addd33e46c7dbac74053736b563818012c0

github client secret validator

view details

Beyang Liu

commit sha 415dd82b00219baa7490a22d8f60233f714e46ed

troubleshooting docs

view details

push time in 9 days

PR opened sourcegraph/sourcegraph

oauth2: debug logging toggled by INSECURE_OAUTH2_LOG_TRACES

Logs will look like this, when INSECURE_OAUTH2_LOG_TRACES=true in the frontend environment:

18:06:37                          frontend | >>>>> HTTP Request: POST https://github.com/login/oauth/access_token                                                                                                                                                                                 
18:06:37                          frontend |       Header: map[Authorization:[Basic xxxxxxxxxxxxxxxxxxxxxx] Content-Type:[application/x-www-form-urlencoded]]                                                                       
18:06:37                          frontend |       Body: code=xxxxxxxxxxx&grant_type=authorization_code                                                                                                                                                                                                          
18:06:37                          frontend | >>>>> HTTP Request: POST https://github.com/login/oauth/access_token                                                                                                                                                                                                                                           
18:06:37                          frontend |       Header: map[Content-Type:[application/x-www-form-urlencoded]]                                                                                                                                                                                                                                            
18:06:37                          frontend |       Body: client_id=xxxxxxxxxxxxx&client_secret=xxxxxxxxxxxxxxxxxxxxxxx&code=xxxxxxxxxxxxxxxxxxxxxxx&grant_type=authorization_code                                                                                                                                                      

+60 -1

0 comment

1 changed file

pr created time in 9 days

create barnchsourcegraph/sourcegraph

branch : bl/oauth2-logging

created branch time in 9 days

PR opened sourcegraph/sourcegraph

Update oauth2 dep

Update to latest master of our fork (have a pending PR to merge to upstream: https://github.com/golang/oauth2/pull/442).

+21 -9

0 comment

2 changed files

pr created time in 10 days

create barnchsourcegraph/sourcegraph

branch : bl/oauth2

created branch time in 10 days

push eventsourcegraph/oauth2

Beyang Liu

commit sha 605770292164aba6b415b7b243a20006ac97b90d

oauth2: return error, error_description, and error_uri when error field is present in token response

view details

push time in 10 days

create barnchsourcegraph/oauth2

branch : debug-20201006

created branch time in 10 days

PR opened golang/oauth2

oauth2: return error if error field is present in access token response

Return an error containing the error, error_description, and error_uri when the error field is present in the access token response, even if the HTTP response code is 2xx.

Fixes #441

+32 -5

0 comment

1 changed file

pr created time in 10 days

issue openedgolang/oauth2

oauth2 swallows token response error if HTTP code is 200

What version of Go are you using (go version)? go version go1.15.2 linux/amd64 What operating system and processor architecture are you using? Ubuntu, x86

The OAuth2 spec states the authorization server should respond with a HTTP 400 if there is an error in the access token response. The error is conveyed in the error, error_description, and error_uri fields in the response.

GitHub OAuth does not exactly obey this spec and can return an error response (examples) with HTTP code 200. In this case, the error from GitHub is swallowed and instead we return a generic error. This makes it difficult to debug exactly what went wrong.

Reproduction steps:

  • Configure GitHub as OAuth2 authorization provider for a web service that uses this package.
  • Change the GitHub client secret so that it no longer matches.
  • Attempt login via GitHub.
  • Note that the error returned by this package is the generic oauth2: server response missing access_token, rather than the actual error returned by GitHub incorrect_client_credentials

Proposed fix: Parse the error field in the access token response. If it is non-empty, then treat the response as an error and return an error containing the error, error_description, and error_uri fields in the access token response. An implementation of this fix is here: https://github.com/sourcegraph/oauth2/commit/605770292164aba6b415b7b243a20006ac97b90d.

created time in 10 days

delete branch sourcegraph/oauth2

delete branch : bl/github-error

delete time in 10 days

create barnchsourcegraph/oauth2

branch : bl/oauth2-error

created branch time in 10 days

create barnchsourcegraph/oauth2

branch : bl/github-error

created branch time in 13 days

push eventsourcegraph/sourcegraph

Beyang Liu

commit sha e323726bf7bce6ef02de1cf3029f8ef843512d2a

SOURCEGRAPH_AUTH_STYLE_PARAMS

view details

push time in 14 days

push eventsourcegraph/sourcegraph

Beyang Liu

commit sha e323726bf7bce6ef02de1cf3029f8ef843512d2a

SOURCEGRAPH_AUTH_STYLE_PARAMS

view details

push time in 14 days

push eventsourcegraph/sourcegraph

Erik Seliger

commit sha ca8413249c7c90108c3b8c5ca4abb719622cdcc5

Properly align diff-stat in changeset nodes like all other states (#13650)

view details

ᴜɴᴋɴᴡᴏɴ

commit sha 1360b5314daac2dd9db3e5c5698bdc7408dfb61b

monitoring: add panels for permissions (#13685)

view details

Quinn Slack

commit sha 1b7c80b11d1a8e6dcbf09f3aa8f089a4e999db85

clean up NavLinks for unauthed users (#13680) When you're not signed in, the NavLinks has unnecessary items: - The external link icon is not necessary for About/Help. It just adds noise. - `Help` -> `Docs` - Remove `Extensions`. It's not important enough to put there. (It's not there for signed-in users even.)

view details

Eric Fritz

commit sha 3697cc1e1a48352cc8b0eabeb2a20ff8e90000ae

codeintel: Load docker images into vm via docker save/load (#13701)

view details

aileenrose

commit sha f5dbf0902da0ea19c092169957a49bab6664d493

Add doctype (#13658)

view details

Eric Fritz

commit sha a8f065d4ecb16f09feff830f36a20aec9814da78

workerutil: Add num_failures column to limit RetryAfter functionality (#13623)

view details

Quinn Slack

commit sha 62f73fa98a321bb0a78134dd837c93293692a068

add "LSIF" to title of LSIF quickstart page (#13660) Avoids confusion about whether this is quickstart for all of Sourcegraph vs. just for LSIF.

view details

Quinn Slack

commit sha 09965489e0a8fec0035f8f8352bf7ad0f213f0d0

make GlobalNavbar a React.FunctionComponent (#13679)

view details

Quinn Slack

commit sha affb013297c3c2e49ef67215eaf96ddca6c85ed7

Refactor nav and search components (#13697)

view details

Robert Lin

commit sha 3c3c1ece954fbd69811036b3fcfc042059a306ad

chore: fix event state check (#13707)

view details

Keegan Carruthers-Smith

commit sha 8313d235af89e53eb9b26de4bb9a795c1617f7ff

ci: build candidate images sooner (#13689) Building the server and frontend images are the slowest steps on master CI runs. This change ensures we run those steps first (and some other slow images earlier as well). The earlier a step runs, the less time it potentially waits (vs other steps) for a free agent.

view details

Robert Lin

commit sha 88972485b6781e44c8b9ecac06056727a15b15a9

chore: adjust renovate action again (#13709)

view details

Robert Lin

commit sha a249d4b524906715107b526ee158cea55097be32

chore: move PR reference to prFooter, remove log job (#13710)

view details

Robert Lin

commit sha 9b6e31ae0beb4d0b7a667486e1d54f68b1f255a7

chore: do not allow codecov bot to trigger renovate (#13711) * chore: do not allow codecov bot to trigger renovate * hack a way to conditionally render pull request link for sourcegraph docker image updates

view details

Ryan Slade

commit sha d00157ae47393b4f7fc25e93d314b70b953cf84d

Sync single external service (#13483) * repo-updater: Sync single external service at a time using workerutil Tests are still failing * repoupdater: Fix server_test * testSyncerSyncWithErrors uses SyncExternalService Also removes a test case that tests a sync of muktiple external services as we no longer support that. * Add EnqueueJobs tests * Removed old references to syncer.Sync Code now compiles but a lot of tests still fail. Added a cleanup function that needs to be called when we stop a worker to unregister prometheus metrics so that it doesn't panic when we start another worker in a subsequent test. * Fix repo-updater tests * Delete orphaned repos * Avoid constraints by changing upsert order * Fix minSyncInterval * WIP test sync * Add a test that syncs multiple external services in sequence * WIP test for orphaned repos * Only delete orphaned repos * repo-updater: Remove stale jobs on startup If a job is processing when repo-updater dies that job would never complete. We would then not requeue the associated external service for syncing. To be safe we delete non locked processing rows on startup. * Typo * db: Mark next_synced_at on service save This will cause a sync to be triggered ASAP for the saved external service. On save, we trigger a call to repo-updater which causes us to enqueue any pending sync jobs. As we've just upated next_sync_at the job for the newly saved repo will be queued. * Rename trigger We no longer trigger a full sync but instead trigger enqueueing pending sync jobs. * Remove orphaned repo code We now have a trigger that does the same thing * Improve orphaned repo test * Test external service deletion * Remove orphaned repo code from syncSubset * Remove leftover debug line * Fix deleting of repos when called via syncSubset * Improve EnqueueSyncJobs docstring Co-authored-by: ᴜɴᴋɴᴡᴏɴ <joe@sourcegraph.com> * Improve error message Co-authored-by: ᴜɴᴋɴᴡᴏɴ <joe@sourcegraph.com> * Check rows.Err() after iteration * Change log line to debug Co-authored-by: ᴜɴᴋɴᴡᴏɴ <joe@sourcegraph.com> * Remove leftover debug line * Add stalled jobs resetter * Pass prometheus register to the SyncWorker * Create ResetterMetrics inline * Remove unused field * Fix docstring * Typo * Missing period * Typo * Add failing test * Resolve name conflicts deterministically * Cleanup name conflicts resolution * Resolve name conflicts during renames * Don't wrap nil error * Don't queue jobs for Phabricator services * Handle large name lists in StoreListReposArgs.Names * Trigger rebuild * Remove stale job cleanup code We use the workerutil resetter now instead * Add failing test * Fix predicate on sourcegraphcom mode * Don't sync deleted external services * When in cloud mode DON'T sync admin added external services Co-authored-by: Asdine El Hrychy <asdine.elhrychy@gmail.com> Co-authored-by: ᴜɴᴋɴᴡᴏɴ <joe@sourcegraph.com>

view details

Keegan Carruthers-Smith

commit sha bd9b03c23199d6ecbe1d0b73c090e163dfea1b6d

workerutil: demote log15.Info to Debug (#13717)

view details

Keegan Carruthers-Smith

commit sha abc47c6afe03b0054f82ed7f3a23848664a6dc8e

codeowners: specify top-level catches earlier (#13719) There were a bunch of rules put in before this old top-level rule. For example internal/workerutil was assigned as owned by Stephen, even though we had a rule for it to be owned by Eric. This was because that rule was before this catch all rule.

view details

Ryan Slade

commit sha b7599f3bf84d85ded03a81360c91e6240f03cb8c

Revert "Sync single external service (#13483)" (#13720) This reverts commit d00157ae47393b4f7fc25e93d314b70b953cf84d.

view details

Beyang Liu

commit sha 5f42de54c1504c58277fb23e6b2cb19bd7d6b6e5

migrations: add back repo name index to speed up search (#13496)

view details

Chris Pine

commit sha 1f5c3f03f8ea1c6e55f11814943a8d1c7510fde3

campaigns: allow user to set git commit author (#13574) * add campaigns git commit author data

view details

push time in 14 days

create barnchsourcegraph/sourcegraph

branch : bl/github-oauth-issue

created branch time in 14 days

created tagsourcegraph/oauth2

tagv1.0.1

Go OAuth2

created time in 14 days

push eventsourcegraph/oauth2

Beyang Liu

commit sha c21c69abeaf41371d387b42c7b8fe91ababfcfeb

SOURCEGRAPH_OAUTH_DEBUG debug logging

view details

push time in 14 days

push eventbeyang/hello-world

Beyang Liu

commit sha 07580ed46acb8dd8296239f67017d7858f65325f

move into subdirectories

view details

push time in 15 days

push eventbeyang/hello-world

Beyang Liu

commit sha 5811300c96a392bf180d9955920769a534598393

hello worlds

view details

push time in 16 days

create barnchbeyang/hello-world

branch : main

created branch time in 16 days

created repositorybeyang/hello-world

created time in 16 days

startedrizsotto/Bear

started time in 16 days

push eventsourcegraph/lsif-clang

Beyang Liu

commit sha be00d0d82dd9e2196a233581fdbdb5343ccbf497

README: fix link

view details

push time in 16 days

Pull request review commentsourcegraph/about

Code search code checks blog post

+---+title: Code search turned static code checker+author: Rijnard van Tonder+authorUrl: https://twitter.com/rvtond+publishDate: 2020-09-30T14:00-07:00+tags: [blog]+slug: code-search-turned-code-checker+heroImage: /blog/XXX.png+published: true+---++<style>+  .gatsby-highlight {+    max-width: 100%;+    width: 40rem;+    margin-left: auto;+    margin-right: auto;+  }+  table {+    width: 40rem;+    border: none;+  }+    table th {+    border: none;+  }+  table td {+    border: none;+  }+  table td:nth-child(2n) {+      text-align: right;+  }+    table th:nth-child(2n) {+      text-align: right;+  }+  table tr:nth-child(2n) {+    background-color: transparent;+  }+</style>++I find static code checkers most valuable when they help teach me better ways to+code in a language or framework. For example, the Go+[staticcheck](https://staticcheck.io/docs/checks#SA6005) tool finds expensive+string comparisons like:++```go+if strings.ToLower(string1) == strings.ToLower(string2) {+  ...+}+```++and suggests instead:++```go+if strings.EqualFold(string1, string2) {+  ...+}+```++I find these short-and-sweet replacements are a great way to learn framework+idioms or library functions, like `strings.EqualFold` in Go.<sup>1</sup> As a+codebase grows, these small inefficiences, inconsistencies, and missed+opportunities compound. Code patterns creep in that affect readability and+performance—[and it matters](https://www.digitalocean.com/blog/how-to-efficiently-compare-strings-in-go/?).++## Productivity workflows: Code checks vs. code search++Code checkers like `staticcheck` typically integrate with continuous integration+(CI) pipelines, or maybe a pre-commit hook in your development environment.+Others, like lint checks, typically integrate with editors. These workflows need+some upfront configuration, the code checks are fixed in place, and productivity+ensues (🤞).++A code search can also find code snippets to replace with functions like+`EqualFold`. In practice though, search engines generally treat code as+plaintext, and can't offer the fidelity of dedicated code checkers. Code+checkers do more work to gather static information of programs, e.g., parsing+syntax into trees and using build outputs like types and dependency graphs. It+takes time to gather this info, it takes knowledge to write checks that use this+info, and it takes time to hook it into your workflow. These constraints take away the+flexibility and speed of search workflows when you're simply looking for changes+to your favorite+`BananaNutChocolateCake Provider`[↗](https://sourcegraph.com/search?q=BananaNutChocolateCake&patternType=literal).++And yet, I can't shake the idea that there's a middle ground between crude+plaintext search and dedicated code checkers. What if the `EqualFold` check I+learned about could be reduced to a simpler and comparably effective _search query_?+Could we find, and maybe eradicate, all the code that should be calling+`EqualFold` instead? And without needing to clone the repos or set up a+fully fledged code checker? And so sprung the idea to explore code checking with+the ease of a flexible, push-button search workflow.++## Pushing code search toward static code checking++Earlier this year Sourcegraph introduced [structural search](blog/going-beyond-regular-expressions-with-structural-code-search/)+to search over code syntax. Structural search uses [comby](https://comby.dev) to implement+a basic building block in traditional code checkers: it interprets programs as+concrete syntax trees, not just plaintext. Sourcegraph search queries now also support patterns with+`or` clauses to search for multiple patterns. Add some file filters, and+it's becomes possible to write configurable code checks as self-contained search queries.+Let's explore this idea!++Here's a search query inspired by+[a check](https://staticcheck.io/docs/checks#S1003) where `strings.Index`+comparisons can be replaced with `strings.Contains`:++```python+language:go+-file:test+-file:vendor++strings.Index(..., ...) > -1++or++strings.Index(..., ...) >= 0++or++strings.Index(..., ...) != -1+```++This query matches all `.go` files, excluding file paths that contain+`test` or `vendor`. It's sensible to exclude `test` and `vendor` paths if we+want to actually propose changes to a project (more on that later). The+patterns `strings.Index(..., ....)` match syntax of `strings.Index` calls,+and the `...` ellipses are special placeholders that match at least two+arguments.<sup>2</sup> The `or` keywords separate the patterns into separate+expressions.++We created a set of the top 100 Go repositories on GitHub (by stars) on+Sourcegraph.com. We can search those by adding `repogroup:go-gh-100` to+the query. Have a look at some of the results:++[🔘 Find ways to improve code in popular Go projects ↗](https://sourcegraph.com/search/console?q=repogroup:go-gh-100%0Alanguage:go%0A-file:test%0A-file:vendor%0A-file:Godeps%0A%0Astrings.Index(...,%20...)%20%3E%20-1%0A%0Aor%0A%0Astrings.Index(...,%20...)%20%3E=%200%0A%0Aor%0A%0Astrings.Index(...,%20...)%20!=%20-1)+<br />+<sup>Side note: our multiline query editor is in a proof-of-concept phase.</sup>++The query finds matches in some of the most popular Go projects in a+couple of seconds. An exhaustive search shows that there are at least 14 matches+at the time of writing. For this flavor of syntactic change, I have a good sense+that these are real hits of code that can be improved (we'll explore+proposing changes to projects later in this blog.)++## Adding more code checks++Because structural search only looks at syntax, it can't yet operate at the+level of a tool like `staticcheck`, which knows more about static properties like type+information and variable scope to implement checks. At the same time,+`staticcheck` isn't a search tool, it's an entire toolset that includes a suite+of pre-written, high-precision checks that's very effective in certain+workflows, like CI. The question is not necessarily whether a search tool can+achieve parity with a tool like `staticcheck`. But given the overlap with+now-expressible search queries, it does prompt: how far can we push structural+code search to find similarly _actionable_ code checks? I.e., checks that match+real cases of code that we can improve.++So, taking inspiration from `staticcheck`, I wanted to see how many of its checks+translate to search queries that I could have high confidence in (i.e., all+patterns find legitimate issues; zero or very-near-zero false positives). I+chose to look at `staticcheck` checks for its clear documentation, which made it+easy to start developing checks. So I attempted to write search patterns+for each [simple static check](https://staticcheck.io/docs/checks).<sup>3</sup>++### Test files as reference patterns++I ran my search queries against `staticcheck`'s own test files to check that they+don't match unintended patterns (false positives) and don't miss real patterns+(false negatives). Each check may have more than one syntactic _variant_, so I+tried to implement patterns for as many variants as I could find in tests. It's+a neat exercise to develop patterns against the reference tests and discover+which variants to cover, all in a self-contained search webapp. Here's an example+where the query matches all the true hits in the test file, annotated with+`// want strings.Contains ...`:+++<img src="https://storage.googleapis.com/sourcegraph-assets/about.sourcegraph.com/blog/2020/multiline-query-editor.png">++[🔘 Example query to match known patterns in test files↗](https://sourcegraph.com/search/console?q=repo:%5Egithub%5C.com/dominikh/go-tools$%20%0Alang:go%0Afile:test%0A%0Astrings.IndexRune(...,%20...)%20%3E%20-1%0A%0Aor%0A%0Astrings.IndexRune(...,%20...)%20%3E=%200%0A%0Aor%0A%0Astrings.IndexRune(...,%20...)%20!=%20-1%0A%0Aor%0A%0Astrings.IndexRune(...,%20...)%20==%20-1%0A%0Aor%0A%0Astrings.IndexRune(...,%20...)%20%3C%200%0A%0Aor%0A%0Astrings.IndexAny(...,%20...)%20%3E%20-1%0A%0Aor%0A%0Astrings.IndexAny(...,%20...)%20%3E=%200%0A%0Aor%0A%0Astrings.IndexAny(...,%20...)%20!=%20-1%0A%0Aor%0A%0Astrings.IndexAny(...,%20...)%20==%20-1%0A%0Aor%0A%0Astrings.IndexAny(...,%20...)%20%3C%200%0A%0Aor%0A%0Astrings.Index(...,%20...)%20%3E%20-1%0A%0Aor%0A%0Astrings.Index(...,%20...)%20%3E=%200%0A%0Aor%0A%0Astrings.Index(...,%20...)%20!=%20-1%0A%0Aor%0A%0Astrings.Index(...,%20...)%20==%20-1%0A%0Aor%0A%0Astrings.Index(...,%20...)%20%3C%200%0A)++Using the test files isn't a guarantee that I've implemented all the checks or+that it's entirely precise, but it adds a lot more confidence than inferring+patterns only from documentation.++### Results++Without making any claims about completeness, I was able to implement+at least one variant for 19 out of 34 checks checks that I feel confident+about. I relied only on patterns in `staticcheck`'s test data to discover syntactic+variants for checks, so I don't know if I covered all the patterns that+`staticcheck` implements in its code.++The majority of checks that I couldn't write required type information to+implement correctly (13 of 34). Other checks I couldn't write required more+complex syntax matching rules (8 of 34). This table roughly quantifies the+expressive needs for implementing checks:++| Description                      | # of checks |+|----------------------------------|-------------|+| Total                            | 34          |+| Works (all variants)             | 11          |+| Works (at least one variant)     | 19          |+| Needs additional syntax matching | 8           |+| Needs type info                  | 13          |++<sup>Note that some checks require type info _and_ additional syntax matching.+Also, one working variant for a check may not require type info, but another+variant for the same check may. I.e., the values overlap and do not sum to+Total.</sup>++Extending search queries to access static properties like type information is a+natural extension for writing better code checks, and an area of [code intelligence](https://docs.sourcegraph.com/user/code_intelligence)+work that we're exploring.++It was tempting to compare more directly with `staticcheck` output by+downloading the 100 Go repositories to disk, and running `staticcheck` on them+to see how the same patterns are found. For `staticcheck` to be effective, the+project typically needs to be built first (my experience was that running+`staticcheck` on individual files can be hit-and-miss, and understandably so). I+didn't like the idea of doing all that work, so I punted. It does nicely motivate+why less precise, but quicker code check workflows appeal.<sup>5</sup>++### Takeaway++Code search can be much more than finding your favorite provider called+`BananaNut.*Cake`. It can be a lightweight workflow for revealing+short-and-sweet ways to make your code better. Perhaps the most powerful idea is+that a universal code search can wholesale find and eradicate the code slips+we're always bound to make.++Starting small, the solution could look like a search query that finds issues in+active and popular Go projects at just the push of a button. Why not start+there? So while we're still working on ways to edit, comment, and annotate+queries, I get giddy about the idea that we can already run a check for+everything at once:++[🔘 Run all the code checks on popular Go projects ↗](https://sourcegraph.com/search/console?q=repogroup:go-gh-100%0Alang:go%0A-file:test%0A-file:vendor%0A-file:Godeps%0Apatterntype:structural%0A%0Astrings.ToLower(...)%20==%20strings.ToLower(...)%20or%0Astrings.IndexRune(...,%20...)%20%3E%20-1%20or%0Astrings.IndexRune(...,%20...)%20%3E=%200%20or%0Astrings.IndexRune(...,%20...)%20!=%20-1%20or%0Astrings.IndexRune(...,%20...)%20==%20-1%20or%0Astrings.IndexRune(...,%20...)%20%3C%200%20or%0Astrings.IndexAny(...,%20...)%20%3E%20-1%20or%0Astrings.IndexAny(...,%20...)%20%3E=%200%20or%0Astrings.IndexAny(...,%20...)%20!=%20-1%20or%0Astrings.IndexAny(...,%20...)%20==%20-1%20or%0Astrings.IndexAny(...,%20...)%20%3C%200%20or%0Astrings.Index(...,%20...)%20%3E%20-1%20or%0Astrings.Index(...,%20...)%20%3E=%200%20or%0Astrings.Index(...,%20...)%20!=%20-1%20or%0Astrings.Index(...,%20...)%20==%20-1%20or%0Astrings.Index(...,%20...)%20%3C%200%20or%0A%0Abytes.Compare(...,%20...)%20==%200%20or%0Abytes.Compare(...,%20...)%20!=%200%20or%0A%0Afor%20:%5B~_%5D,%20:%5B~_%5D%20=%20range%20or%0A%0Afor%20true%20%7B...%7D%20or%0A%0A:%5Bs.%5D%5B:len(:%5Bs%5D)%5D%20or%0A%0Atime.Now().Sub(...)%20or%0A%0Aif%20strings.HasPrefix(:%5Bstr.%5D,%20:%5Bprefix.%5D)%20%7B%0A%20%20:%5Bstr.%5D%20=%20:%5Bstr.%5D%5Blen(:%5Bprefix%5D):%5D%0A%7D%0A%0Aor%0A%0Afor%20:%5Bi.%5D%20:=%200;%20:%5Bi.%5D%20%3C%20:%5Bn.%5D;%20:%5Bi.%5D%20%20%20%7B%0A%20%20:%5Bbs.%5D%5B:%5Bi%5D%5D%20=%20:%5Bbs.%5D%5B:%5Boffset.%5D%20:%5Bi.%5D%5D%0A%7D%0A%0Aor%0A%0Amake(...,%20:%5Bx%5D,%20:%5Bx%5D)%20or%20%0Amake(map%5B:%5B%5Bw%5D%5D%5D:%5B%5Bw%5D%5D,%200)%20or%0Amake(chan%20int,%200)%0A%0Aor%0A%0Afunc%20:%5Bfn.%5D(...)%20%7B%0A%20%20...return%0A%7D%20%0A%0Aor%20%0A%0Afunc()%20%7B%0A%20%20...return%0A%7D%0A%0A.Sub(time.Now())%0A%0Aor%0A%0Afmt.Println(%22%25s%22,%20%22...%22)%0A%0Aor%0A%0Aerrors.New(fmt.Sprintf(...))%0A%0Aor%0A%0Afor%20:%5B~_%5D,%20:%5B_.%5D%20:=%20range%20%5B%5Drune(...)%0A%0Aor%0A%0Asort.Sort(sort.IntSlice(...))%20or%0Asort.Sort(sort.StringSlice(...))%20or%0Asort.Sort(sort.StringSlice(...))%20%0A%0Aor%0A%0Aif%20:%5B~_%5D,%20ok%20:=%20:%5Bm.%5D%5B:%5Bk%5D%5D;%20ok%20%7B%0A%09:%5Bm.%5D%5B:%5Bk%5D%5D%20=%20append(:%5Bm.%5D%5B:%5Bk%5D%5D,%20%22:%5Bv1%5D%22,%20%22:%5Bv2%5D%22)%0A%7D%20else%20%7B%0A%09:%5Bm.%5D%5B:%5Bk%5D%5D%20=%20%5B%5Dstring%7B%22:%5Bv1%5D%22,%20%22:%5Bv2%5D%22%7D%0A%7D%0A%0Aor%0A%0Aif%20:%5B~_%5D,%20ok%20:=%20:%5Bm.%5D%5B:%5Bk%5D%5D;%20ok%20%7B%0A%09:%5Bm.%5D%5B:%5Bk%5D%5D%20=%20append(:%5Bm.%5D%5B:%5Bk%5D%5D,%20%22:%5Bv1%5D%22)%0A%7D%20else%20%7B%0A%09:%5Bm.%5D%5B:%5Bk%5D%5D%20=%20%5B%5Dstring%7B%22:%5Bv1%5D%22%7D%0A%7D%0A%0Aor%0A%0Aselect%20%7B%0A%09case%20%3C-time.After(0):%0A%7D%0A%0Aor%0A%0Afmt.Print(fmt.Sprintf(%22...%22,%20...))%20or%0Afmt.Println(fmt.Sprintf(%22...%22,%20...))%20or%0Afmt.Fprint(nil,%20fmt.Sprintf(%22...%22,%20...))%20or%0Afmt.Fprintln(nil,%20fmt.Sprintf(%22...%22,%20...))%20or%0Afmt.Sprint(fmt.Sprintf(%22...%22,%20...))%20or%0Afmt.Sprintln(fmt.Sprintf(%22...%22,%20...))%0A%0Aor%20%0A%0Afmt.Sprintf(%22%25s%22,%20%22...%22))+<br />+<sup>Side note: patterns run in order and results return early after finding "enough" matches.</sup>++The great thing about code checks as queries is that it's easy to simply+delete patterns that we may not find as valuable. When I explored some+individual queries, it was reassuring to discover that _no_ patterns occur+in any of the Go repositories. For example,+[S1035](https://staticcheck.io/docs/checks#S1035) checks that there are unneeded+`http.CanonicalHeaderKey` calls on the first argument of certain functions:++```python+headers.Add(http.CanonicalHeaderKey(...), ...) or+headers.Del(http.CanonicalHeaderKey(...)) or+headers.Get(http.CanonicalHeaderKey(...)) or+headers.Set(http.CanonicalHeaderKey(...), ...)+```++There are [no matches ↗](https://sourcegraph.com/search/console?q=repogroup:go-gh-100%0A-file:test%0A-file:vendor%0Alang:go%0A%0Aheaders.Add(http.CanonicalHeaderKey(...),%20...)%20or%0Aheaders.Del(http.CanonicalHeaderKey(...))%20or%0Aheaders.Get(http.CanonicalHeaderKey(...))%20or%0Aheaders.Set(http.CanonicalHeaderKey(...),%20...)) for this pattern in the Go repositories, but there are [some matches ↗](https://sourcegraph.com/search/console?q=repogroup:go-gh-100%0A-file:test%0Alang:go%0A%0Aheaders.Add(http.CanonicalHeaderKey(...),%20...)%20or%0Aheaders.Del(http.CanonicalHeaderKey(...))%20or%0Aheaders.Get(http.CanonicalHeaderKey(...))%20or%0Aheaders.Set(http.CanonicalHeaderKey(...),%20...)) in vendored files, when we we remove the `-file:vendor` field.+++### Proposing changes++The most exciting part of finding real hits with any code checking tool is that+we discover an _actionable_ way to improve the code! I've narrowed down the+search query so that it (hopefully) finds only real and uncontroversial patterns+to improve and contribute to active Go projects.++Part of an effective contribution means that we need to avoid matches in files+that are tests, vendored, or external dependencies. While anyone can use the+results to make contributions, it's important to be mindful of contributor+guidelines, clearly communicate and motivate proposed changes in pull requests,+and validate that the change passes a project's tests or CI checks. And, while the+query does exclude common test and vendored files, it's best to check that+matches occur in files that really are part of the project.++If you're interested in potentially turning the results of [the query](https://sourcegraph.com/search/console?q=repogroup:go-gh-100%0Alang:go%0A-file:test%0A-file:vendor%0A-file:Godeps%0Apatterntype:structural%0A%0Astrings.ToLower(...)%20==%20strings.ToLower(...)%20or%0Astrings.IndexRune(...,%20...)%20%3E%20-1%20or%0Astrings.IndexRune(...,%20...)%20%3E=%200%20or%0Astrings.IndexRune(...,%20...)%20!=%20-1%20or%0Astrings.IndexRune(...,%20...)%20==%20-1%20or%0Astrings.IndexRune(...,%20...)%20%3C%200%20or%0Astrings.IndexAny(...,%20...)%20%3E%20-1%20or%0Astrings.IndexAny(...,%20...)%20%3E=%200%20or%0Astrings.IndexAny(...,%20...)%20!=%20-1%20or%0Astrings.IndexAny(...,%20...)%20==%20-1%20or%0Astrings.IndexAny(...,%20...)%20%3C%200%20or%0Astrings.Index(...,%20...)%20%3E%20-1%20or%0Astrings.Index(...,%20...)%20%3E=%200%20or%0Astrings.Index(...,%20...)%20!=%20-1%20or%0Astrings.Index(...,%20...)%20==%20-1%20or%0Astrings.Index(...,%20...)%20%3C%200%20or%0A%0Abytes.Compare(...,%20...)%20==%200%20or%0Abytes.Compare(...,%20...)%20!=%200%20or%0A%0Afor%20:%5B~_%5D,%20:%5B~_%5D%20=%20range%20or%0A%0Afor%20true%20%7B...%7D%20or%0A%0A:%5Bs.%5D%5B:len(:%5Bs%5D)%5D%20or%0A%0Atime.Now().Sub(...)%20or%0A%0Aif%20strings.HasPrefix(:%5Bstr.%5D,%20:%5Bprefix.%5D)%20%7B%0A%20%20:%5Bstr.%5D%20=%20:%5Bstr.%5D%5Blen(:%5Bprefix%5D):%5D%0A%7D%0A%0Aor%0A%0Afor%20:%5Bi.%5D%20:=%200;%20:%5Bi.%5D%20%3C%20:%5Bn.%5D;%20:%5Bi.%5D%20%20%20%7B%0A%20%20:%5Bbs.%5D%5B:%5Bi%5D%5D%20=%20:%5Bbs.%5D%5B:%5Boffset.%5D%20:%5Bi.%5D%5D%0A%7D%0A%0Aor%0A%0Amake(...,%20:%5Bx%5D,%20:%5Bx%5D)%20or%20%0Amake(map%5B:%5B%5Bw%5D%5D%5D:%5B%5Bw%5D%5D,%200)%20or%0Amake(chan%20int,%200)%0A%0Aor%0A%0Afunc%20:%5Bfn.%5D(...)%20%7B%0A%20%20...return%0A%7D%20%0A%0Aor%20%0A%0Afunc()%20%7B%0A%20%20...return%0A%7D%0A%0A.Sub(time.Now())%0A%0Aor%0A%0Afmt.Println(%22%25s%22,%20%22...%22)%0A%0Aor%0A%0Aerrors.New(fmt.Sprintf(...))%0A%0Aor%0A%0Afor%20:%5B~_%5D,%20:%5B_.%5D%20:=%20range%20%5B%5Drune(...)%0A%0Aor%0A%0Asort.Sort(sort.IntSlice(...))%20or%0Asort.Sort(sort.StringSlice(...))%20or%0Asort.Sort(sort.StringSlice(...))%20%0A%0Aor%0A%0Aif%20:%5B~_%5D,%20ok%20:=%20:%5Bm.%5D%5B:%5Bk%5D%5D;%20ok%20%7B%0A%09:%5Bm.%5D%5B:%5Bk%5D%5D%20=%20append(:%5Bm.%5D%5B:%5Bk%5D%5D,%20%22:%5Bv1%5D%22,%20%22:%5Bv2%5D%22)%0A%7D%20else%20%7B%0A%09:%5Bm.%5D%5B:%5Bk%5D%5D%20=%20%5B%5Dstring%7B%22:%5Bv1%5D%22,%20%22:%5Bv2%5D%22%7D%0A%7D%0A%0Aor%0A%0Aif%20:%5B~_%5D,%20ok%20:=%20:%5Bm.%5D%5B:%5Bk%5D%5D;%20ok%20%7B%0A%09:%5Bm.%5D%5B:%5Bk%5D%5D%20=%20append(:%5Bm.%5D%5B:%5Bk%5D%5D,%20%22:%5Bv1%5D%22)%0A%7D%20else%20%7B%0A%09:%5Bm.%5D%5B:%5Bk%5D%5D%20=%20%5B%5Dstring%7B%22:%5Bv1%5D%22%7D%0A%7D%0A%0Aor%0A%0Aselect%20%7B%0A%09case%20%3C-time.After(0):%0A%7D%0A%0Aor%0A%0Afmt.Print(fmt.Sprintf(%22...%22,%20...))%20or%0Afmt.Println(fmt.Sprintf(%22...%22,%20...))%20or%0Afmt.Fprint(nil,%20fmt.Sprintf(%22...%22,%20...))%20or%0Afmt.Fprintln(nil,%20fmt.Sprintf(%22...%22,%20...))%20or%0Afmt.Sprint(fmt.Sprintf(%22...%22,%20...))%20or%0Afmt.Sprintln(fmt.Sprintf(%22...%22,%20...))%0A%0Aor%20%0A%0Afmt.Sprintf(%22%25s%22,%20%22...%22)) into open source+contributions, e-mail me at <rijnard@sourcegraph.com>, I can help.

I am in favor of this idea. It would be a great way to engage new users. I do not know if there is a way to enable a private Slack like ours have a public channel that anyone can join. Creating a separate community Slack channel poses the question of whether or not this would be visible enough to members of the engineering team.

What would be ideal is if we could enable anyone to join a community channel without approval, that channel is visible to members of the team, and we can restrict visibility of more sensitive channels to only Sourcegraph teammates.

rvantonder

comment created time in 16 days

PullRequestReviewEvent

Pull request review commentsourcegraph/about

Code search code checks blog post

+---+title: Code search turned static code checker+author: Rijnard van Tonder+authorUrl: https://twitter.com/rvtond+publishDate: 2020-09-30T14:00-07:00+tags: [blog]+slug: code-search-turned-code-checker+heroImage: /blog/XXX.png+published: true+---++<style>+  .gatsby-highlight {+    max-width: 100%;+    width: 40rem;+    margin-left: auto;+    margin-right: auto;+  }+  table {+    width: 40rem;+    border: none;+  }+    table th {+    border: none;+  }+  table td {+    border: none;+  }+  table td:nth-child(2n) {+      text-align: right;+  }+    table th:nth-child(2n) {+      text-align: right;+  }+  table tr:nth-child(2n) {+    background-color: transparent;+  }+</style>++I find static code checkers most valuable when they help teach me better ways to+code in a language or framework. For example, the Go+[staticcheck](https://staticcheck.io/docs/checks#SA6005) tool finds expensive+string comparisons like:++```go+if strings.ToLower(string1) == strings.ToLower(string2) {+  ...+}+```++and suggests instead:++```go+if strings.EqualFold(string1, string2) {+  ...+}+```++I find these short-and-sweet replacements are a great way to learn framework+idioms or library functions, like `strings.EqualFold` in Go.<sup>1</sup> As a+codebase grows, these small inefficiences, inconsistencies, and missed+opportunities compound. Code patterns creep in that affect readability and+performance—[and it matters](https://www.digitalocean.com/blog/how-to-efficiently-compare-strings-in-go/?).++## Productivity workflows: Code checks vs. code search++Code checkers like `staticcheck` typically integrate with continuous integration+(CI) pipelines, or maybe a pre-commit hook in your development environment.+Others, like lint checks, typically integrate with editors. These workflows need+some upfront configuration, the code checks are fixed in place, and productivity+ensues (🤞).++A code search can also find code snippets to replace with functions like+`EqualFold`. In practice though, search engines generally treat code as+plaintext, and can't offer the fidelity of dedicated code checkers. Code+checkers do more work to gather static information of programs, e.g., parsing+syntax into trees and using build outputs like types and dependency graphs. It+takes time to gather this info, it takes knowledge to write checks that use this+info, and it takes time to hook it into your workflow. These constraints take away the+flexibility and speed of search workflows when you're simply looking for changes+to your favorite+`BananaNutChocolateCake Provider`[↗](https://sourcegraph.com/search?q=BananaNutChocolateCake&patternType=literal).++And yet, I can't shake the idea that there's a middle ground between crude+plaintext search and dedicated code checkers. What if the `EqualFold` check I+learned about could be reduced to a simpler and comparably effective _search query_?+Could we find, and maybe eradicate, all the code that should be calling+`EqualFold` instead? And without needing to clone the repos or set up a+fully fledged code checker? And so sprung the idea to explore code checking with+the ease of a flexible, push-button search workflow.++## Pushing code search toward static code checking++Earlier this year Sourcegraph introduced [structural search](blog/going-beyond-regular-expressions-with-structural-code-search/)+to search over code syntax. Structural search uses [comby](https://comby.dev) to implement+a basic building block in traditional code checkers: it interprets programs as+concrete syntax trees, not just plaintext. Sourcegraph search queries now also support patterns with+`or` clauses to search for multiple patterns. Add some file filters, and+it's becomes possible to write configurable code checks as self-contained search queries.+Let's explore this idea!++Here's a search query inspired by+[a check](https://staticcheck.io/docs/checks#S1003) where `strings.Index`+comparisons can be replaced with `strings.Contains`:++```python+language:go+-file:test+-file:vendor++strings.Index(..., ...) > -1++or++strings.Index(..., ...) >= 0++or++strings.Index(..., ...) != -1+```++This query matches all `.go` files, excluding file paths that contain+`test` or `vendor`. It's sensible to exclude `test` and `vendor` paths if we+want to actually propose changes to a project (more on that later). The+patterns `strings.Index(..., ....)` match syntax of `strings.Index` calls,+and the `...` ellipses are special placeholders that match at least two+arguments.<sup>2</sup> The `or` keywords separate the patterns into separate+expressions.++We created a set of the top 100 Go repositories on GitHub (by stars) on+Sourcegraph.com. We can search those by adding `repogroup:go-gh-100` to+the query. Have a look at some of the results:++[🔘 Find ways to improve code in popular Go projects ↗](https://sourcegraph.com/search/console?q=repogroup:go-gh-100%0Alanguage:go%0A-file:test%0A-file:vendor%0A-file:Godeps%0A%0Astrings.Index(...,%20...)%20%3E%20-1%0A%0Aor%0A%0Astrings.Index(...,%20...)%20%3E=%200%0A%0Aor%0A%0Astrings.Index(...,%20...)%20!=%20-1)+<br />+<sup>Side note: our multiline query editor is in a proof-of-concept phase.</sup>++The query finds matches in some of the most popular Go projects in a+couple of seconds. An exhaustive search shows that there are at least 14 matches+at the time of writing. For this flavor of syntactic change, I have a good sense+that these are real hits of code that can be improved (we'll explore+proposing changes to projects later in this blog.)++## Adding more code checks++Because structural search only looks at syntax, it can't yet operate at the+level of a tool like `staticcheck`, which knows more about static properties like type+information and variable scope to implement checks. At the same time,+`staticcheck` isn't a search tool, it's an entire toolset that includes a suite+of pre-written, high-precision checks that's very effective in certain+workflows, like CI. The question is not necessarily whether a search tool can+achieve parity with a tool like `staticcheck`. But given the overlap with+now-expressible search queries, it does prompt: how far can we push structural+code search to find similarly _actionable_ code checks? I.e., checks that match+real cases of code that we can improve.++So, taking inspiration from `staticcheck`, I wanted to see how many of its checks+translate to search queries that I could have high confidence in (i.e., all+patterns find legitimate issues; zero or very-near-zero false positives). I+chose to look at `staticcheck` checks for its clear documentation, which made it+easy to start developing checks. So I attempted to write search patterns+for each [simple static check](https://staticcheck.io/docs/checks).<sup>3</sup>++### Test files as reference patterns++I ran my search queries against `staticcheck`'s own test files to check that they+don't match unintended patterns (false positives) and don't miss real patterns+(false negatives). Each check may have more than one syntactic _variant_, so I+tried to implement patterns for as many variants as I could find in tests. It's+a neat exercise to develop patterns against the reference tests and discover+which variants to cover, all in a self-contained search webapp. Here's an example+where the query matches all the true hits in the test file, annotated with+`// want strings.Contains ...`:+++<img src="https://storage.googleapis.com/sourcegraph-assets/about.sourcegraph.com/blog/2020/multiline-query-editor.png">++[🔘 Example query to match known patterns in test files↗](https://sourcegraph.com/search/console?q=repo:%5Egithub%5C.com/dominikh/go-tools$%20%0Alang:go%0Afile:test%0A%0Astrings.IndexRune(...,%20...)%20%3E%20-1%0A%0Aor%0A%0Astrings.IndexRune(...,%20...)%20%3E=%200%0A%0Aor%0A%0Astrings.IndexRune(...,%20...)%20!=%20-1%0A%0Aor%0A%0Astrings.IndexRune(...,%20...)%20==%20-1%0A%0Aor%0A%0Astrings.IndexRune(...,%20...)%20%3C%200%0A%0Aor%0A%0Astrings.IndexAny(...,%20...)%20%3E%20-1%0A%0Aor%0A%0Astrings.IndexAny(...,%20...)%20%3E=%200%0A%0Aor%0A%0Astrings.IndexAny(...,%20...)%20!=%20-1%0A%0Aor%0A%0Astrings.IndexAny(...,%20...)%20==%20-1%0A%0Aor%0A%0Astrings.IndexAny(...,%20...)%20%3C%200%0A%0Aor%0A%0Astrings.Index(...,%20...)%20%3E%20-1%0A%0Aor%0A%0Astrings.Index(...,%20...)%20%3E=%200%0A%0Aor%0A%0Astrings.Index(...,%20...)%20!=%20-1%0A%0Aor%0A%0Astrings.Index(...,%20...)%20==%20-1%0A%0Aor%0A%0Astrings.Index(...,%20...)%20%3C%200%0A)++Using the test files isn't a guarantee that I've implemented all the checks or+that it's entirely precise, but it adds a lot more confidence than inferring+patterns only from documentation.++### Results

I think it would be interesting to show a small table comparing the definitions of these checks in staticcheck (presumably long/complex) vs. comby.

rvantonder

comment created time in 17 days

Pull request review commentsourcegraph/about

Code search code checks blog post

+---+title: Code search turned static code checker+author: Rijnard van Tonder+authorUrl: https://twitter.com/rvtond+publishDate: 2020-09-30T14:00-07:00+tags: [blog]+slug: code-search-turned-code-checker+heroImage: /blog/XXX.png+published: true+---++<style>+  .gatsby-highlight {+    max-width: 100%;+    width: 40rem;+    margin-left: auto;+    margin-right: auto;+  }+  table {+    width: 40rem;+    border: none;+  }+    table th {+    border: none;+  }+  table td {+    border: none;+  }+  table td:nth-child(2n) {+      text-align: right;+  }+    table th:nth-child(2n) {+      text-align: right;+  }+  table tr:nth-child(2n) {+    background-color: transparent;+  }+</style>++I find static code checkers most valuable when they help teach me better ways to+code in a language or framework. For example, the Go+[staticcheck](https://staticcheck.io/docs/checks#SA6005) tool finds expensive+string comparisons like:++```go+if strings.ToLower(string1) == strings.ToLower(string2) {+  ...+}+```++and suggests instead:++```go+if strings.EqualFold(string1, string2) {+  ...+}+```++I find these short-and-sweet replacements are a great way to learn framework+idioms or library functions, like `strings.EqualFold` in Go.<sup>1</sup> As a+codebase grows, these small inefficiences, inconsistencies, and missed+opportunities compound. Code patterns creep in that affect readability and+performance—[and it matters](https://www.digitalocean.com/blog/how-to-efficiently-compare-strings-in-go/?).++## Productivity workflows: Code checks vs. code search++Code checkers like `staticcheck` typically integrate with continuous integration+(CI) pipelines, or maybe a pre-commit hook in your development environment.+Others, like lint checks, typically integrate with editors. These workflows need+some upfront configuration, the code checks are fixed in place, and productivity+ensues (🤞).++A code search can also find code snippets to replace with functions like+`EqualFold`. In practice though, search engines generally treat code as+plaintext, and can't offer the fidelity of dedicated code checkers. Code+checkers do more work to gather static information of programs, e.g., parsing+syntax into trees and using build outputs like types and dependency graphs. It+takes time to gather this info, it takes knowledge to write checks that use this+info, and it takes time to hook it into your workflow. These constraints take away the+flexibility and speed of search workflows when you're simply looking for changes+to your favorite+`BananaNutChocolateCake Provider`[↗](https://sourcegraph.com/search?q=BananaNutChocolateCake&patternType=literal).++And yet, I can't shake the idea that there's a middle ground between crude+plaintext search and dedicated code checkers. What if the `EqualFold` check I+learned about could be reduced to a simpler and comparably effective _search query_?+Could we find, and maybe eradicate, all the code that should be calling+`EqualFold` instead? And without needing to clone the repos or set up a+fully fledged code checker? And so sprung the idea to explore code checking with+the ease of a flexible, push-button search workflow.++## Pushing code search toward static code checking++Earlier this year Sourcegraph introduced [structural search](blog/going-beyond-regular-expressions-with-structural-code-search/)+to search over code syntax. Structural search uses [comby](https://comby.dev) to implement+a basic building block in traditional code checkers: it interprets programs as+concrete syntax trees, not just plaintext. Sourcegraph search queries now also support patterns with+`or` clauses to search for multiple patterns. Add some file filters, and+it's becomes possible to write configurable code checks as self-contained search queries.+Let's explore this idea!++Here's a search query inspired by+[a check](https://staticcheck.io/docs/checks#S1003) where `strings.Index`+comparisons can be replaced with `strings.Contains`:++```python+language:go+-file:test+-file:vendor++strings.Index(..., ...) > -1++or++strings.Index(..., ...) >= 0++or++strings.Index(..., ...) != -1+```++This query matches all `.go` files, excluding file paths that contain+`test` or `vendor`. It's sensible to exclude `test` and `vendor` paths if we+want to actually propose changes to a project (more on that later). The+patterns `strings.Index(..., ....)` match syntax of `strings.Index` calls,+and the `...` ellipses are special placeholders that match at least two+arguments.<sup>2</sup> The `or` keywords separate the patterns into separate+expressions.++We created a set of the top 100 Go repositories on GitHub (by stars) on+Sourcegraph.com. We can search those by adding `repogroup:go-gh-100` to+the query. Have a look at some of the results:++[🔘 Find ways to improve code in popular Go projects ↗](https://sourcegraph.com/search/console?q=repogroup:go-gh-100%0Alanguage:go%0A-file:test%0A-file:vendor%0A-file:Godeps%0A%0Astrings.Index(...,%20...)%20%3E%20-1%0A%0Aor%0A%0Astrings.Index(...,%20...)%20%3E=%200%0A%0Aor%0A%0Astrings.Index(...,%20...)%20!=%20-1)+<br />+<sup>Side note: our multiline query editor is in a proof-of-concept phase.</sup>++The query finds matches in some of the most popular Go projects in a+couple of seconds. An exhaustive search shows that there are at least 14 matches+at the time of writing. For this flavor of syntactic change, I have a good sense+that these are real hits of code that can be improved (we'll explore+proposing changes to projects later in this blog.)++## Adding more code checks++Because structural search only looks at syntax, it can't yet operate at the+level of a tool like `staticcheck`, which knows more about static properties like type+information and variable scope to implement checks. At the same time,+`staticcheck` isn't a search tool, it's an entire toolset that includes a suite+of pre-written, high-precision checks that's very effective in certain+workflows, like CI. The question is not necessarily whether a search tool can+achieve parity with a tool like `staticcheck`. But given the overlap with+now-expressible search queries, it does prompt: how far can we push structural+code search to find similarly _actionable_ code checks? I.e., checks that match+real cases of code that we can improve.++So, taking inspiration from `staticcheck`, I wanted to see how many of its checks+translate to search queries that I could have high confidence in (i.e., all+patterns find legitimate issues; zero or very-near-zero false positives). I+chose to look at `staticcheck` checks for its clear documentation, which made it+easy to start developing checks. So I attempted to write search patterns+for each [simple static check](https://staticcheck.io/docs/checks).<sup>3</sup>++### Test files as reference patterns++I ran my search queries against `staticcheck`'s own test files to check that they+don't match unintended patterns (false positives) and don't miss real patterns+(false negatives). Each check may have more than one syntactic _variant_, so I+tried to implement patterns for as many variants as I could find in tests. It's+a neat exercise to develop patterns against the reference tests and discover+which variants to cover, all in a self-contained search webapp. Here's an example+where the query matches all the true hits in the test file, annotated with+`// want strings.Contains ...`:+++<img src="https://storage.googleapis.com/sourcegraph-assets/about.sourcegraph.com/blog/2020/multiline-query-editor.png">++[🔘 Example query to match known patterns in test files↗](https://sourcegraph.com/search/console?q=repo:%5Egithub%5C.com/dominikh/go-tools$%20%0Alang:go%0Afile:test%0A%0Astrings.IndexRune(...,%20...)%20%3E%20-1%0A%0Aor%0A%0Astrings.IndexRune(...,%20...)%20%3E=%200%0A%0Aor%0A%0Astrings.IndexRune(...,%20...)%20!=%20-1%0A%0Aor%0A%0Astrings.IndexRune(...,%20...)%20==%20-1%0A%0Aor%0A%0Astrings.IndexRune(...,%20...)%20%3C%200%0A%0Aor%0A%0Astrings.IndexAny(...,%20...)%20%3E%20-1%0A%0Aor%0A%0Astrings.IndexAny(...,%20...)%20%3E=%200%0A%0Aor%0A%0Astrings.IndexAny(...,%20...)%20!=%20-1%0A%0Aor%0A%0Astrings.IndexAny(...,%20...)%20==%20-1%0A%0Aor%0A%0Astrings.IndexAny(...,%20...)%20%3C%200%0A%0Aor%0A%0Astrings.Index(...,%20...)%20%3E%20-1%0A%0Aor%0A%0Astrings.Index(...,%20...)%20%3E=%200%0A%0Aor%0A%0Astrings.Index(...,%20...)%20!=%20-1%0A%0Aor%0A%0Astrings.Index(...,%20...)%20==%20-1%0A%0Aor%0A%0Astrings.Index(...,%20...)%20%3C%200%0A)++Using the test files isn't a guarantee that I've implemented all the checks or+that it's entirely precise, but it adds a lot more confidence than inferring+patterns only from documentation.++### Results++Without making any claims about completeness, I was able to implement+at least one variant for 19 out of 34 checks checks that I feel confident+about. I relied only on patterns in `staticcheck`'s test data to discover syntactic+variants for checks, so I don't know if I covered all the patterns that+`staticcheck` implements in its code.++The majority of checks that I couldn't write required type information to+implement correctly (13 of 34). Other checks I couldn't write required more+complex syntax matching rules (8 of 34). This table roughly quantifies the+expressive needs for implementing checks:++| Description                      | # of checks |+|----------------------------------|-------------|+| Total                            | 34          |+| Works (all variants)             | 11          |+| Works (at least one variant)     | 19          |+| Needs additional syntax matching | 8           |+| Needs type info                  | 13          |++<sup>Note that some checks require type info _and_ additional syntax matching.+Also, one working variant for a check may not require type info, but another+variant for the same check may. I.e., the values overlap and do not sum to+Total.</sup>++Extending search queries to access static properties like type information is a+natural extension for writing better code checks, and an area of [code intelligence](https://docs.sourcegraph.com/user/code_intelligence)+work that we're exploring.++It was tempting to compare more directly with `staticcheck` output by+downloading the 100 Go repositories to disk, and running `staticcheck` on them+to see how the same patterns are found. For `staticcheck` to be effective, the+project typically needs to be built first (my experience was that running+`staticcheck` on individual files can be hit-and-miss, and understandably so). I+didn't like the idea of doing all that work, so I punted. It does nicely motivate+why less precise, but quicker code check workflows appeal.<sup>5</sup>++### Takeaway++Code search can be much more than finding your favorite provider called

Maybe a table showing relative advantages/disadvantages of staticcheck vs seach+comby would be evocative here.

Potential rows:

  • Accuracy (syntax vs semantics)
  • Scalability
  • Ease of definition
  • Ease of execution
rvantonder

comment created time in 17 days

Pull request review commentsourcegraph/about

Code search code checks blog post

+---+title: Code search turned static code checker+author: Rijnard van Tonder+authorUrl: https://twitter.com/rvtond+publishDate: 2020-09-30T14:00-07:00+tags: [blog]+slug: code-search-turned-code-checker+heroImage: /blog/XXX.png+published: true+---++<style>+  .gatsby-highlight {+    max-width: 100%;+    width: 40rem;+    margin-left: auto;+    margin-right: auto;+  }+  table {+    width: 40rem;+    border: none;+  }+    table th {+    border: none;+  }+  table td {+    border: none;+  }+  table td:nth-child(2n) {+      text-align: right;+  }+    table th:nth-child(2n) {+      text-align: right;+  }+  table tr:nth-child(2n) {+    background-color: transparent;+  }+</style>++I find static code checkers most valuable when they help teach me better ways to+code in a language or framework. For example, the Go+[staticcheck](https://staticcheck.io/docs/checks#SA6005) tool finds expensive+string comparisons like:++```go+if strings.ToLower(string1) == strings.ToLower(string2) {+  ...+}+```++and suggests instead:++```go+if strings.EqualFold(string1, string2) {+  ...+}+```++I find these short-and-sweet replacements are a great way to learn framework+idioms or library functions, like `strings.EqualFold` in Go.<sup>1</sup> As a+codebase grows, these small inefficiences, inconsistencies, and missed+opportunities compound. Code patterns creep in that affect readability and+performance—[and it matters](https://www.digitalocean.com/blog/how-to-efficiently-compare-strings-in-go/?).++## Productivity workflows: Code checks vs. code search++Code checkers like `staticcheck` typically integrate with continuous integration+(CI) pipelines, or maybe a pre-commit hook in your development environment.+Others, like lint checks, typically integrate with editors. These workflows need+some upfront configuration, the code checks are fixed in place, and productivity+ensues (🤞).++A code search can also find code snippets to replace with functions like+`EqualFold`. In practice though, search engines generally treat code as+plaintext, and can't offer the fidelity of dedicated code checkers. Code+checkers do more work to gather static information of programs, e.g., parsing+syntax into trees and using build outputs like types and dependency graphs. It+takes time to gather this info, it takes knowledge to write checks that use this+info, and it takes time to hook it into your workflow. These constraints take away the+flexibility and speed of search workflows when you're simply looking for changes+to your favorite+`BananaNutChocolateCake Provider`[↗](https://sourcegraph.com/search?q=BananaNutChocolateCake&patternType=literal).++And yet, I can't shake the idea that there's a middle ground between crude

To me, this intro leaves out a huge advantage of search over staticcheck, which is scalability. With search+comby, you can execute your find-replace over the entire universe of code, whereas staticcheck requires manual setup/effort for each repository you want to cover. This difference in scale/automaticness means if I'm someone who added a new API, I would totally reach for search+comby to update existing usages of that API to take advantage of my new function. Not as easy with staticcheck, because other repository maintainers have to set it up first. Big difference in friction.

rvantonder

comment created time in 17 days

Pull request review commentsourcegraph/about

Code search code checks blog post

+---+title: Code search turned static code checker+author: Rijnard van Tonder+authorUrl: https://twitter.com/rvtond+publishDate: 2020-09-30T14:00-07:00+tags: [blog]+slug: code-search-turned-code-checker+heroImage: /blog/XXX.png+published: true+---++<style>+  .gatsby-highlight {+    max-width: 100%;+    width: 40rem;+    margin-left: auto;+    margin-right: auto;+  }+  table {+    width: 40rem;+    border: none;+  }+    table th {+    border: none;+  }+  table td {+    border: none;+  }+  table td:nth-child(2n) {+      text-align: right;+  }+    table th:nth-child(2n) {+      text-align: right;+  }+  table tr:nth-child(2n) {+    background-color: transparent;+  }+</style>++I find static code checkers most valuable when they help teach me better ways to
I find static code checkers most valuable when they teach me better ways to
rvantonder

comment created time in 18 days

PullRequestReviewEvent
PullRequestReviewEvent

push eventsourcegraph/sourcegraph

Beyang Liu

commit sha f86335240f77a006f6887a7c057073c299d69428

CODENOTIFY: beyang subscriptions (#14396)

view details

push time in 17 days

delete branch sourcegraph/sourcegraph

delete branch : bl/codenotify

delete time in 17 days

PR merged sourcegraph/sourcegraph

CODENOTIFY: beyang subscriptions

<!-- Reminder: Have you updated the changelog and relevant docs (user docs, architecture diagram, etc) ? -->

+9 -2

2 comments

7 changed files

beyang

pr closed time in 17 days

PR opened sourcegraph/sourcegraph

CODENOTIFY: beyang subscriptions

<!-- Reminder: Have you updated the changelog and relevant docs (user docs, architecture diagram, etc) ? -->

+9 -2

0 comment

7 changed files

pr created time in 18 days

create barnchsourcegraph/sourcegraph

branch : bl/codenotify

created branch time in 18 days

PullRequestReviewEvent

Pull request review commentsourcegraph/about

Blog: A different way to think about code ownership

+---+title: "A different way to think about code ownership"+author: Nick Snyder+authorUrl: https://twitter.com/nickdsnyder+publishDate: 2020-10-05+tags: [blog]+slug: a-different-way-to-think-about-code-ownership+heroImage: https://sourcegraphstatic.com/blog/codenotify-survey-results.png+published: true+---++We’re [experimenting](https://github.com/sourcegraph/sourcegraph/pull/13838) with a new way to think about code ownership, and have built a new tool called [Codenotify](https://github.com/sourcegraph/codenotify). Codenotify enables developers to easily subscribe to file changes in a Git repository without creating the expectation that changes to those files are blocked on the subscriber’s review.++## Defining the problem++We host all of [our code on GitHub](https://github.com/sourcegraph/), so up until now we have been using a GitHub [CODEOWNERS](https://docs.github.com/en/free-pro-team@latest/github/creating-cloning-and-archiving-repositories/about-code-owners) file in each repository. When someone opens a pull request, GitHub will look at the CODEOWNERS file to determine which reviewers to automatically add.++As our team has grown over the past year, I started to observe patterns that made me think that the way we assigned code reviewers was flawed.++- Pull requests that touched multiple files of different kinds and in multiple places would sometimes end up with a large list of automatically added reviews.+- Reviewers wouldn’t review all the changes they were notified about because it wasn’t clear if their personal review was relevant (for example: they may have gotten notified because they are on the team that owns the code, but aren’t the best reviewer for that particular change).+- Authors (especially newer teammates) were getting slowed down because it was not clear whether they needed to wait for all automatically added reviewers or not. This was exacerbated by the fact that some teammates used CODEOWNERS as a way to be notified of changes they cared about, even if they didn’t have the intention or ability to promptly review all those changes.++My diagnosis was that there were two problems:++1. The low signal to noise ratio of automatically assigned reviews was slowing the team down.+1. The design of CODEOWNERS (its name, its UI treatment, it automatically adding code reviewers) created a gatekeeper culture to code review that discouraged a [high agency](https://about.sourcegraph.com/company/values#high-agency) mindset.++## Validation++I had a hypothesis—that we needed something other than CODEOWNERS—but it needed validation. Was the team feeling the same pains and observing the same patterns as I was? Was continuing to use CODEOWNERS a viable solution moving forward?++To get the sentiment of the team, [I opened a pull request that proposed deleting CODEOWNERS with my reasons](https://github.com/sourcegraph/sourcegraph/pull/11718) and asked the entire team to submit anonymous feedback. Here were the results:++<div class="text-center">+  <img src="https://sourcegraphstatic.com/blog/codenotify-survey-results.png" alt="50% yes, 31% maybe, 19% no">+</div>++In the free-form comments, responders provided extra context about their support, concerns, and suggestions for the proposal. Here are a few representative examples:++- "I've honestly not gotten any value out of CODEOWNERS files. I never look into them, I never trust the auto-add-as-reviewer feature in GitHub PRs."+- "Would this result in people who wrote more LOC being enlisted into more reviews?"+- "I use CODEOWNERS to get notified of PRs"+- "Replace with [OWNERS](https://chromium.googlesource.com/chromium/src/+/master/docs/code_reviews.md) (something people will actually update)"+- "I think removing CODEOWNERS makes it more difficult for new developers without historical context to determine who the right reviewers are."++It was clear we needed _something_, but CODEOWNERS didn’t do exactly what we needed and we didn’t have the ability to change its behavior.++## Exploring alternatives++There was enough evidence that a problem existed to justify exploring alternatives.++I found [Fullstory’s blog post about how they ran into similar problems with CODEOWNERS and ended up creating a bot](https://bionic.fullstory.com/taming-github-codeowners-with-bots/). I would have loved to experiment with using their bot, but unfortunately it was not open source.++A few of our teammates, including myself, worked at Google so we were familiar with the [OWNERS](https://chromium.googlesource.com/chromium/src/+/master/docs/code_reviews.md) format used internally and in the Chromium project. This would have been an improvement over CODEOWNERS because multiple files are easier to maintain than a single large file and [Chromium’s implementation explicitly attempts to minimize the number of reviewers](https://chromium.googlesource.com/chromium/tools/depot_tools/+/master/owners.py#607), but I was worried this would continue to create a gatekeeper culture around modifying code.++I wanted a solution that would:++1. Allow developers to subscribe to the code changes that they cared about without creating an expectation that they would review all those changes.+1. Allow authors to use their judgement and agency to intentionally select reviewers whose feedback they think would be valuable (for example: Who wrote the code being edited? Who reviewed the code being edited? Who provides valuable feedback in a timely manner?).+1. Be easy to understand and maintain in a large and constantly evolving codebase.++Neither CODEOWNERS nor OWNERs were designed with all these requirements in mind, so I started to think about what a new tool might look like, and I ended up with [Codenotify](https://github.com/sourcegraph/codenotify).++## Designing Codenotify++As the name implies, Codenotify is designed around the concept of notifications, not about establishing "ownership" of code or mandating "reviewers". Codenotify works on the command line and is codehost agnostic at its core, but it is also packaged as a GitHub Action so it is easy for repositories on GitHub, like ours, to adopt. The GitHub Action sends notifications to subscribers by mentioning them in a comment instead of adding them to the official "reviewers" or "assignees" lists on GitHub.++Notification rules are stored in [CODENOTIFY files](https://sourcegraph.com/github.com/sourcegraph/codenotify/-/blob/README.md#codenotify-files) in any directory. This makes rules more maintainable and less brittle to moving code because the CODENOTIFY files are adjacent to the files they have rules for.++Codenotify rules and files are familiar in syntax to CODEOWNERS, but simpler to understand because each rule is additive, not hierarchical. This means that you can understand how a single rule behaves without needing to consider any other rules (unlike OWNERS or CODEOWNERS).++## Starting our experiment

Maybe add a small section here with a CTA to readers to try it out. Can link to https://github.com/sourcegraph/codenotify.

nicksnyder

comment created time in 18 days

PullRequestReviewEvent

startedsourcegraph/codenotify

started time in 18 days

push eventbeyang/google-keys

Beyang Liu

commit sha 59f3f2f21298ed164db5cb8cfd40751c8222fa5c

spacing

view details

Beyang Liu

commit sha 6308ecb7a13f5a8911e7b18e235d602227a0bf93

fix google keys

view details

push time in 19 days

push eventsourcegraph/about

Beyang Liu

commit sha 216d7b4dca6c14cb07b8132f24d32996a54c3bbd

fix formatting

view details

push time in 19 days

push eventsourcegraph/about

Beyang Liu

commit sha c7721c94bedd9cee15294459803115c320c4dabf

podcast transcript cleanup

view details

push time in 19 days

push eventsourcegraph/about

Beyang Liu

commit sha c9b172262a2e850b3847a5a593a622c8e0fea706

podcast: ep12

view details

push time in 19 days

push eventsourcegraph/ghdump

Beyang Liu

commit sha a5484a8c6894ac6f63a54b163f2d4c8a4c3c8d60

add back all 100k repos

view details

push time in 20 days

push eventbeyang/google-keys

Beyang Liu

commit sha 75cc03c2f90e16a74e4a73f823517c0bf3e34609

fix google keys

view details

push time in 22 days

startedbellard/quickjs

started time in 22 days

startedwspl/go-quickjs

started time in 22 days

startedelsaland/elsa

started time in 22 days

more