profile
viewpoint
Beto Dealmeida betodealmeida @preset-io Bodega Bay, CA https://blog.taoetc.org/ Writing open source software since 2003.

betodealmeida/gsheets-db-api 164

A Python DB-API and SQLAlchemy dialect to Google Spreasheets

betodealmeida/jellyjampreserve 14

JellyJamPreserve is a Raspberry Pi project using Jack Timemachine (http://plugin.org.uk/timemachine/) to record audio in a studio.

betodealmeida/consolelog 8

Log to browser console from Python

betodealmeida/druid-dbapi 6

Python DB-API and SQLAlchemy dialect for Druid.

betodealmeida/cassette-tape-player 4

A pure Javascript music player shaped like a cassette tape

betodealmeida/magicdate 3

Convert fuzzy date to a datetime object.

betodealmeida/dbapihelper 2

A helper library for writing DB API modules

betodealmeida/incubator-superset 2

Apache Superset (incubating) is a modern, enterprise-ready business intelligence web application

betodealmeida/este-sou-eu 1

A service for identify, authentication and authorization

betodealmeida/gardenlights 1

Garden lights controller

Pull request review commentapache/superset

feat: allow assets to be managed externally

+# Licensed to the Apache Software Foundation (ASF) under one+# or more contributor license agreements.  See the NOTICE file+# distributed with this work for additional information+# regarding copyright ownership.  The ASF licenses this file+# to you under the Apache License, Version 2.0 (the+# "License"); you may not use this file except in compliance+# with the License.  You may obtain a copy of the License at+#+#   http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing,+# software distributed under the License is distributed on an+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+# KIND, either express or implied.  See the License for the+# specific language governing permissions and limitations+# under the License.+"""Add columns for external management++Revision ID: 5fd49410a97a+Revises: c53bae8f08dd+Create Date: 2022-01-19 07:34:20.594786++"""++# revision identifiers, used by Alembic.+revision = "5fd49410a97a"+down_revision = "c53bae8f08dd"++import sqlalchemy as sa+from alembic import op+++def upgrade():+    with op.batch_alter_table("dashboards") as batch_op:+        batch_op.add_column(+            sa.Column("is_managed_externally", sa.Boolean(), nullable=True)

Fixed it by using server_default. Now the new column is populated with false values, and the same happens when a new resource is created.

betodealmeida

comment created time in 8 hours

PullRequestReviewEvent

push eventbetodealmeida/incubator-superset

Stephen Liu

commit sha 22896f28037c377e40d833e5c7ab2de8418d6c17

feat: add chart description in info tooltip (#17207) * feat: add chart list description * fix: text overflow * fix: text-overflow with line-height

view details

Adam Dobrawy

commit sha f53a267ffb741036afb9cf2fb2bb8368fa64a149

Correction of proper names format in README (#18087)

view details

ofekisr

commit sha 810cfc13db666ecb2da8d9e66b06ff6d2524c764

fix can't use examples helpers on non app context based environment (#18086)

view details

Beto Dealmeida

commit sha 9e2bc72fb9687eb1d4563fd02853fea6b94a978d

chore: split CLI into multiple files (#18082) * chore: split CLI into multiple files * Update tests * Who fixes the fixtures? * Add subcommands dynamically * Rebase

view details

Emily Wu

commit sha f505094835d43365696144eb8bf8f9a1d5977994

fix misspelling (#18097)

view details

Josue Lugaro

commit sha 035638c95869d0d4b7c9d44f13e88e3535ecdf4c

refactor: sqleditorleftbar to functional (#17807) * Working on converting sqleditorleftbar to functional component * Creating draft PR to address bug * Still working on solving re rendering bug * infinite rerender fix * Creating draft PR to address bug * Cleaning up in preparation for push * Made changes suggested by Elizabeth * Fixed issues as per Lindsey's comment Co-authored-by: Arash <arash.afghahi@gmail.com>

view details

Adam Dobrawy

commit sha 5ce79ff8563b0610e21b2f11275438b23a74a81f

fix(docs): fix path of image for "Create New Chart" (#18089)

view details

Adam Dobrawy

commit sha c3a17a72289f036bbb31443f2541bf625c8450e0

Migrate Checkbox story to tsx - see #18100 (#18101) Looks good!

view details

Adam Dobrawy

commit sha a5e805ef2a77857193b2bd55adaefef0f6115e87

refactor: migrate RowCountLabel to TypeScript & added story (#18105) * enable superbook for explore component * migrate RowCountLabel to TypeScript * add storybook for RowCountLabel

view details

Daniel Vaz Gaspar

commit sha 5fb8b0a13a3b50f279007d1934356dcc85298cb5

fix: logging warning on dataframe (don't use python's warnings) (#18111) * fix: logging warning on dataframe (don't use python's warnings) * lint

view details

Elizabeth Thompson

commit sha 1f8129241ba197a25795f1a9459996df9b696a4c

update changelog and updating for 1.4.0 (#18083)

view details

Beto Dealmeida

commit sha d5cc55ade407325e3e4e8f4c3a25761572cdd6d3

feat: allow assets to be managed externally

view details

Beto Dealmeida

commit sha d8f65923caa0ed922e0d8b8571bb74ace6f91fda

Use server_default

view details

push time in 8 hours

push eventbetodealmeida/incubator-superset

Beto Dealmeida

commit sha 2ea9c7ca48501cd507d60391a69ac6ab67d0438f

Set a User Agent

view details

push time in 10 hours

PR opened apache/superset

chore: use pkg_resources for cleaner config

<!--- Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/ Example: fix(dashboard): load charts correctly -->

SUMMARY

<!--- Describe the change below, including rationale and design decisions -->

I was looking at our config.py and noticed we can do a better job in picking up the location of the superset directory and a couple JSON files. The functions for resource management from pkg_resources provide a cleaner interface that works across packages and editable installs.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

<!--- Skip this if not applicable -->

TESTING INSTRUCTIONS

<!--- Required! What steps can be taken to manually verify the changes? -->

ADDITIONAL INFORMATION

<!--- Check any relevant boxes with "x" --> <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->

  • [ ] Has associated issue:
  • [ ] Required feature flags:
  • [ ] Changes UI
  • [ ] Includes DB Migration (follow approval process in SIP-59)
    • [ ] Migration is atomic, supports rollback & is backwards-compatible
    • [ ] Confirm DB migration upgrade and downgrade tested
    • [ ] Runtime estimates and downtime expectations provided
  • [ ] Introduces new feature or API
  • [ ] Removes existing feature or API
+9 -4

0 comment

1 changed file

pr created time in 11 hours

push eventapache/superset

Beto Dealmeida

commit sha f71ed8b2d70e5937e1048a87076d5c4d66403fc6

chore: use pkg_resources for cleaner config

view details

push time in 11 hours

create barnchapache/superset

branch : use_pkg_resources

created branch time in 11 hours

Pull request review commentapache/superset

feat: a simple client

+# Licensed to the Apache Software Foundation (ASF) under one+# or more contributor license agreements.  See the NOTICE file+# distributed with this work for additional information+# regarding copyright ownership.  The ASF licenses this file+# to you under the Apache License, Version 2.0 (the+# "License"); you may not use this file except in compliance+# with the License.  You may obtain a copy of the License at+#+#   http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing,+# software distributed under the License is distributed on an+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+# KIND, either express or implied.  See the License for the+# specific language governing permissions and limitations+# under the License.++"""+A simple client for running SQL queries against Superset:++    >>> from superset.client import SupersetClient+    >>> client = SupersetClient("http://localhost:8088/", "admin", "admin")+    >>> print(client.databases)+    [<Database "examples" (postgres)>]++    >>> examples = client.databases[0]+    >>> sql = "SELECT platform, rank FROM video_game_sales LIMIT 2"+    >>> print(examples.run_query(sql))+      platform  rank+    0      Wii     1+    1      NES     2++Data is returned in a Pandas Dataframe.++"""++from typing import List, Union++import pandas as pd+import requests+from bs4 import BeautifulSoup+from yarl import URL++from superset.utils.core import shortid+++class Database:+    """+    A database configured in Superset.+    """++    def __init__(  # pylint: disable=too-many-arguments+        self,+        baseurl: Union[str, URL],+        database_id: int,+        name: str,+        backend: str,+        session: requests.Session,+        csrf_token: str,+    ):+        self.baseurl = URL(baseurl)+        self.database_id = database_id+        self.name = name+        self.backend = backend+        self.session = session+        self.csrf_token = csrf_token++    def run_query(self, sql: str, limit: int = 1000) -> pd.DataFrame:+        """+        Run a SQL query, returning a Pandas dataframe.+        """+        url = self.baseurl / "superset/sql_json/"+        data = {+            "client_id": shortid()[:10],+            "database_id": self.database_id,+            "json": True,+            "runAsync": False,+            "schema": None,+            "sql": sql,+            "sql_editor_id": "1",+            "tab": "Untitled Query 2",+            "tmp_table_name": "",+            "select_as_cta": False,+            "ctas_method": "TABLE",+            "queryLimit": limit,+            "expand_data": True,+        }+        headers = {+            "X-CSRFToken": self.csrf_token,+            "Accept": "application/json",+            "Content-Type": "application/json",+        }++        response = self.session.post(url, json=data, headers=headers)+        payload = response.json()+        return pd.DataFrame(payload["data"])++    def __repr__(self) -> str:+        return f'<Database "{self.name}" ({self.backend})>'+++class SupersetClient:  # pylint: disable=too-few-public-methods++    """+    A client for running queries against Superset.+    """++    def __init__(self, baseurl: Union[str, URL], username: str, password: str):+        # convert to URL if necessary+        self.baseurl = URL(baseurl)+        self.username = username

You're right, I definitely want to support the other authentication mechanisms. I started with username because it's the standard auth mechanism and one of the most common, and we can maintain backwards compatibility it while adopting an auth interface.

What I have in mind for the future is:

def __init__(
    self,
    baseurl: Union[str, URL],
    username: Optional[str] = None,
    password: Optional[str] = None,
    auth: Optional[AuthInterface] = None,
):
    if username:
        auth = UsernamePasswordAuth(username, password)

This way we can introduce more authentication mechanisms without introducing breaking changes.

betodealmeida

comment created time in 12 hours

PullRequestReviewEvent

Pull request review commentapache/superset

feat: a simple client

+# Licensed to the Apache Software Foundation (ASF) under one+# or more contributor license agreements.  See the NOTICE file+# distributed with this work for additional information+# regarding copyright ownership.  The ASF licenses this file+# to you under the Apache License, Version 2.0 (the+# "License"); you may not use this file except in compliance+# with the License.  You may obtain a copy of the License at+#+#   http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing,+# software distributed under the License is distributed on an+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+# KIND, either express or implied.  See the License for the+# specific language governing permissions and limitations+# under the License.++"""+A simple client for running SQL queries against Superset:++    >>> from superset.client import SupersetClient+    >>> client = SupersetClient("http://localhost:8088/", "admin", "admin")+    >>> print(client.databases)+    [<Database "examples" (postgres)>]++    >>> examples = client.databases[0]+    >>> sql = "SELECT platform, rank FROM video_game_sales LIMIT 2"+    >>> print(examples.run_query(sql))+      platform  rank+    0      Wii     1+    1      NES     2++Data is returned in a Pandas Dataframe.++"""++from typing import List, Union++import pandas as pd+import requests+from bs4 import BeautifulSoup+from yarl import URL++from superset.utils.core import shortid+++class Database:+    """+    A database configured in Superset.+    """++    def __init__(  # pylint: disable=too-many-arguments+        self,+        baseurl: Union[str, URL],+        database_id: int,+        name: str,+        backend: str,+        session: requests.Session,+        csrf_token: str,+    ):+        self.baseurl = URL(baseurl)+        self.database_id = database_id+        self.name = name+        self.backend = backend+        self.session = session+        self.csrf_token = csrf_token++    def run_query(self, sql: str, limit: int = 1000) -> pd.DataFrame:+        """+        Run a SQL query, returning a Pandas dataframe.+        """+        url = self.baseurl / "superset/sql_json/"+        data = {+            "client_id": shortid()[:10],+            "database_id": self.database_id,+            "json": True,+            "runAsync": False,+            "schema": None,+            "sql": sql,+            "sql_editor_id": "1",+            "tab": "Untitled Query 2",+            "tmp_table_name": "",+            "select_as_cta": False,+            "ctas_method": "TABLE",+            "queryLimit": limit,+            "expand_data": True,+        }+        headers = {+            "X-CSRFToken": self.csrf_token,+            "Accept": "application/json",+            "Content-Type": "application/json",

Oh, great idea! You mean the version of Superset (or SHA)?

betodealmeida

comment created time in a day

PullRequestReviewEvent

PR opened apache/superset

feat: a simple client

<!--- Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/ Example: fix(dashboard): load charts correctly -->

SUMMARY

<!--- Describe the change below, including rationale and design decisions -->

Users often ask how to run queries against Superset programmatically, and the answer is not trivial.

I added a small client called SupersetClient that takes care of the weird auth we use for SQL Lab (/superset/sql_json), simplifying the process.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

<!--- Skip this if not applicable -->

N/A

TESTING INSTRUCTIONS

<!--- Required! What steps can be taken to manually verify the changes? -->

$ pip install 'superset[client]'
>>> from superset.client import SupersetClient
>>> client = SupersetClient("http://localhost:8088/", "admin", "admin")
>>> print(client.databases)
[<Database "examples" (postgres)>]
>>> examples = client.databases[0]
>>> sql = "SELECT platform, rank FROM video_game_sales LIMIT 2"
>>> print(examples.run_query(sql))
  platform  rank
0      Wii     1
1      NES     2

ADDITIONAL INFORMATION

<!--- Check any relevant boxes with "x" --> <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->

  • [ ] Has associated issue:
  • [ ] Required feature flags:
  • [ ] Changes UI
  • [ ] Includes DB Migration (follow approval process in SIP-59)
    • [ ] Migration is atomic, supports rollback & is backwards-compatible
    • [ ] Confirm DB migration upgrade and downgrade tested
    • [ ] Runtime estimates and downtime expectations provided
  • [ ] Introduces new feature or API
  • [ ] Removes existing feature or API
+151 -1

0 comment

2 changed files

pr created time in a day

create barnchbetodealmeida/incubator-superset

branch : python_sdk

created branch time in a day

issue commentpydap/pydap

accessing pydap.org results in DNS error

The domain hasn't expired, and I haven't changed anything. I'm not super familiar with using a custom domain with readthedocs, but I can point it somewhere else if the IP has changed.

tomkralidis

comment created time in a day

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentapache/superset

feat: allow assets to be managed externally

+# Licensed to the Apache Software Foundation (ASF) under one+# or more contributor license agreements.  See the NOTICE file+# distributed with this work for additional information+# regarding copyright ownership.  The ASF licenses this file+# to you under the Apache License, Version 2.0 (the+# "License"); you may not use this file except in compliance+# with the License.  You may obtain a copy of the License at+#+#   http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing,+# software distributed under the License is distributed on an+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+# KIND, either express or implied.  See the License for the+# specific language governing permissions and limitations+# under the License.+"""Add columns for external management++Revision ID: 5fd49410a97a+Revises: c53bae8f08dd+Create Date: 2022-01-19 07:34:20.594786++"""++# revision identifiers, used by Alembic.+revision = "5fd49410a97a"+down_revision = "c53bae8f08dd"++import sqlalchemy as sa+from alembic import op+++def upgrade():+    with op.batch_alter_table("dashboards") as batch_op:+        batch_op.add_column(+            sa.Column("is_managed_externally", sa.Boolean(), nullable=True)

Good point... I'm not sure why Alembic didn't add default=False, since it's defined in the model. Let me add it manually, thanks!

betodealmeida

comment created time in 2 days

PullRequestReviewEvent

issue commentbetodealmeida/shillelagh

Requests missing from dependencies

Ah, nice catch! We should add it as a dependency.

mrshu

comment created time in 2 days

startedmultiprocessio/datastation

started time in 2 days

PR opened apache/superset

Sync exposures

<!--- Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/ Example: fix(dashboard): load charts correctly -->

SUMMARY

<!--- Describe the change below, including rationale and design decisions -->

Stacked on https://github.com/apache/superset/pull/18098.

Syncs dashboards to DBT as exposures.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

<!--- Skip this if not applicable -->

TESTING INSTRUCTIONS

<!--- Required! What steps can be taken to manually verify the changes? -->

ADDITIONAL INFORMATION

<!--- Check any relevant boxes with "x" --> <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->

  • [ ] Has associated issue:
  • [ ] Required feature flags:
  • [ ] Changes UI
  • [ ] Includes DB Migration (follow approval process in SIP-59)
    • [ ] Migration is atomic, supports rollback & is backwards-compatible
    • [ ] Confirm DB migration upgrade and downgrade tested
    • [ ] Runtime estimates and downtime expectations provided
  • [ ] Introduces new feature or API
  • [ ] Removes existing feature or API
+463 -11

0 comment

10 changed files

pr created time in 2 days

create barnchbetodealmeida/incubator-superset

branch : sync_exposures

created branch time in 2 days

PR opened apache/superset

feat: command to sync DBT to Superset

<!--- Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/ Example: fix(dashboard): load charts correctly -->

SUMMARY

<!--- Describe the change below, including rationale and design decisions -->

This PR introduces a new command to sync metadata from DBT to Superset. The command reads the profile and manifest files, creating/updating databases and datasets in Superset based on them.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

<!--- Skip this if not applicable -->

Given this ~/.dbt/profiles.yml:

superset_examples:
  outputs:

    dev:
      type: postgres
      threads: 1
      host: localhost
      port: 5432
      user: beto
      pass: ''
      dbname: examples_dev
      schema: public
      meta:
        superset:
          cache_timeout: 300  # arbitrary metadata for our DB

  target: dev

The file messages_channels.sql:

SELECT
  messages.ts,
  channels.name,
  messages.text
FROM
  {{ source ('public', 'messages') }} messages
  JOIN {{ source ('public', 'channels') }} channels ON messages.channel_id = channels.id

And schema.yaml:

version: 2

sources:
  - name: public
    tables:
      - name: messages
        description: 'Messages in the Slack channel'
      - name: channels
        description: 'Information about Slack channels'

metrics:
  - name: cnt 
    label: ''
    model: ref('messages_channels')
    description: ''
    type: count
    sql: '*'

We can run:

$ superset sync dbt \
> ~/Projects/dbt-examples/superset_examples/target/manifest.json \
> --project superset_examples \
> --target dev  # not needed, default is already "dev"

This will (1) create (or update) the a new database connection based on Postgres:

Screenshot 2022-01-19 at 14-46-25 Superset

It will also (2) create/update three datasets owner by the admin:

Screen Shot 2022-01-19 at 2 47 47 PM

(Note that the dataset description comes from the DBT config.)

It will also populate metrics:

Screenshot 2022-01-19 at 14-48-38 Superset

TESTING INSTRUCTIONS

<!--- Required! What steps can be taken to manually verify the changes? -->

  1. Create a DBT project.
  2. Run superset sync dbt /path/to/project/target/manifest.json --project PROJECT --target TARGET

Check that everything is imported correctly.

Currently, this only works for Postgres, but adding other profile types is straightforward.

ADDITIONAL INFORMATION

<!--- Check any relevant boxes with "x" --> <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->

  • [ ] Has associated issue:
  • [ ] Required feature flags:
  • [ ] Changes UI
  • [ ] Includes DB Migration (follow approval process in SIP-59)
    • [ ] Migration is atomic, supports rollback & is backwards-compatible
    • [ ] Confirm DB migration upgrade and downgrade tested
    • [ ] Runtime estimates and downtime expectations provided
  • [ ] Introduces new feature or API
  • [ ] Removes existing feature or API
+366 -11

0 comment

9 changed files

pr created time in 2 days

push eventbetodealmeida/incubator-superset

Beto Dealmeida

commit sha 4035ccb433afa059a95d15a3f83178c20b03e316

feat: command to sync DBT to Superset

view details

push time in 2 days

Pull request review commentapache/superset

feat: allow assets to be managed externally

 def name(self) -> str:     params = Column(String(1000))     perm = Column(String(1000))     schema_perm = Column(String(1000))+    is_managed_externally = Column(Boolean, default=False)+    external_url = Column(Text, nullable=True)

The problem is that external_url should really be optional — depending on the source a resource might be managed externally, without having an external URL associated with it. So it might be possible to have is_managed_externally=true while at the same time external_url=null.

betodealmeida

comment created time in 2 days

PullRequestReviewEvent

push eventbetodealmeida/incubator-superset

Beto Dealmeida

commit sha 50b3ffefc5c57f1e3f49b3a177cadbe8e2651882

WIP

view details

push time in 2 days

push eventbetodealmeida/incubator-superset

Beto Dealmeida

commit sha 9e2bc72fb9687eb1d4563fd02853fea6b94a978d

chore: split CLI into multiple files (#18082) * chore: split CLI into multiple files * Update tests * Who fixes the fixtures? * Add subcommands dynamically * Rebase

view details

Beto Dealmeida

commit sha 48b558306472a8ab593a4df98c0f349599711abd

WIP

view details

push time in 2 days

create barnchbetodealmeida/incubator-superset

branch : dbt_sync

created branch time in 2 days

push eventapache/superset

Beto Dealmeida

commit sha 9e2bc72fb9687eb1d4563fd02853fea6b94a978d

chore: split CLI into multiple files (#18082) * chore: split CLI into multiple files * Update tests * Who fixes the fixtures? * Add subcommands dynamically * Rebase

view details

push time in 2 days

more