profile
viewpoint
Ophir LOJKINE lovasoa @auto-grid @enexflow Paris, France ophir.dev Senior Software engineer @auto-grid, previously worked @Qwant

lovasoa/bad_json_parsers 346

Exposing problems in json parsers of several programming languages.

gap-decoder/gapdecoder 61

Google Arts And Culture Downloader. Python script to download high-resolution images from google arts & culture.

j6s/ophir.php 22

PHP script that converts ODT to HTML

adamboche/python-marshmallow-union 8

Union fields for marshmallow.

al34034/Qwant-Gnome-shell-integration 7

A Gnome-shell extension to be able to search Qwant directly from the overview

lovasoa/bin2png 5

Embed binary data inside an HTML file in an efficient way.

Fantasya/Bornes-Ecossaises 1

TA - Shotten Toten

liufan2312/tgs-salt-identification 1

Code for TGS salt identification competition on kaggle

lovasoa/anylang 1

Type text in any alphabet. Anylang is a javascript library that converts a phonetic transcription of a text in a language to a text written in the alphabet of the target language. Currently works with hebrew (with vowels) and russian.

issue commentgap-decoder/gapdecoder

No Module Named: 'aiohttp'

You can also use dezoomify-rs, which should be faster and for which you won't have to manually handle dependencies

jaaschutz

comment created time in 6 hours

pull request commentsql-js/sql.js

Change exec(...) to return column names for empty result sets

See https://sql.js.org/documentation/Database.html#%5B%22iterateStatements%22%5D

frankier

comment created time in 17 hours

PR closed sql-js/sql.js

Reviewers
Change exec(...) to return column names for empty result sets

Currently exec(...) will not put anything in its results array for a SELECT query which returns no results. This is not ideal. Say I run 3 SELECT queries, and 1 returns an empty result set. How do I know which query isn't included in the results array? With this change, you can always tell what the size of the results array will be just by inspecting the query. I have also changed getColumnNames(...) to work for empty result sets by using sqlite3_column_count instead of sqlite3_data_count.

+87 -16

6 comments

5 changed files

frankier

pr closed time in 17 hours

pull request commentsql-js/sql.js

Change exec(...) to return column names for empty result sets

Now that #412 and #429 have been merged, one can achieve the same results as the proposed updated exec, in a few lines of code, without needing a breaking change to the API :

function exec(db, sql, params) {
  const results = [];
  for (const statement of db.iterateStatements(sql)) {
    statement.bind(params);
    const columns = statement.getColumnNames();
    const values = [];
    while (statement.step()) values.push(statement.get());
    results.push({ columns, values });
  }
  return results;
}
frankier

comment created time in 17 hours

pull request commentsql-js/sql.js

Multiple query enhanced feedback

It's on NPM : https://www.npmjs.com/package/sql.js :smiley:

cpainterwakefield

comment created time in 17 hours

push eventsql-js/sql.js

ophir

commit sha 334565562c5d3dd07418def2d4236881015ca5fc

Simplify the documentation and remove console.log from tests

view details

push time in 17 hours

push eventsql-js/sql.js

ophir

commit sha a73bc4067d4fe76feace69e16dbe3d1ba53d80c6

v1.4.0

view details

push time in 17 hours

created tagsql-js/sql.js

tagv1.4.0

A javascript library to run SQLite on the web.

created time in 17 hours

push eventsql-js/sql.js

cpainterwakefield

commit sha e20bb74d70d04cd57339d50b5947dc2f800ce136

Add db.iterateStatements (#429) db.iterateStatements allows giving sql.js an SQL string and iterating over statements objects created from that string * Initial working version of statement iteration. * Tests written and running green. * Resolved linter issues. * Modified approach based on PR feedback; simple testing works, automated tests and documentation to be written. * Testing and documentation written. * Undid prior commit (accidentally committed change from sql-wasm.js to sql-wasm-debug.js) * Applied all suggested modifications. * Documentation fixes. * Improve the documentation of db#iterateStatements * Add @implements annotations for StatementIterator * Reformat test code * Fix the type definition of StatementIterator.StatementIteratorResult Co-authored-by: ophir <pere.jobs@gmail.com>

view details

push time in 17 hours

PR merged sql-js/sql.js

Multiple query enhanced feedback

Added ability to iterate over statements in a multi-statement SQL string. Also added Statement methods accessing SQLite's sqlite3_sql() and sqlite3_normalized_sql() functions.

+300 -3

12 comments

6 changed files

cpainterwakefield

pr closed time in 17 hours

PullRequestReviewEvent

push eventcpainterwakefield/sql.js

ophir

commit sha dbc9edcca4ec2b314ae94a1df69f8049f1b12bad

Fix the type definition of StatementIterator.StatementIteratorResult

view details

push time in 17 hours

push eventcpainterwakefield/sql.js

ophir

commit sha 2d7902e74b10c9e591feed4450a0795ed8212e9e

Add @implements annotations for StatementIterator

view details

ophir

commit sha d9392c13fbfcba6bc4ad6592b6240065bd0cc482

Reformat test code

view details

push time in 17 hours

push eventcpainterwakefield/sql.js

ophir

commit sha f829a876770939285039cba550cb42c6d1373131

Improve the documentation of db#iterateStatements

view details

push time in 18 hours

issue commentLucioFranco/tokio-compat-02

tokio-compat-02 does not work for any library that uses tokio::time::delay_for

Yes, the panic message help. But the panic may never happen during development, and happen for the first time in production, depending on how your dependencies use delay_for. I think a good solution would be to warn about this problem in the README and in the documentation, and to advise users to use not this library in production.

lovasoa

comment created time in 19 hours

issue commentLucioFranco/tokio-compat-02

tokio-compat-02 does not work for any library that uses tokio::time::delay_for

Is this really resolved ? The user has no way to know whether a given method in a library will call delay_for, and thus cannot know where to apply .compat()

lovasoa

comment created time in 19 hours

pull request commentsql-js/sql.js

Multiple query enhanced feedback

I resolved all the github conversations that you solved.

cpainterwakefield

comment created time in a day

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return stmt;     }; +    /** Iterate over multiple SQL statements in a SQL string+     @param {string} sql a string of SQL+     @return {StatementIterator} the resulting statement iterator+     */+    Database.prototype["iterateStatements"] = function iterateStatements(sql) {+        return new StatementIterator(sql, this);+    };++    /* Internal methods to implement statement iteration */++    Database.prototype.advanceIterator = function advanceIterator(iter) {+        if (iter.sqlPtr === null) {+            throw "Invalid iterator";+        }+        if (iter.activeStatement !== null) {+            iter.activeStatement["free"]();+            iter.activeStatement = null;+        }+        if (!this.db) {+            this.finalizeIterator(iter);+            throw "Database closed";

Unfortunately this would be a breaking change. This is one we will have to do at some point, but let's keep that for later.

cpainterwakefield

comment created time in a day

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return res;     }; +    /**+     * @classdesc+     * An iterator over multiple SQL statements in a string,+     * preparing and returning a Statement object for the next SQL+     * statement on each iteration.+     *+     * You can't instantiate this class directly, you have to use a+     * {@link Database} object in order to create a statement iterator+     *+     * {@see Database#iterateStatements}+     *+     * **Warning**: When you close a database (using db.close()),+     * using any statement iterators created by the database will+     * result in undefined behavior.+     *+     * @example+     * // loop over and execute statements in string sql+     * for (let statement of db.iterateStatements(sql) {+     *     statement.step();+     *     // get results, etc.+     *     // do not call statement.free() manually, each statement is freed+     *     // before the next one is parsed+     * }+     *+     * // capture any bad query exceptions with feedback+     * // on the bad sql+     * let it = db.iterateStatements(sql);+     * try {+     *     for (let statement of it) {+     *         statement.step();+     *     }+     * } catch(e) {+     *     console.log(+     *         `The SQL string "${it.getRemainingSQL()}" ` ++     *         `contains the following error: ${e}`+     *     );+     * }+     *+     * @constructs StatementIterator+     * @memberof module:SqlJs+     * @param {string} sql A string containing multiple SQL statements+     * @param {Database} db The database from which this iterator was created+     */+    function StatementIterator(sql, db) {+        this.db = db;+        var sz = lengthBytesUTF8(sql) + 1;+        this.sqlPtr = _malloc(sz);+        if (this.sqlPtr === null) {+            throw new Error("Unable to allocate memory for the SQL string");+        }+        stringToUTF8(sql, this.sqlPtr, sz);+        this.nextSqlPtr = this.sqlPtr;+        this.nextSqlString = null;+        this.activeStatement = null;+    }++    /**+     * @typedef {{+        value:Statement,

When the iterator is done, we return { done: true } which doesn't match this definition. I think the most useful type definition we can give is {done : true, value: undefined} | {done: false, value: Statement}.

cpainterwakefield

comment created time in a day

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return res;     }; +    /**+     * @classdesc+     * An iterator over multiple SQL statements in a string,+     * preparing and returning a Statement object for the next SQL+     * statement on each iteration.+     *+     * You can't instantiate this class directly, you have to use a+     * {@link Database} object in order to create a statement iterator+     *+     * {@see Database#iterateStatements}+     *+     * **Warning**: When you close a database (using db.close()),+     * using any statement iterators created by the database will+     * result in undefined behavior.
cpainterwakefield

comment created time in a day

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return res;     }; +    /**+     * @classdesc+     * Represents an iterator over multiple SQL statements in a string,+     * preparing and returning a Statement object for the next SQL+     * statement on each iteration.+     *+     * You can't instantiate this class directly, you have to use a+     * {@link Database} object in order to create a statement.+     *+     * **Warning**: When you close a database (using db.close()),+     * using any statement iterators created by the database will+     * result in undefined behavior.

It looks like it has been fixed. So we should remove the warning.

cpainterwakefield

comment created time in a day

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return res;     }; +    /**+     * @classdesc+     * Represents an iterator over multiple SQL statements in a string,+     * preparing and returning a Statement object for the next SQL+     * statement on each iteration.+     *+     * You can't instantiate this class directly, you have to use a+     * {@link Database} object in order to create a statement.+     *+     * **Warning**: When you close a database (using db.close()),+     * using any statement iterators created by the database will+     * result in undefined behavior.

@cpainterwakefield Has this been corrected ?

cpainterwakefield

comment created time in a day

PullRequestReviewEvent

created taglovasoa/dezoomify-extension

tagv0.4.4

A browser extension to detect zoomable images in web pages and downloading them with dezoomify

created time in 2 days

delete tag lovasoa/dezoomify-extension

delete tag : v0.4.4

delete time in 2 days

delete tag lovasoa/dezoomify-extension

delete tag : v1.3.1

delete time in 2 days

push eventlovasoa/dezoomify-extension

ophir

commit sha 09db7df6045dd88f1b6b56602e7209626a7e3c1f

fix duplicated id in github workflow file

view details

push time in 2 days

push eventlovasoa/dezoomify

ophir

commit sha 3142ef1022c9a44af80fc8a1c8ce8dc8ad90a181

Better error handling for seadragon

view details

push time in 2 days

push eventlovasoa/dezoomify-extension

ophir

commit sha 7aa3740b1618898f8c359138c58e035b956c2e60

v0.4.4

view details

push time in 2 days

created taglovasoa/dezoomify-extension

tagv0.4.4

A browser extension to detect zoomable images in web pages and downloading them with dezoomify

created time in 2 days

push eventlovasoa/dezoomify-extension

ophir

commit sha ec686050e0447d2c4e4258fbf1248c0614887ea6

Pass seadragon image tile URLs to dezoomify Fixes #8 See https://github.com/lovasoa/dezoomify/commit/6209c7e9a8585200f7f14bffa29fa6c36b1730c7

view details

ophir

commit sha b52feeaf1de10a4939b9401b8b9a54e0452eb9a3

Use the new dezoomify.ophir.dev URL

view details

ophir

commit sha 72e157369000a87a37c510ee7443678130e210a2

Merge branch 'master' of github.com:lovasoa/dezoomify-extension

view details

push time in 2 days

issue closedlovasoa/dezoomify-extension

deepzoom image files are always inferred as .dzi files, whereas they sometimes are .xml

See : https://github.com/lovasoa/dezoomify/issues/403

Example

For the following page:

  • https://patrimoine-numerique.ville-valenciennes.fr/ark:/29755/B_596066101_DAG_INC

the zoomable image URL is erroneously detected as

  • https://patrimoine-numerique.ville-valenciennes.fr/in/dz/attach_import_fb0a174e-7f77-4e63-ac94-924b4aad5fdc.dzi instead of
  • https://patrimoine-numerique.ville-valenciennes.fr/in/dz/attach_import_fb0a174e-7f77-4e63-ac94-924b4aad5fdc.xml

closed time in 2 days

lovasoa

issue commentlovasoa/dezoomify

DZI-problem at badigit.comune.bologna.it/mappe/Tsfoglia01.asp

I updated dezoomify so you should now be able to give it directly the tile url http://badigit.comune.bologna.it/mappe/303/GeneratedImages/dzc_output_files/11/2_2.jpg

mapper2

comment created time in 2 days

push eventlovasoa/dezoomify

ophir

commit sha 6209c7e9a8585200f7f14bffa29fa6c36b1730c7

Try both dzi and xml when given a seadraon tile url See: - https://github.com/lovasoa/dezoomify/issues/462 - https://github.com/lovasoa/dezoomify-extension/issues/8 - https://github.com/lovasoa/dezoomify/issues/480

view details

push time in 2 days

issue commentlovasoa/dezoomify

DZI-problem at badigit.comune.bologna.it/mappe/Tsfoglia01.asp

dezoomify-extension assumes the extension is .dzi whereas it is in fact .xml.

In your case : http://badigit.comune.bologna.it/mappe/303/GeneratedImages/dzc_output.xml

mapper2

comment created time in 2 days

issue closedlovasoa/dezoomify

DZI-problem at badigit.comune.bologna.it/mappe/Tsfoglia01.asp

I am not sure if this is a new site issue or just a DZI issue, but maps at http://badigit.comune.bologna.it/mappe/Tsfoglia01.asp?start=37 are not downloadable

http://badigit.comune.bologna.it/mappe/303/GeneratedImages/dzc_output_files/11/2_2.jpg "An error occured while trying to dezoomify your image. This is probably because dezoomify did not manage to find a zoomable image at the address you gave."

I've tried looking at the newtwork traffic but no luck

Great collection of historical maps of the Bologna region of Italy

Best

closed time in 2 days

mapper2

issue commentlovasoa/dezoomify

DZI-problem at badigit.comune.bologna.it/mappe/Tsfoglia01.asp

Duplicate of https://github.com/lovasoa/dezoomify-extension/issues/8

mapper2

comment created time in 2 days

push eventsql-js/sql.js

Ophir LOJKINE

commit sha 0cfeaef669f90da6b5cb6ce1de2ee0b60048e058

Update README.md

view details

push time in 2 days

push eventsql-js/sql.js

Ophir LOJKINE

commit sha 97cf240919ecc2a568df47dd265ecaaff6797a29

Update README.md

view details

push time in 2 days

issue commentimage-rs/jpeg-decoder

jpeg-decoder is slower than libjpeg-turbo

see my pull request that uses simd for this function: https://github.com/image-rs/jpeg-decoder/pull/146

Shnatsel

comment created time in 2 days

issue commentsql-js/sql.js

Update database not working

@askmewhat: If you are using node.js, you probably want to use a native sqlite3 binding, and not sql.js. Updates will just work.

phamvansy

comment created time in 3 days

push eventlovasoa/dezoomify-rs

ophir

commit sha 48a8b2be93e642f8ca3fef46aec39bbbc7d13c4a

Switch back to tokio 0.2 See https://github.com/LucioFranco/tokio-compat-02/issues/1

view details

push time in 4 days

issue commentLucioFranco/tokio-compat-02

tokio-compat-02 does not work for any library that uses tokio::time::delay_for

real_main().compat().await prevents the user from using tokio 0.3 anywhere in his code. There is no point in doing that instead of just sticking to tokio 0.2, is it ?

lovasoa

comment created time in 4 days

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return stmt;     }; +    /** Iterate over multiple SQL statements in a SQL string+     @param {string} sql a string of SQL+     @return {StatementIterator} the resulting statement iterator+     */+    Database.prototype["iterateStatements"] = function iterateStatements(sql) {+        return new StatementIterator(sql, this);+    };++    /* Internal method to implement statement iteration */++    Database.prototype["advanceIterator"] = function advanceIterator(iter) {

There is no notion of visibility in javascript, but I think it's okay to consider Database::statements visible at the level of the library.

cpainterwakefield

comment created time in 4 days

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return stmt;     }; +    /** Iterate over multiple SQL statements in a SQL string+     @param {string} sql a string of SQL+     @return {StatementIterator} the resulting statement iterator+     */+    Database.prototype["iterateStatements"] = function iterateStatements(sql) {+        return new StatementIterator(sql, this);+    };++    /* Internal method to implement statement iteration */++    Database.prototype["advanceIterator"] = function advanceIterator(iter) {

The code now doesn't access any private code in Database, but it accesses private fields in StatementIterator. Let's move it to StatementIterator.next

cpainterwakefield

comment created time in 4 days

PullRequestReviewEvent

issue commentLucioFranco/tokio-compat-02

tokio-compat-02 does not work with reqwest

The problem seems to come from the use of tokio02::time::delay_for. Here is a more simple reproduction :

use tokio_compat_02::FutureExt;

fn main() {
    let duration = std::time::Duration::from_secs(1);
    let rt = tokio03::runtime::Builder::new_multi_thread().build().unwrap();
    rt.block_on(async move {
        tokio02::time::delay_for(duration).compat().await;
    });
}
[package]
name = "hello_world"
version = "0.1.0"
edition = "2018"

[dependencies]
tokio03 = { package = "tokio", version = "0.3.0", features = ["full"] }
tokio02 = { package = "tokio", version = "0.2.22", features = ["full"] }
tokio-compat-02 = "0.1.0"
lovasoa

comment created time in 4 days

issue openedLucioFranco/tokio-compat-02

tokio-compat-02 does not work with reqwest

Hello !

reqwest is compatible with tokio 0.2, so I would have expected this crate to make it work with tokio 0.3. However, it looks like adding a timeout to requests makes reqwest require the runtime synchronously during send, not giving a chance to tokio_compat_02 to even receive the future.

Is there a way to work around that ? Does this problem exist with other libraries ? Could this be documented in tokio-compat-02 ?

Reproducible error

The following code panics at runtime :

main.rs

use tokio_compat_02::FutureExt;
use std::error::Error;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>>{
    let client = reqwest::Client::builder()
        .timeout(std::time::Duration::from_secs(1))
        .build().unwrap();
    let body = client.get("http://postman-echo.com/get?x=y")
        .send().compat().await?
        .text().compat().await?;

    println!("body = {:?}", body);
    Ok(())
}

Removing the timeout makes the code work.

Cargo.toml

[package]
name = "hello_world"
version = "0.1.0"
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
tokio = { version = "0.3.0", features = ["full"] }
tokio-compat-02 = "0.1.0"
reqwest = "0.10.8"

Backtrace

  17:     0x564a4305d1df - tokio::time::driver::handle::Handle::current::hf8848d51f6549010
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/time/driver/handle.rs:24
  18:     0x564a430bdf8f - tokio::time::driver::registration::Registration::new::h732389a1ee4f517d
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/time/driver/registration.rs:18
  19:     0x564a430bd865 - tokio::time::delay::delay_until::h113830ca763fcab4
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/time/delay.rs:19
  20:     0x564a430bd8e9 - tokio::time::delay::delay_for::hd299671a400d1562
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/time/delay.rs:56
  21:     0x564a42d2168c - core::ops::function::FnOnce::call_once::hfda903b17901e465
                               at /home/ophir/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227
  22:     0x564a42bf5661 - core::option::Option<T>::map::ha53e7fd9a36c6a7d
                               at /home/ophir/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/option.rs:437
  23:     0x564a42c01075 - reqwest::async_impl::client::Client::execute_request::h088b54c49985a55b
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.10.8/src/async_impl/client.rs:1049
  24:     0x564a42ba194d - reqwest::async_impl::request::RequestBuilder::send::h487b0aca5a9d9e2a
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.10.8/src/async_impl/request.rs:410
  25:     0x564a42b4d644 - hello_world::main::{{closure}}::hbae9112fb1cfa227
                               at /tmp/hello_world/src/main.rs:9

<details>

thread 'main' panicked at 'there is no timer running, must be called from the context of Tokio runtime', /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/time/driver/handle.rs:25:14
stack backtrace:
   0:     0x564a431202d0 - std::backtrace_rs::backtrace::libunwind::trace::h577ea05e9ca4629a
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/../../backtrace/src/backtrace/libunwind.rs:96
   1:     0x564a431202d0 - std::backtrace_rs::backtrace::trace_unsynchronized::h50b9b72b84c7dd56
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/../../backtrace/src/backtrace/mod.rs:66
   2:     0x564a431202d0 - std::sys_common::backtrace::_print_fmt::h6541cf9823837fac
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/sys_common/backtrace.rs:79
   3:     0x564a431202d0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hf64fbff071026df5
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/sys_common/backtrace.rs:58
   4:     0x564a4313ff5c - core::fmt::write::h9ddafa4860d8adff
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/core/src/fmt/mod.rs:1082
   5:     0x564a4311ae57 - std::io::Write::write_fmt::h1d2ee292d2b65481
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/io/mod.rs:1514
   6:     0x564a431225b0 - std::sys_common::backtrace::_print::ha25f9ff5080d886d
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/sys_common/backtrace.rs:61
   7:     0x564a431225b0 - std::sys_common::backtrace::print::h213e8aa8dc5405c0
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/sys_common/backtrace.rs:48
   8:     0x564a431225b0 - std::panicking::default_hook::{{closure}}::h6482fae49ef9d963
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panicking.rs:200
   9:     0x564a431222fc - std::panicking::default_hook::he30ad7589e0970f9
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panicking.rs:219
  10:     0x564a43122c13 - std::panicking::rust_panic_with_hook::haa1ed36ada4ffb03
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panicking.rs:569
  11:     0x564a431227e9 - std::panicking::begin_panic_handler::{{closure}}::h7001af1bb21aeaeb
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panicking.rs:476
  12:     0x564a4312075c - std::sys_common::backtrace::__rust_end_short_backtrace::h39910f557f5f2367
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/sys_common/backtrace.rs:153
  13:     0x564a431227a9 - rust_begin_unwind
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panicking.rs:475
  14:     0x564a4313e811 - core::panicking::panic_fmt::h4e2659771ebc78eb
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/core/src/panicking.rs:85
  15:     0x564a4313e5a3 - core::option::expect_failed::hd5a1da3a6b6bbb34
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/core/src/option.rs:1213
  16:     0x564a430b3141 - core::option::Option<T>::expect::h9d7d15f4679d0b8f
                               at /home/ophir/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/option.rs:333
  17:     0x564a4305d1df - tokio::time::driver::handle::Handle::current::hf8848d51f6549010
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/time/driver/handle.rs:24
  18:     0x564a430bdf8f - tokio::time::driver::registration::Registration::new::h732389a1ee4f517d
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/time/driver/registration.rs:18
  19:     0x564a430bd865 - tokio::time::delay::delay_until::h113830ca763fcab4
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/time/delay.rs:19
  20:     0x564a430bd8e9 - tokio::time::delay::delay_for::hd299671a400d1562
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.22/src/time/delay.rs:56
  21:     0x564a42d2168c - core::ops::function::FnOnce::call_once::hfda903b17901e465
                               at /home/ophir/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227
  22:     0x564a42bf5661 - core::option::Option<T>::map::ha53e7fd9a36c6a7d
                               at /home/ophir/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/option.rs:437
  23:     0x564a42c01075 - reqwest::async_impl::client::Client::execute_request::h088b54c49985a55b
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.10.8/src/async_impl/client.rs:1049
  24:     0x564a42ba194d - reqwest::async_impl::request::RequestBuilder::send::h487b0aca5a9d9e2a
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.10.8/src/async_impl/request.rs:410
  25:     0x564a42b4d644 - hello_world::main::{{closure}}::hbae9112fb1cfa227
                               at /tmp/hello_world/src/main.rs:9
  26:     0x564a42b53803 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h972b8e20ad86aad6
                               at /home/ophir/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:79
  27:     0x564a42b45f56 - tokio::park::thread::CachedParkThread::block_on::{{closure}}::h6a9c914bfaeb5dc7
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.3.0/src/park/thread.rs:263
  28:     0x564a42b4b108 - tokio::coop::with_budget::{{closure}}::hc97446a975be66bb
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.3.0/src/coop.rs:121
  29:     0x564a42b518b2 - std::thread::local::LocalKey<T>::try_with::h3016f5ea79a3c140
                               at /home/ophir/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:265
  30:     0x564a42b514d2 - std::thread::local::LocalKey<T>::with::he5206ae44ad18c49
                               at /home/ophir/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:241
  31:     0x564a42b45874 - tokio::coop::with_budget::haba85bd914ceb7e7
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.3.0/src/coop.rs:114
  32:     0x564a42b45874 - tokio::coop::budget::h4908e07ff8677677
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.3.0/src/coop.rs:98
  33:     0x564a42b45874 - tokio::park::thread::CachedParkThread::block_on::h5b6c83a18a451fac
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.3.0/src/park/thread.rs:263
  34:     0x564a42b4e51c - tokio::runtime::enter::Enter::block_on::h99bcb772d987584f
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.3.0/src/runtime/enter.rs:151
  35:     0x564a42b50e34 - tokio::runtime::thread_pool::ThreadPool::block_on::h3fe142ffd66093f5
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.3.0/src/runtime/thread_pool/mod.rs:80
  36:     0x564a42b4e0f2 - tokio::runtime::Runtime::block_on::h502481ad5895c8e6
                               at /home/ophir/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.3.0/src/runtime/mod.rs:422
  37:     0x564a42b4e355 - hello_world::main::hcdeaf6a10d54c5c7
                               at /tmp/hello_world/src/main.rs:4
  38:     0x564a42b523db - core::ops::function::FnOnce::call_once::hffc66f07dc4708d9
                               at /home/ophir/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227
  39:     0x564a42b50f2e - std::sys_common::backtrace::__rust_begin_short_backtrace::h4ff564984328cd04
                               at /home/ophir/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:137
  40:     0x564a42b4e5d1 - std::rt::lang_start::{{closure}}::h0e55bbab106b5b7e
                               at /home/ophir/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:66
  41:     0x564a431230e1 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h6a3209f124be2235
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/core/src/ops/function.rs:259
  42:     0x564a431230e1 - std::panicking::try::do_call::h88ce358792b64df0
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panicking.rs:373
  43:     0x564a431230e1 - std::panicking::try::h6311c259678e50fc
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panicking.rs:337
  44:     0x564a431230e1 - std::panic::catch_unwind::h56c5716807d659a1
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panic.rs:379
  45:     0x564a431230e1 - std::rt::lang_start_internal::h73711f37ecfcb277
                               at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/rt.rs:51
  46:     0x564a42b4e5a7 - std::rt::lang_start::h4c47e7eb2de15c5f
                               at /home/ophir/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:65
  47:     0x564a42b4e3fa - main
  48:     0x7f006f2800b3 - __libc_start_main
  49:     0x564a42b450be - _start
  50:                0x0 - <unknown>

</details>

created time in 4 days

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return res;     }; +    /**+     * @classdesc+     * An iterator over multiple SQL statements in a string,+     * preparing and returning a Statement object for the next SQL+     * statement on each iteration.+     *+     * You can't instantiate this class directly, you have to use a+     * {@link Database} object in order to create a statement iterator+     *+     * {@see Database#iterateStatements}+     *+     * **Warning**: When you close a database (using db.close()),+     * using any statement iterators created by the database will+     * result in undefined behavior.+     *+     * @example+     * // loop over and execute statements in string sql+     * for (let statement of db.iterateStatements(sql) {+     *     statement.step();+     *     // get results, etc.+     *     // do not call statement.free()
     *     // do not call statement.free() manually, each statement is freed before the next one is parsed
cpainterwakefield

comment created time in 4 days

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return res;     }; +    /**+     * @classdesc+     * An iterator over multiple SQL statements in a string,+     * preparing and returning a Statement object for the next SQL+     * statement on each iteration.+     *+     * You can't instantiate this class directly, you have to use a+     * {@link Database} object in order to create a statement iterator+     *+     * {@see Database#iterateStatements}+     *+     * **Warning**: When you close a database (using db.close()),+     * using any statement iterators created by the database will+     * result in undefined behavior.+     *+     * @example+     * // loop over and execute statements in string sql+     * for (let statement of db.iterateStatements(sql) {+     *     statement.step();+     *     // get results, etc.+     *     // do not call statement.free()+     * }+     *+     * // capture any bad query exceptions with feedback+     * // on the bad sql+     * let it = db.iterateStatements(sql);+     * try {+     *     for (let statement of it) {+     *         statement.step();+     *     }+     * } catch(e) {+     *     console.log(e);  // error message+     *     console.log(" occurred while executing:");+     *     console.log(it.getRemainingSQL());
     *     console.log(`The SQL string "${it.getRemainingSQL()}" contains the following error: ${e}`);
cpainterwakefield

comment created time in 4 days

PullRequestReviewEvent

pull request commentsql-js/sql.js

Multiple query enhanced feedback

I marked everything that was resolved as such. Once the remaining open points are fixed, we can merge.

cpainterwakefield

comment created time in 4 days

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return stmt;     }; +    /** Iterate over multiple SQL statements in a SQL string+     @param {string} sql a string of SQL+     @return {StatementIterator} the resulting statement iterator+     */+    Database.prototype["iterateStatements"] = function iterateStatements(sql) {+        return new StatementIterator(sql, this);+    };++    /* Internal method to implement statement iteration */++    Database.prototype["advanceIterator"] = function advanceIterator(iter) {+        var stack = stackSave();+        var pzTail = stackAlloc(4);+        setValue(apiTemp, 0, "i32");+        setValue(pzTail, 0, "i32");+        var returnCode = sqlite3_prepare_v2(+            this.db,+            iter.nextSql,+            -1,+            apiTemp,+            pzTail+        );+        var tail = UTF8ToString(getValue(pzTail, "i32"));

I think we want to call handleError in a try block, with a catch and a finally

cpainterwakefield

comment created time in 4 days

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

+exports.test = function(SQL, assert) {+  // Create a database+  var db = new SQL.Database();++  // Multiline SQL+  var sqlstr = "CREATE TABLE test (x text, y integer);\n"+    + "INSERT INTO test\n"+    + "VALUES ('hello', 42), ('goodbye', 17);\n"+    + "SELECT * FROM test;\n"+    + " -- nothing here";+  var sqlstart = "CREATE TABLE test (x text, y integer);"++  // Manual iteration+  // Get an iterator+  var it = db.iterateStatements(sqlstr);++  // Get first item+  var x = it.next();+  assert.equal(x.done, false, "Valid iterator object produced");+  assert.equal(x.value.getSQL(), sqlstart, "Statement is for first query only");+  assert.equal(it.getRemainingSQL(), sqlstr.slice(sqlstart.length), "Remaining sql retrievable");++  // execute the first query+  x.value.step();++  // get and execute the second query+  x = it.next();+  assert.equal(x.done, false, "Second query found");+  x.value.step();++  // get and execute the third query+  x = it.next();+  assert.equal(x.done, false, "Third query found");+  x.value.step();+  assert.deepEqual(x.value.getColumnNames(), ['x', 'y'], "Third query is SELECT");++  // check for additional queries+  x = it.next();+  assert.equal(x.done, true, "Done reported after last query");++  // no more iterations should be allowed+  assert.throws(function(){ it.next(); }, "Invalid iterator", "Cannot iterate past end");++  db.run("DROP TABLE test;");++  // for...of+  var count = 0;+  for (let statement of db.iterateStatements(sqlstr)) {+      statement.step();+      count = count + 1;+  }+  assert.equal(count, 3, "For loop iterates correctly");++  var badsql = "SELECT * FROM test;garbage in, garbage out";++  // bad sql will stop iteration+  assert.throws(function(){+      for (let statement of db.iterateStatements(badsql)) {+          statement.step();+      }+  }, "Bad SQL stops iteration");

We should probably be more precise in our error assertion.

var it = db.iterateStatements("SELECT 1 as x; pile of garbage; ");
assert.deepEqual({x : 1}, it.next().value.getAsObject());
assert.throws(it.next.bind(it));
assert.deepEqual({value: undefined, done: true}, it.next());
cpainterwakefield

comment created time in 4 days

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return stmt;     }; +    /** Iterate over multiple SQL statements in a SQL string+     @param {string} sql a string of SQL+     @return {StatementIterator} the resulting statement iterator+     */+    Database.prototype["iterateStatements"] = function iterateStatements(sql) {+        return new StatementIterator(sql, this);+    };++    /* Internal methods to implement statement iteration */++    Database.prototype.advanceIterator = function advanceIterator(iter) {+        if (iter.sqlPtr === null) {+            throw "Invalid iterator";+        }+        if (iter.activeStatement !== null) {+            iter.activeStatement["free"]();+            iter.activeStatement = null;+        }+        if (!this.db) {+            this.finalizeIterator(iter);+            throw "Database closed";+        }+        var stack = stackSave();+        var pzTail = stackAlloc(4);+        setValue(apiTemp, 0, "i32");+        setValue(pzTail, 0, "i32");+        var returnCode = sqlite3_prepare_v2_sqlptr(+            this.db,+            iter.nextSqlPtr,+            -1,+            apiTemp,+            pzTail+        );+        if (returnCode === SQLITE_OK) {+            iter.nextSqlPtr = getValue(pzTail, "i32");+        }+        stackRestore(stack);+        if (returnCode === SQLITE_OK) {+            var pStmt = getValue(apiTemp, "i32");+            if (pStmt === NULL) {+                this.finalizeIterator(iter);+                return { done: true };+            }+            iter.activeStatement = new Statement(pStmt, this);+            this.statements[pStmt] = iter.activeStatement;+            return { value: iter.activeStatement, done: false };+        }+        iter.nextSqlString = UTF8ToString(iter.nextSqlPtr);

What is the point of setting the private nextSqlString field here ?

cpainterwakefield

comment created time in 4 days

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return stmt;     }; +    /** Iterate over multiple SQL statements in a SQL string+     @param {string} sql a string of SQL+     @return {StatementIterator} the resulting statement iterator+     */+    Database.prototype["iterateStatements"] = function iterateStatements(sql) {+        return new StatementIterator(sql, this);+    };++    /* Internal methods to implement statement iteration */++    Database.prototype.advanceIterator = function advanceIterator(iter) {+        if (iter.sqlPtr === null) {+            throw "Invalid iterator";+        }+        if (iter.activeStatement !== null) {+            iter.activeStatement["free"]();+            iter.activeStatement = null;+        }+        if (!this.db) {+            this.finalizeIterator(iter);+            throw "Database closed";
            throw new Error("Database closed");
cpainterwakefield

comment created time in 4 days

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return stmt;     }; +    /** Iterate over multiple SQL statements in a SQL string+     @param {string} sql a string of SQL+     @return {StatementIterator} the resulting statement iterator+     */+    Database.prototype["iterateStatements"] = function iterateStatements(sql) {+        return new StatementIterator(sql, this);+    };++    /* Internal methods to implement statement iteration */++    Database.prototype.advanceIterator = function advanceIterator(iter) {+        if (iter.sqlPtr === null) {+            throw "Invalid iterator";
            throw new Error("Some more descriptive error message");
cpainterwakefield

comment created time in 4 days

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return res;     }; +    /**+     * @classdesc+     * An iterator over multiple SQL statements in a string,+     * preparing and returning a Statement object for the next SQL+     * statement on each iteration.+     *+     * You can't instantiate this class directly, you have to use a+     * {@link Database} object in order to create a statement iterator+     *+     * {@see Database#iterateStatements}+     *+     * **Warning**: When you close a database (using db.close()),+     * using any statement iterators created by the database will+     * result in undefined behavior.+     *+     * @example+     * // loop over and execute statements in string sql+     * for (let statement of db.iterateStatements(sql) {+     *     statement.step();+     *     // get results, etc.+     *     // do not call statement.free()+     * }+     *+     * // capture any bad query exceptions with feedback+     * // on the bad sql+     * let it = db.iterateStatements(sql);+     * try {+     *     for (let statement of it) {+     *         statement.step();+     *     }+     * } catch(e) {+     *     console.log(e);  // error message+     *     console.log(" occurred while executing:");+     *     console.log(it.getRemainingSQL());+     * }+     *+     * @constructs StatementIterator+     * @memberof module:SqlJs+     * @param {string} sql A string containing multiple SQL statements+     * @param {Database} db The database from which this iterator was created+     */+    function StatementIterator(sql, db) {+        this.db = db;+        var sz = lengthBytesUTF8(sql) + 1;+        this.sqlPtr = _malloc(sz);+        if (this.sqlPtr === null) {+            throw "Bad malloc";
            throw new Error("Bad malloc");
cpainterwakefield

comment created time in 4 days

PullRequestReviewEvent

pull request commentsql-js/sql.js

Multiple query enhanced feedback

My new code doesn't use pzTail at all, and I don't call it "lastSql" now, as that is confusing (you helped my clarify my thinking on this). Instead, the iterator now has a method "getRemainingSql()", which simply reports the value pointed to by the nextSql pointer. This works, and makes more sense; if a user wants to see the remaining SQL after each statement is prepared, they can do so (not sure why they would, but it is there). In my case, if there is an exception, I can access the remaining SQL to report that "the error occurred in here somewhere".

Perfect, this sounds like what we want

cpainterwakefield

comment created time in 4 days

pull request commentsql-js/sql.js

Multiple query enhanced feedback

If you want to access your "lastSql" after an error, you can attach it to the error type you are throwing... But I don't see the interest, since SQLite doesn't make any guarantee concerning where pzTail is going to end after an invalid SQL statement. What do you want to do with that string ?

cpainterwakefield

comment created time in 4 days

pull request commentsql-js/sql.js

Multiple query enhanced feedback

I just made some testing with your implementation. It looks like db.iterateStatements("\\") returns an infinite iterator (instead of an iterator that raises an error and then finishes).

cpainterwakefield

comment created time in 5 days

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return stmt;     }; +    /** Iterate over multiple SQL statements in a SQL string+     @param {string} sql a string of SQL+     @return {StatementIterator} the resulting statement iterator+     */+    Database.prototype["iterateStatements"] = function iterateStatements(sql) {+        return new StatementIterator(sql, this);+    };++    /* Internal method to implement statement iteration */++    Database.prototype["advanceIterator"] = function advanceIterator(iter) {+        var stack = stackSave();+        var pzTail = stackAlloc(4);+        setValue(apiTemp, 0, "i32");+        setValue(pzTail, 0, "i32");+        var returnCode = sqlite3_prepare_v2(+            this.db,+            iter.nextSql,+            -1,+            apiTemp,+            pzTail+        );+        var tail = UTF8ToString(getValue(pzTail, "i32"));

We free the string and invalidate the iterator before returning {done:true} to the user.

cpainterwakefield

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

+exports.test = function(SQL, assert, done) {+  // Create a database+  var db = new SQL.Database();++  // Multiline SQL, with comments+  var sqlstr = "CREATE TABLE test (x text, y integer);\n"+    + "INSERT INTO test -- here's a single line comment\n"+    + "VALUES ('hello', 42), ('goodbye', 17);\n"+    + "/* Here's a multiline \n"+    + "   comment */ \n"+    + "SELECT * FROM test;\n"+    + " -- nothing here";++  // Get an iterator+  var it = db.iterateStatements(sqlstr);++  // Get first item+  var x = it.next();+  assert.equal(x.done, false, "Valid iterator object produced");+  assert.equal(x.value.getSQL(), "CREATE TABLE test (x text, y integer);", "Statement is for first query only");++  // execute the first query+  x.value.step();+  x.value.free()++  // get and execute the second query+  x = it.next();+  assert.equal(x.done, false, "Second query found");+  x.value.step();+  x.value.free();++  // get and execute the third query+  x = it.next();+  assert.equal(x.done, false, "Third query found");+  x.value.step();+  assert.equal(x.value.get(), ['hello', 42], "Third query row results correct");+  x.value.free();++  // check for additional queries+  x = it.next();+  assert.equal(x.done, true, "No more queries reported");++  x = it.next();+  assert.equal(x.done, true, "Advancing done iterator does nothing");+};++if (module == require.main) {+	const target_file = process.argv[2];+  const sql_loader = require('./load_sql_lib');+  sql_loader(target_file).then((sql)=>{+    require('test').run({+      'test database': function(assert, done){+        exports.test(sql, assert, done);
      'test database': function(assert){
        exports.test(sql, assert);
cpainterwakefield

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

+exports.test = function(SQL, assert, done) {

Taking a done callback makes the test asynchronous. Never calling done prevents the failing test to be reported. This is probably why you didn't notice the tests were failing.

exports.test = function(SQL, assert) {
cpainterwakefield

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

+exports.test = function(SQL, assert, done) {+  // Create a database+  var db = new SQL.Database();++  // Multiline SQL, with comments+  var sqlstr = "CREATE TABLE test (x text, y integer);\n"+    + "INSERT INTO test -- here's a single line comment\n"+    + "VALUES ('hello', 42), ('goodbye', 17);\n"+    + "/* Here's a multiline \n"+    + "   comment */ \n"+    + "SELECT * FROM test;\n"+    + " -- nothing here";++  // Get an iterator+  var it = db.iterateStatements(sqlstr);++  // Get first item+  var x = it.next();+  assert.equal(x.done, false, "Valid iterator object produced");+  assert.equal(x.value.getSQL(), "CREATE TABLE test (x text, y integer);", "Statement is for first query only");++  // execute the first query+  x.value.step();+  x.value.free()++  // get and execute the second query+  x = it.next();+  assert.equal(x.done, false, "Second query found");+  x.value.step();+  x.value.free();++  // get and execute the third query+  x = it.next();+  assert.equal(x.done, false, "Third query found");+  x.value.step();+  assert.equal(x.value.get(), ['hello', 42], "Third query row results correct");

The tests are failing because of this line.

cpainterwakefield

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return stmt;     }; +    /** Iterate over multiple SQL statements in a SQL string+     @param {string} sql a string of SQL+     @return {StatementIterator} the resulting statement iterator+     */+    Database.prototype["iterateStatements"] = function iterateStatements(sql) {+        return new StatementIterator(sql, this);+    };++    /* Internal method to implement statement iteration */++    Database.prototype["advanceIterator"] = function advanceIterator(iter) {+        var stack = stackSave();+        var pzTail = stackAlloc(4);+        setValue(apiTemp, 0, "i32");+        setValue(pzTail, 0, "i32");+        var returnCode = sqlite3_prepare_v2(+            this.db,+            iter.nextSql,+            -1,+            apiTemp,+            pzTail+        );+        var tail = UTF8ToString(getValue(pzTail, "i32"));

What I need is not method to convert the remaining text pointer to a string, but a mechanism to retrieve the most recently attempted SQL.

The most recently attempted SQL is the remaining text.

[...] After the call to sqlite3_prepare_v2(), I have 1 pointer stored in pzTail. Whether the call succeeds or fails, pzTail points to the portion of the SQL string after the statement that sqlite attempted to make a prepared statement from. If we succeed, then we use that for the next iteration. If we succeed, we can also retrieve the SQL from the prepared statement using sqlite3_sql(). However, if we fail, there is no statement, so no way to retrieve the attempted SQL except by using the original string and subtracting pzTail from it.

This is not what the SQLite documentation says. SQLite will try to parse what you give, and stop when it either successfully parses a statement or encounters an error. If the string you provided does not start with a valid sql statement, I think there is nothing you can do but just retrieve the full remaining text. SQLite does not make any guarantee to parse invalid statements correctly. Or at least that's what I understand from the documentation: https://sqlite.org/c3ref/prepare.html

I guess I could store 2 pointers and then a method to convert that stretch of bytes to a string.

You need two pointers: one to the full string to be able to free it once the iteration is over, and one to the currently remaining part of the string (pzTail). There is no notion of an end of an invalid sql statement.

cpainterwakefield

comment created time in 5 days

PullRequestReviewEvent

push eventlovasoa/dezoomify-rs

ophir

commit sha 3d730ab7e13c3f7813f83a7746045992eef1818c

Update dependencies

view details

push time in 5 days

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

+exports.test = function(SQL, assert, done) {+  // Create a database+  var db = new SQL.Database();++  // Multiline SQL, with comments+  var sqlstr = "CREATE TABLE test (x text, y integer);\n"+    + "INSERT INTO test -- here's a single line comment\n"+    + "VALUES ('hello', 42), ('goodbye', 17);\n"+    + "/* Here's a multiline \n"+    + "   comment */ \n"+    + "SELECT * FROM test;\n"

We should probably also test error handling

cpainterwakefield

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return stmt;     }; +    /** Iterate over multiple SQL statements in a SQL string+     @param {string} sql a string of SQL+     @return {StatementIterator} the resulting statement iterator+     */+    Database.prototype["iterateStatements"] = function iterateStatements(sql) {+        return new StatementIterator(sql, this);+    };++    /* Internal method to implement statement iteration */++    Database.prototype["advanceIterator"] = function advanceIterator(iter) {+        var stack = stackSave();+        var pzTail = stackAlloc(4);+        setValue(apiTemp, 0, "i32");+        setValue(pzTail, 0, "i32");+        var returnCode = sqlite3_prepare_v2(+            this.db,+            iter.nextSql,+            -1,+            apiTemp,+            pzTail+        );+        var tail = UTF8ToString(getValue(pzTail, "i32"));

What you are currently doing is allocating memory, converting your string to utf8, copying it to the allocated memory, giving a pointer to the allocated memory to sqlite (all of this is handled by emscripten in cwrap), and then, when it returns, converting back the end of the string to utf16, and starting all over again.

What we should actually be doing is allocating the string manually at the beginning, keep a simple pointer of the remaining text to be parsed, and updating it as we go. In the end, we manually free the entire string.

We can add a method to convert the remaining text pointer to a string. In the current design, lastSql is not documented, so you could not rely on it anyway.

cpainterwakefield

comment created time in 5 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return stmt;     }; +    /** Iterate over multiple SQL statements in a SQL string+     @param {string} sql a string of SQL+     @return {StatementIterator} the resulting statement iterator+     */+    Database.prototype["iterateStatements"] = function iterateStatements(sql) {+        return new StatementIterator(sql, this);+    };++    /* Internal method to implement statement iteration */++    Database.prototype["advanceIterator"] = function advanceIterator(iter) {+        var stack = stackSave();+        var pzTail = stackAlloc(4);+        setValue(apiTemp, 0, "i32");+        setValue(pzTail, 0, "i32");+        var returnCode = sqlite3_prepare_v2(+            this.db,+            iter.nextSql,+            -1,+            apiTemp,+            pzTail+        );+        var tail = UTF8ToString(getValue(pzTail, "i32"));+        iter.lastSql = iter.nextSql.substr(+            0,+            iter.nextSql.length - tail.length+        );+        iter.nextSql = tail;+        stackRestore(stack);+        if (returnCode === SQLITE_OK) {

I do not understand. handleError(returnCode); happy_path() would do the exact same thing as if (returnCode === ok) { happy_path() } manual_error_handling(), it would just be clearer.

cpainterwakefield

comment created time in 5 days

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return stmt;     }; +    /** Iterate over multiple SQL statements in a SQL string+     @param {string} sql a string of SQL+     @return {StatementIterator} the resulting statement iterator+     */+    Database.prototype["iterateStatements"] = function iterateStatements(sql) {+        return new StatementIterator(sql, this);+    };++    /* Internal method to implement statement iteration */++    Database.prototype["advanceIterator"] = function advanceIterator(iter) {+        var stack = stackSave();+        var pzTail = stackAlloc(4);+        setValue(apiTemp, 0, "i32");+        setValue(pzTail, 0, "i32");+        var returnCode = sqlite3_prepare_v2(+            this.db,+            iter.nextSql,+            -1,+            apiTemp,+            pzTail+        );+        var tail = UTF8ToString(getValue(pzTail, "i32"));

You shouldn't need to keep track of lastSql at all. You can use only nextSql and have it be a simple opaque pointer. You will need to create a new cwrap for sqlite3_prepare_v2 that takes a number instead of a string.

cpainterwakefield

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return stmt;     }; +    /** Iterate over multiple SQL statements in a SQL string+     @param {string} sql a string of SQL+     @return {StatementIterator} the resulting statement iterator+     */+    Database.prototype["iterateStatements"] = function iterateStatements(sql) {

This should be a simple

if (typeof Symbol == "function" && typeof Symbol.iterator == "symbol") {
  StatementIterator.prototype[Symbol.iterator] = function iterator() {
    return this;
  }
}
cpainterwakefield

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return stmt;     }; +    /** Iterate over multiple SQL statements in a SQL string+     @param {string} sql a string of SQL+     @return {StatementIterator} the resulting statement iterator+     */+    Database.prototype["iterateStatements"] = function iterateStatements(sql) {+        return new StatementIterator(sql, this);+    };++    /* Internal method to implement statement iteration */++    Database.prototype["advanceIterator"] = function advanceIterator(iter) {

Ok for keeping that on the database then.

cpainterwakefield

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentsql-js/sql.js

Multiple query enhanced feedback

 Module["onRuntimeInitialized"] = function onRuntimeInitialized() {         return rowObject;     }; +    /** Get the SQL string used in preparing this statement.+     @return {string} The SQL string+     */+    Statement.prototype["getSQL"] = function getSQL() {+        return sqlite3_sql(this.stmt);+    };++    /** Get the SQLite's normalized version of the SQL string used in

Then maybe we can add an @example and a link to SQLite's documentation in the jsdoc.

cpainterwakefield

comment created time in 5 days

PullRequestReviewEvent

issue commentShopify/sarama

sarama.PacketDecodingError=kafka : error decoding packet: message of length 1213486160 too large or too small

  • 1213486160 is the decimal representation of 0x48 0x54 0x54 0x50 which is HTTP in ascii
  • What happened is probably this: A client expecting a binary payload (such as kafka) received an HTTP response instead. It may either be talking to the wrong host and port, or there is an HTTP proxy (such as kubernetes ingress) where there shouldn't be one.
programmer4m

comment created time in 5 days

more