profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/elixir-sqlite/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
elixir-sqlite elixir-sqlite Elixir Sqlite

elixir-sqlite/sqlite_ecto2 117

Sqlite3 adapter for Ecto 2.2.x

elixir-sqlite/sqlitex 115

An Elixir wrapper around esqlite. Allows access to sqlite3 databases.

elixir-sqlite/ecto_sqlite3 89

An Ecto SQLite3 adapter.

elixir-sqlite/exqlite 63

An SQLite3 driver for Elixir

elixir-sqlite/elixir_sqlite 3

Elixir NIF for sqlite

startedelixir-sqlite/exqlite

started time in 3 days

startedelixir-sqlite/ecto_sqlite3

started time in 4 days

startedelixir-sqlite/ecto_sqlite3

started time in 4 days

issue commentelixir-sqlite/exqlite

Add Windows CI

Oh this is awesome.

kevinlang

comment created time in 9 days

issue commentelixir-sqlite/exqlite

Add Windows CI

Looks like this is unblocked now.

https://github.com/erlef/setup-beam/issues/51

kevinlang

comment created time in 9 days

issue closedelixir-sqlite/ecto_sqlite3

Ecto expectations for map fields

Howdy,

I'm trying to use the polymorphic_embed library with the SQLite3 ecto adapter and I'm running into an error loading the map field. It works correctly with the ecto postgres adapter. It seems like ecto is expecting JSON fields to be deserialized from a string to a struct.

1) test returns the oldest 25 order transitions by default (Tai.Orders.SearchTransitionsTest)                                                                                                      
     apps/tai/test/tai/orders/search_transitions_test.exs:4                                                                                                                                          
     ** (FunctionClauseError) no function clause matching in PolymorphicEmbed.do_get_polymorphic_module_from_map/3                                                                                   
                                                                                                                                                                                                     
     The following arguments were given to PolymorphicEmbed.do_get_polymorphic_module_from_map/3:                                                                                                    
                                                                                                                                                                                                     
         # 1                                                                                                                                                                                         
         "{\"last_received_at\":\"2021-07-23T06:42:10.671042Z\",\"last_venue_timestamp\":\"2021-07-23T06:42:10.671036Z\",\"__type__\":\"cancel\"}"                                                   
                                                                                                                                                                                                     
         # 2                                                                                                                                                                                         
         "__type__"                                                                                                                                                                                  
                                                                                                                                                                                                     
         # 3                                                                                                                                                                                         
         [%{identify_by_fields: [], module: Tai.Orders.Transitions.AcceptCreate, type: "accept_create"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.VenueCreateError, type: "venue_crea
te_error"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.RescueCreateError, type: "rescue_create_error"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.Open, type: "open"}, %
{identify_by_fields: [], module: Tai.Orders.Transitions.PendCancel, type: "pend_cancel"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.AcceptCancel, type: "accept_cancel"}, %{identify_b
y_fields: [], module: Tai.Orders.Transitions.VenueCancelError, type: "venue_cancel_error"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.RescueCancelError, type: "rescue_cancel_error"},
 %{identify_by_fields: [], module: Tai.Orders.Transitions.Cancel, type: "cancel"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.PendAmend, type: "pend_amend"}, %{identify_by_fields: [],
 module: Tai.Orders.Transitions.AcceptAmend, type: "accept_amend"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.VenueAmendError, type: "venue_amend_error"}, %{identify_by_fields: [], m
odule: Tai.Orders.Transitions.RescueAmendError, type: "rescue_amend_error"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.Amend, type: "amend"}, %{identify_by_fields: [], module: Tai.Or
ders.Transitions.Fill, type: "fill"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.PartialFill, type: "partial_fill"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.Expire, t
ype: "expire"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.Reject, type: "reject"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.Skip, type: "skip"}]                      
                                                                                                                                                                                                     
     Attempted function clauses (showing 1 out of 1):                                                                                                                                                
                                                                                                                                                                                                     
         defp do_get_polymorphic_module_from_map(%{} = attrs, type_field, types_metadata)                                                                                                            
                                                                                                                                                                                                     
     code: search_order_transitions = Tai.Orders.search_transitions(order.client_id, nil)                                                                                                            
     stacktrace:                                                                                                                                                                                     
       (polymorphic_embed 1.6.3) lib/polymorphic_embed.ex:286: PolymorphicEmbed.do_get_polymorphic_module_from_map/3                                                                                 
       (polymorphic_embed 1.6.3) lib/polymorphic_embed.ex:248: PolymorphicEmbed.load/3                                                                                                               
       (ecto 3.6.2) lib/ecto/type.ex:894: Ecto.Type.process_loaders/3                                                                                                                                
       (ecto 3.6.2) lib/ecto/repo/queryable.ex:406: Ecto.Repo.Queryable.struct_load!/6                                                                                                               
       (ecto 3.6.2) lib/ecto/repo/queryable.ex:238: anonymous fn/5 in Ecto.Repo.Queryable.preprocessor/3                                                                                             
       (elixir 1.11.4) lib/enum.ex:1411: Enum."-map/2-lists^map/1-0-"/2
       (ecto 3.6.2) lib/ecto/repo/queryable.ex:229: Ecto.Repo.Queryable.execute/4
       (ecto 3.6.2) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
       test/tai/orders/search_transitions_test.exs:8: (test)

I've created a branch to reproduce to problem

https://github.com/fremantle-industries/tai/tree/sqlite3-polymorphic-embed-invalid-map-field

$ mix test test/tai/orders/search_transitions_test.exs

The embed field is defined in the OrderTransition schema

https://github.com/fremantle-industries/tai/blob/sqlite3-polymorphic-embed-invalid-map-field/apps/tai/lib/tai/orders/order_transition.ex#L20

closed time in 9 days

rupurt

issue commentelixir-sqlite/ecto_sqlite3

Ecto expectations for map fields

Closing as @mathieuprog has merged a fix. Cheers @warmwaffles @kevinlang.

rupurt

comment created time in 9 days

startedelixir-sqlite/ecto_sqlite3

started time in 10 days

issue commentelixir-sqlite/ecto_sqlite3

Ecto expectations for map fields

We probably need to make if function similarly.

rupurt

comment created time in 10 days

issue commentelixir-sqlite/ecto_sqlite3

Ecto expectations for map fields

Yep, I think (1) would not work. I'm not too familiar with ParamaterizedTypes, but reading through briefly I now understand that ultimately your library is its own "type", so we wouldn't have any loader to process it anyway.

I agree that (2) seems the way to go, if we do not want to make changes to exqlite (I defer to @warmwaffles on that one).


@warmwaffles I don't think it is an issue with the MySQL driver, based on this file: https://github.com/elixir-ecto/myxql/blob/fdb147dba07d7699c4af23448996ae7f4663bce6/lib/myxql/protocol/values.ex

There I see that the driver is encoding the json column response directly into an Elixir map, so the ParamaterizedType would get that value to process on load.

rupurt

comment created time in 10 days

issue commentelixir-sqlite/ecto_sqlite3

Support Ecto.Changeset.foreign_key_constraint/3 ?

I've added some documentation about this limitation that points back to this issue.

jjcarstens

comment created time in 10 days

push eventelixir-sqlite/ecto_sqlite3

Kevin Lang

commit sha 5f5cd892a89928c8c514cc086b9444d9c4e46890

typo

view details

push time in 10 days

push eventelixir-sqlite/ecto_sqlite3

Kevin Lang

commit sha ba3489bf403a97a2d4bd2234d454d93f92bbc87a

add note about foreign key constraints for #42

view details

push time in 10 days

issue commentelixir-sqlite/ecto_sqlite3

Ecto expectations for map fields

If it is a limitation (purposely) of exqlite, then we can convert the json string into a map. I don't see how solution (1) works.

rupurt

comment created time in 10 days

issue closedelixir-sqlite/ecto_sqlite3

Support Ecto.Changeset.foreign_key_constraint/3 ?

I have a relational setup that I would like to just give me an {:error, changeset} tuple back for when the foreign record doesn't exist. However, no matter what I put I always get this error:

14:02:56.563 [error] Task #PID<0.559.0> started from #PID<0.557.0> terminating
** (Ecto.ConstraintError) constraint error when attempting to insert struct:

    *  (foreign_key_constraint)

If you would like to stop this constraint violation from raising an
exception and instead add it as an error to your changeset, please
call `foreign_key_constraint/3` on your changeset with the constraint
`:name` as an option.

The changeset defined the following constraints:

    * intervals_schedule_id_fkey (foreign_key_constraint)

...

If I'm understanding right, SQLite knows there is a constraint here, but its unnamed and so all we get back to help is an empty * (foreign_key_constraint) clue. And since there is no name to match on, Ecto can't reconcile it and things just crash. I've even tried using the name: "" option, but I think Ecto just ignores the empty string.

Is it even possible to get around this or are we limited by the abilities of SQLite here? I do understand I can use the foreign_key: :off option in the adapter to just skip validations entirely which is currently my workaround, but I would love to get the {:error, changeset} return instead.

Caveat: I am in no way a DB expert and this could be 100% user error. My migrations are simple and would love to know if Im just doing things wrong:

create table(:schedules) do
  add :name, :string
end

create table(:intervals) do
  add :schedule, references(:schedules)
end

closed time in 10 days

jjcarstens

issue commentelixir-sqlite/ecto_sqlite3

Support Ecto.Changeset.foreign_key_constraint/3 ?

For reference, here is what we get back from SQLite3 and thus why we cannot really do anything:

sqlite> pragma foreign_keys=on;
sqlite> create table a(b integer not null primary key); 
sqlite> create table c(d references a(b));
sqlite> insert into c values(1);
Error: FOREIGN KEY constraint failed

No mention of which constraint failed. See https://sqlite.org/forum/forumpost/b17fdd0d59 for examples on other types of constraints.

jjcarstens

comment created time in 10 days

issue commentelixir-sqlite/ecto_sqlite3

Support Ecto.Changeset.foreign_key_constraint/3 ?

Yeah this is a SQLite3 issue. See this issue for more details: https://github.com/elixir-sqlite/ecto_sqlite3/issues/28

We have a hack that works for some cases. I believe I tried adapting that hack for foreign keys in the above issue, but failed.

I encourage you to post a request on the SQLite3 forum asking for some sort of API from SQLite3 so we can support this! :smile:

In the meantime, I'll try thinking about whether there may be other hacks we can do. I'll play around a bit an re-open this if I find a new approach. I'll also update the docs to make this a known limitation.

jjcarstens

comment created time in 10 days

issue openedelixir-sqlite/ecto_sqlite3

Support Ecto.Changeset.foreign_key_constraint/3 ?

I have a relational setup that I would like to just give me an {:error, changeset} tuple back for when the foreign record doesn't exist. However, no matter what I put I always get this error:

14:02:56.563 [error] Task #PID<0.559.0> started from #PID<0.557.0> terminating
** (Ecto.ConstraintError) constraint error when attempting to insert struct:

    *  (foreign_key_constraint)

If you would like to stop this constraint violation from raising an
exception and instead add it as an error to your changeset, please
call `foreign_key_constraint/3` on your changeset with the constraint
`:name` as an option.

The changeset defined the following constraints:

    * intervals_schedule_id_fkey (foreign_key_constraint)

...

If I'm understanding right, SQLite knows there is a constraint here, but its unnamed and so all we get back to help is an empty * (foreign_key_constraint) clue. And since there is no name to match on, Ecto can't reconcile it and things just crash. I've even tried using the name: "" option, but I think Ecto just ignores the empty string.

Is it even possible to get around this or are we limited by the abilities of SQLite here? I do understand I can use the foreign_key: :off option in the adapter to just skip validations entirely which is currently my workaround, but I would love to get the {:error, changeset} return instead.

Caveat: I am in no way a DB expert and this could be 100% user error. My migrations are simple and would love to know if Im just doing things wrong:

create table(:schedules) do
  add :name, :string
end

create table(:intervals) do
  add :schedule, references(:schedules)
end

created time in 10 days

issue commentelixir-sqlite/ecto_sqlite3

Ecto expectations for map fields

@mathieuprog this is the same issue I posted in the polymorphic_embed repo https://github.com/mathieuprog/polymorphic_embed/issues/39. When you get a chance do you mind reporting back how we should move forward with it?

rupurt

comment created time in 10 days

issue commentelixir-sqlite/ecto_sqlite3

Ecto expectations for map fields

I'm curious if this is also an issue with the MySQL adapter. I based a lot of the sqlite3 adapter off of it.

rupurt

comment created time in 11 days

issue commentelixir-sqlite/ecto_sqlite3

Ecto expectations for map fields

I'm fairly certain this is due to differences between postgrex and exqlite handle json.

exqlite is pretty low level and only handles the 5 actual SQLite data types. This means that we do the mapping from the JSON string into a map in the Ecto adapter layer (i.e., this library).

postgrex, by comparison, handles the full set of Postgres data types (which includes e.g., jsonb) and does the conversion into the Elixir equivalents in the driver before it reaches the Ecto layer. I think the myxql adapter does hte same.

My idea is that is the cause of the issue. This library is not letting the data returned from the query go through the full Ecto "pipeline" and allowing our loaders to be run. Specifically these lines at lib/ecto/adapters/sqlite3.ex (note that we do the JSON decode here):

  @impl Ecto.Adapter
  def loaders({:map, _}, type) do
    [&Codec.json_decode/1, &Ecto.Type.embedded_load(type, &1, :json)]
  end

  @impl Ecto.Adapter
  def loaders({:array, _}, type) do
    [&Codec.json_decode/1, type]
  end

  @impl Ecto.Adapter
  def loaders(:map, type) do
    [&Codec.json_decode/1, type]
  end

The fix is either:

  1. Update the polymorphic_embed library to run the associated loaders first before it does its thing
  2. Update the polymorphic_embed library to decode the JSON string if it detects it gets a string and not a map before it does its thing
  3. Update exqlite to have more conversion scope

Either (1) or (2) I think is the preferred solution.exqlite tries to stay as true to SQLite3 actual interface as possible - and SQLite3 simply does not return anything other than a string representation of the json column, as it only has 5 actual data types.

I hope that makes sense. You may need to coordinate with the maintainer of that library for the fix. I'll keep this issue open for now, though I don't think there is anything on our end we can do.

rupurt

comment created time in 13 days

issue openedelixir-sqlite/ecto_sqlite3

Ecto expectations for map fields

Howdy,

I'm trying to use the polymorphic_embed library with the SQLite3 ecto adapter and I'm running into an error loading the embed map field. It works correctly with the ecto postgres adapter.

1) test returns the oldest 25 order transitions by default (Tai.Orders.SearchTransitionsTest)                                                                                                      
     apps/tai/test/tai/orders/search_transitions_test.exs:4                                                                                                                                          
     ** (FunctionClauseError) no function clause matching in PolymorphicEmbed.do_get_polymorphic_module_from_map/3                                                                                   
                                                                                                                                                                                                     
     The following arguments were given to PolymorphicEmbed.do_get_polymorphic_module_from_map/3:                                                                                                    
                                                                                                                                                                                                     
         # 1                                                                                                                                                                                         
         "{\"last_received_at\":\"2021-07-23T06:42:10.671042Z\",\"last_venue_timestamp\":\"2021-07-23T06:42:10.671036Z\",\"__type__\":\"cancel\"}"                                                   
                                                                                                                                                                                                     
         # 2                                                                                                                                                                                         
         "__type__"                                                                                                                                                                                  
                                                                                                                                                                                                     
         # 3                                                                                                                                                                                         
         [%{identify_by_fields: [], module: Tai.Orders.Transitions.AcceptCreate, type: "accept_create"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.VenueCreateError, type: "venue_crea
te_error"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.RescueCreateError, type: "rescue_create_error"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.Open, type: "open"}, %
{identify_by_fields: [], module: Tai.Orders.Transitions.PendCancel, type: "pend_cancel"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.AcceptCancel, type: "accept_cancel"}, %{identify_b
y_fields: [], module: Tai.Orders.Transitions.VenueCancelError, type: "venue_cancel_error"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.RescueCancelError, type: "rescue_cancel_error"},
 %{identify_by_fields: [], module: Tai.Orders.Transitions.Cancel, type: "cancel"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.PendAmend, type: "pend_amend"}, %{identify_by_fields: [],
 module: Tai.Orders.Transitions.AcceptAmend, type: "accept_amend"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.VenueAmendError, type: "venue_amend_error"}, %{identify_by_fields: [], m
odule: Tai.Orders.Transitions.RescueAmendError, type: "rescue_amend_error"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.Amend, type: "amend"}, %{identify_by_fields: [], module: Tai.Or
ders.Transitions.Fill, type: "fill"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.PartialFill, type: "partial_fill"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.Expire, t
ype: "expire"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.Reject, type: "reject"}, %{identify_by_fields: [], module: Tai.Orders.Transitions.Skip, type: "skip"}]                      
                                                                                                                                                                                                     
     Attempted function clauses (showing 1 out of 1):                                                                                                                                                
                                                                                                                                                                                                     
         defp do_get_polymorphic_module_from_map(%{} = attrs, type_field, types_metadata)                                                                                                            
                                                                                                                                                                                                     
     code: search_order_transitions = Tai.Orders.search_transitions(order.client_id, nil)                                                                                                            
     stacktrace:                                                                                                                                                                                     
       (polymorphic_embed 1.6.3) lib/polymorphic_embed.ex:286: PolymorphicEmbed.do_get_polymorphic_module_from_map/3                                                                                 
       (polymorphic_embed 1.6.3) lib/polymorphic_embed.ex:248: PolymorphicEmbed.load/3                                                                                                               
       (ecto 3.6.2) lib/ecto/type.ex:894: Ecto.Type.process_loaders/3                                                                                                                                
       (ecto 3.6.2) lib/ecto/repo/queryable.ex:406: Ecto.Repo.Queryable.struct_load!/6                                                                                                               
       (ecto 3.6.2) lib/ecto/repo/queryable.ex:238: anonymous fn/5 in Ecto.Repo.Queryable.preprocessor/3                                                                                             
       (elixir 1.11.4) lib/enum.ex:1411: Enum."-map/2-lists^map/1-0-"/2
       (ecto 3.6.2) lib/ecto/repo/queryable.ex:229: Ecto.Repo.Queryable.execute/4
       (ecto 3.6.2) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
       test/tai/orders/search_transitions_test.exs:8: (test)

I've created a branch to reproduce to problem

https://github.com/fremantle-industries/tai/tree/sqlite3-polymorphic-embed-invalid-map-field

$ mix test test/tai/orders/search_transitions_test.exs

created time in 14 days

startedelixir-sqlite/ecto_sqlite3

started time in 15 days

startedelixir-sqlite/ecto_sqlite3

started time in 16 days

startedelixir-sqlite/ecto_sqlite3

started time in 18 days

startedelixir-sqlite/ecto_sqlite3

started time in 18 days

startedelixir-sqlite/ecto_sqlite3

started time in 18 days

push eventelixir-sqlite/ecto_sqlite3

mradke

commit sha 8c9bc137ab1e01a5cdd207994f90dcb2fcb4a049

Fixes link to limitations in Readme.md (#40)

view details

push time in 19 days

PR merged elixir-sqlite/ecto_sqlite3

Fixes link to limitations in Readme.md

I clicked on the link, and the wrong section was opened :-)

+1 -1

1 comment

1 changed file

mradke

pr closed time in 19 days

pull request commentelixir-sqlite/ecto_sqlite3

Fixes link to limitations in Readme.md

thanks!

mradke

comment created time in 19 days